Extend & Generalize an Existing Multi-Source Funding Opportunity Acquisition System (Python)

We are looking for an experienced Python engineer with web scraping / structured data acquisition / source integration experience to extend and generalize an existing multi-source funding opportunity acquisition system.

This is NOT a greenfield crawler project.

We already have an existing Python implementation with:

source acquisition runner;
write-gated importer;
downstream promotion of validated discoveries;
source registry and routing configuration;
normalized candidate contracts;
stable source/candidate identities;
SHA-256 content hashing;
deduplication and idempotency concepts;
event classification and triage;
automated tests;
several working deterministic source adapters;
an existing n8n environment.

Your task is to reuse, consolidate, extend and generalize the current implementation, not rebuild it from scratch.

A detailed Technical Pricing Pack is attached. Please review it before bidding. The attachment contains the full architecture context, source corpus, acceptance requirements, sample contracts, test expectations and Definition of Done.

Main Goal

Build a maintainable Multi-Source Opportunity Acquisition Framework v0.1 that can:

regularly monitor our full approved source corpus;
detect genuinely new funding opportunities;
distinguish new calls from results, amendments, information pages, archives and already known opportunities;
normalize source-specific data into the existing acquisition contract;
preserve source identity, provenance and raw-content identity;
deduplicate repeated discoveries;
detect changes to previously known opportunities;
preserve relevant PDF/DOCX attachment links;
prepare valid opportunities for the existing downstream system;
allow most future sources to be added through configuration rather than new custom Python code;
expose a stable operational entry point suitable for n8n.
Mandatory High-Value Source Areas

The v0.1 scope explicitly includes:

1. NGO.pl / fundusze.ngo.pl

We already have qualification work and a previously validated discovery path.

NGO.pl is treated as a secondary aggregator/discovery source.

Expected principle:

NGO.pl discovery -> official-source resolution where possible -> normalized candidate

It must remain part of the acquisition system.

2. Witkac.pl

Witkac is an important primary/public application platform.

Existing repository work includes:

known public contest index;
public discovery evidence;
incremental monitoring specification;
polite-fetch rules;
resolver/implementation specifications;
URL/contest identity handling.

The selected freelancer should reuse this existing work rather than research Witkac from zero.

Known access limitations must be handled safely and explicitly.

3. Corporate foundations / private grant operators

We already have a dedicated source corpus.

The selected freelancer will receive a 46-row 2026-qualified/verified corporate-foundation dataset containing fields such as:

official domain;
grants/program page;
news page;
regulations;
results;
application platform;
RSS/newsletter;
evidence and status.

A wider 84-row corpus also exists as supporting context.

The objective is not to create 46 separate scrapers.

The freelancer should determine which sources can use:

generic HTML;
RSS;
list/detail patterns;
common foundation/private-operator adapters;
Witkac/application-platform resolution;
configuration-only onboarding;

and which genuinely require custom adapters.

Every row in the verified 46-source dataset must be accounted for in the final coverage matrix.

Existing Source Universe

Existing work also covers or researches sources such as:

NIW;
Warsaw ETO;
BIP families / BIP Logonet;
Fundusze Europejskie;
EEA Grants;
ARiMR / gov.pl;
PARP;
BIPLO;
Grantona;
dotacje.io;
Atlas Dotacji;
generic HTML sources;
RSS/sitemap sources.

At project start we will provide a frozen source acceptance corpus.

We are deliberately not limiting this project to 10 or 20 URLs.

Every source in the frozen corpus must receive an explicit final status, for example:

SUPPORTED_EXISTING_ADAPTER
SUPPORTED_GENERIC_CONFIG
SUPPORTED_NEW_CUSTOM_ADAPTER
BLOCKED_EXTERNAL_LIMITATION
OUT_OF_SCOPE_BY_EXPLICIT_AGREEMENT

Sources must not simply be skipped.

Generalization

Where technically reasonable, we want reusable source classes instead of one scraper per website.

Expected classes may include:

RSS / Atom;
RSS + detail page;
HTML list -> detail;
generic configurable HTML;
BIP patterns;
registry/tenant-driven sources;
official programme/call pages;
JSON/API;
Witkac/application-platform style sources;
corporate foundation grant/news/program pages;
PDF/DOCX attachment discovery.

The desired future workflow is:

add source to registry -> choose source class -> configure -> validate -> test -> enable

instead of writing another scraper.

Classification & Deduplication

A newly fetched URL is not automatically a new opportunity.

The system must distinguish:

new real call -> INGEST_READY;
ambiguous possible call -> REVIEW_REQUIRED;
results -> NOT_A_NEW_CALL;
information/archive page -> NOT_A_NEW_CALL;
known unchanged call -> idempotent no-op;
known changed call -> update/change signal;
aggregator article -> discovery + official-source resolution.

Deduplication must reuse or extend existing stable IDs, canonical URLs, hashes and source identities.

It must also handle:

repeated scheduled runs;
URL variants;
Witkac identity/fragment variants;
aggregator -> official-source relationships;
changed content;
the same opportunity discovered from multiple listing paths.
Safety & Reliability

Expected safeguards include:

request timeouts;
response-size limits;
allowed-domain validation;
safe redirect handling;
bounded concurrency;
request budgets;
sensible retry/backoff;
no uncontrolled crawling;
partial source failures must not crash the entire batch;
anti-bot/403/JS-only limitations must be visible;
no secrets in logs;
no new paid infrastructure without prior approval.

Repeated execution must be idempotent and must not create duplicate records.

n8n Integration

The wider system already uses n8n.

We need a small stable integration point:

n8n trigger -> acquisition runner -> normalize/classify/deduplicate -> existing persistence -> run summary

This project does not include redesigning the entire n8n architecture.

Testing & Documentation

Expected deliverables include:

regression tests for existing adapters;
adapter contract tests;
registry validation;
generic source-class tests;
NGO.pl tests;
Witkac resolver/identity/monitor tests;
corporate-foundation onboarding tests;
idempotency/dedup/change tests;
source-failure tests;
attachment-discovery tests;
proof that several new sources can be added through configuration only.

Documentation should include:

final architecture;
supported source classes;
source registry documentation;
adapter contract;
official-source resolution rules;
how to add a source without changing Python;
how to add a custom adapter;
failure/recovery notes;
final coverage matrix for the complete source corpus;
separate coverage matrix for all 46 verified corporate-foundation sources;
NGO.pl and Witkac support/limitations.
Out of Scope

This project does not include:

organization/client intake;
organization profiling;
full semantic extraction of grant requirements from PDFs;
matching organizations to opportunities;
rewriting the downstream decision engine;
application writing;
new frontend development;
database redesign;
rebuilding the acquisition system from scratch.
Before Bidding

Please read the attached Technical Pricing Pack carefully. It contains significantly more technical detail than this listing.

After reviewing it, please confirm:

whether the full scope is clear;
how you would generalize the existing implementation without rebuilding working adapters;
how you would approach NGO.pl;
how you would reuse the existing Witkac work;
how you would cover the 46-source corporate-foundation dataset without creating 46 custom scrapers;
how you would handle deduplication, change detection and idempotency;
your realistic delivery time;
estimated hours;
whether your quoted price covers the complete agreed v0.1 scope;
anything described in the listing or attachment that is not included in your price.

We want to agree the complete scope and price before development starts