Stabilize Nightly Airflow ETL
Budget / Salary€250–750
TypeFreelance project
LocationRemote
Posted12 minutes ago
Our Airflow-driven nightly ETL has been crashing four times a week for the past ten days. Each morning someone manually restarts the failed tasks, yet the root cause remains unknown. The stack is plain and simple—Airflow, Python, and Postgres—and I do not want a full rewrite. I need the existing workflow made dependable, observable, and well-documented.
Scope of work
• Diagnose why tasks error out or hang and implement targeted fixes so the DAG finishes unaided every night.
• Add sensible alerts, metrics, and log retention so future anomalies surface instantly with enough context to act. Popular choices such as Airflow built-in email/SMS, Slack webhooks, and Grafana/Prometheus are all acceptable as long as they fit smoothly into the current setup.
• Produce concise documentation covering architecture, failure modes, recovery steps, and any code changes, written so a teammate can maintain the pipeline without guesswork.
Acceptance criteria
1. The DAG completes successfully five consecutive weeknights in a staging run, then five nights in production.
2. Any subsequent failure raises an alert within minutes, citing the failing task and stack trace.
3. Documentation is committed next to the codebase and reviewed with me before project sign-off.
You will need at least three years of hands-on data engineering experience and enough CET time-zone overlap for real-time troubleshooting calls. If you can start Monday, message me with a brief overview of similar stability projects you have handled and which monitoring tools you plan to lean on.
Scope of work
• Diagnose why tasks error out or hang and implement targeted fixes so the DAG finishes unaided every night.
• Add sensible alerts, metrics, and log retention so future anomalies surface instantly with enough context to act. Popular choices such as Airflow built-in email/SMS, Slack webhooks, and Grafana/Prometheus are all acceptable as long as they fit smoothly into the current setup.
• Produce concise documentation covering architecture, failure modes, recovery steps, and any code changes, written so a teammate can maintain the pipeline without guesswork.
Acceptance criteria
1. The DAG completes successfully five consecutive weeknights in a staging run, then five nights in production.
2. Any subsequent failure raises an alert within minutes, citing the failing task and stack trace.
3. Documentation is committed next to the codebase and reviewed with me before project sign-off.
You will need at least three years of hands-on data engineering experience and enough CET time-zone overlap for real-time troubleshooting calls. If you can start Monday, message me with a brief overview of similar stability projects you have handled and which monitoring tools you plan to lean on.
Apply on Freelancer →
Project sourced from Freelancer.com. Applications happen directly on the original platform — we never collect your data.