Resolve 3 Dysfunctional Airflow DAGs (Databricks)
Budget / Salary₹12,500–37,500
TypeFreelance project
LocationRemote
Posted2 hours ago
Title: Urgent: Debug & Fix 3 Failing Airflow DAGs (Databricks Notebook Orchestration)
Description:
I have 3 Airflow DAGs already deployed and visible in our Airflow instance, but they are currently failing on execution. I need a freelancer to diagnose the root cause of each failure and provide fixes so all three run successfully.
The three DAGs:
[Profile Load DAG] — [brief description if you have one]
[Load DAG] — [brief description if you have one]
[PAU DAG] — [brief description if you have one]
Each DAG orchestrates a sequence of Databricks notebook tasks (batch acquire → config-driven processing steps → batch complete), reading from PostgreSQL-backed control/checkpoint tables.
What I need:
Review the failure logs/error traces for each of the 3 DAGs in Airflow
Identify the root cause of each failure (could be config mismatches, permission issues, path/naming issues, parameter mismatches between tasks, or notebook-level bugs)
Propose and implement fixes
Confirm all 3 DAGs run end-to-end successfully after the fix
Access provided: Airflow UI access, Databricks workspace access, relevant notebook and DAG code — shared privately with the selected freelancer.
Ideal candidate: Experience with Apache Airflow (DAG debugging, task logs, Variable/connection troubleshooting), Databricks notebooks (PySpark/Python), and SQL-based control/checkpoint table patterns.
Description:
I have 3 Airflow DAGs already deployed and visible in our Airflow instance, but they are currently failing on execution. I need a freelancer to diagnose the root cause of each failure and provide fixes so all three run successfully.
The three DAGs:
[Profile Load DAG] — [brief description if you have one]
[Load DAG] — [brief description if you have one]
[PAU DAG] — [brief description if you have one]
Each DAG orchestrates a sequence of Databricks notebook tasks (batch acquire → config-driven processing steps → batch complete), reading from PostgreSQL-backed control/checkpoint tables.
What I need:
Review the failure logs/error traces for each of the 3 DAGs in Airflow
Identify the root cause of each failure (could be config mismatches, permission issues, path/naming issues, parameter mismatches between tasks, or notebook-level bugs)
Propose and implement fixes
Confirm all 3 DAGs run end-to-end successfully after the fix
Access provided: Airflow UI access, Databricks workspace access, relevant notebook and DAG code — shared privately with the selected freelancer.
Ideal candidate: Experience with Apache Airflow (DAG debugging, task logs, Variable/connection troubleshooting), Databricks notebooks (PySpark/Python), and SQL-based control/checkpoint table patterns.
Apply on Freelancer →
Project sourced from Freelancer.com. Applications happen directly on the original platform — we never collect your data.