Databricks Data Cleaning & Loading

I have several relational database tables that need to be extracted into Databricks, cleaned, and reshaped before being loaded into a separate database / data-warehouse. The entire workflow should be scripted in Python (PySpark and SQL) and version-controlled in GitHub, with development ideally done from VS Code so that I can review and extend the notebooks locally.

Key points you’ll handle
• Connect to the source database from Databricks and pull the required tables.
• Use PySpark DataFrame APIs and SQL to remove duplicates, harmonise column types, apply business rules, and create the final fact and dimension structures.
• Write the transformed data back to the target warehouse in an efficient, partition-friendly format (Parquet or Delta Lake preferred).
• Push all notebooks / scripts and any helper modules to a GitHub repo with clear commit messages so that CI can be added later.

Acceptance criteria
– All transformations run end-to-end in Databricks without manual intervention.
– Resulting tables match the agreed schema and row counts between source and target reconcile within 0.1 %.
– Code is readable, modular, and PEP 8 compliant; Git history shows logical commits.

If this sounds straightforward to you and you’re comfortable with Databricks, PySpark, SQL, GitHub, and VS Code, let’s get started—clean code and reliable loads are what matter most.