Databricks Data Engineering Support

I’m looking for hands-on help building and optimising data-engineering workflows inside Databricks. My immediate focus is on ingesting data that arrives in two ways—batch loads from relational databases and real-time streams—then transforming, testing and storing it so downstream analytics teams can trust the output.

Here is what I need from you:

• Design and implement robust ingestion pipelines that pull incremental data from the source databases and process the streaming feeds with minimal latency.
• Develop scalable transformation logic inside notebooks (PySpark / SQL) and package reusable jobs so they can be scheduled and monitored through Databricks Workflows.
• Advise on cluster configuration, performance tuning and cost-efficient autoscaling as the data volumes grow.
• Document the solution clearly so future engineers can onboard quickly, and leave me with repeatable deployment steps (ideally via Databricks repos or CI/CD).

If you have a proven track record delivering similar data-engineering projects in Databricks and you’re comfortable juggling both relational and streaming inputs, I’d love to work together. Please share a brief note about your relevant experience and any links to comparable work you can show.