Experienced Azure Data Engineer Required.

Freelance Azure Data Engineer

Project Type

Personal Project – Freelance / Part-time

Role

Azure Data Engineer – Azure Databricks

Experience

3–5 years of relevant hands-on experience preferred.

Project Overview

We are looking for an experienced Azure Data Engineer to support a data engineering project involving data ingestion, transformation, validation, and processing using Microsoft Azure services.

The project involves working with structured and semi-structured data, including healthcare/insurance-related data.

Required Technical Skills

- Azure Databricks
- PySpark / Python
- Strong SQL / SQL Server
- Azure Data Factory (ADF)
- Azure Data Lake Storage Gen2 (ADLS Gen2)
- Delta Lake
- Databricks SQL
- ETL / ELT development
- Incremental and historical data processing
- Delta MERGE / Upsert operations
- JSON parsing and nested JSON handling
- Data deduplication
- Data validation and source-to-target reconciliation
- Performance optimization in Databricks
- Git / basic CI-CD knowledge

Preferred Domain Experience

Experience in any of the following is preferred:

- Healthcare
- Insurance
- Member/Customer data
- Claims
- Policy/Benefits data
- FHIR / HL7
- Healthcare data integration

Responsibilities

- Develop and maintain Azure Databricks notebooks using PySpark.
- Build data ingestion pipelines using ADF and Azure services.
- Process data from SQL Server and other sources.
- Implement historical and incremental data processing.
- Create and maintain Delta tables.
- Implement MERGE/Upsert logic for incremental data.
- Parse and transform nested JSON data.
- Perform data cleansing, deduplication, and transformation.
- Perform source-to-target data validation and reconciliation.
- Troubleshoot data pipeline and Databricks issues.
- Optimize Spark jobs and SQL queries where required.
- Document the implemented solution and provide knowledge transfer.

Ideal Candidate

The ideal candidate should have strong hands-on experience rather than only theoretical knowledge.

The candidate should be comfortable explaining and implementing a complete flow such as:

SQL Server / Event-based Source
→ ADF / Databricks
→ ADLS Gen2
→ Delta Lake
→ Transformation & Validation
→ Unified Target

Engagement

- Freelance / Part-time
- Remote
- Project-based
- Duration: To be discussed
- Working hours: Flexible, based on project requirements
- Payment: Hourly or milestone-based, depending on experience

How to Apply

Please share:

1. Total years of Azure Data Engineering experience
2. Azure Databricks experience
3. PySpark experience
4. SQL/SQL Server experience
5. ADF and ADLS Gen2 experience
6. Healthcare/Insurance project experience, if any
7. Availability (hours per week)
8. Expected hourly/project rate
9. Brief description of similar projects handled

Location: Pune, Maharashtra
Work mode: On-site / In-person
Candidate must be available to work from my location in Pune.