Experienced Data Engineer for Healthcare Application

Data Engineer to design, build, and operate scalable data solutions supporting healthcare applications, analytics, product capabilities, and external data integrations. This engineer will work closely with Data Engineering, Product, Application Engineering, Analytics, and business stakeholders to build reliable pipelines and data models while maintaining strong standards for data quality, security, performance, and production reliability.

The ideal candidate is a strong hands-on engineer with deep Python and SQL experience, distributed data-processing knowledge, cloud experience—preferably Microsoft Azure—and the ability to independently design and troubleshoot production data systems.

Key Responsibilities
- Design, build, and maintain scalable batch and data-integration pipelines for structured and semi-structured healthcare and business data.
- Develop production-quality solutions using Python and advanced SQL/T-SQL.
- Build distributed data-processing workloads using Apache Spark / PySpark and Databricks.
- Design reliable ingestion, transformation, validation, and external data-export processes.
- Develop and maintain data models that make data reliable and understandable for applications, analytics, and business consumers.
- Implement strong data-quality controls, reconciliation, schema validation, error handling, monitoring, retries, and observability.
- Optimize SQL, Spark, and Databricks workloads for performance, scalability, and cost.
- Design solutions that are fault tolerant, maintainable, testable, and production ready.
- Build and integrate REST APIs and data interfaces where required.
- Create design documents, evaluate technologies, and develop proof-of-concepts for new solutions.
- Support production systems, investigate incidents, perform root-cause analysis, and implement long-term fixes.
- Write unit, integration, and functional tests.
- Build and maintain cloud infrastructure using Terraform / Infrastructure as Code.
- Participate in CI/CD, code reviews, architecture reviews, and engineering-standard development.

Required Technical Skills
Strong / core requirements: Python, SQL/T-SQL, data structures and software-design patterns, ETL/ELT and data-pipeline architecture, data modeling, data quality, performance tuning, cloud data platforms, REST APIs/data integrations, Git and CI/CD, and Terraform/IaC.
A strong candidate should also understand stored procedures, triggers, indexing/query optimization, scalable pipeline design, failure recovery, idempotency, schema evolution, testing, monitoring, and production support.

Preferred Technology Stack
Azure: Azure Data Lake Storage, Azure Data Factory, Azure Databricks, Azure SQL/SQL Server, Key Vault, Entra ID/RBAC, Azure monitoring services.
Data engineering: Databricks, Apache Spark, PySpark, Delta Lake, SQL Server, orchestration/workflow technologies and modern lakehouse/data-platform patterns.
Infrastructure: Terraform, CI/CD and automated deployment practices.
Azure experience would be preferred, although strong AWS or GCP experience combined with the ability to transition into Azure would also be relevant.

Healthcare Experience — Preferred
Healthcare experience would be valuable, particularly working with:
Claims • Eligibility • Members • Providers • Healthcare data exchanges • HIPAA/PHI • HL7 • FHIR • X12
Useful X12 knowledge could include 837 claims, 835 remittance, 270/271 eligibility and 834 enrollment.