Data Engineer -- 2
Budget / SalaryHourly project
TypeFreelance project
LocationRemote
Posted1 hour ago
Onsite – In person, client site – Washington DC
Long Term
Salary Range : $100- $150/hour
Joining period : Immediate
About the Role
We are seeking a highly skilled Data Engineer to join our onsite team. In this role, you will design, build, and optimize complex, highly scalable big data pipelines and infrastructure on AWS to support advanced regulatory analytics, surveillance systems, and market research.
Key Responsibilities
Pipeline Development: Design, develop, and maintain high-performance ETL/ELT data pipelines capable of ingesting and transforming massive volumes of structured and unstructured financial data.
Big Data Processing: Utilize PySpark, Apache Spark, and AWS EMR to process billions of records efficiently, ensuring low latency and high reliability.
Cloud Architecture: Architect and optimize solutions within FINRA’s AWS data lake and data warehousing environment (S3, Athena, Glue, Snowflake, Presto/Trino).
Data Quality & Governance: Implement rigorous data quality checks, validation frameworks, and observability pipelines to ensure data lineage, consistency, and strict regulatory compliance.
Collaboration: Partner cross-functionally with data scientists, economists, software engineers, and regulatory stakeholders to translate complex business requirements into robust technical solutions.
CI/CD & DevOps: Deploy scalable infrastructure using CI/CD pipelines (Jenkins, Bitbucket, Git) and infrastructure-as-code (Terraform, CloudFormation).
Performance Optimization: Troubleshoot production data pipelines, tune PySpark/SQL queries for optimal compute performance, and manage AWS resource costs.
Required Qualifications
Experience: 5+ years of hands-on, dedicated Data Engineering experience in a production environment.
Programming Languages: Expert-level proficiency in Python (specifically for data engineering) and advanced SQL.
Big Data Ecosystem: Deep, hands-on experience with Apache Spark (PySpark) and distributed computing architecture.
Cloud Platforms: Strong implementation experience within the AWS ecosystem, specifically with EMR, S3, Glue, Athena, IAM, and Step Functions.
Data Architecture: Proven experience working with data lakes, modern data warehousing (e.g., Snowflake, Redshift), and data formats (Parquet, Apache Iceberg, Delta Lake).
Education: Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a closely related discipline.
Work Authorization: Eligible to work in US
Preferred Qualifications
Prior domain experience in financial services, capital markets, payroll, or regulatory environments.
Experience with streaming data technologies such as Apache Kafka or Amazon Kinesis.
AWS Certifications (e.g., AWS Certified Data Analytics, AWS Certified Solutions Architect).
Familiarity with AI-assisted developer tooling (GitHub Copilot, Amazon Q) and integrating AI into engineering workflows.
Excellent communication skills with the ability to explain complex technical concepts to non-technical financial stakeholders.
Long Term
Salary Range : $100- $150/hour
Joining period : Immediate
About the Role
We are seeking a highly skilled Data Engineer to join our onsite team. In this role, you will design, build, and optimize complex, highly scalable big data pipelines and infrastructure on AWS to support advanced regulatory analytics, surveillance systems, and market research.
Key Responsibilities
Pipeline Development: Design, develop, and maintain high-performance ETL/ELT data pipelines capable of ingesting and transforming massive volumes of structured and unstructured financial data.
Big Data Processing: Utilize PySpark, Apache Spark, and AWS EMR to process billions of records efficiently, ensuring low latency and high reliability.
Cloud Architecture: Architect and optimize solutions within FINRA’s AWS data lake and data warehousing environment (S3, Athena, Glue, Snowflake, Presto/Trino).
Data Quality & Governance: Implement rigorous data quality checks, validation frameworks, and observability pipelines to ensure data lineage, consistency, and strict regulatory compliance.
Collaboration: Partner cross-functionally with data scientists, economists, software engineers, and regulatory stakeholders to translate complex business requirements into robust technical solutions.
CI/CD & DevOps: Deploy scalable infrastructure using CI/CD pipelines (Jenkins, Bitbucket, Git) and infrastructure-as-code (Terraform, CloudFormation).
Performance Optimization: Troubleshoot production data pipelines, tune PySpark/SQL queries for optimal compute performance, and manage AWS resource costs.
Required Qualifications
Experience: 5+ years of hands-on, dedicated Data Engineering experience in a production environment.
Programming Languages: Expert-level proficiency in Python (specifically for data engineering) and advanced SQL.
Big Data Ecosystem: Deep, hands-on experience with Apache Spark (PySpark) and distributed computing architecture.
Cloud Platforms: Strong implementation experience within the AWS ecosystem, specifically with EMR, S3, Glue, Athena, IAM, and Step Functions.
Data Architecture: Proven experience working with data lakes, modern data warehousing (e.g., Snowflake, Redshift), and data formats (Parquet, Apache Iceberg, Delta Lake).
Education: Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a closely related discipline.
Work Authorization: Eligible to work in US
Preferred Qualifications
Prior domain experience in financial services, capital markets, payroll, or regulatory environments.
Experience with streaming data technologies such as Apache Kafka or Amazon Kinesis.
AWS Certifications (e.g., AWS Certified Data Analytics, AWS Certified Solutions Architect).
Familiarity with AI-assisted developer tooling (GitHub Copilot, Amazon Q) and integrating AI into engineering workflows.
Excellent communication skills with the ability to explain complex technical concepts to non-technical financial stakeholders.
Apply on Freelancer →
Project sourced from Freelancer.com. Applications happen directly on the original platform — we never collect your data.