Intermediate Data Engineer

Intermediate Data Engineer
Minimum Requirements:

Matric (Grade 12)

Masters in Software Engineering, Data Engineering, Computer Science or related field

2-3 years of relevant work experience

Strong Scala or Java background

Knowledge of AWS, GCP, Azure, or other cloud platform

Understanding of data modeling principles

Ability to work with complex data models

Experience with relational and NoSQL databases (e.g. Postgres, ElasticSearch/OpenSearch, graph databases such as Neptune or neo4j)

Experience with technologies that power analytics (Spark, Hadoop, Kafka, Docker, Kubernetes) or other distributed computing systems

Knowledge of API development and machine learning deployment

Responsibilities:

Build and scale data infrastructure that powers real-time data processing of billions of records in a streaming architecture

Build scalable data ingestion and machine learning inference pipelines

Build general-purpose APIs to deliver data science outputs to multiple business units

Scale up production systems to handle increased demand from new products, features, and users

Provide visibility into the health of our data platform (comprehensive view of data flow, resources usage, data lineage, etc) and optimize cloud costs

Automate and handle the life-cycle of the systems and platforms that process our data

Originally posted on Himalayas