Project overview
Project is a business intelligence type of solution dealing with data from different sources: logs, site visitors, CRM lists or purchase history data, etc. An EMR based solution is used to analyze and develop person-based marketing campaigns that can be deployed to any of the media platforms. Tech stack: AWS: S3, EC2, Elastic Beanstalk, Aurora RDS, EMR DB: MySQL (using Aurora DB from AWS) Hadoop: Spark on top of AWS EMR ETL: Apache Nifi correction: AWS: S3, EC2, SQS, Aurora RDS, EMR Required: Experience as a Data Engineer Knowledge of database design Experience in Data warehousing, ETL, Data Quality, batch and stream processing. Knowledge of Big Data stack - Hadoop, Hive, Spark, Kafka, etc. Strong knowledge of SQL Strong knowledge of object-oriented programming and design patterns Knowledge of Scala, or proficient in a typed or functional language Knowledge of NoSQL databases Bachelor or Masters degree in Engineering/Computer Science/Math or related discipline Desired: Experience with Google Cloud, AWS or Azure Experience with Big Query Experience with Kubernetes Experience with Scrum Experience with BDD Knowledge of Machine Learning