Modern Data Stack ELT Pipeline

END-TO-END MODERN DATA STACK (MDS) DATA ENGINEERING PIPELINE

Technology Stack:
AWS RDS (MySQL) → Fivetran → Snowflake → Power BI

PROJECT OVERVIEW

Designed and implemented an end-to-end cloud data engineering pipeline to ingest, transform, model, and visualize e-commerce data using modern data stack technologies.

Built a production-style ELT pipeline where transactional data is stored in MySQL hosted on AWS RDS and automatically ingested into Snowflake using Fivetran with incremental CDC-based synchronization.

DATA ARCHITECTURE

AWS RDS MySQL
↓
Fivetran – Incremental CDC Ingestion
↓
Snowflake RAW Layer
↓
Streams + Tasks + SQL Transformations
↓
Snowflake STAGING Layer
↓
Star Schema / Dimensional Modeling
↓
Snowflake ANALYTICS Layer
↓
Power BI DirectQuery Dashboard

KEY IMPLEMENTATION

• Configured AWS RDS MySQL as the transactional source system.

• Implemented Fivetran MySQL CDC / Binary Log replication for automated incremental data ingestion.

• Designed a multi-layer Snowflake data warehouse architecture.

• Created RAW, STAGING, and ANALYTICS layers for structured data processing.

• Used Snowflake Streams to capture incremental data changes.

• Used Snowflake Tasks to automate scheduled data transformations.

• Implemented SQL-based data cleansing, validation, and business transformations.

• Designed a Star Schema with FACT_SALES, DIM_DATE, DIM_CUSTOMER, DIM_PRODUCT, and DIM_REGION.

• Configured AWS S3 integration for secure bulk data staging and data movement.

• Connected Snowflake to Power BI using DirectQuery.

• Created KPI calculations and business dashboards using DAX.

DATA LAYERS

RAW Layer:
Stores source data ingested through Fivetran with minimal modification.

STAGING Layer:
Performs data cleaning, validation, standardization, and transformation using SQL.

ANALYTICS Layer:
Contains the dimensional Star Schema optimized for reporting and business analytics.

BUSINESS OUTCOME

The pipeline automates the movement of transactional data from the source database to the analytics layer, reducing manual data processing and enabling near-real-time reporting through Power BI.

This project demonstrates practical experience with cloud data integration, CDC, ELT architecture, Snowflake data warehousing, dimensional modeling, workflow automation, and business intelligence reporting.

PROJECT DETAILS

Project: End-to-End Modern Data Stack (MDS) Pipeline

Developer: Ramkumar G

Technologies:
AWS RDS | MySQL | Fivetran | Snowflake | AWS S3 | SQL | Snowflake Streams | Snowflake Tasks | Power BI | DAX