Sales Data Pipeline Development
Budget / Salary$30–250
TypeFreelance project
LocationRemote
Posted1 hour ago
I need a small sales data pipeline (proof of concept) that runs directly on Windows 10/11. NO Docker. Delivery in 4 days, $100-$300 fixed, 2 milestones.
DATA FLOW
C++ simulator -> MQTT (Mosquitto) -> C# .NET collector -> SQLite -> Python loader -> Hadoop HDFS -> sales reports
WHAT TO BUILD
1) C++ sales simulator
- Simulates 10 stores and 50 products, sends each sale as JSON to MQTT (QoS 1)
- Each sale: message_id, store_id, product_id, quantity, price, timestamp (UTC)
- Settings: number of messages and speed. Stops after N messages (default 50,000)
- Sends a few invalid messages on purpose (for testing)
2) C# .NET 8 collector
- Subscribes to MQTT (e.g. MQTTnet) with a persistent session, so no data is lost when it restarts
- Checks every message: good rows -> "sales" table, bad rows -> "rejected" table
- No duplicates (unique message_id)
3) SQLite database
- Tables: sales, rejected, load_log
4) Python loader -> Hadoop HDFS
- Copies new SQLite rows to HDFS as CSV, in folders by date: /sales/dt=YYYY-MM-DD/ (UTC)
- Safe to run twice: no duplicates, no missing rows
- Lock, so two loaders cannot run at the same time
5) Hadoop sales reports
- Hadoop Streaming jobs with Python mapper/reducer. Output 5 CSV files:
1. Total sales per day
2. Sales per store per day
3. Top 10 products by revenue
4. Sales by hour (UTC)
5. Average basket value per store
6) Verify script
- One command that checks: sent = SQLite = HDFS = report totals. Prints OK or FAIL.
TECHNICAL REQUIREMENTS
- NO Docker, NO containers of any kind
- On Windows: Mosquitto (Windows installer), .NET 8, Python 3.11+, C++ with CMake (+ vcpkg)
- Hadoop 3.3.x single node in WSL2 Ubuntu (Hadoop does not run well on Windows directly; WSL2 is OK)
- All ports bound to 127.0.0.1 only (MQTT 1883, Hadoop web UI 9870)
- All times in UTC
- All settings in one config file. No hard-coded paths or passwords.
- Setup scripts (PowerShell + bash) and a clear step-by-step README
- Unit tests for the collector and the loader
MILESTONES
Milestone 1 (40%) - Day 2:
C++ simulator + Mosquitto + C# collector + SQLite working.
50,000 messages sent = 50,000 rows stored (sales + rejected), 0 duplicates.
Milestone 2 (60%) - Day 4:
Hadoop setup, Python loader, 5 reports, verify script, final README, short screen recording of a full run.
ACCEPTANCE - I will run everything on my own Windows machine before I release payment
- Fresh install works by following only the README, without Docker
- Collector stopped for 60 seconds, then started again: 0 messages lost
- Loader run twice: 0 duplicates in HDFS
- Verify script prints OK, and the 5 report CSVs are correct
DELIVERY
- One ZIP per milestone: full source code, build and setup scripts, README, sample config
- Full ownership of the code goes to me
HOW TO BID
- Start your bid with the word PIPELINE so I know you read this
- Tell me your experience with MQTT and Hadoop (links to similar work if possible)
- Confirm you can deliver in 4 days without Docker
- Your fixed price
DATA FLOW
C++ simulator -> MQTT (Mosquitto) -> C# .NET collector -> SQLite -> Python loader -> Hadoop HDFS -> sales reports
WHAT TO BUILD
1) C++ sales simulator
- Simulates 10 stores and 50 products, sends each sale as JSON to MQTT (QoS 1)
- Each sale: message_id, store_id, product_id, quantity, price, timestamp (UTC)
- Settings: number of messages and speed. Stops after N messages (default 50,000)
- Sends a few invalid messages on purpose (for testing)
2) C# .NET 8 collector
- Subscribes to MQTT (e.g. MQTTnet) with a persistent session, so no data is lost when it restarts
- Checks every message: good rows -> "sales" table, bad rows -> "rejected" table
- No duplicates (unique message_id)
3) SQLite database
- Tables: sales, rejected, load_log
4) Python loader -> Hadoop HDFS
- Copies new SQLite rows to HDFS as CSV, in folders by date: /sales/dt=YYYY-MM-DD/ (UTC)
- Safe to run twice: no duplicates, no missing rows
- Lock, so two loaders cannot run at the same time
5) Hadoop sales reports
- Hadoop Streaming jobs with Python mapper/reducer. Output 5 CSV files:
1. Total sales per day
2. Sales per store per day
3. Top 10 products by revenue
4. Sales by hour (UTC)
5. Average basket value per store
6) Verify script
- One command that checks: sent = SQLite = HDFS = report totals. Prints OK or FAIL.
TECHNICAL REQUIREMENTS
- NO Docker, NO containers of any kind
- On Windows: Mosquitto (Windows installer), .NET 8, Python 3.11+, C++ with CMake (+ vcpkg)
- Hadoop 3.3.x single node in WSL2 Ubuntu (Hadoop does not run well on Windows directly; WSL2 is OK)
- All ports bound to 127.0.0.1 only (MQTT 1883, Hadoop web UI 9870)
- All times in UTC
- All settings in one config file. No hard-coded paths or passwords.
- Setup scripts (PowerShell + bash) and a clear step-by-step README
- Unit tests for the collector and the loader
MILESTONES
Milestone 1 (40%) - Day 2:
C++ simulator + Mosquitto + C# collector + SQLite working.
50,000 messages sent = 50,000 rows stored (sales + rejected), 0 duplicates.
Milestone 2 (60%) - Day 4:
Hadoop setup, Python loader, 5 reports, verify script, final README, short screen recording of a full run.
ACCEPTANCE - I will run everything on my own Windows machine before I release payment
- Fresh install works by following only the README, without Docker
- Collector stopped for 60 seconds, then started again: 0 messages lost
- Loader run twice: 0 duplicates in HDFS
- Verify script prints OK, and the 5 report CSVs are correct
DELIVERY
- One ZIP per milestone: full source code, build and setup scripts, README, sample config
- Full ownership of the code goes to me
HOW TO BID
- Start your bid with the word PIPELINE so I know you read this
- Tell me your experience with MQTT and Hadoop (links to similar work if possible)
- Confirm you can deliver in 4 days without Docker
- Your fixed price
Apply on Freelancer →
Project sourced from Freelancer.com. Applications happen directly on the original platform — we never collect your data.