Scale Email-Sending Platform with Laravel & ClickHouse
Budget / Salary$3,000–5,000
TypeFreelance project
LocationRemote
Posted2 hours ago
Title: Senior engineer — scale a Laravel + ClickHouse email-sending platform to 1M+/day
Description:
We run a Laravel/PHP email-sending platform (MySQL, ClickHouse) that delivers through ESP/SMTP relays (Amazon SES, SendGrid, Mailgun, SparkPost). It already has a full sending control plane — per-server warmup quotas, a 3-layer volume throttle, reservation accounting, provider cooldowns, SMTP classification, MX classification. It works today at low volume. We need a senior engineer to re-architect the hot paths so it reliably scales from current volume to 100k → 250k → 500k → ~1,000,000+ messages/day without violating per-domain / per-provider / warmup limits.
Concrete work areas:
Rate-accounting & distributed concurrency — we have both a file-based and a Redis-cache-reservation path; benchmark contention/correctness under multiple worker hosts and move hot counting to atomic, shardable accounting.
Recipient selection — replace a growing per-wave exclusion query with an indexed dispatch-ledger/cursor design.
Dispatch throughput — replace serial ~200-recipient waves with a design that keeps workers saturated without flooding the queue.
ClickHouse ingestion — replace one-insert-per-event with buffered/bulk batch inserts (this is a first-class requirement).
Queue backend + multi-host worker topology, MySQL indexing/EXPLAIN at 30M+ rows, load testing and observability.
Required (must have all):
Strong PHP/Laravel (queues, jobs, Eloquent)
Distributed systems / high concurrency: correct concurrent rate limiting, distributed locks, atomic counters (Redis/Lua), reservation & idempotency
Laravel queue/worker scaling (database vs Redis vs SQS), Supervisor, multi-host fleets
MySQL at scale: indexing, EXPLAIN, hot-row contention, tens of millions of rows
ClickHouse in production — MergeTree schema, batched/bulk ingestion, avoiding part-per-insert (must have operated it, not just queried it)
Proven experience operating a 500k–1M+/day email-sending system (this is the key differentiator)
Load testing + production observability
Nice to have: PowerMTA/Halon/MailerQ (only relevant if we later run our own MTA — not required now); deliverability fundamentals; ex-ESP engineer (SendGrid/Twilio, Mailgun, SES, SparkPost).
How this starts: a paid ~20–40 hour architecture + benchmark review first — assess the production system, benchmark the bottlenecks, and deliver a target architecture + migration plan + implementation estimate. Implementation is a separate, larger engagement awarded after that review.
To bid, you MUST answer these two questions (bids that don't will be ignored):
How would you batch our ClickHouse event ingestion (sends/opens/clicks/bounces) to avoid part-per-insert at ~1M events/day, and how do you handle failures/duplicates?
How would you design a distributed rate limiter so 20 workers across 3 hosts never exceed a per-provider hourly cap, with correct behavior if a worker dies mid-send?
Please link a specific project where you scaled email sending or high-throughput ClickHouse ingestion. Do not bid if your approach relies on rotating domains/IPs or "guaranteed inbox" tactics — we protect long-term sender reputation.
Description:
We run a Laravel/PHP email-sending platform (MySQL, ClickHouse) that delivers through ESP/SMTP relays (Amazon SES, SendGrid, Mailgun, SparkPost). It already has a full sending control plane — per-server warmup quotas, a 3-layer volume throttle, reservation accounting, provider cooldowns, SMTP classification, MX classification. It works today at low volume. We need a senior engineer to re-architect the hot paths so it reliably scales from current volume to 100k → 250k → 500k → ~1,000,000+ messages/day without violating per-domain / per-provider / warmup limits.
Concrete work areas:
Rate-accounting & distributed concurrency — we have both a file-based and a Redis-cache-reservation path; benchmark contention/correctness under multiple worker hosts and move hot counting to atomic, shardable accounting.
Recipient selection — replace a growing per-wave exclusion query with an indexed dispatch-ledger/cursor design.
Dispatch throughput — replace serial ~200-recipient waves with a design that keeps workers saturated without flooding the queue.
ClickHouse ingestion — replace one-insert-per-event with buffered/bulk batch inserts (this is a first-class requirement).
Queue backend + multi-host worker topology, MySQL indexing/EXPLAIN at 30M+ rows, load testing and observability.
Required (must have all):
Strong PHP/Laravel (queues, jobs, Eloquent)
Distributed systems / high concurrency: correct concurrent rate limiting, distributed locks, atomic counters (Redis/Lua), reservation & idempotency
Laravel queue/worker scaling (database vs Redis vs SQS), Supervisor, multi-host fleets
MySQL at scale: indexing, EXPLAIN, hot-row contention, tens of millions of rows
ClickHouse in production — MergeTree schema, batched/bulk ingestion, avoiding part-per-insert (must have operated it, not just queried it)
Proven experience operating a 500k–1M+/day email-sending system (this is the key differentiator)
Load testing + production observability
Nice to have: PowerMTA/Halon/MailerQ (only relevant if we later run our own MTA — not required now); deliverability fundamentals; ex-ESP engineer (SendGrid/Twilio, Mailgun, SES, SparkPost).
How this starts: a paid ~20–40 hour architecture + benchmark review first — assess the production system, benchmark the bottlenecks, and deliver a target architecture + migration plan + implementation estimate. Implementation is a separate, larger engagement awarded after that review.
To bid, you MUST answer these two questions (bids that don't will be ignored):
How would you batch our ClickHouse event ingestion (sends/opens/clicks/bounces) to avoid part-per-insert at ~1M events/day, and how do you handle failures/duplicates?
How would you design a distributed rate limiter so 20 workers across 3 hosts never exceed a per-provider hourly cap, with correct behavior if a worker dies mid-send?
Please link a specific project where you scaled email sending or high-throughput ClickHouse ingestion. Do not bid if your approach relies on rotating domains/IPs or "guaranteed inbox" tactics — we protect long-term sender reputation.
Apply on Freelancer →
Project sourced from Freelancer.com. Applications happen directly on the original platform — we never collect your data.