Project overview
About the Client: Our client is modernizing the brokerage ecosystem. They are a diversified financial services firm that is replacing the legacy infrastructure used across capital markets. They started from scratch by building a completely cloud-native clearing and custody system designed for today’s complex, global market. Their proprietary prime brokerage platform adds significant efficiency to the market while focusing on minimizing risk, redundancy, and cost for clients. Their goal is to create a single source-of-truth platform for every asset class, in every country, and in any currency. By combining highly skilled product and engineering talent with seasoned finance professionals, our client is building the essentials to compete in today’s fast-paced markets. About the Role: Our client is seeking a Kafka Expert to support and optimize their data streaming infrastructure. The ideal candidate will have extensive experience in Kafka, including broker management, connector clusters, Kafka management UI, service and developer authentication, and transformers, with a proven ability to architect and manage large-scale streaming systems. This role includes working closely with the engineering team to improve the performance, scalability, and maintainability of their Kafka environment while integrating it with related technologies. Key Responsibilities: • Manage and optimize Kafka brokers, connector clusters, and associated management tools/UI. • Develop, test, and maintain Kafka transformers for real-time data processing. • Ensure high availability, reliability, and monitoring of Kafka infrastructure. • Collaborate with cross-functional teams to integrate Kafka with tools like Snowflake for data warehousing and analytics. • Work on containerized deployments leveraging Kubernetes. • Contribute to automation workflows using Argo or Airflow. • Evaluate and integrate other streaming technologies as needed to enhance system capabilities. Required Skills & Experience: • Proven expertise with Apache Kafka (brokers, connectors, UI tools, transformers). • Hands-on experience with Kafka Clusters and best practices for scaling and managing them. • Proficiency with related technologies: Snowflake, Kubernetes. • Strong analytical and problem-solving skills. • Experience working remotely with geographically distributed teams. Preferred Skills: • Experience with other streaming technologies (e.g., Red Panda, Apache Flink, Pulsar, or Spark Streaming). • Familiarity with cloud environments (AWS Required). • Understanding of DevOps practices, CI/CD pipelines, and monitoring tools. • Proficiency with related technologies: Argo, Airflow.