Build Scalable LLM Training Infrastructure

via Freelancer ·

Budget / SalaryHourly project
TypeFreelance project
LocationRemote
Posted1 hour ago
We are seeking a highly experienced Distributed AI Infrastructure Engineer to design, develop, and optimize large-scale AI training systems.

The ideal candidate will have deep expertise in distributed computing, GPU architecture, high-performance networking, and large language model (LLM) training infrastructure.

This role involves solving complex engineering challenges related to multi-node GPU synchronization, distributed fault tolerance, memory optimization, and high-performance AI workloads.

Key Responsibilities

Design and implement distributed training infrastructure supporting 100+ GPUs across multiple compute nodes.

Develop and optimize large-scale LLM training pipelines using PyTorch, DeepSpeed, FSDP, and Megatron-LM.

Implement advanced parallelism strategies, including tensor, pipeline, and data parallelism.

Optimize NCCL communication, GPU memory utilization, and inter-node network performance.

Build fault-tolerant training systems with automated checkpointing and recovery.

Investigate distributed deadlocks, GPU synchronization failures, and performance bottlenecks.

Develop scalable GPU orchestration solutions using Kubernetes, Slurm, or Ray.

Implement real-time monitoring and performance analysis for distributed AI workloads.

Ensure training consistency, reliability, and efficient resource utilization.

Required Qualifications

5+ years of experience in distributed systems, machine learning infrastructure, or high-performance computing.

Strong proficiency in Python, C++, and PyTorch.

Advanced understanding of CUDA, NCCL, GPU memory management, and distributed communication.

Hands-on experience with multi-node, multi-GPU training.

Experience with NVIDIA GPU architectures, InfiniBand, and high-speed networking.

Strong knowledge of Linux, Docker, Kubernetes, and cloud infrastructure.

Proven ability to troubleshoot complex distributed training failures.

Preferred Qualifications

Experience training or fine-tuning models with 70B+ parameters.

Experience with NVIDIA H100/H200 GPU clusters.

Knowledge of custom CUDA kernels and Triton.

Experience optimizing large-scale AI training efficiency.

Familiarity with distributed checkpointing, ZeRO optimization, and hybrid parallelism.

Technical Challenge

The successful candidate should be capable of diagnosing and resolving complex distributed training failures involving GPU communication deadlocks, inconsistent checkpoints, network interruptions, and unexpected model convergence issues.

Our Goal

Build a highly scalable, fault-tolerant AI training infrastructure capable of efficiently training next-generation large language models while minimizing computational costs and maximizing system reliability.
java wireless finite element analysis cuda machine learning (ml) network administration ai (artificial intelligence) hw/sw deep learning distributed systems large language models (llms)
Apply on Freelancer →

Project sourced from Freelancer.com. Applications happen directly on the original platform — we never collect your data.