Senior Backend Engineer, Inference Platform

Together AI is building an inference platform for generative AI models. This Senior Backend Engineer role focuses on optimizing global request routing, load balancing, auto-scaling, and prefix caching across thousands of GPUs. The engineer will work with state-of-the-art accelerators (H100, H200, GB200) and collaborate with research teams to bring new model architectures into production. The role involves contributing to open source projects like SGLang, vLLM, and NVIDIA Dynamo.

5+ years building large-scale distributed systems and API microservices. Strong background in efficiency, scalability, and stability. Excellent understanding of OS concepts: multi-threading, memory management, networking, storage. Expert-level programming in Rust, Go, Python, or TypeScript. Knowledge of LLMs and production serving is a plus. Experience with SGLang, vLLM, NVIDIA Dynamo, Kubernetes, GPU software stacks (CUDA, Triton, NCCL), and HPC technologies (InfiniBand, NVLink, MPI) is valued.