Senior IA Inference Engineer (x/f/m)

<h2 class="pb-2 pt-4 heading-xl text-foreground"><strong class="font-semibold text-foreground">Set a new pulse for healthcare!</strong></h2> <div class="whitespace-pre-wrap break-words font-normal first:pt-0 last:pb-0 py-1 @sm:pt-2 @sm:pb-[10px] @sm:leading-relaxed text-base leading-relaxed text-foreground">We are looking for a <strong class="font-semibold text-foreground">Senior Inference Engineer</strong> to join the <strong class="font-semibold text-foreground">ML Platform (TIGER)</strong> team in <strong class="font-semibold text-foreground">AI &amp; Machine Learning</strong>.</div> <div class="whitespace-pre-wrap break-words font-normal first:pt-0 last:pb-0 py-1 @sm:pt-2 @sm:pb-[10px] @sm:leading-relaxed text-base leading-relaxed text-foreground">Your mission will be to power Doctolib's AI-driven healthcare products — from Consultation Assistant to Phone Assistant and Health Companion — by owning the self-hosted LLM serving platform that keeps the most sensitive patient data within infrastructure Doctolib controls. You will build and operate the shared foundations that every AI feature team relies on: deploying, tuning, and scaling open-weight models across our GPU fleet, contributing directly to faster, safer, and more cost-efficient AI for care teams and patients across Europe.</div> <div class="whitespace-pre-wrap break-words font-normal first:pt-0 last:pb-0 py-1 @sm:pt-2 @sm:pb-[10px] @sm:leading-relaxed text-base leading-relaxed text-foreground">Working in the tech team at Doctolib means building innovative products and features to improve the daily lives of care teams and patients.</div> <h2 class="pb-2 pt-4 heading-xl text-foreground"><strong class="font-semibold text-foreground">What you'll do</strong></h2> <div class="whitespace-pre-wrap break-words font-normal first:pt-0 last:pb-0 py-1 @sm:pt-2 @sm:pb-[10px] @sm:leading-relaxed text-base leading-relaxed text-foreground">Your responsibilities include but are not limited to:</div> <ul class="pb-2 flex flex-col gap-1 list-disc pl-6 text-foreground text-base leading-relaxed"> <li class="break-words text-foreground text-base leading-relaxed">Build the shared LLM serving foundations with MLOps engineers — define the serving stack, GPU capacity planning, and deployment automation for open-weight models (Gemma, Mistral, Llama, Qwen, and their fine-tuned variants) on Ray Serve / Anyscale, while shaping the evolution toward Kubernetes-based orchestration</li> <li class="break-words text-foreground text-base leading-relaxed">Own the model registry and promotion path — establish clear processes for versioning, validating, and promoting models into production, benchmark performance and output quality under realistic workloads, and maintain the curated model catalog together with the Agents Platform team</li> <li class="break-words text-foreground text-base leading-relaxed">Optimize inference performance and efficiency — operate and tune serving engines (vLLM, Triton Inference Server, TensorRT-LLM) to reduce time-to-first-token, inter-token latency, and cost per token through batching, caching, quantization, and parallelism strategies across our GPU fleet (L4 to H200)</li> <li class="break-words text-foreground text-base leading-relaxed">Own production reliability and disaster recovery — define SLOs, run load tests before launches, implement safe rollouts, rollbacks, and autoscaling, and participate in on-call and incident response</li> <li class="break-words text-foreground text-base leading-relaxed">Make serving performance and costs measurable — build Datadog dashboards, alerting, and cost reporting to track latency, GPU utilization, and inference spend, and identify opportunities to improve resource efficiency</li> <li class="break-words text-foreground text-base leading-relaxed">Keep the inference stack secure, compliant, and evolving — work with Security and Privacy on network isolation, data residency, and auditability, while continuously assessing new models, serving engines (such as SGLang), and GPU generations</li> </ul> <h2 class="pb-2 pt-4 heading-xl text-foreground"><strong class="font-semibold text-foreground">Who you are</strong></h2> <div class="whitespace-pre-wrap break-words font-normal first:pt-0 last:pb-0 py-1 @sm:pt-2 @sm:pb-[10px] @sm:leading-relaxed text-base leading-relaxed text-foreground">Before you read on: if you don't have the exact profile described below, but you feel this job description matches your skill set, we still encourage you to apply.</div> <div class="whitespace-pre-wrap break-words font-normal first:pt-0 last:pb-0 py-1 @sm:pt-2 @sm:pb-[10px] @sm:leading-relaxed text-base leading-relaxed text-foreground"><strong class="font-semibold text-foreground">You'll be a great fit if you:</strong></div> <ul class="pb-2 flex flex-col gap-1 list-disc pl-6 text-foreground text-base leading-relaxed"> <li class="break-words text-foreground text-base leading-relaxed">Have 5+ years of software or computer engineering experience, including hands-on production experience with at least one LLM serving engine (vLLM, SGLang, TensorRT-LLM, TGI, or Triton) — ideally having chosen, introduced, tuned, or scaled one in production, not just run it with defaults</li> <li class="break-words text-foreground text-base leading-relaxed">Are proficient in Python and Terraform, with a solid understanding of transformer inference mechanics: prefill vs. decode, KV cache, batching strategies, and why a model is compute- or memory-bound on a given GPU</li> <li class="break-words text-foreground text-base leading-relaxed">Have strong experience running GPU workloads on Kubernetes or a managed Ray / Anyscale platform, and a performance engineering mindset — you design reproducible benchmarks, read profiler output, and back decisions with numbers</li> <li class="break-words text-foreground text-base leading-relaxed">Are comfortable owning production reliability: observability, incident response, and on-call are a natural part of the role</li> <li class="break-words text-foreground text-base leading-relaxed">Have excellent communication and collaboration skills, high agency, a pragmatic build-vs-buy approach, and are comfortable working in a scope that is still taking shape</li> <li class="break-words text-foreground text-base leading-relaxed">Are fluent in English</li> </ul> <div class="whitespace-pre-wrap break-words font-normal first:pt-0 last:pb-0 py-1 @sm:pt-2 @sm:pb-[10px] @sm:leading-relaxed text-base leading-relaxed text-foreground"><strong class="font-semibold text-foreground">It would be fantastic if you:</strong></div> <ul class="pb-2 flex flex-col gap-1 list-disc pl-6 text-foreground text-base leading-relaxed"> <li class="break-words text-foreground text-base leading-relaxed">Have experience with distributed serving frameworks (KServe, llm-d, NVIDIA Dynamo), including disaggregated prefill/decode or KV-cache-aware routing, or with serving Mixture-of-Experts, multimodal models, or many LoRA adapters on shared base models</li> <li class="break-words text-foreground text-base leading-relaxed">Have GPU kernel experience or deep profiling skills (CUDA, Triton, Nsight), or have contributed to open-source inference projects</li> <li class="break-words text-foreground text-base leading-relaxed">Have experience in regulated environments (healthcare, finance, public sector) and with frameworks such as GDPR, HDS, ISO 27001, or SOC 2</li> </ul> <h2 class="pb-2 pt-4 heading-xl text-foreground"><strong class="font-semibold text-foreground">Our stack</strong></h2> <ul class="pb-2 flex flex-col gap-1 list-disc pl-6 text-foreground text-base leading-relaxed"> <li class="break-words text-foreground text-base leading-relaxed"><strong class="font-semibold text-foreground">Programming languages:</strong> Python, plus comfort reading/debugging a systems language (C++, Rust, or Go)</li> <li class="break-words text-foreground text-base leading-relaxed"><strong class="font-semibold text-foreground">Cloud providers:</strong> GCP / AWS / Azure</li> <li class="break-words text-foreground text-base leading-relaxed"><strong class="font-semibold text-foreground">Serving orchestration:</strong> Ray Serve / Anyscale (GCP), with Kubernetes (GKE) + GPU scheduling as an emerging parallel target</li> <li class="break-words text-foreground text-base leading-relaxed"><strong class="font-semibold text-foreground">LLM serving engines (in production):</strong> vLLM, Triton Inference Server, TensorRT-LLM (also evaluating SGLang and other emerging engines)</li> <li class="break-words text-foreground text-base leading-relaxed"><strong class="font-semibold text-foreground">Models:</strong> Self-hosted open-weight LLMs (Gemma, Mistral, Llama, Qwen), fine-tuned in-house</li> <li class="break-words text-foreground text-base leading-relaxed"><strong class="font-semibold text-foreground">Experiment tracking &amp; model registry:</strong> MLflow</li> <li class="break-words text-foreground text-base leading-relaxed"><strong class="font-semibold text-foreground">GenAI gateway:</strong> LiteLLM</li> <li class="break-words text-foreground text-base leading-relaxed"><strong class="font-semibold text-foreground">LLM evaluation &amp; observability:</strong> Braintrust</li> <li class="break-words text-foreground text-base leading-relaxed"><strong class="font-semibold text-foreground">GPUs:</strong> NVIDIA L4, A10G, L40S, RTX 6000 Pro, A100 (40GB/80GB), H200</li> <li class="break-words text-foreground text-base leading-relaxed"><strong class="font-semibold text-foreground">Container / Infra as code:</strong> Docker, Terraform, Helm, Kustomize</li> <li class="break-words text-foreground text-base leading-relaxed"><strong class="font-semibold text-foreground">Observability:</strong> Datadog</li> <li class="break-words text-foreground text-base leading-relaxed"><strong class="font-semibold text-foreground">Data warehouse / storage:</strong> GCP BigQuery / GCS / AWS S3</li> <li class="break-words text-foreground text-base leading-relaxed">And any other tool you deem relevant!</li> </ul> <h2 class="pb-2 pt-4 heading-xl text-foreground"><strong class="font-semibold text-foreground">Life at Doctolib Tech</strong></h2> <ul class="pb-2 flex flex-col gap-1 list-disc pl-6 text-foreground text-base leading-relaxed"> <li class="break-words text-foreground text-base leading-relaxed">Our solutions are built on a single fully cloud-native platform that supports web and mobile app interfaces, multiple languages, and is adapted to country and healthcare specialty requirements.</li> <li class="break-words text-foreground text-base leading-relaxed">Our stack is composed of Rails, TypeScript, Java, Python, Kotlin, Swift, and React Native.</li> <li class="break-words text-foreground text-base leading-relaxed">We leverage AI ethically across our products to empower patients and health professionals. Discover our AI vision <a class="break-all font-semibold transition-all duration-200 ease-in-out hover:underline text-highlight hover:text-highlight-400 active:text-highlight-dark" href="https://careers.doctolib.com/ai-careers/" target="_blank">here</a>.</li> </ul> <div class="whitespace-pre-wrap break-words font-normal first:pt-0 last:pb-0 py-1 @sm:pt-2 @sm:pb-[10px] @sm:leading-relaxed text-base leading-relaxed text-foreground"><em>Want to learn more about our tech culture and environment? Visit the <a class="break-all font-semibold transition-all duration-200 ease-in-out hover:underline text-highlight hover:text-highlight-400 active:text-highlight-dark" href="https://careers.doctolib.com/tech-doctolib/" target="_blank">Doctolib Tech site</a>.</em></div> <h2 class="pb-2 pt-4 heading-xl text-foreground"><strong class="font-semibold text-foreground">What we offer</strong></h2> <ul class="pb-2 flex flex-col gap-1 list-disc pl-6 text-foreground text-base leading-relaxed"> <li class="break-words text-foreground text-base leading-relaxed">Free comprehensive health insurance (basic package) for you an