IBM Research and Red Hat deployed a 753B open model on H100 GPUs, serving thousands of concurrent coding agents at 5-10x lower cost than commercial APIs.

Optimize inference for large language models (LLMs) with llm-d, reducing redundant computation and lowering AI workload costs.

Scale agentic AI with llm-d to achieve full infrastructure sovereignty across diverse hardware.

The llm-d router's default configuration is built on token-aware routing: keep each request on the cache-warm endpoint until a calibrated token-load limit is exceeded, then route by load alone,…

Optimize inference for large language models (LLMs) with llm-d, reducing redundant computation and lowering AI workload costs.

YouTube video by Red Hat

llm-d extends vLLM's Hybrid Memory Allocator across KV offloading to CPU and storage and KV-aware routing, making the offload connector HMA-aware - for 1.8–1.9x faster KV loads and about 115% higher…

Benchmarking llm-d's prefix-cache-aware routing across anonymized GPU pools on the NxtGen sovereign cloud, showing how one routing layer improves throughput and TTFT across single-vendor and…

llm-d v0.7 shifts focus from proving capabilities to making them deployable, with changes across deployment tooling, hardware support, documentation, and continuous integration.

llm-d v0.7 shifts focus from proving capabilities to making them deployable, with changes across deployment tooling, hardware support, documentation, and continuous integration.

Open Source Distributed AI Inference (llm-d/vLLM) Meetup Boston/Cambridge Hosted by Google Cloud, Red Hat AI, and the llm-d Community Date: Thursday, May 28th…

Open Source Distributed AI Inference (llm-d/vLLM) Meetup Boston/Cambridge Hosted by Google Cloud, Red Hat AI, and the llm-d Community Date: Thursday, May 28th…

In this video, Sally from SIG Observability demonstrates how to achieve comprehensive observability for Large Language Model (LLM) deployments using llm-d distributed tracing and vLLM NIXL metrics. …

How migrating from a simple vLLM deployment to a robust MLOps platform utilizing KServe, llm-d's intelligent routing, and vLLM solved significant scaling and operational challenges in LLM deployment…

How migrating from a simple vLLM deployment to a robust MLOps platform utilizing KServe, llm-d's intelligent routing, and vLLM solved significant scaling and operational challenges in LLM deployment…

How migrating from a simple vLLM deployment to a robust MLOps platform utilizing KServe, llm-d's intelligent routing, and vLLM solved significant scaling and operational challenges in LLM deployment t...

YouTube video by CNCF [Cloud Native Computing Foundation]

We are thrilled to announce that llm-d has officially been accepted as a Cloud Native Computing Foundation (CNCF) Sandbox project! As generative AI transitions from research labs to production…

We are thrilled to announce that llm-d has officially been accepted as a Cloud Native Computing Foundation (CNCF) Sandbox project! As generative AI transitions from research labs to production…

Deep technical sessions. Live demos. Real conversations. If you're deploying, or scaling LLM inference, this is the room to be in. Join Red Hat AI, IBM,…

In this technical demo, we explore how llm-d optimizes distributed inference by using Precise Prefix Cache-Aware Routing and how you can gain full visibility into these decisions using Distributed…

YouTube video by llm-d Project