vLLM

What is it?

vLLM is an open-source inference server developed by UC Berkeley that serves large language models with state-of-the-art throughput and low latency. It runs on NVIDIA, AMD, and Intel GPUs and exposes an OpenAI-compatible REST API.

What does it do?

vLLM maximises GPU utilisation using PagedAttention — a memory manager that treats the KV cache like virtual memory — and continuous batching, which keeps the GPU saturated by dynamically inserting new requests into in-flight batches. The result is 2–4× higher throughput than naive inference loops at the same tail latency.

Where is it used?

vLLM powers the inference layer of many production AI products: chat assistants, RAG pipelines, code-generation APIs, and on-premise LLM deployments for regulated industries. It supports Llama, Mistral, Qwen, DeepSeek, Gemma, and most Hugging Face causal-LM architectures out of the box.

When & why it emerged

vLLM was released in 2023 to address the inefficiency of early Hugging Face Transformers inference, where GPU memory was fragmented and throughput collapsed under concurrent load. PagedAttention borrowed ideas from operating-system virtual memory and became the industry reference implementation for LLM serving.

Why we use it at Internative

We deploy vLLM for clients who need to host open-weight LLMs on their own GPUs — either for data-sovereignty compliance, cost predictability at high volume, or latency budgets that cloud APIs cannot meet. It integrates cleanly with our LangChain and FastAPI service stacks.