How do I serve GLM-5 with acceptable latency under load?

To serve GLM-5 with acceptable latency under load, you need to control (1) concurrency, (2) context length, and (3) batching/KV-cache pressure. GLM-5 supports very large context and output limits (the official docs list 200K context and 128K max output), but using those limits by default is the fastest way to blow up p95 latency and GPU memory under real traffic. The “acceptable latency” playbook is: cap tokens, stream outputs, keep prompts compact (use retrieval), and tune your inference server so it doesn’t over-batch or over-allocate KV cache. If you’re using vLLM (recommended in the GLM-5 repo), vLLM’s own docs emphasize tuning knobs like gpu_memory_utilization, max_num_seqs, and max_num_batched_tokens because they directly control KV cache allocation and preemption behavior. See: GLM-5 overview, Migrate to GLM-5, and vLLM’s Optimization and Tuning plus Parallelism and Scaling.

A practical “production tuning checklist” that engineers actually use looks like this:

• Cap request shape (most important):

  • Set max_output_tokens based on your UI (don’t allow huge outputs by default).

  • Enforce a max context budget per request (e.g., 8–20k tokens for most endpoints).

  • Use streaming so users see tokens immediately (Z.ai documents streaming support for GLM-5): Streaming.

• Use parallelism appropriately:

  • If the model doesn’t fit on one GPU, use tensor parallel across multiple GPUs as described in vLLM’s parallelism docs: Parallelism and Scaling.

  • Scale out by running multiple replicas behind a load balancer once a single node is stable.

• Tune vLLM for p95 (these knobs matter more than “more CPUs”):

  • Increase gpu_memory_utilization to allocate more KV cache (up to safe limits): Optimization and Tuning.

  • Reduce max_num_seqs and/or max_num_batched_tokens to prevent latency spikes from oversized batches: same vLLM doc.

  • If you hit OOM, reduce max_model_len and batch limits (vLLM calls this out explicitly): Conserving Memory.

This is where load testing pays off: fix a representative prompt distribution, run a constant QPS test, and plot p50/p95 latency vs. max_num_seqs and gpu_memory_utilization. You’ll usually find a “knee” where throughput improves but p95 explodes; choose settings just before that knee.

For Milvus.io-style workloads (docs Q&A, developer assistance), the best latency optimization is to avoid long prompts in the first place by using retrieval. Store your documentation in a vector database such as Milvus or Zilliz Cloud, retrieve top-k chunks (often 5–10) with metadata filters (version, product area), and feed only those chunks into GLM-5. This reduces average context length dramatically, which reduces KV cache pressure and makes batching more stable. The difference is noticeable: shorter prompts let you increase concurrency without OOM and without preemptions. Treat this as an architectural requirement, not a “nice to have”: RAG is a latency feature as much as it is an accuracy feature.