The best chunk size for GLM-5 RAG prompts is the one that maximizes retrieval precision while preserving enough local context for the model to answer without guessing. In practice, most developer-doc RAG systems land in the range of 300–800 tokens per chunk, with 50–150 tokens overlap, but the “best” value depends on your content shape: API references, conceptual guides, code snippets, and tables behave differently. For GLM-5 specifically, the large context window makes it tempting to increase chunk size, but chunk size is primarily a retrieval quality problem, not a context-limit problem. If your chunks are too large, vector similarity search tends to retrieve broad sections that include irrelevant text (hurting precision and wasting tokens). If chunks are too small, you lose definitions and preconditions (hurting answer completeness). Your goal is “one chunk contains one coherent idea plus its necessary constraints.” You can confirm GLM-5’s long-context capability in the official docs (200K context / 128K output): Migrate to GLM-5.
A practical way to choose chunk size is to tune using document types and query types, rather than a single global setting. Here’s a useful baseline split for developer documentation indexed in Milvus or Zilliz Cloud:
API reference pages: smaller chunks (250–450 tokens). Keep endpoints/parameters together.
How-to guides: medium chunks (400–800 tokens). Preserve step sequences and prerequisites.
Conceptual overviews: medium-to-large chunks (600–1,000 tokens) if sections are well-structured.
Code examples: chunk by example block, not by token count. Keep one example intact.
Tables/config matrices: chunk as “table + caption + nearby explanation,” or pre-render to text with consistent formatting.
Also store metadata per chunk: doc_type, section_heading, version, lang. Then, at retrieval time, you can filter or boost by doc_type depending on the question. For example, a “how do I configure X?” question should prefer howto chunks over overview chunks. This often improves accuracy more than any single change to chunk length. If you add re-ranking, chunk size can be slightly larger without losing precision, but don’t use re-ranking as an excuse to index messy chunks.
Finally, measure instead of guessing. Create a small evaluation set (50–200 real questions from search logs or support tickets) and compute: (1) retrieval recall (does top-k include an answer-containing chunk?), (2) context efficiency (how many retrieved tokens are actually used?), and (3) grounded answer quality (does the answer match retrieved text?). If recall is low, your chunks may be too small or poorly segmented; if efficiency is low, your chunks may be too large or too repetitive. Because you’re building on Milvus.io, you’ll likely have strong headings and versioned docs—use them to chunk by semantic boundaries instead of raw token windows. Once retrieval is solid, GLM-5’s job becomes much easier: it composes answers from relevant chunks rather than trying to infer missing details. That’s the “best chunk size” in practice: the one that makes retrieval consistently fetch the right pieces with minimal noise.