Context Rot happens because large language models have finite attention capacity, even when the context window is large. While modern models can technically accept tens or hundreds of thousands of tokens, their attention is not perfectly uniform across that entire range. As more tokens are added, earlier information becomes harder for the model to weigh correctly relative to newer inputs.
Another cause is prompt structure drift. In long conversations, instructions, examples, retrieved documents, and user messages are often appended without strong structure. Over time, this creates ambiguity about what is authoritative. For instance, an early system instruction may say “Answer concisely,” but later retrieved text may include verbose explanations. The model may start following the tone and structure of the later text instead of the original instruction, even though both are still present.
Context Rot is also amplified by repeated retrieval and accumulation. In RAG systems, each turn may fetch more documents and append them to the prompt. Without pruning or ranking, irrelevant or partially relevant chunks dilute the signal of the truly important ones. This is why production systems often store knowledge externally in a vector database like Milvus or Zilliz Cloud and retrieve only the most relevant chunks per query, rather than keeping everything in the prompt forever.
For more resources, click here: https://milvus.io/blog/keeping-ai-agents-grounded-context-engineering-strategies-that-prevent-context-rot-using-milvus.md