To effectively monitor a production semantic search system, focus on three categories of metrics: search quality, system performance, and user behavior. These metrics help identify issues, optimize relevance, and ensure the system meets user needs. Each category provides distinct insights into different aspects of the system’s operation and impact.
For search quality, start with precision@k (how many top-k results are relevant) and recall@k (how many relevant items are in the top-k results). These metrics directly measure relevance but require labeled data. For example, if users search for “affordable wireless headphones,” precision@5 tells you if 3 out of 5 results match that intent. Include query latency (time to return results) and error rates (failed searches) to catch performance bottlenecks. Also, track query diversity—if 80% of searches return the same 10 results, your system might be overly narrow. Use embedding-based metrics like cosine similarity drift to detect if semantic representations degrade over time (e.g., due to model updates or data shifts).
For system performance, monitor infrastructure metrics like CPU/memory usage, indexing latency (time to add new documents), and throughput (queries per second). For example, a spike in indexing latency could indicate issues with scaling your vector database. Track cache hit rate to optimize costs—if 60% of repeated queries use cached results, you’re saving compute resources. Also, measure embedding generation time, especially if you’re using a large language model (LLM) to create vectors. If generating embeddings for 1,000 documents takes 10 minutes today but 30 minutes tomorrow, investigate model or hardware issues.
For user behavior, analyze click-through rates (CTR) on search results and session duration after a search. Low CTR on top results (e.g., 20% clicks on position 1) might indicate poor relevance. Track query reformulation rate—if 40% of users rephrase the same search, your system isn’t understanding intent. Use A/B testing to compare metrics between algorithm versions. For example, if switching from BM25 to a dense retriever increases CTR by 15%, it’s a win. Finally, log long-tail queries (e.g., “how to fix error code 0xE1A8B2”) to identify gaps in your document corpus or embedding model’s knowledge.
By combining these metrics, you’ll maintain a system that’s fast, accurate, and aligned with user needs. Prioritize based on your use case—an e-commerce platform might focus on CTR and conversion rates, while an internal knowledge base would emphasize precision@10 and query reformulation rates. Regularly review and adjust thresholds as your data and requirements evolve.