Needle In Haystack: Definition and Examples

The Needle In a Haystack (NIAH) test is a benchmark designed to evaluate large language models' (LLMs) ability to locate and extract a specific piece of information intentionally placed within a large textual context. The principle is simple: a precise fact (the needle) is inserted at different positions in a very long document (the haystack), and then the model is asked to retrieve that information.

This test has become an industry standard for measuring the real-world performance of extended context windows. Indeed, a model may claim to support 100,000 tokens of context, but if its ability to retrieve information significantly degrades when that information is placed in the middle of the text, that context window is practically less usable than advertised. The NIAH test reveals these weaknesses by systematically varying the needle's position and context length.

Results are typically presented as a two-dimensional heatmap, with the needle's depth in the document on one axis and the total context length on the other. This allows visualization of a model's weak spots—for example, many models show degraded performance when the information is in the middle of the text, a phenomenon known as "lost in the middle."

For prompt engineering practitioners, understanding a model's NIAH results is essential. It allows them to structure prompts strategically: place critical information at the beginning or end of the context, break long documents into shorter segments, or use explicit recall techniques to guide the model's attention to important elements.