For production AI use cases, web search at inference time is often the weakest link in the pipeline. In this post, I discuss why that’s often the case and how to fix it. This post specifically applies to web search as implemented for applications that do not need real-time data, but just up-to-date, post-knowledge-cutoff, citable documents.
Why inference-time web search is overrated
People who say web search is a good tool for agents have Stockholm syndrome.
Google is a cesspool and has been for several years. It might seem like agents would be able to wade through it a bit more effectively, since they can run many parallel web search queries. Yet I find that search is often a source of unnecessary latency, bad information, and even hallucinations.
Web search as implemented for agents today has three major issues:
- Just-in-time discovery is hard.
- Information derived from webpage contents is unreliable.
- Progressive disclosure of webpages is poor.
First, it’s often challenging to find the right information at any given moment. Agents are extensively trained on deep research tasks, so their search queries are lengthy and sophisticated. But high quality information is often poorly discoverable. There is usually a significantly lower volume of it on the internet in the first place, and high quality sources tend to focus less on discoverability than sites whose business model depends on traffic. For example, there’s only one official site for the County of San Mateo’s parks department and it is poorly optimized for machine readability. Yet there are exponentially more travel blogs and social media posts about the same parks, all with impeccable SEO and often with agent-friendly features. Furthermore, after reading dozens of deep research traces recently, I’ve often noticed agents searching for information they already know exists rather than trying to discover genuinely new ideas. That’s just recall with extra steps!
Secondly, webpage contents are unreliable. They change often, and agents can inadvertently retrieve a cached version that is out of date. It’s also difficult for agents to tell how fresh webpage contents are; I’ve frequently seen agents incorporate information from 10 or even 20 years ago into their response. And, need I say it, webpages don’t have to tell the truth. Because of their post-training, agents tend to be overly trusting of information they find, just like they tend to trust you when you say you’re authorized to take a certain action. While I have seen agents question unreliable information, I’ve also seen agents just roll with it when they find something a human would notice is obviously incorrect.
Finally, progressive disclosure as implemented today does not work well. In most implementations (most notably Google’s AI Overview but also Claude Code), the main agent context never sees the full webpage. That’s sensible at first blush because some webpages are massive and they often contain irrelevant content, SEO, and junk HTML or JS. So the main agent usually gets a summary written by a smaller model. Your agent is basically playing telephone, making this system a major source of hallucinations. Common forms of this type of hallucination include citing a page that ultimately does not back up the claim, citing pages that have been taken down, and (my personal favorite) quoting prices for out of stock or already-sold listings. We know that agents shouldn’t ingest full webpages immediately, but the current system is untenable.
As a bonus issue, web search is insecure. It’s a potential attack vector for prompt injection and data exfiltration attacks. It also requires network egress.
Use a database of documents instead
Instead of giving your agent access to web search at inference, you can run queries at scale in advance of inference to collect large numbers of high quality documents; classify, transform, and annotate them; make them discoverable to your agents via a custom tool with progressive disclosure; and add fresh data via batch or streaming ingestion.
Yes, you’re still searching the web, but by doing it in advance instead of at inference time, you get more deterministic, exhaustive, and high quality search results.
This method borrows from a cool technique used in many deep research benchmarks. Instead of allowing agents to query the live internet, which creates unequal conditions across runs, they use a large fixed set of documents.
This method is superior in a few ways:
- Discoverability is enhanced because a larger share of documents are relevant and those documents have clean, high quality metadata.
- Information quality is improved because documents can be vetted offline.
- Fast progressive disclosure can be supported via querying of metadata, including summaries or semantic search over contents.
I’ll walk through some practical advice for each step. Let’s say you’re building a system that needs to supply information about all software products used in the United States.
Search in advance. The process starts with an exhaustive search for relevant documents. Whether a document is relevant to your intended use case depends on that use case, of course. Eventually as you search more and more, you will find diminishing returns as fewer and fewer documents add new information to the pool. Make sure to implement deduplication and logging so you know exactly how documents were discovered and when. You should also save the full contents. You can also optionally transform documents from their original state into records at whatever level of granularity you’d like to search. For example, you might want to extract individual software products from documents as their own records, with links back to the source document.
Classify, transform, and annotate. Storing documents on their own is helpful, but the more you can hydrate each record with extra information, the better your eventual results will be. I use Jev to quickly score documents on hundreds of dimensions, principally to determine trustworthiness and relevance. Later you can go back and compare results across different models if you want, but Jev makes this step extremely fast and cheap. For example, you can ask dozens of questions about a YouTube video transcript that references a software product to determine if the video was posted by the company that makes the product, if the product is featured as a sponsorship, if the video is principally about that product or if it compares alternatives, and so on. For relevance, you can decompose what “relevance” means and ask many questions about it. In the case of the software product database, you might ask if a product is mentioned at all, what context it’s mentioned in, if it’s mentioned positively, if its features are described, how many features are described, and so on. I also recommend annotating with short summaries and/or bullet point summaries, using a model like GPT 5.6 Luna or Claude Haiku 5.5. You should also clean up the full contents as much as possible (e.g. removing redundant HTML, headers/footers, video timestamps, etc.). Maintain pagination information if possible and relevant.
Make them discoverable. Instead of trying to query the raw db, have your agents use a tool you provide. I recommend two features: require agents to query summaries and/or metadata first (depending on your use case), and write query results to files instead of dumping the results into the model’s context window. For example, your tool or harness might enforce that your agent must check out the publish date, document type (e.g. webpage, video, whitepaper), and summary to decide whether it is useful before querying for it. (Since you’re basically caching the internet, you’ll want to have rules for staleness as well to help the agent understand what “fresh” means in the context of your use case.) Then, write retrieved documents into a folder of references, which allows the agent to search over their contents, check the documents’ lengths, read parts of them, and otherwise use their contents more intelligently and progressively. This also helps them better quote sources, since they can rely on the document contents rather than their memory. If an agent decides to cite a source, then your harness should enforce that it has read the text directly (you can design your own rules for this - you may want to enforce that the agent reads all or just part of the source).
Add fresh data. Most retrieval systems benefit from fresh data, but you’ll want to decide how often you actually need to ingest that information and how you’ll find new queries. Ingestion can also change the behavior of your system because you’re providing new documents to query, so you’ll probably want to run evals after large ingests. Batch ingestion on a nightly or weekly schedule can be a good choice if documents related to your use case don’t appear frequently, or if the state of things is unlikely to change within that timeframe. If you’re building a pipeline that requires whitepapers from a specific set of conferences, you might just schedule a batch ingest after they’re released for each conference. For the software products use case, nightly is probably fine, since people rarely need up-to-the-minute information about new software products. Hourly could also work. If your system does need up-to-the-minute information, then probably you should just use web search (lol), but you could consider a streaming architecture for ingestion if you prefer the approach described in this post. A fast architecture for discovering new documents is to monitor common sources like Hacker News, Reddit, Product Hunt, and other similar sites (depending on your use case), then classify them with Jev, check for duplicates, and ingest them again either streaming or in batches.
Bottom line
Web search at inference sucks. Databases ftw. I hope you give this architecture a try!