JRL Intelligence
← All posts
By J. David Miller ·

Search in the Agentic Age, Part 4: How to Optimize Your Website Content for AI Search

Remember the ghost in the back room? Say you took part 3 to heart. You updated your website and spelled out exactly why you're the best ghost removal team in town. So how does the AI learn you're the one to call?

Think of it as four gates your page likely has to get through: crawled, indexed, retrieved, and named or cited. Miss one, and the gates after it may never open.

Gate 1: Crawled

Crawling is how bots from Google, Bing and ChatGPT search read your website. A crawler follows links from page to page, copies what it finds and brings it back to the search engine. If your site blocks a crawler, it can't copy you. OpenAI says sites that opt out of its search crawler won't be shown in ChatGPT search answers.

Crawlers come back on their own schedule. Google says recrawling can take a few days to a few weeks, so a change you make today may take a while to show up.

You can ask them to come back sooner. You don't have to just wait:

  • Google Search Console lets you ask Google to recrawl a page you've updated, and to submit a sitemap, a list of every page on your site.
  • Bing Webmaster Tools does the same for Bing. That matters beyond Bing, because OpenAI says ChatGPT search sometimes partners with other search providers, and its help page for Enterprise and Edu accounts names Bing.
  • IndexNow is a free way to tell Bing and other participating search engines automatically whenever a page changes. A page you submit to one participating search engine is shared with the others. Google doesn't use it. On many website platforms, it's a single setting.

A request asks for a recrawl but doesn't guarantee one. Google says plainly that requesting a crawl doesn't guarantee a page will appear in results quickly, or at all. And as of November 2026, ChatGPT, Claude, Perplexity and Grok didn't list a way to submit your site. With those, the job is simply not to block them.

Make sure you aren't blocking them. Every website can have a small text file called robots.txt. It tells each bot, by name, what it's allowed to read. To block a bot, you add a line telling it to stay out. To unblock it, you delete that line. You can see yours by typing your web address followed by /robots.txt.

Some companies run more than one bot. OpenAI, for example, has one bot that collects material that may be used to train its models and a separate one for ChatGPT search, and it says each setting is independent. Blocking the training bot doesn't remove you from ChatGPT search. Blocking the search bot does.

Your website host or security service may also make these choices for you. Some, including Cloudflare, now let site owners block AI bots with a single setting, and as of November 2026, Cloudflare's defaults had already changed more than once that year. Settings change, so check yours rather than assume.

Put it in words a crawler can read. Crawlers read text best. Information locked inside a photo, a scanned PDF or a login page is harder to use. A medical practice that posts its insurance list as a scanned image, or an HVAC company whose price sheet is a photo, may be hiding the very answers customers ask about.

Gate 2: Indexed

Indexing means filing your pages so they can be found later. Think of a library: crawling brings the book into the building, and indexing catalogs it. Google says it can look at individual sections, or passages, of a page, and match the concepts in a question to the concepts on a page. Many AI systems likely store those passages as numbers that capture meaning.

Being crawled doesn't guarantee being indexed. A search engine can copy a page and still decide not to file it. Common reasons:

  • A "noindex" setting left on by mistake. It tells search engines to skip the page. It's sometimes switched on while a site is being built and never switched off.
  • Duplicate pages. If two pages say nearly the same thing, the search engine may file only one, and it may not be the one you want.
  • Thin content. A page with a heading and two sentences gives the search engine little reason to file it.

You can check. In Google Search Console, the URL Inspection tool shows whether a page is indexed and, if not, why. Bing Webmaster Tools has a similar check.

There's one more rule for AI answers. Google says a page has to be indexed, and eligible to show a snippet, the short preview in regular search, before it can appear as a link in its AI answers. A setting that hides your snippet can keep you out.

Here's how that looks in practice. A practice adds a new page about hearing aids. It's clear, specific and useful, but until it's indexed, it can't be found. An HVAC company has three nearly identical pages for furnace repair, one for each town it serves. The search engine may file only one, and that might be the town it serves least.

Gate 3: Retrieved

Retrieval is the moment the AI pulls source material to answer a question. It may not search your question just once. Google says its AI Mode breaks a question into subtopics and runs many searches at the same time. One search may land on your website. Another may land on your reviews, your listings or an article about you.

Take a parent who asks, "Who treats ringing in the ears in kids near me?" That one question might become several searches: what causes ringing in the ears, which doctors treat children, which practices are nearby, and what their patients say about them. A homeowner who asks, "Who can fix a furnace that won't start this weekend?" might set off searches for furnace repair, weekend service, the town and reviews.

The AI likely ranks what comes back and builds its answer from the few passages that fit best. It likely isn't reading your whole site.

Say someone asks, "Who can get rid of the strange noises in my attic?" One search may pull a passage from your site that says, "We investigate and clear unexplained activity in homes and businesses," even though "noises" and "attic" never appear in it. Another may check what your customers say about you.

That's why part 3's steps matter:

  • Each section should stand on its own. Retrieval pulls pieces, not whole pages. A clear heading followed by a direct answer is more likely to be pulled than an answer buried in the fourth paragraph.
  • Answer the narrower questions. A page that only says "We treat ear problems" has nothing to offer the search about children. A page that says "We treat ringing in the ears in children and adults" does.
  • Keep it in text. A passage the AI can't read is a passage it can't pull.

Gate 4: Named or cited

Being cited means the AI names your business in its answer, or links to your page as a source. Being named is likely what counts most. A link is just one way in.

The spots are few. The AI gives one answer, and for a local question, it usually names just a few businesses. In Pew's March 2025 data, most Google AI summaries cited three or more sources, and only 1% cited just one. You're competing for a few spots, not a page of ten links.

Answers also change. In BrightLocal's September 2026 study, the same local question asked again often named different businesses. Different tools draw on different sources. The question can be split into different searches each time. And the AI writes its answer by predicting words, so the wording, and the businesses it names, can shift from one run to the next. One test tells you little. Several tests tell you more.

So what makes a passage more likely to be cited? Likely the same things a careful person would look for: a clear, specific answer that matches the question, backed up by reviews and listings that tell the same story. A practice whose website says its doctors treat children with ringing in the ears, and whose reviews mention exactly that, gives the AI an answer and a reason to trust it. An HVAC company whose website, Google profile and Yelp page all say "weekend furnace repair" gives the AI three sources that agree.

Try this

You can see how your site does at each gate in about 20 minutes.

  1. Crawled: Type your web address followed by /robots.txt. If you see a line that names a search bot, such as Googlebot, Bingbot or OAI-SearchBot, and says "Disallow," ask whoever manages your site why it's there.
  2. Indexed: In Google Search Console, use URL Inspection on your most important service page. Is it indexed? If not, what's the reason?
  3. Retrieved: Write down one real question a customer asked this week. Break it into the smaller searches the AI might run. Does your site answer each one?
  4. Named or cited: Ask that question in two or three AI tools, twice each. Write down who gets named, whether you were named or linked, and what the AI said about you.
  5. Pick the gate where you fell short, and fix one thing this week.

Coming next

That's the mechanism behind part 3. Retrieval needs something specific to match. In part 5, we'll look at who should build this content, and how to get ready for what comes next.

So which of the four gates do you think your website is most likely to miss?

This is part 4 of a 5-part series. Read part 1, part 2 and part 3.

Sources

  • OpenAI, Overview of OpenAI crawlers (OAI-SearchBot and GPTBot; each setting is independent): developers.openai.com
  • OpenAI Help Center, ChatGPT search (sometimes partners with other search providers): help.openai.com
  • OpenAI Help Center, ChatGPT search for Enterprise and Edu (names Bing as its search provider): help.openai.com
  • Google Search Central, Ask Google to recrawl your URLs: developers.google.com
  • Google Search Central, AI features and your website (indexed and snippet-eligible; query fan-out): developers.google.com
  • Google Search Central, A guide to Google Search ranking systems (passage ranking, neural matching): developers.google.com
  • Google, AI Mode and query fan-out: blog.google
  • IndexNow, documentation (submitted URLs shared with participating search engines): indexnow.org
  • Cloudflare, "Your site, your rules: new AI traffic options for all customers," July 1, 2026: blog.cloudflare.com
  • PPC Land, Cloudflare revises its planned AI crawler defaults, September 2026: ppc.land
  • Pew Research Center, "Google users are less likely to click on links when an AI summary appears in the results," July 22, 2025: pewresearch.org
  • BrightLocal, "What 200k local AI searches tell us about what businesses get recommended," September 16, 2026: brightlocal.com