How do you make a website readable by AI search?
Answer engines quote pages they can fetch as plain HTML, not as a running app. If the facts live only after JavaScript, most crawlers never see them. Question-shaped headings, tables, definition lists and a clear robots policy are what get extracted. A simpler static page often beats a richer client render.
What do answer-engine crawlers actually fetch?
Most answer engines start with a GET of the URL, then parse the HTML they receive. They keep title, meta description, visible headings, lists, tables, definition lists and short paragraphs. They usually skip login walls, infinite scroll, content behind tabs that never appear in the first response, and text drawn only on a canvas.
They also skip much of what a browser would run later. If the first HTML is an empty shell and the article arrives from an API after hydration, the crawler has nothing to quote. Server-rendered HTML, or HTML that already contains the answer in the initial payload, is the reliable path. A brochure site with static files is often enough; a single-page app is not, unless the server still emits the same copy.
Why is JavaScript-only content invisible to most of them?
Crawlers are not a full browser session. Some run a short render; many do not wait for client routing, lazy chunks or consent banners. If the H1, the FAQ answers and the numbers only exist after React or Vue mount, they are absent from the document the model later cites.
The trade-off is real. Client rendering can make an app feel faster for people. For AI search it hides the page. Prefer HTML that already contains the answer, then enhance. If you must ship an app, prerender the public URLs or use a static export for the articles you want quoted. Do not put the only copy of a fact inside a component that never server-renders.
| Source in the page | Usually fetched | Usually skipped |
|---|---|---|
| First HTML response with headings and tables | Yes | No |
| Text inserted only after client hydration | No | Yes |
| Content behind a login or paywall | No | Yes |
| robots.txt Disallow for that path | No | Yes |
| Plain FAQ answers in the HTML | Yes | No |
How should robots.txt and llms.txt treat AI crawlers?
robots.txt is still the blunt instrument. If you disallow a user-agent, responsible crawlers should stay out. If you allow them, they will fetch what the file permits. Treat AI bots as named agents: allow the public help and docs you want quoted; disallow account, checkout, search-result and personal pages. Do not rely on security through obscurity; robots.txt is public.
llms.txt is a convention, not a law. It is a markdown file at the site root that points crawlers at the pages you consider canonical: docs, pricing, definitions. It does not replace HTML. It does not force a model to read you. It is a map. Keep it short, link real URLs, and make sure those URLs already contain extractable answers. If a simpler robots allow on /docs is enough, you do not need a second file.
- Answer engine
- A system that cites or summarises web pages in a reply, rather than only listing links.
- Initial HTML
- The markup returned on the first GET, before client scripts run.
- llms.txt
- An optional root markdown file that lists preferred URLs for language models to read.
- Extractable block
- A heading, table, list or definition pair a parser can lift without running your app.
Why do question-shaped headings and tables get quoted?
Models match a user's question to a heading that already looks like that question. An H2 that ends with a question mark is a retrieval key. The paragraphs under it should answer immediately, in British English, without requiring the rest of the site. Tables and definition lists survive extraction because they have structure: columns, terms, rows. Marketing adjectives do not.
Write the answer in the first sentence of the section. Name trade-offs: a static article may rank in AI answers while a richer dashboard never will. Mention GROX at most where it is the product under discussion; this page is about crawlability, not a pitch. If a spreadsheet of facts is the source of truth, publish a HTML table, not a screenshot. Screenshots are pictures of text, not text.
- Put the user's question in the heading, ending with a question mark.
- Answer in the first paragraph; keep later paragraphs for mechanism and limits.
- Use a real table when comparing fetch versus skip, not a decorative grid.
- Keep FAQ answers self-contained so they still make sense off the page.
Common questions
Do I need JavaScript for AI search engines to read my site?
No. Most answer engines quote the first HTML response. If the facts are already in that markup, scripts are optional. JavaScript-only articles are often invisible. Server-render or statically generate the pages you want cited, then add interactivity if people need it.
What should robots.txt say about AI crawlers?
Name the bots you recognise and allow only public URLs you want quoted, such as docs and pricing. Disallow account, checkout and personalised search. robots.txt is advice, not a lock. Sensitive pages still need authentication. Keep the file small and check it after deploys.
Is llms.txt required for a site to be quoted?
No. It is an optional map of canonical pages. Crawlers that ignore it will still use ordinary HTML if robots.txt allows the URL. A clear article with question headings and tables matters more than a second index file. Use llms.txt only if it points at pages that already contain the answers.
Why do tables and FAQ headings get copied into answers?
Parsers lift structure: a question heading, a two-to-four column table, a term and definition. Those blocks survive when surrounding prose is trimmed. Write answers that stand alone, in plain sentences, so a quoted fragment remains true without the rest of the article.
For how one conversational surface routes work across tools, see GROX and the Help Centre.