1 minute readReference note
Why can a page be accessible to crawlers but absent from AI answers?
In brief
Crawler access is one stage between a published asset and an AI answer. For retrieval-based use, check discovery, processing, and selection for the request. Inspect the product's other active information paths as well: learned knowledge, other sources, tool results, and context.
Check the public response and visible text, relevant platform controls, available index evidence, internal references, and the question the asset answers. Compare current and historical descriptions across the broader source record. Where retrieval details are available, inspect selected passages and source versions. Record displayed citations as one part of that evidence and check additional retrieval records where provided.
Act on the diagnosed stage. Repair a genuine access block, improve discovery of a useful asset, clarify its facts or entity relationships, and address competing incorrect accounts. Use provider documentation for the platform controls being changed and test the relevant answer again. Keep access, observed retrieval, source interpretation, and the generated claim as separate checkpoints within one source-to-answer investigation.