vibecheck0.3.0
AI readinessaeoai-crawlers-blockedwarning

robots.txt blocks AI crawlers in Vanilla JS

If your robots.txt disallows AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and friends — then assistants can’t read your content and won’t cite you in their answers. This is often accidental: a blanket Disallow, or a default that shipped with a template, quietly cuts you out of the fastest-growing discovery channel. (If the block is deliberate, that’s a valid choice — just make sure it is.)

Framework fixes
GeneralNext.jsVanilla JS

Symptoms

  • robots.txt has a Disallow: / for * or for AI bot user-agents
  • Your content never appears in AI assistant answers
  • Assistants say they can’t access the page
  • VibeCheck lists the specific bots your robots.txt blocks

How VibeCheck catches it

In your widget · Problems

warningaeorobots.txt blocks AI crawlers

To your coding agent · MCP

agent › get_detected_issues
→ { detector: "aeo", issue: "robots.txt blocks AI crawlers", threshold: "robots.txt has Disallow: / for * or a known AI bot (GPTBot, ClaudeBot, PerplexityBot, …)" }

The same string in your widget and in your agent’s context — no screenshot, no copy-paste.

Root causes

  • A blanket Disallow: / left over from staging
  • A template default that blocks AI user-agents
  • Blocking AI bots specifically without intending to

The fix for Vanilla JS

Edit the static robots.txt at your web root. Remove any Disallow: / that isn’t intentional and scope rules to private paths.

html
User-agent: *
Allow: /
Disallow: /admin/

Steps

  1. Open robots.txt and find any Disallow: / affecting * or AI bots
  2. Allow the AI crawlers you want to appear in answers
  3. Scope remaining Disallow rules to private paths only

See the general, framework-agnostic fix →

FAQ

Which user-agents are the AI crawlers?
The common ones include GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google’s AI training), and CCBot (Common Crawl). VibeCheck names the specific ones your robots.txt blocks.
Should I allow AI crawlers?
It’s a business decision. If you want visibility and citations in AI answers, allow them. If you want to keep content out of AI training/answers, block them deliberately. The point is to choose, not to block by accident.
Does blocking Google-Extended affect normal Google Search?
No. Google-Extended controls AI/Gemini training and AI features; it’s separate from Googlebot’s normal indexing. You can allow Googlebot and still block Google-Extended if you prefer.