vibecheck0.3.0
AI readinessaeoai-crawlers-blockedwarning

robots.txt blocks AI crawlers in Next.js

If your robots.txt disallows AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and friends — then assistants can’t read your content and won’t cite you in their answers. This is often accidental: a blanket Disallow, or a default that shipped with a template, quietly cuts you out of the fastest-growing discovery channel. (If the block is deliberate, that’s a valid choice — just make sure it is.)

Framework fixes
GeneralNext.jsVanilla JS

Symptoms

  • robots.txt has a Disallow: / for * or for AI bot user-agents
  • Your content never appears in AI assistant answers
  • Assistants say they can’t access the page
  • VibeCheck lists the specific bots your robots.txt blocks

How VibeCheck catches it

In your widget · Problems

warningaeorobots.txt blocks AI crawlers

To your coding agent · MCP

agent › get_detected_issues
→ { detector: "aeo", issue: "robots.txt blocks AI crawlers", threshold: "robots.txt has Disallow: / for * or a known AI bot (GPTBot, ClaudeBot, PerplexityBot, …)" }

The same string in your widget and in your agent’s context — no screenshot, no copy-paste.

Root causes

  • A blanket Disallow: / left over from staging
  • A template default that blocks AI user-agents
  • Blocking AI bots specifically without intending to

The fix for Next.js

Generate robots.txt from app/robots.ts with MetadataRoute.Robots so the allow/disallow rules live in code and stay reviewable.

app/robots.tsts
import type { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
  return {
    rules: [{ userAgent: '*', allow: '/', disallow: '/admin/' }],
    sitemap: 'https://acme.com/sitemap.xml',
  }
}

Next.js docs →

Steps

  1. Open robots.txt and find any Disallow: / affecting * or AI bots
  2. Allow the AI crawlers you want to appear in answers
  3. Scope remaining Disallow rules to private paths only

See the general, framework-agnostic fix →

FAQ

Which user-agents are the AI crawlers?
The common ones include GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google’s AI training), and CCBot (Common Crawl). VibeCheck names the specific ones your robots.txt blocks.
Should I allow AI crawlers?
It’s a business decision. If you want visibility and citations in AI answers, allow them. If you want to keep content out of AI training/answers, block them deliberately. The point is to choose, not to block by accident.
Does blocking Google-Extended affect normal Google Search?
No. Google-Extended controls AI/Gemini training and AI features; it’s separate from Googlebot’s normal indexing. You can allow Googlebot and still block Google-Extended if you prefer.