# robots.txt blocks AI crawlers in Next.js

> How to fix robots.txt blocks ai crawlers in Next.js — with the exact fix and copy-paste code.

_Category: AI readiness · Detector `aeo` · Check `ai-crawlers-blocked` · Severity: warning_

If your `robots.txt` disallows AI crawlers — `GPTBot`, `ClaudeBot`, `PerplexityBot`, `Google-Extended` and friends — then assistants can’t read your content and won’t cite you in their answers. This is often accidental: a blanket `Disallow`, or a default that shipped with a template, quietly cuts you out of the fastest-growing discovery channel. (If the block is deliberate, that’s a valid choice — just make sure it is.)

## The fix for Next.js

Generate `robots.txt` from `app/robots.ts` with `MetadataRoute.Robots` so the allow/disallow rules live in code and stay reviewable.

_app/robots.ts_

```ts
import type { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
  return {
    rules: [{ userAgent: '*', allow: '/', disallow: '/admin/' }],
    sitemap: 'https://acme.com/sitemap.xml',
  }
}
```

[Next.js docs](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots)

### Steps

1. Open `robots.txt` and find any `Disallow: /` affecting `*` or AI bots
2. Allow the AI crawlers you want to appear in answers
3. Scope remaining `Disallow` rules to private paths only

## How VibeCheck detects it

The `aeo` detector flags this live in the browser and reports it to the widget's Problems list — and to your coding agent over MCP.

- **Issue string:** `robots.txt blocks AI crawlers`
- **Threshold:** robots.txt has Disallow: / for * or a known AI bot (GPTBot, ClaudeBot, PerplexityBot, …)

## FAQ

### Which user-agents are the AI crawlers?

The common ones include `GPTBot` and `OAI-SearchBot` (OpenAI), `ClaudeBot` (Anthropic), `PerplexityBot` (Perplexity), `Google-Extended` (Google’s AI training), and `CCBot` (Common Crawl). VibeCheck names the specific ones your `robots.txt` blocks.

### Should I allow AI crawlers?

It’s a business decision. If you want visibility and citations in AI answers, allow them. If you want to keep content out of AI training/answers, block them deliberately. The point is to choose, not to block by accident.

### Does blocking `Google-Extended` affect normal Google Search?

No. `Google-Extended` controls AI/Gemini training and AI features; it’s separate from `Googlebot`’s normal indexing. You can allow `Googlebot` and still block `Google-Extended` if you prefer.

See the general, framework-agnostic fix: https://vibecheck.wcgw.fun/fix/ai-crawlers-blocked.md

---

Fix guide from VibeCheck — https://vibecheck.wcgw.fun/fix/ai-crawlers-blocked. Full site index for LLMs: https://vibecheck.wcgw.fun/llms.txt