# robots.txt blocks AI crawlers

> If robots.txt disallows GPTBot, ClaudeBot or PerplexityBot, assistants can’t read or cite you. Allow the AI crawlers you want to appear in answers.

_Category: AI readiness · Detector `aeo` · Check `ai-crawlers-blocked` · Severity: warning_

If your `robots.txt` disallows AI crawlers — `GPTBot`, `ClaudeBot`, `PerplexityBot`, `Google-Extended` and friends — then assistants can’t read your content and won’t cite you in their answers. This is often accidental: a blanket `Disallow`, or a default that shipped with a template, quietly cuts you out of the fastest-growing discovery channel. (If the block is deliberate, that’s a valid choice — just make sure it is.)

## Symptoms

- `robots.txt` has a `Disallow: /` for `*` or for AI bot user-agents
- Your content never appears in AI assistant answers
- Assistants say they can’t access the page
- VibeCheck lists the specific bots your `robots.txt` blocks

## How VibeCheck detects it

The `aeo` detector flags this live in the browser and reports it to the widget's Problems list — and to your coding agent over MCP.

- **Issue string:** `robots.txt blocks AI crawlers`
- **Threshold:** robots.txt has Disallow: / for * or a known AI bot (GPTBot, ClaudeBot, PerplexityBot, …)

## Root causes

- A blanket `Disallow: /` left over from staging
- A template default that blocks AI user-agents
- Blocking AI bots specifically without intending to

## The fix

Decide deliberately. If you want to be read and cited by assistants, ensure `robots.txt` allows the AI crawlers (or at least doesn’t `Disallow: /` for them). Scope any `Disallow` to genuinely private paths rather than blocking the whole site or all AI agents. If you intend to block them, keep it — just confirm it’s intentional.

### Steps

1. Open `robots.txt` and find any `Disallow: /` affecting `*` or AI bots
2. Allow the AI crawlers you want to appear in answers
3. Scope remaining `Disallow` rules to private paths only

_robots.txt — allow AI crawlers, block only private paths_

```html
User-agent: *
Allow: /
Disallow: /admin/

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

Sitemap: https://acme.com/sitemap.xml
```

## Framework-specific fixes

### Next.js

Generate `robots.txt` from `app/robots.ts` with `MetadataRoute.Robots` so the allow/disallow rules live in code and stay reviewable.

_app/robots.ts_

```ts
import type { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
  return {
    rules: [{ userAgent: '*', allow: '/', disallow: '/admin/' }],
    sitemap: 'https://acme.com/sitemap.xml',
  }
}
```

[Next.js docs](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots)

### Vanilla JS

Edit the static `robots.txt` at your web root. Remove any `Disallow: /` that isn’t intentional and scope rules to private paths.

```html
User-agent: *
Allow: /
Disallow: /admin/
```

## FAQ

### Which user-agents are the AI crawlers?

The common ones include `GPTBot` and `OAI-SearchBot` (OpenAI), `ClaudeBot` (Anthropic), `PerplexityBot` (Perplexity), `Google-Extended` (Google’s AI training), and `CCBot` (Common Crawl). VibeCheck names the specific ones your `robots.txt` blocks.

### Should I allow AI crawlers?

It’s a business decision. If you want visibility and citations in AI answers, allow them. If you want to keep content out of AI training/answers, block them deliberately. The point is to choose, not to block by accident.

### Does blocking `Google-Extended` affect normal Google Search?

No. `Google-Extended` controls AI/Gemini training and AI features; it’s separate from `Googlebot`’s normal indexing. You can allow `Googlebot` and still block `Google-Extended` if you prefer.

## Related problems

- [Missing robots.txt](https://vibecheck.wcgw.fun/fix/missing-robots-txt.md) — Search visibility
- [Missing or invalid sitemap.xml](https://vibecheck.wcgw.fun/fix/missing-sitemap.md) — Search visibility
- [Missing llms.txt](https://vibecheck.wcgw.fun/fix/missing-llms-txt.md) — AI readiness
- [No agent interface (MCP) advertised](https://vibecheck.wcgw.fun/fix/missing-mcp-discovery.md) — AI readiness

---

Fix guide from VibeCheck — https://vibecheck.wcgw.fun/fix/ai-crawlers-blocked. Full site index for LLMs: https://vibecheck.wcgw.fun/llms.txt