# Missing robots.txt

> A robots.txt tells crawlers what to index and where your sitemap is. Add one so crawling is intentional, not guessed.

_Category: Search visibility · Detector `seo` · Check `robots-missing` · Severity: info_

`robots.txt` is the first file most crawlers request. It lets you allow or disallow paths and — importantly — point crawlers at your sitemap. It is optional but recommended: without it, crawling is left to defaults, and you lose an easy place to advertise your sitemap and manage crawl access.

## Symptoms

- `/robots.txt` returns a `404`
- No place to point crawlers at your sitemap
- No control over which paths crawlers should skip

## How VibeCheck detects it

The `seo` detector flags this live in the browser and reports it to the widget's Problems list — and to your coding agent over MCP.

- **Issue string:** `Missing robots.txt`
- **Threshold:** /robots.txt is not a 2xx text/plain response

## Root causes

- No `robots.txt` was ever added to the project
- The SPA returns `index.html` for `/robots.txt` instead of plain text
- The file exists but is served with the wrong content type

## The fix

Add a `/robots.txt` served as `text/plain`. For most sites, allow all crawling and reference your sitemap. Only disallow paths you genuinely want kept out of the index (and remember `robots.txt` does not make a page private — it only requests crawlers skip it).

### Steps

1. Create a `robots.txt` that allows crawling and lists your sitemap
2. Serve it at `/robots.txt` with a `text/plain` content type
3. Disallow only paths you truly want uncrawled

_robots.txt_

```bash
User-agent: *
Allow: /

Sitemap: https://acme.com/sitemap.xml
```

## Framework-specific fixes

### React

A Vite SPA has no server, so a `robots.txt` is just a static file. Drop it in `public/` and Vite copies it to the build root, served at `/robots.txt` as `text/plain`.

```bash
public/robots.txt   # served at /robots.txt
```

### Next.js

Add `app/robots.ts` exporting a default function returning a `MetadataRoute.Robots` object. Next serves it at `/robots.txt`.

_app/robots.ts_

```ts
import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: '*', allow: '/' },
    sitemap: 'https://acme.com/sitemap.xml',
  }
}
```

[Next.js docs](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots)

### Vue

In Nuxt, place `robots.txt` in the `public/` directory, or use the `@nuxtjs/robots` module to generate it from config. A plain Vite + Vue app is the same as React — a static file in `public/`.

```bash
public/robots.txt   # served at /robots.txt
```

[Vue docs](https://nuxt.com/modules/robots)

### Svelte

Serve `robots.txt` from a SvelteKit endpoint (or place a static file in `static/`).

_src/routes/robots.txt/+server.ts_

```ts
export function GET() {
  const body = 'User-agent: *\nAllow: /\n\nSitemap: https://acme.com/sitemap.xml\n'
  return new Response(body, { headers: { 'Content-Type': 'text/plain' } })
}
```

### Vanilla JS

Place a static `robots.txt` in your public directory so it is served at `/robots.txt`.

```bash
public/robots.txt   # served at /robots.txt
```

## FAQ

### Does `robots.txt` make a page private?

No. It only requests that compliant crawlers skip a path. The page is still reachable by URL. To keep content private, use authentication; to keep it out of search, use `noindex`.

### What is the minimum useful `robots.txt`?

Allow all crawling and point to your sitemap: “`User-agent: *`”, “`Allow: /`”, and a “`Sitemap:`” line. That is enough for most sites.

### Can `robots.txt` block AI crawlers?

Yes — you can disallow specific AI user-agents. If you want to be cited by AI answer engines, do the opposite and allow them; VibeCheck’s AEO audit checks for that.

### Why does VibeCheck flag `robots.txt` as missing when the file exists?

The probe treats `/robots.txt` as missing unless it responds 2xx with a `text/plain` content type. An SPA dev server that returns `index.html` for unknown paths reads as “not a real `robots.txt`” — serve it as plain text to clear the check.

### Should I disallow anything by default?

No. Start with “`Allow: /`” and only disallow paths you genuinely want kept out of the index — a staging area, internal search results, or duplicate print views. Over-broad `Disallow` rules are a common way to accidentally de-index a whole site.

## Related problems

- [Missing or invalid sitemap.xml](https://vibecheck.wcgw.fun/fix/missing-sitemap.md) — Search visibility
- [robots.txt blocks AI crawlers](https://vibecheck.wcgw.fun/fix/ai-crawlers-blocked.md) — AI readiness
- [Page is set to “noindex”](https://vibecheck.wcgw.fun/fix/accidental-noindex.md) — Search visibility
- [No agent interface (MCP) advertised](https://vibecheck.wcgw.fun/fix/missing-mcp-discovery.md) — AI readiness

---

Fix guide from VibeCheck — https://vibecheck.wcgw.fun/fix/missing-robots-txt. Full site index for LLMs: https://vibecheck.wcgw.fun/llms.txt