# Missing robots.txt in React

> How to fix missing robots.txt in React — with the exact fix and copy-paste code.

_Category: Search visibility · Detector `seo` · Check `robots-missing` · Severity: info_

`robots.txt` is the first file most crawlers request. It lets you allow or disallow paths and — importantly — point crawlers at your sitemap. It is optional but recommended: without it, crawling is left to defaults, and you lose an easy place to advertise your sitemap and manage crawl access.

## The fix for React

A Vite SPA has no server, so a `robots.txt` is just a static file. Drop it in `public/` and Vite copies it to the build root, served at `/robots.txt` as `text/plain`.

```bash
public/robots.txt   # served at /robots.txt
```

### Steps

1. Create a `robots.txt` that allows crawling and lists your sitemap
2. Serve it at `/robots.txt` with a `text/plain` content type
3. Disallow only paths you truly want uncrawled

## How VibeCheck detects it

The `seo` detector flags this live in the browser and reports it to the widget's Problems list — and to your coding agent over MCP.

- **Issue string:** `Missing robots.txt`
- **Threshold:** /robots.txt is not a 2xx text/plain response

## FAQ

### Does `robots.txt` make a page private?

No. It only requests that compliant crawlers skip a path. The page is still reachable by URL. To keep content private, use authentication; to keep it out of search, use `noindex`.

### What is the minimum useful `robots.txt`?

Allow all crawling and point to your sitemap: “`User-agent: *`”, “`Allow: /`”, and a “`Sitemap:`” line. That is enough for most sites.

### Can `robots.txt` block AI crawlers?

Yes — you can disallow specific AI user-agents. If you want to be cited by AI answer engines, do the opposite and allow them; VibeCheck’s AEO audit checks for that.

### Why does VibeCheck flag `robots.txt` as missing when the file exists?

The probe treats `/robots.txt` as missing unless it responds 2xx with a `text/plain` content type. An SPA dev server that returns `index.html` for unknown paths reads as “not a real `robots.txt`” — serve it as plain text to clear the check.

### Should I disallow anything by default?

No. Start with “`Allow: /`” and only disallow paths you genuinely want kept out of the index — a staging area, internal search results, or duplicate print views. Over-broad `Disallow` rules are a common way to accidentally de-index a whole site.

See the general, framework-agnostic fix: https://vibecheck.wcgw.fun/fix/missing-robots-txt.md

---

Fix guide from VibeCheck — https://vibecheck.wcgw.fun/fix/missing-robots-txt. Full site index for LLMs: https://vibecheck.wcgw.fun/llms.txt