# Missing robots.txt in Svelte

> How to fix missing robots.txt in Svelte — with the exact fix and copy-paste code.

_Category: Search visibility · Detector `seo` · Check `robots-missing` · Severity: info_

`robots.txt` is the first file most crawlers request. It lets you allow or disallow paths and — importantly — point crawlers at your sitemap. It is optional but recommended: without it, crawling is left to defaults, and you lose an easy place to advertise your sitemap and manage crawl access.

## The fix for Svelte

Serve `robots.txt` from a SvelteKit endpoint (or place a static file in `static/`).

_src/routes/robots.txt/+server.ts_

```ts
export function GET() {
  const body = 'User-agent: *\nAllow: /\n\nSitemap: https://acme.com/sitemap.xml\n'
  return new Response(body, { headers: { 'Content-Type': 'text/plain' } })
}
```

### Steps

1. Create a `robots.txt` that allows crawling and lists your sitemap
2. Serve it at `/robots.txt` with a `text/plain` content type
3. Disallow only paths you truly want uncrawled

## How VibeCheck detects it

The `seo` detector flags this live in the browser and reports it to the widget's Problems list — and to your coding agent over MCP.

- **Issue string:** `Missing robots.txt`
- **Threshold:** /robots.txt is not a 2xx text/plain response

## FAQ

### Does `robots.txt` make a page private?

No. It only requests that compliant crawlers skip a path. The page is still reachable by URL. To keep content private, use authentication; to keep it out of search, use `noindex`.

### What is the minimum useful `robots.txt`?

Allow all crawling and point to your sitemap: “`User-agent: *`”, “`Allow: /`”, and a “`Sitemap:`” line. That is enough for most sites.

### Can `robots.txt` block AI crawlers?

Yes — you can disallow specific AI user-agents. If you want to be cited by AI answer engines, do the opposite and allow them; VibeCheck’s AEO audit checks for that.

### Why does VibeCheck flag `robots.txt` as missing when the file exists?

The probe treats `/robots.txt` as missing unless it responds 2xx with a `text/plain` content type. An SPA dev server that returns `index.html` for unknown paths reads as “not a real `robots.txt`” — serve it as plain text to clear the check.

### Should I disallow anything by default?

No. Start with “`Allow: /`” and only disallow paths you genuinely want kept out of the index — a staging area, internal search results, or duplicate print views. Over-broad `Disallow` rules are a common way to accidentally de-index a whole site.

See the general, framework-agnostic fix: https://vibecheck.wcgw.fun/fix/missing-robots-txt.md

---

Fix guide from VibeCheck — https://vibecheck.wcgw.fun/fix/missing-robots-txt. Full site index for LLMs: https://vibecheck.wcgw.fun/llms.txt