vibecheck0.3.0
Search visibilityseorobots-missinginfo

Missing robots.txt in Vanilla JS

robots.txt is the first file most crawlers request. It lets you allow or disallow paths and — importantly — point crawlers at your sitemap. It is optional but recommended: without it, crawling is left to defaults, and you lose an easy place to advertise your sitemap and manage crawl access.

Framework fixes
GeneralReactNext.jsVueSvelteVanilla JS

Symptoms

  • /robots.txt returns a 404
  • No place to point crawlers at your sitemap
  • No control over which paths crawlers should skip

How VibeCheck catches it

In your widget · Problems

infoseoMissing robots.txt

To your coding agent · MCP

agent › get_detected_issues
→ { detector: "seo", issue: "Missing robots.txt", threshold: "/robots.txt is not a 2xx text/plain response" }

The same string in your widget and in your agent’s context — no screenshot, no copy-paste.

Root causes

  • No robots.txt was ever added to the project
  • The SPA returns index.html for /robots.txt instead of plain text
  • The file exists but is served with the wrong content type

The fix for Vanilla JS

Place a static robots.txt in your public directory so it is served at /robots.txt.

bash
public/robots.txt   # served at /robots.txt

Steps

  1. Create a robots.txt that allows crawling and lists your sitemap
  2. Serve it at /robots.txt with a text/plain content type
  3. Disallow only paths you truly want uncrawled

See the general, framework-agnostic fix →

FAQ

Does robots.txt make a page private?
No. It only requests that compliant crawlers skip a path. The page is still reachable by URL. To keep content private, use authentication; to keep it out of search, use noindex.
What is the minimum useful robots.txt?
Allow all crawling and point to your sitemap: “User-agent: *”, “Allow: /”, and a “Sitemap:” line. That is enough for most sites.
Can robots.txt block AI crawlers?
Yes — you can disallow specific AI user-agents. If you want to be cited by AI answer engines, do the opposite and allow them; VibeCheck’s AEO audit checks for that.
Why does VibeCheck flag robots.txt as missing when the file exists?
The probe treats /robots.txt as missing unless it responds 2xx with a text/plain content type. An SPA dev server that returns index.html for unknown paths reads as “not a real robots.txt” — serve it as plain text to clear the check.
Should I disallow anything by default?
No. Start with “Allow: /” and only disallow paths you genuinely want kept out of the index — a staging area, internal search results, or duplicate print views. Over-broad Disallow rules are a common way to accidentally de-index a whole site.