Three robots.txt truths
1. It lives at exactly one address: yourdomain.com/robots.txt — a file anywhere else does nothing. 2. It controls crawling, not indexing: a blocked page can still appear in results (as a bare link) if others link to it; to remove a page from the index use a noindex tag instead — and never block a page you also need crawled to see the noindex. 3. It is a request, not a lock: well-behaved crawlers obey it, bad bots ignore it, so never list secret paths — the file is public and reads like a map of what you hid.
Frequently Asked Questions
Should I block my admin area?
Yes for crawl budget and tidiness (WordPress sites block /wp-admin/ but allow admin-ajax.php, as in the default example here) — but remember it is not security.
Related Tools
SEO Analyzer — Analyze any page's on-page SEO — title, meta, headings, images, links.
Open Graph Checker — See how your link will look when shared on Facebook, X and chat apps.
Redirect Checker — Trace a URL's redirect chain — every hop and status code.
Meta Tag Generator — Generate title, description, Open Graph and Twitter tags in one go.
SERP Preview Tool — Preview your Google result — desktop and mobile — before you publish.
Robots.txt Tester — Test whether a crawler may fetch a URL under your robots.txt rules.
XML Sitemap Generator — Turn a list of URLs into a valid sitemap.xml — download included.
Keyword Density Checker — Count single words and 2–3 word phrases in any text, with percentages.
UTM Builder — Build campaign URLs with UTM parameters for clean analytics.
URL Slug Generator — Turn any headline into a clean, SEO-friendly URL slug.
Schema Markup Generator — Generate JSON-LD schema — Article, Product, FAQ, LocalBusiness, WebSite.
Hreflang Tag Generator — Generate hreflang link tags for multi-language and regional pages.
YouTube Keyword Tool — Get live YouTube autocomplete suggestions for any seed keyword.