Easily generate valid robots.txt rules to instruct Googlebot and other search engines on exactly what they should—and shouldn't—index on your site.
Helps search engines discover all pages on your site.
Distribute high-value software libraries, custom wrappers, and API wrappers with secure, seat-locked digital rights management. Manage activations, client seats, and usage telemetry natively.
Explore Software DRMPackage deployment protocols, automation scripts, and server runbooks into version-controlled, collaborative SEM manuals. Standardize system setups for engineering teams.
Explore SEM RunbooksUnlock collaborative workspaces. Track the lifecycle of 16 distinct software and knowledge asset types, deploy shared team keys, push CLI packages, and audit system credentials in one unified dashboard.
Every day, thousands of automated bots crawl the internet. Some of these bots are highly desirable (like Googlebot, which indexes your site for Google Search). Others are less desirable (like scraping bots or aggressive AI training spiders).
A robots.txt file is the internet standard (known as the Robots Exclusion Protocol) for politely asking these bots what they are allowed to look at.
The User-Agent string identifies the specific bot you are talking to.
User-agent: * means the rule applies to all bots.User-agent: Googlebot means the rule applies only to Google's web crawler.The Disallow directive tells the bot which paths it should not crawl.
For example, Disallow: /admin/ prevents the bot from crawling any URL that starts with /admin/.
The Allow directive overrides a Disallow rule for a specific subdirectory.
For instance, you might Disallow: /assets/ to save crawl budget on large files, but Allow: /assets/public/ so a specific folder is still indexed.
You can include the absolute URL to your XML sitemap in the robots.txt file. This is highly recommended as it acts as a direct map for search engines to discover all your important pages.
Example: Sitemap: https://www.yourdomain.com/sitemap.xml
The most common setup for a public website.
User-agent: * Disallow:
Crucial for staging servers to prevent Google from indexing your unfinished site.
User-agent: * Disallow: /
User-agent: * Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /private/
Recently, many sites have opted to block AI companies from scraping their content for training data.
User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: anthropic-ai Disallow: /
A robots.txt file is a text document placed in your website's root directory that tells web crawlers (like Googlebot) which pages they are allowed to index.
It must always be placed at the absolute root of your domain name (e.g. `https://yourdomain.com/robots.txt`). Files placed in subfolders will not be parsed by search engines.
Use the Disallow directive: `User-agent: * Disallow: /private-directory/`.
Quickly generate standard .gitignore files by selecting your framework, language, and IDE.
Build and validate robust HTTP CSP headers visually without memorizing the syntax.
Generate and preview standard HTML meta tags and OpenGraph tags for perfect social media sharing.
These tools are just the beginning. Create a free AIMD account to build your ultimate developer profile, launch custom communities, and organize your entire knowledge base in one beautifully unified platform. Say goodbye to scattered links and fragmented workflows.
Create Free AccountQuickly generate standard .gitignore files by selecting your framework, language, and IDE.
Build and validate robust HTTP CSP headers visually without memorizing the syntax.
Generate and preview standard HTML meta tags and OpenGraph tags for perfect social media sharing.