Robots.txt
The robots.txt file is a plain text file placed at the root of a website that provides instructions to search engine crawlers — telling them which pages or sections of your site they should and should not crawl.
What is Robots.txt?
The robots.txt file is a plain text file placed at the root of a website (yoursite.com/robots.txt) that provides instructions to search engine crawlers — telling them which pages or sections of your site they should and should not crawl.
It is part of the Robots Exclusion Protocol (REP), a standard web convention that well-behaved crawlers like Googlebot follow voluntarily. It is not a security mechanism — it controls crawling, not access.
How Robots.txt Works
How robots.txt works: The file contains directives specifying which user agents (crawlers) the rules apply to, and what they should or shouldn't crawl. Common uses for robots.txt include: blocking duplicate or low-value pages from being crawled (parameter-based URLs, admin sections, internal search results), preventing crawlers from consuming server resources on non-public areas, pointing crawlers to the sitemap location, and managing crawl budget on large sites.
Important distinction: Robots.txt controls crawling (whether Google visits a page), not indexing (whether Google includes a page in search results). To prevent a page from appearing in search results, use the meta robots noindex tag or the X-Robots-Tag HTTP header.
In Webflow, the robots.txt file can be customized in Project Settings under the Integrations tab. By default, Webflow's robots.txt disallows crawling of the staging (webflow.io) domain, which is correct behavior — you don't want staging pages indexed.
At Appsrow, we review and configure robots.txt as part of every technical SEO setup.
Transform your website with expert Webflow development
From brand identity to Webflow development and marketing, we handle it all. Trusted by 50+ global startups and teams.