Keeping a page out of Google: robots.txt or noindex?
Want to keep a page out of Google, such as a thank-you page, a test page or an old promotion? Many people think of the robots.txt file. But robots.txt doesn't keep a page out of Google. That's what noindex is for. In this article you'll learn the difference, how to use both and what to do with a page you've deleted. We use example.com as an example.
The difference in short
| robots.txt | noindex | |
|---|---|---|
| What it says | "Don't fetch this page." | "Don't include this page in the search results." |
| Does it keep a page out of Google? | No. If others link to the page, the address can still appear, without a description. | Yes, once Google has visited the page again. |
| Where is it? | A single file at https://example.com/robots.txt | In the page itself, or as a header the server sends along |
| What do you use it for? | Steering crawl traffic, for example skipping your own site's search result pages | Pages that stay accessible but don't belong in Google |
Neither of them is a security measure. Anyone can read your robots.txt, and a page with noindex remains accessible. If something really needs to be protected, put it behind a password. See Password protecting a directory in DirectAdmin.
Using noindex
Put this line in the <head> of the page:
<meta name="robots" content="noindex">
For files without HTML, such as a PDF, the server can send the same instruction as a header: X-Robots-Tag: noindex. The effect is the same.
Usually you don't have to type this yourself:
- AI Builder: open the SEO section, choose the page and under More for this page turn off the This page visible in Google switch. The page stays accessible but gets a noindex. Then publish again.
- WordPress: an SEO plugin has a per-page setting to stop the page from being indexed. The Discourage search engines from indexing this site checkbox under Settings, Reading applies to the whole site.
- Other website builders: look for a setting like "hide from search engines".
Creating or editing robots.txt
On a hosting package, robots.txt is located in your domain's web folder, in DirectAdmin the public_html folder. You create or edit the file via File Manager or FTP, see Opening File Manager: the folders in your package. An example:
User-agent: * Disallow: /zoeken Disallow: /winkelmand/ Sitemap: https://example.com/sitemap.xml
User-agent: *applies to all search robots.Disallow:with a path forbids everything that starts with that path.Disallow: /forbids the entire site.- If you also create a
User-agent: Googlebotgroup, Google only follows that group and ignores the*group. So repeat all the rules there that also apply to Google. - Don't block folders with stylesheets, scripts or images that your page needs.
- Having no robots.txt is not a problem: then everything is allowed. If the address returns a server error, Google temporarily stops fetching.
WordPress generates its own robots.txt as long as there is no file in the web folder. The AI Builder places its own robots.txt with a Sitemap: line when publishing.
Detailed explanation with screenshots: Sitemap and robots.txt in the AI Builder and Sitemap and robots.txt in WordPress and your own site in the Getting Found Online guide.
Google usually caches your robots.txt for up to 24 hours. In Search Console, under Settings, you'll find the robots.txt report: which version Google last fetched, with errors and warnings, and a button to have it fetched again.
A deleted page: 404 or 410, not a soft 404
If you've deleted a page, the address should return the status code 404 (not found) or 410 (gone). On the next visit, Google will then remove the address from the search results. If the page has a successor, redirect the old address with a 301 to that new page, see Site Redirection: redirecting a domain or page.
A soft 404 is a page that says it doesn't exist, or is nearly empty, but still returns the code 200 (found). Google often recognises this and won't include such a page. In Search Console you'll see it as the reason Soft 404. Redirecting all deleted pages to your homepage can also count as a soft 404. So make a page that's really gone return a real 404 or 410.
Does a page need to disappear from the results quickly, for example because it accidentally contained personal data? Then use Removals in Search Console. This works temporarily, for about six months. So also make sure the page has a 404, 410, password or noindex.
Further reading
- Google: Block indexing with noindex (English)
- Google: Introduction to robots.txt (English)
- Google: robots.txt report
- Blocking files via .htaccess
Using WordPress? In the lesson Findability: the basics of the free WordPress course you can read more about robots.txt and the Discourage search engines checkbox.
Want to learn more? Take the free Getting Found Online course, lesson 3.3 robots.txt: what to block and what not to block.