Editing your robots.txt file can de-index your website

Adding the incorrect configuration to your robots.txt file can remove you website from search engines.

A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site.

For a more in-depth view on robots.txt please read this article from Google.

Below is an example of a typical robots.txt file.


User-agent: *

Disallow: /super-secret

Creating a Robot.txt modification

We suggest checking out the contents of your existing robots.txt file. You can find this at

URL Rewrites

To create/update update your robots.txt follow the steps below;

  1. Enter the local URL that you wish to apply robots.txt. This URL must end in robots.txt
  2. Enter your robots.txt rules.