Robots.txt
Editing your robots.txt file can de-index your website
Adding the incorrect configuration to your robots.txt file can remove you website from search engines.
A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site.
For a more in-depth view on robots.txt please read this article from Google.
Below is an example of a typical robots.txt file.
# HTTPS www.updatable.com
User-agent: *
Sitemap: https://www.updatable.com/sitemap.xml
Sitemap: https://www.updatable.com/sitemaps/https-index-uk-archive.xml
Sitemap: https://www.updatable.com/sitemaps/https-index-uk-news.xml
Sitemap: https://www.updatable.com/dev/sitemap/sitemapindex.xml
Disallow: /super-secret
Creating a Robot.txt modification
We suggest checking out the contents of your existing robots.txt file. You can find this at https://www.yourdomain.com/robots.txt
To create/update update your robots.txt follow the steps below;
- Enter the local URL that you wish to apply robots.txt. This URL must end in robots.txt
- Enter your robots.txt rules.