I want to upload a robots.txt file into the root directory

A robots.txt file is a file that informs search engines which files/directories of the site you do not want to be indexed. If you do not add a robots.txt file then your entire site will be indexed. See Google Help Page for more information on this topic.

Are there any parts of your site that you do not want to be indexed? If not, you do not need to add a robots.txt file to your site.

If you get a message saying that your URL is blocked because of a robots.txt file, please ignore this. Google has simply tried to index the preview URL of one of your pages. A preview URL is what you see in your browser window when you open your site in the Yola Site Builder and then preview it. Yola prevents Google from crawling these URLs as Google should not index unpublished sites.

This message simply serves to indicate that everything is working correctly. This will not interfere with your search engine ranking or Google Webmaster's ability to return data about your site. As long as your site is published and you have submitted the correct URL, Google will be able to crawl your site.

If you get a "Page not Found" or 404 error this probably means that there was a problem with your DNS server (the server of your domain name registrar) or that you had made a DNS change around the time that Google tried to index your site. Google makes an attempt approximately every 5 days, so you simply need to wait for the Google spiders to come around again and everything should be fine.

Was this article helpful?
0 out of 3 found this helpful