SEO tips – Sitemap and robots.txt
Wednesday, December 23rd, 2009SEO tips – Sitemap and robots.txt
Adding a robots.txt file to your server is a way of telling search engines who to index and what not to index. It is a also a great idea to add your sitemap location as well.
To create a sitemap there are many freely available you can find them by searching for “sitemap generator” but a nice simple one can be found here enter your website address and download the XML version. You will then need to load it to the server using FTP software. We use filezilla which is free and easy to use.
If you want to invite a search bot to index your entire website here is the code:
___________________
# robots.txt courtesy of http://www.cirencester-website-design.co.uk/index.html
User-agent: *
Disallow:
Disallow: /add the forward slash followed by the directory names you want to exclude (one per line) i.e. /customer_data
Sitemap: http://www.yourwebsite.co.uk/sitemap.xml
___________________
This simple code tells all search engine bots that nothing is disallowed and the location of our site map. Using a text editor (see resources) copy and paste the above and change our address to your own website