# This file is to help prevent the crawling and indexing of certain parts of your site by web crawlers and spiders. # By telling these "robots" where not to go on your site, you save bandwidth and server resources. Sitemap: http://www.legacycharter.com/sitemap.xml # Deny key bots User-agent: grub-client User-agent: grub User-agent: larbin User-agent: wget Disallow: / # For all robots User-agent: * Crawl-delay: 10 # Directories Disallow: /cgi/ Disallow: /cgi-bin/ Disallow: /img/ Disallow: /images/ Disallow: /incl/ Disallow: /includes/ Disallow: /stats/