What specific robot txt tag officially stops google from posting your links on their web site?

Also which specific robot txt tag prevents web crawlers, especially deep web from caching your pages and indexing them?
Also which robot txt deletes any cached page google and web crawlers already have?

Popular Posts:

Related Posts:

2 Responses to “What specific robot txt tag officially stops google from posting your links on their web site?”

  • Tigger:

    You can block them with your spam filter and Firewall

    Locate the offensive ip address and put this in your denial of service box on the firewall and spam filter

  • Nate D:

    To block Google from all files:

    User-Agent: Googlebot
    Disallow: /

    To block all agents from a folder:

    User-agent: *
    Disallow: /folder1/

    To block from a file:

    Disallow: /private_file.html

    Your robot.txt file is just a guide, it can not delete cached pages.
    Though you can get a Webmaster Tools account with Google and tell them to delete certain pages.
    http://www.google.com/webmasters/tools/

Leave a Reply

Featured Products

Archive
Gruvisoft Donations