Now that is well worth doing, if you have any respect for the people who look at your web site. I'm assuming you have a site that is not designed to rip people off, ram advertising at them, etc.
It says in this excellent document...
39. When a website is first visited (by any ISP customer) the pages are not inspected. Instead, a request is queued to fetch the site's "robots.txt" file; viz: a file maintained by the website owner which tells web crawlers and other automated systems which parts of the website should not be indexed or processed.
40. Once the robots.txt file (if any) has been fetched, it will be cached. The cache retention period will be value set by the website using standard HTTP cache-control mechanisms, or for one month if no period is specified. The minimum period that the file will be cached for is two hours.
41. The robots.txt file will be inspected and URLs that fall within forbidden areas of the website will not be processed by the Phorm system.
42. This mechanism, which will permit website owners to opt their pages out of the Phorm system, does not seem to have been previously described in any of Phorm's documentation. They were unable to provide an explanation as to why this had not previously been disclosed.
It is pretty damned clear why they kept quiet about it. I shall be fixing my web site so people who view it are not spied on.
Another thing that has not been mentioned, and you can see why, is that if anyone thinks BT won't also pass all this data to the Home Office, MI5, MI6 and the CIA, they are fools. That is the most likely reason the Home Office refuses to prevent this from being done to us.
Munzly
Useful data - I have taken note.