The other day someone contacted me to say that the Direct Marketing Association had quietly ditched its Your Choice Preference Scheme. This doesn't seem to be the case – what seems to have caused confusion is that they fail to mention the existence of the scheme on their page about preference services. You'd almost expect this to be the case; it's a dummy scheme anyway.

While searching the domain dma.org.uk for strings such as "your choice" and "junk mail" I did find that the DMA website tells all search engines not to index the website – with the exception of Google. If you use DuckDuckGo, for instance, you get results like this:

You get similar results if you use other alternatives for Google. The reason these search engines aren't allowed to display results is that the DMA website instructs them not to do so using a robot.txt file:

User-agent: Googlebot
Allow: /
Disallow /user/kerry-mailbird-co-uk

User-agent: *
Disallow: /
Disallow /user/kerry-mailbird-co-uk

The first three lines tell the Googlebot that it's allowed to index all pages on the site, apart from the page /user/kerry-mailbird-co-uk. The second block tells all other search engines that they're not allowed to index any page and that they're also not allowed to index the page /user/kerry-mailbird-co-uk. (The last line is obviously redundant as the preceding rule already stated that nothing may be indexed – apparently it's rather important that Kerry Holden is hidden from view ;)

I suspect the DMA only wanted to prevent the website is listed on sites such as the Wayback Machine – it wouldn't be the first time they're trying to get rid of old and perhaps embarrassing content. If so they could have simply told their robot not to index the site:

User-agent: ia_archiver
Disallow: /

Or maybe they feel that people should only use Google?