The History of the Do Not Track Header

Last month, both the FTC and Commerce Department published privacy reports that mentioned the possibility of a Do Not Track mechanism. Most people, even those who follow privacy issues, didn't really understand how such a mechanism would work, or where the idea came from. The goal of this lengthy blog post is to try and shed a bit of light on that.

The History of Do Not Track

In 2007, several public interest groups, including the World Privacy Forum, CDT and EFF, asked the FTC to create a Do Not Track list for online advertising. In a very savvy move, these groups named their scheme such that it instantly evoked the massively popular Do Not Call list. That is, even if the average person did not know how the Do Not Track list worked, it would sound like a good idea.

The public interest proposal would have required that online advertisers, not consumers, submit their information to the FTC, which would compile a machine readable list of the domain names used by those companies to place cookies or otherwise track consumers. Browser vendors and 3rd party software makers could then subscribe to this list, and effectively block many forms of tracking. It sounded like a great idea, but, it went nowhere, and as the Google Trends chart below shows, it was largely forgotten by the media until 2010.



What happened to bring Do Not Track back to life? FTC Chairman Jon Leibowitz.

On July 27 2010, the Senate Commerce Committee held a hearing on the topic of online privacy. In his oral testimony at the hearing, Leibowitz stated that the commission was exploring the idea of proposing a "do-not-track" list (he appears to have gone off the official script, as the phrase "do not track" does not appear in his formal written remarks.)

Once the concept (even in the abstract) of Do Not Track has been brought back to life, journalists covering the story assumed that it was the public interest groups' proposal that was now actively being considered by policymakers. However, over the space of a few months, a completely different mechanism, one which relies on web browsers sending a header, seemed to gain momentum.

This seems to have caught many in industry and the press off guard. No one knows where the idea came from, or how it managed to displace the previous public interest groups' effort. The purpose of this blog post is to try and clear that up.


Opt Out Cookies

For more than a decade, the major online advertisers have offered "opt out" mechanisms, through which consumers could signal to the companies that they did not want to receive advertisements targeted based on their online browsing habits. These opt outs worked via cookies (one specific to each ad network), which a consumer could either obtain by visiting each advertising network's website, or (if the company was a member of the Network Advertising Initiative (NAI), from the NAI website.

While certainly a step in the right direction when they were first offered, the opt out cookies have numerous flaws, the most important of which, is that as cookies, they are deleted whenever consumers attempts to protect their privacy and erase other tracking cookies. Quite simply, using the built in browser controls, consumers cannot instruct their browser to "keep the opt out cookies, but delete everything else." Consumers thus have to re-obtain these opt out cookies each time they delete their cookies, or, perhaps more likely the case, privacy conscious consumers gave up on the formal opt outs, and instead relied on frequent cookie deletion as a more reliable means to opt out.

In March 2009, Google released a browser add-on that made Google's own doubleclick.net behavioral advertising opt out cookie permanent. Thus, with the add-on installed, users could freely delete their cookies whenever they wanted without accidentally removing Google's opt out cookie. While this was a great move on Google's part, there were more than 100 other advertising networks, and so even if Google's opt out cookie persisted, these other opt out cookies would be erased whenever a consumer took steps to protect their privacy.

My Targeted Advertising Cookie Opt-Out (TACO) add-on

A few days after Google released their opt out tool I bumped into security researcher Dan Kaminsky at a conference. I'm afraid I don't remember the specifics of our conversation anymore, but generally, we spoke about flaws in the opt out system, Google's new tool, and possible technical alternatives to cookie based opt outs, including a browser header.

Soon after (and likely inspired by) my conversation with Dan, I downloaded Google's tool (which the company had released under an open source license) and modified it to include the opt out cookies for several other behavioral advertising networks. I published my TACO add-on and within days hundreds of people downloaded and installed it.

A few days later, Dan emailed me, and urged me to include a browser header in TACO -- not because it would have any immediate impact (since no ad network would look for it), but because it would be a clear expression of user intent:

The reality is you can be tracked no matter what you do or don't set. However -- humor me: Just add an "X-No-Track: user-opt-out=explicit" header to all HTTP requests, and add window.tracking-opt-out=explicit to every DOM.

Oh, and put a comment in the source above it, calling it the Holy Hand Grenade :)

Trust me :)

At the time, I dismissed Dan's suggestion. I wanted to build a tool that would actually improve user privacy, and since cookies were the only way for consumers to opt out, I thought my time was best spent improving that experience. However, on the TACO home page, I noted that a header mechanism would be a far superior replacement for opt out cookies:

The use of individual opt-out cookies for each advertising company is sub-optimial (in fact, the current situation totally sucks). We shouldn't have to identify and seek out each company that might track us in order to opt out. This tool currently supports 90 different advertising networks, some of which require multiple cookies (for different domain names). As a result, this tool installs 90+ opt-out cookies into the browser (they're all generic, and contain no unique, or identifiable information). Since there are still quite a few networks that the tool does not support, it is quite easy to see that the tool could eventually install 100 or more cookies in a user's browser. This solution simply does not scale.

In an ideal universe, we would be able to set a single cookie in the browser stating our preference to be not tracked, without needing to first identify individual advertising networks. Consider, after all, the approach taken with the hugely successful do not call list. You add yourself to a single list, which all telemarketers are then required to honor.

However, for privacy reasons, cookies cannot be accessed by websites hosted in domains different than those that set the original cookie. That is, if google.com sets a cookie in your browser, microsoft.com won't be able to read it if you visit their site. For 99% of cookies (such as the session cookies used to authenticate your login to Facebook), this is a really good idea. However, for a universal opt-out cookie, this presents significant problems.

As a result, cookies are the wrong technology for a universal opt-out mechanism.

One alternative approach would be to permit the browser to send an opt-out HTTP header, which it could then transmit to every web server which the user connected. Such a scenario would require that Microsoft, Mozilla, Apple and Google sit down to design such a technical spec. It would also require that the big advertising networks agree to honor such a HTTP header based method for opt-out.

I spent much of the summer of 2009 immersed in the world of online advertising. This included numerous conference calls with attorneys at advertising networks, and evenings spent on the web, locating new advertising networks with opt out cookie I could clone, and add to TACO. This lead to several updates of my increasingly popular tool, which eventually grew to include more than 100 different opt out cookies.

However, it was never my intention to maintain a browser plugin (even a successful one) -- I am a researcher and an activist, and so my goal in creating TACO was primarily to poke the advertising industry in the eye. As such, within weeks of creating TACO, I reached out to the folks at Mozilla, and begged them to take TACO off my hands by building similar functionality into the Firefox browser.

While several individuals at Mozilla were receptive to the idea of TACO (and had installed it onto their own computers), they weren't so in love with the idea of shipping 100 different opt out cookies with their browser, or having to maintain and update the list for new add networks. Quite simply, TACO was an inelegant kludge, and didn't scale. In March of 2009, Mozilla's VP of Engineering Mike Shaver emailed me to state his own preference for a header:

Could we not just standardize/promote a header like X-Tracking-Opt-Out, and ask the tracking groups to honour it? Simpler to specify, simpler to update (the null case, in fact), forward-effective as new ad networks add support, and separated from the implications and implementation of cookies.

The Do Not Track Header

The header approach suffered from a serious chicken and egg problem. No ad network was willing to look for, or respect the header (primarily because no one was sending the header). Likewise, because no one was looking for the header, the browser vendors weren't ready to add support for it to their products.

In July of 2009, I decided to try and solve this problem. My friend and research collaborator Sid Stamm helped me to put together a prototype Firefox add-on that added two headers to outgoing HTTP requests:

X-Behavioral-Ad-Opt-Out: 1
X-Do-Not-Track: 1

The reason I opted for two headers was that many advertising firms' opt outs only stop their use of behavioral data to customize advertising. That is, even after you opt out, they continue to track you. There are a handful of firms though that do promise to no longer track you when you opt out. One big problem is that it is very difficult for consumers to figure out which company is doing what -- since they all use the term opt out.

I assumed that any header-based system would be voluntary, and so by using two different headers, I would be able to play nicely with whatever a firm was willing to do. That is, if a firm currently agreed to opt consumers out of all tracking, then the firm could look for the Do Not Track header, but if the firm refused to provide a tracking opt out, they could at least agree to respect a behavioral advertising opt out header.

In mid July 2009, the Future of Privacy Forum organized a meeting and conference call in which I pitched the header concept to a bunch of industry players, public interest groups, and other interested parties. I was perhaps slightly over-dramatic when I told them that the "day of reckoning was coming", for opt out cookies, and that it was time to embrace a header based mechanism. I told them that I planned to add the headers (enabled by default) to my TACO add-on in a future release, after which, I would be able to argue that hundreds of thousands of consumers were sending this signal that the advertising firms were ignoring.

In the end, none of the advertising firms showed any interest in the header. A couple months later, I started working at the Federal Trade Commission, and ultimately decided against including the header in TACO, as I thought it might rock the boat at my new job.

In mid 2010, when the FTC Chairman breathed life back into the discussion of Do Not Track, the header I had implemented and lobbied for somehow managed to catch the attention of privacy advocates, public interest groups, regulators and even browser vendors. Ultimately, the Behavioral Advertising Opt Out header seems to have been discarded, and instead, focus has shifted to a single header to communicate a user's preference to not be tracked.

The policy of Do Not Track

The technology behind implementing the Do Not Track header is trivially easy - it took Sid Stamm just a few minutes to whip up the first prototype. The far more complex problem relates to the policy questions of what advertising networks do when they receive the header. This is something that is very much still up in the air (particularly since no ad network has agreed to look for or respect the header).

Over the last few months, a number of privacy experts, including Arvind Narayanan and Jonathan Mayer at Stanford, Lee Tien and Peter Eckersley at the Electronic Frontier Foundation, and Harlan Yu at Princeton University have worked to come up with a solid proposal that will help to shape this more complex part of the debate.

If industry (or the FTC, Commerce and Congress) ultimately settle on the header based approach, there will likely be an intense lobbying effort on industry's part to define what firms must do when they receive the header. Specifically, they will seek to retain as much data as possible, even when they receive the header. As such, the devil will be in the details, and unfortunately, these details will likely be lost on many members of Congress and the press.

TACO 2.0 released

Update: Mozilla has approved TACO 2.0. All current TACO users should see a prompt to update the add-on the next time they restart Firefox.

I am happy to announce the release of version 2.0 of the Targeted Advertising Cookie Opt-out (TACO) Firefox add-on.

This version has been completely rewritten from scratch, primarily by Daniel Witte @ Mozilla Corp. It also includes opt-out cookies for 6 additional advertising companies: Snoobi, comScore VoiceFive, Hurra, Criteo, Coremetrics and EyeWonder.

I am waiting for the nice folks at Mozilla to read through the code and then approve it. If you lack patience, and simply cannot wait, TACO 2.0 can be installed by clicking here. Otherwise, wait a few days until Mozilla approves it, and then the 100,000 or so existing TACO users should receive an automatic update to this new version.

A total rewrite

The original TACO was essentially a fork of Google's Advertising Cookie Opt Out Plugin. Google's original tool included one cookie -- I simply modified it to include an additional 100 or so opt out cookies.

The problem is that Google's original code wasn't all that good -- it would reload all of the opt-out cookies each time a new window/tab was opened, and then force them to be reloaded again every 10 minutes, even if none of the opt-out cookies had changed.

Perhaps this isn't such a big deal for a tool that is designed to install a single opt-out cookie. However, it clearly didn't scale well.

Unfortunately, my Javascript skills are pretty horrible, and so I really wasn't up to the task of rewriting TACO by myself. Luckily, Daniel Witte, Mozilla's resident cookie guru offered to lend a hand, and eventually rewrote the entire add-on from scratch.

This new version is considerably faster, and no longer re-installs 100+ cookies into the browser each time a new tab/window is opened nor does it reinstall them again every 10 minutes after that.

Blocking third party cookies

One of the biggest complaint from TACO power-users was that the tool would not function when the user had configured the browser to block all 3rd party cookies (a suggested practice, and one which both Safari and Chrome do by default). I am happy to announce that TACO now plays nicely with blocked 3rd party cookies, and so the paranoid amongst you should feel free to go ahead and block them without having to worry about it breaking TACO.

A note about EyeWonder

Finally, blog-readers may remember that I recently pointed to EyeWonder's non-existent broken opt-out as an example of one the worst practices in the industry.

After 9 days, it looks like the company finally designed and implemented fixed the opt-out, and so users of TACO 2.0 are automatically opted out of all of EyeWonder's behavioral advertising.

My response to Safecount

Thank you to all the people who emailed me their thoughts, and those who left comments on my previous blog post regarding Safecount's request that their cookie not be included in TACO.

After thinking things through, I sent Tom Kelly, the company's COO this response:

Hi Tom,

The feedback I received on my blog was not particularly supportive of your request.

I have thought things through, and decided to do the following:

1. I have added a note to the TACO home page, which states:

"Safecount argues that they are not a behavioral advertising company. However, they are a member of the Network Advertising Initiative, and do collect detailed data on the browsing and ad-viewing habits of Internet users. Furthermore, this data is often collected with no notice provided to the user on the web page where Safecount's tracking code has been embedded. "

2. If you or your engineers would like to spend a day or two creating the code necessary to enhance TACO, which will provide users with a list of the companies whose opt-out cookies are available and or active, and a way for users to disable individual opt-outs, I would be happy to look over such a patch, and if it is decent, consider applying it to the mainline TACO codebase.

Such a feature would be nice, but frankly, it isn't important enough for me to spend my own time developing it. However, just to be clear, even if such an ability to disable individual opt-out cookies existed within TACO, I would have them all turned _on_ by default. That is, users would need to go into a preferences window, scroll down through 60 or so company names (since Safecount is not at the beginning of the alphabet), and then choose to disable your opt-out cookie.

As you know very well (and in fact, your business model depends upon it), few consumers ever take the time to dig through preference windows or look into privacy policies in order to learn about particular company's activities. Thus, were such a feature to exist, I highly doubt if more than a handful of consumers would ever make use of it.

In any case - I would be happy to consider such a patch, but I suspect that it probably isn't worth your engineers' time to work on it.

Cheers

Chris

Safecount: Please opt us out of TACO

This afternoon, I received an interesting set of emails from Tom Kelly, the Chief Operating Officer at Safecount.

Hi Christopher -

A colleague forwarded us a link to your Taco download page where we were surprised to see Safecount listed with the likes of many ad networks.

While we, and I, find your development efforts to be interesting, and nicely in line with the entrepreneurial spirit of the web, some of the classifications on your page are quite mis-leading to consumers.

Safecount is a research company and we occasionally invite certain website visitors randomly to volunteer their opinions. We don't sell any products, we don't target anyone with advertising based on behavior or attitude, and we only work with publishers who give us permission to perform research on their sites.

That's the danger of generic 4th party cookie blocking, it ends up blocking web efforts OTHER than ad revenue, behavioral targeting profiteers. Maybe you'll consider removing Safecount from your list.

Respectfully,

- tom

After asking him if I could post his email to my blog, he followed up with this:

Sure thing, Chris. My point is that, while Safecount does place cookies on user's browsers based on certain ads they've seen:

A) we don't use that info to target any marketing or advertising to them - we're not a behavioral targeting group
B) we're 100% transparent in the cookies we do place

As a matter of fact, one can go to www.safcount.net and view ALL of the info we have for their computer (not personal info). There they can also delete that data and tell us how often they'd agree to be invited to take a quick survey, including "never". We're as much about control and transparency as I think you are.

Thanks, Chris.

- tom

It has been nearly four months since the first version of TACO was first released. The latest version supports 84 different behavioral advertising firms, has been downloaded nearly 250,000 times, and is in daily use by nearly 80,000 users. That means that my tool is responsible for 6.7 million opt-out cookies (actually, it's more, due to the fact that some networks require multiple cookies for different advertising domains). Holy cow!

In those four months, this is the first time that an advertising industry executive has asked me to remove his company's opt-out cookie from TACO, and so I am honestly not quite sure how to react.

My initial reaction is to say no, for the following reasons:

1. I have created TACO for fun, as a side project. I don't charge for TACO, and I have a day job (well, actually, several). I really don't have time to evaluate each advertising company one by one to figure out if the company engages in a good or bad activity. If consumers want that level of analysis, they are free to use the "complete" or "selective" opt-out tools provided by PrivacyChoice -- which is run by a former Yahoo! advertising executive who has Seen the Light, Loves Privacy And Who You Should Totally Trust (TM).

2. Picking individual advertising industry companies who should or should not be included in TACO is a slippery slope, which will open me up to criticism, and accusations of abuse of power. TACO currently includes every generic, non-identifiable opt-out http cookie of all the online advertising industry companies that I know about. This is an easy standard to adhere to, and should protect me from accusations of bias.

3. Safecount, WPP (the mega advertising firm which owns it), the Network Advertising Initiative and others are free to make their own competitors to TACO which provide users with more choice, which provide users with less choice, which make it more or less difficult to opt out, or which make you dinner and do your laundry. TACO is open source, so they are even free to fork my code, and save themselves the weekend of coding it will take to create it from scratch.

4. Safecount is an advertising industry firm, which uses long term cookies to track the browsing and other activities of end-users. The company might not be in the behavioral advertising business, but it is certainly in the collection of consumer data business, which is still creepy.

5. Safecount has provided consumers with the ability to opt-out of its data collection/use, but then objects when tools like TACO actually make it easy for consumers to opt-out. 99% of consumers have never heard of the company, and so wouldn't even know to visit their opt-out page in the first place.

6. If the company is really "as much about control and transparency" as I am, they could switch from an opt out model to an opt in model. Let consumers who value the survey taking experience choose to have data on their browsing across multiple websites collected and analyzed. If the company switched to this model, the opt-out mechanism provided by TACO would be moot.

7. Likewise, while consumers can "go to www.safcount.net and view ALL of the info we have for their computer (not personal info)," this simply isn't good enough. It is totally unrealistic to expect consumers to visit the websites of 90-100 different advertising firms to "view the data collected on them", evaluate it, consider each company's 20+ page privacy policy, and then evaluate the kind of business and data relationship that they'd like to have with that firm.

Consumers don't opt-out of telemarketing from individual advertising firms after evaluating each firm's policy on calling during dinner hours -- No. They sign up for a single do-not call list, and are then free of the annoyance. We need the same for the online advertising industry. A single opt out for all data collection and usage.

After writing this all down, I think I am even more convinced that leaving Safecount in the list of opt-outs provided by TACO is a good idea.

However, I suppose a reasonable case can be made that the company is not a behavioral advertising firm -- and so I am open to at least changing the language on the TACO page to note that Safecount is merely an advertising firm that collects detailed information on the browsing and web viewing activity of Internet users.

Blog readers -- do you have any thoughts on this? Please leave a comment.

One Man's War on Advertising

I can happily report that two more companies, Blue Kai and Media Math, have in the last few days modified their advertising systems to now use non-identifiable opt-out cookies. Hurrah.

I've received unofficial word that at least one other company is making a similar switch. One by one, most of the online advertising companies are realizing that identifiable opt out cookies are bad for consumers, and more importantly, really bad PR for them.

Expect a new version of TACO in the next few days, incorporating these new opt-out cookies.

As the weeks go on, I wouldn't be surprised to see that initial list of 17 bad advertisers shrink.... to just two: Microsoft and Yahoo. These firms are the largest of the offenders, are slow unwieldy corporations which are unable to turn on a dime, and in some cases, simply don't see why they should be forced to stop tracking users.

If we do get to a point where only these mega advertising titans are refusing to provide consumers with an anonymous way of opting out of tracking and targeting, progress may depend upon legislators showing a bit of interest in the topic.

---------------

In other news, Jim Harper @ CATO took another good-natured another whack at me this week.

Writing at the Tech Liberation Front, he stated that:

Chris is deeply focused on advertisers and his dislike of being tracked by advertisers. Though it is not absolute, I have a preference against tracking by anyone other than sites that I know, like, and trust. I’m no more worried about advertisers than any entity that would track my surfing - and there are many.

To say that I am solely focused on advertisers is unfair and incorrect. I wish to avoid being tracked by all parties, be they Facebook, the RIAA, online advertisers, or the US Government (remember, after all, the name of this blog). However, while the advertising industry has collectively provided consumers with a mechanism to opt-out of their more creepy practices (albeit one that is difficult to use), other would-be-watchers have not.

If Mr. Harper is aware of an opt-out cookie that I can load into my browser to opt myself out of the National Security Agency's illegal monitoring of domestic Internet traffic, I hope that he will let me know.

I will be the first to admit that privacy on the Internet sucks. I wish it were better. I wish we didn't have evil telecom companies who believe that they can monetize their customers' web browsing habits. I wish our government, even with the new President, respected the Fourth Amendment.

I am not claiming that my TACO add-on is perfect, or that it solves all of the privacy issues on the web. It is a specific technical solution for a particular policy problem. It is not a comprehensive solution to all the woes of web privacy -- it is just a way, I believe, for the little guy to reclaim a tiny little bit of that long forgotten right to be left alone.

Later on, Mr. Harper asserts that I am calling for new legislation:

With the right law in place, Chris appears to believe, “[t]he Federal Trade Commission and Congress would likely take an interest” when advertisers tried to skirt opt-out cookies, using other technologies to glean information about Web surfers’ interests.

Perhaps I was unclear in my previous post, but I do not believe that new legislation is required to go after advertisers who continue to engage in targeted advertising even after the user has opted out. The FTC has a clear legal mandate to go after those who engage in deceptive and unfair business practices. Advertisers who ignored their own opt-out cookies would seem to be engaging in an unfair and deceptive way. I would argue that the FTC already has all the authority it needs to go after such firms.