Roger Clarke's Web-Site
© Xamax Consultancy Pty Ltd, 1995-2013
|Identity Matters||Other Topics||Waltzing Matilda||What's New|
Principal, Xamax Consultancy Pty Ltd, Canberra
Visiting Fellow, Department of Computer Science, Australian National University
Version of 1 June 1998, but revised several times, most recently 13 Jan 2001
(original of mid-1996; this version of 2 Feb 1997; significant updates in Aug 1997, incl. Appendix 3 created and later revised; some legal references added in late 1999; 'new-age cookies' references updated on 22, 28 Oct 2000, some corrections to the history in App. 1 on 13 Jan 2001)
© Xamax Consultancy Pty Ltd, 1996-2001
This paper is at http://www.rogerclarke.com/II/Cookies.html
New RFCs were published in early October 2000, which are intended to address some of the serious problems created by cookies. For details, see below.
Cookies are a particular means of maintaining 'state' information, so that a web-server 'remembers' where its conversation with a particular web-browser is up to.
This document provides an outline of 'cookies', an analysis of their privacy and consumer rights implications, an explanation of how marketers and others can use them responsibly, and a review of developments in the area. References are provided.
The document was originally developed as a case study to support projects on 'understanding cyberculture' and 'encouraging cyberculture'.
Netscape's web technology includes a feature called 'cookies'. A cookie is a record stored on a user's machine, as a result of a web-server instructing a web-browser to do so. It is sent to an appropriate web-server along with a request for pages.
The process is as follows:
The matching logic applied in step 4 above is somewhat complex. The server nominates:
The parameters may therefore be very specific (a particular URL) or very general (any document on any web-server in a particular domain).
This is very loose, because a site can set cookies to be delivered to many different sites. For example, within the .gov.au domain (which covers the Australian federal government plus the governments of all 8 States and Territories), the web-servers used by many agencies set cookies for .gov.au, when they should be set for at worst the specific agency, and really only for the specific page or sub-site. If a government wanted to establish surveillance over people's electronic interactions with governments, that would be an effective, but thoroughly surreptitious/underhand, way of doing so.
In short, there's considerable scope for data in cookies to be accessed by multiple web-servers in relation to multiple pages, rather than just for a specific, narrowly-defined purpose.
Details are provided in Netscape's specification for Cookies.
Cookies have a variety of potential applications, some of which are highly likely to be attractive to users. In particular, it could form the basis for a profile managed (and protected) by the user, containing membership-numbers, login-ids and passwords, user-interface preferences, and search-related data (such as areas of interest and common synonyms).
They may find it convenient to merge their knowledge of the user with that of other agencies, and perhaps also that of consumer marketing companies.
Law enforcement agencies can be expected to take an interest users' cookies, and to gain access to their contents. Through access to the record-identifier, a search warrant could be executed on-line, and without the user's knowledge.
In many jurisdictions, even in 'the free world', law enforcement agencies have legislative right of access to information held by telecommunications companies (telcos) without any explicit third-party authorisation such as a search warrant. The Australian authority is s.88(3)(g) of the Telecommunications Act 1991 (Cth). This empowers law enforcement agencies to request (though not to demand) disclosure of information from telcos (i.e. Telstra, Optus, Vodafone, and shortly many more companies), where the information is 'reasonably necessary for an investigation'.
There is speculation that law enforcement agencies may shortly seek to extend the application of this section to information held by ISPs. It will require a skilled and patient lawyer to unravel the mess of law, in order to analyse the situation. For those who want to try, the relevant section is available on AustLII.
Access to cookie-derived data in these circumstances may be beneficial from the viewpoint of efficient conduct of government business, and law and order; but not from the viewpoint of a miscreant being set upon by the authorities. Productivity enhancement and the exercise of control over terrorism are attractive uses; surveillance and repression much less so.
Cookies provide marketers with the ability to maintain user-profile data, and to do so on the user's own machine.
Moreover, corporations that form strategic partnerships could exchange their record-identifiers, and hence pool their user-profile information.
Those who subscribe to the 'efficiency of marketing communications' hypothesis may think these are 'good things'. The vast majority of the net-community would appear to think otherwise.
Generally, cookies are set because a designer quite consciously makes it happen. During late 1996, however, it became apparent that a range of web-pages set cookies without their designers' knowledge. This included several sites which, for ideological and/or strategic reasons, had strong pro-privacy policies, and were highly embarrassed to discover that they were setting cookies. (I'm being coy, and not mentioning the sites I caught out, because I'm entirely satisfied that it was accidental).
The reason for this seemingly incongruous situation is that:
Over the last couple of years, it's become apparent that the primary reason for the flood of non-functional cookies is, surprise, surprise, Microsoft's incompetence: version 4 of MS's IIS web-server has a serious bug in it. Morover, there are reasons not to move to later versions too quickly (i.e. a series of other MS blunders), and hence many sites and their users are still stuck with this nonsense in late 2000!).
Many members of net-communities react badly to the use of, and even the very concept of, cookies. Author Simson L. Garfinkel, firstname.lastname@example.org, summed up the stronger version of their opposition as follows: "I basically equate cookies to the notion of a store being able to tattoo a bar code on your forehead, and then laser-scan you every time you come through the doors".
The factors that underlie this level of concern appear to be the following:
In the language used in some other privacy-invasive technologies (such as Caller-ID in telephony), permission for write-access is 'per-call opt-out', with no 'global opt-out' capability. This approach is at the opposite extreme from the consumer-friendly alternative of 'global opt-in' (which involves having a parameter that defaults to denial of access, but can be changed to permit access), supplemented by 'per-call opt-in' (which enables the user to leave the global setting as 'deny access', but over-ride the parameter on those occasions when they want the cookie to be written).
Although a few people were aware of them as early as the first quarter of 1995, the attention of the net-community was only drawn to cookies somewhat later, around the first quarter of 1996. This is briefly discussed in Appendix 1. The warnings became more mainstream by mid-1996; see http://catless.ncl.ac.uk/Risks/18.19.html#subj3, and http://catless.ncl.ac.uk/Risks/18.40.html#subj9.
By late 1996, there was copious media-coverage of the feature, including http://catless.ncl.ac.uk/Risks/18.63.html#subj10.
By December 1996, advice was beginning to appear in various places as to how to cope with cookies. See, in particular, http://catless.ncl.ac.uk/Risks/18.65.html#subj10, http://catless.ncl.ac.uk/Risks/18.67.html#subj4, and http://catless.ncl.ac.uk/Risks/18.68.html#subj11.
The most intriguing recommendation as to how to deal with them is at http://catless.ncl.ac.uk/Risks/18.70.html#subj10. This suggests that the Netscape implementation actually contains an undocumented ability to deny cookies at a global level, at least under Unix. If so, perhaps Netscape can recover the damage that the issue has done to their previous reputation as 'good guys' of the Internet.
Appendix 2 summarises the currently recommended ways of dealing with cookies.
Each of the work-arounds is capable of being neutralised by a determined provider of server-browser technology, and especially by a consortium of commercial interests, e.g. CommerceNet.
To date, this only aware of only one site that has been claimed to refuse access to a browser that does not permit cookies to be set; but the (limited) tests performed could not force a refusal.
It is feasible to apply the existing, highly deficient class of cookies in a responsible manner. Appendix 3 explains how.
I've yet to find a good reference that examines the law as it relates to cookies.
Sites worth searching include:
The Cookie feature is a prototype of a potentially very useful form of net-behaviour. In its current form, however, it is highly intrusive, and mostly undertaken surreptitiously. The vast majority of net-users are unaware of it, and most of those who know about it don't know what to do about it. At present, it's trivially simple to prevent or subvert; but commercial interests may cause that to change. It is feasible for managers of web-sites to apply cookies repsonsibly.
Cookies were an innovation of Netscape's sometime in 1995. They were apparently supported by Netscape Navigator 1.0 (but nobody realised), but began to be used when Netscape 2.0 was released. It appears that they were documented for developers, but not for anyone else. In short, an intrusive enhancement to the web was slipped in surreptitiously.
Most of us who were active in Internet and web policy matters only became aware of the existence of cookies in mid-February 1996. Public concerns rose rapidly, for the very good reasons outlined in this document.
Shortly afterwards, in February 1997, a more general mechanism to support state-maintenance was proposed as RFC 2109 'HTTP State Management Mechanism' (by Dave Kristol of Bell Labs and Lou Montulli, then of Netscape).
Dave had to fight a long, slow battle to get the need for a responsible cookie-architecture onto IETF's agenda. Despite my raising it directly with Tim Berners-Lee, W3C avoided the matter entirely, reflecting the increasing constraints on its freedom of action arising from it desire to avoid upsetting its corporate sponsors.
At last, Dave's efforts paid dividends. The revised document,was published in early October 2000 as a 'Proposed Standard', RFC2965 'HTTP State Management Mechanism' (25 pp., by Dave Kristol, Bell Labs and Lou Montulli, now of Epinions.com).
It's now up to all of us to put pressure on IETF and W3C to adopt the formal proposal; and on all web-server and web-browser providers to implement cookies in the responsible manner proposed.
In addition, the concerns about the existing cookie mechanism were addressed in a 'Best Current Practice document, BCP 44, RFC2964 'Use of HTTP State Management' (7 pp., by K. Moore, University of Tennessee and N. Freed, Innosoft).
Note that a corespondent, Jeff Hodges, has pointed out to me, "The specification of HTTP/1.1 as commonly implemented and presently used in practice on the Internet is ambiguous". It would be very helpful to everyone if IETF could consolidate the many RFCs involved, including 2109 (State Management), 2616 (HTTP 1.1), 2617 (Authentication), 2817 (HTTP over TLS), 2935 (OTP), and now 2964 (Use of State Management) and 2965 (State Management) - and, if they're still alive, maybe also 2069 (Digest Access Authentication), 2186 (Cache Protocol), 2518 (Webdav), 2818 (HTTP over TLS).
I've not yet assessed RFCs 2964 and 2965 against the consumer requirements laid out in this document; but it was developed with many of the problems in mind. I hope to get an assessment up in this location some time soon.
Cookies are a relatively primitive and restrictive form of interference with the client workstation compared with Java. When I originally wrote this piece in 1996, I expected cookies to be quickly superseded by Java applets, and end up being of largely historical interest.
In fact, Java implementations have had enormous problems, and Java applets frequently kill web-browsers, variously because of incompetent applet-programmers and poor-quality interpreters embedded in the web-browser. So cookies are still around, and are still greatly-used.
The idea may have originated in 1992-93, within MIT's Kerberos security scheme - see http://catless.ncl.ac.uk/Risks/14.36.html#subj6 (although the purpose of the original 'magic cookie' was much more specific than Netscape's general purpose tool). (The Webopaedia entry at http://www.sandybay.com/pc-web/cookie.htm traces it to right back to the name of a Unix token).
Cookies were discussed in Netscape Technical Note 20019 of 28 July 1995. This points to an undated, short, preliminary specification.
It appears that cookies were addressed in the documentation supporting Netscape 2.0, although at least some versions of the Netscape Cookies and Privacy FAQ state that the feature was supported from Netscape 1.0 onwards. Correspondents have reported being aware of the facility (based primarily on informal sources) as early as mid-1994.
Along with a lot of other people, my first knowledge of them was in mid-February 1996 ('Leading Web Browsers May Violate Privacy of Users' Computers, Activities' by Lee Gomes, San Jose Mercury News, February 13, 1996, distributed by Phil Agre on his rre service, and referring to earlier but unspecified discussions on newsgroups, and to an article during the same week in 'The Times' of London).
Cookies appear to have become a mainstream device as Netscape Navigator 3.0 swept the market, in mid-1996. They were also implemented in Microsoft's Internet Explorer 3.0. (IE wasn't of much consequence at that stage. As always, MS came from behind and used heavy-handed methods to knock off the opposition).
A paper on the evolution of cookies is at http://portal.research.bell-labs.com/~dmk/cookie-ver.html.
To force the browser to get your approval before recording a cookie:
To prevent cookies-records being written:
Note that web-browsers commonly store cookies in main-memory. Hence the technique of preventing cookies being written to disk probably will not prevent the browser from storing and transmitting cookies during the (possibly very long) period between browser-startups.
In mid-December 1996, a product called PGPcookie.cutter was announced, for availability in January 1997 for $US19.95. See http://www.pgp.com/newsroom/prel7.cgi. Other products have since followed.
Use Netscape Navigator Version 4, which is stated to provide additional means to enable a user to manage cookie-settings. Possibly MS Internet Explorer Version 4 has extended capabilities too.
Designers of web-sites can apply cookies in a manner that addresses the interests of consumers. The principles are as follows:
This analysis has used various of the sources mentioned above, and has greatly benefitted from discussions on the link e-list during the periods January and July/August 1997, archived at http://online.anu.edu.au/mail-archives/link/. It also drew heavily on that wonderful resource Risks Forum, archived and searchable at http://catless.ncl.ac.uk/Risks.
The content and infrastructure for these community service pages are provided by Roger Clarke through his consultancy company, Xamax.
From the site's beginnings in August 1994 until February 2009, the infrastructure was provided by the Australian National University. During that time, the site accumulated close to 30 million hits. It passed 40 million by the end of 2012.
Sponsored by Bunhybee Grasslands, the extended Clarke Family, Knights of the Spatchcock and their drummer
Xamax Consultancy Pty Ltd
ACN: 002 360 456
78 Sidaway St, Chapman ACT 2611 AUSTRALIA
Tel: +61 2 6288 1472, 6288 6916
Created: 30 January 1997 - Last Amended: 13 January 2001 by Roger Clarke - Site Last Verified: 15 February 2009
This document is at www.rogerclarke.com/II/Cookies.html