Roger Clarke's Web-Site


© Xamax Consultancy Pty Ltd,  1995-2016

Roger Clarke's 'Evaluation of PIA Guidelines'

An Evaluation of Privacy Impact Assessment Guidance Documents

Published in International Data Privacy Law 1, 2 (March 2011) 111-120

PrePrint of 3 November 2010

Roger Clarke **

© Xamax Consultancy Pty Ltd, 2009-11

Available under an AEShareNet Free
for Education licence or a Creative Commons 'Some
Rights Reserved' licence.

This document is at


Privacy Impact Assessments (PIAs) have become mainstream in many jurisdictions since the mid-1990s. Considerable experience has been gained, and the features of effective PIA processes are capable of being described and communicated. A range of guidance documents have been published, but they vary considerably in their quality. This paper draws on the literature and professional experience in order to present a list of criteria whereby the quality of a PIA guidance document can be judged. It then applies the criteria to a dozen documents published by government agencies in ten jurisdictions.


1. Introduction

Privacy impact assessment (PIA) is a systematic process for evaluating the potential effects on privacy of a project, initiative or proposed system or scheme. (Clarke 2009), in the Computer Law & Security Review, traces the development of the concept, starting in the 1970s.

PIAs became progressively more common from the mid-1990s, particularly in (in alphabetical order) Australia, Canada, New Zealand and the U.S.A. A related and parallel development in Europe has been what have been referred to as 'pre-decisional assessments', which gave rise to 'prior checking' requirements under EU law (ICO 2007b, Appendix H).

A range of guides have been published, to assist organisations in performing PIAs and in achieving the intended benefits for the organisation and the affected public alike. Warren et al. (2008), also in CLSR, describes the process whereby the UK Information Commissioner's guidance document was prepared.

There is, however, considerable variation in the comprehensiveness and quality of existing guidance documents. The purposes of this paper are firstly to present a set of best practice criteria whereby the quality of PIA guidance documents can be assessed, and secondly to apply the list of criteria to a substantial set of existing documents.

The list has been developed through two processes. Firstly, it is grounded in the body of literature cited in (Clarke 2009). Secondly, it draws on the author's consultancy work over a 15-year period, which has included a substantial number of PIA assignments, the conduct of PIA training courses, and the development of PIA guidance documents.

The paper commences by providing a working definition of a PIA and identifying key characterisics that distinguish the PIA process from other procedures. It then presents the criterion list, structured under ten major headings. A dozen PIA guidance documents issued by government agencies in ten different jurisdictions are then evaluated. The quality of the documents is shown to be highly variable.

2. Nature of a PIA

A variety of definitions of the term 'PIA' were catalogued in Clarke (2009). The following encapsulates the key features:

Privacy impact assessment (PIA) is a systematic process that identifies and evaluates, from the perspectives of all stakeholders, the potential effects on privacy of a project, initiative or proposed system or scheme, and includes a search for ways to avoid or mitigate negative privacy impacts

A PIA needs to be distinguished from a range of other procedures undertaken by organisations. Important among them are the following:

These distinctions are reflected in the list of criteria presented below.

3. Criteria for Evaluating PIA Guidance Documents

The effectiveness of a guidance document depends on a range of factors. Some features may be desirable, or at least helpful. For example, the document may assist organisations by outlining the categories of benefits that can arise from a PIA process, and by identifying the various kinds of trigger that may give rise to PIAs being conducted. Those factors, are not, however, a guidance document's critical success factors.

This section identifies the key factors and discusses the most important aspects that need to be considered when evaluating a PIA guidance document's quality. For each criterion on the list, discussion is provided and subsidiary aspects are identified.

(1) Status of the Guidance Document

In relation to any particular project, performance of a PIA may be obligatory, or conditionally obligatory depending on some factor such as the significance of its apparent privacy impacts or a declaration by a Minister or an oversight agency. Alternatively, performance of a PIA may be a condition of funding or approval, or may be a recommended course of action, or may be entirely voluntary. Another variant is obligatory performance of a preliminary Privacy Issues Analysis or threshhold assessment, in order to provide a proper basis for the decision as to whether a PIA needs to be performed, and, if so, what scope the PIA process needs to have.

Whatever the degree of compulsion or choice may be, it is important that the status of available PIA guidance documents be clear. The use of the PIA guidance document may be:

A guidance document's status may of course vary, depending on such factors as the nature of the project or of the sponsoring organisation.

(2) Discoverability of the Guidance Document

This criterion refers to the ease with which organisations that would benefit from the guidance are able to discover its existence. Relevant considerations include the following:

(3) Applicability of the Guidance Document

The document needs to indicate a wide scope of activities to which it is applicable. Projects to create or amend systems that handle personal data are a primary focus, including both automated and manual business processes and machine-readable and merely human-readable data-storage. Broader schemes and programmes need to be encompassed, which are likely to have implications not only for data privacy but also for the privacy of personal behaviour, personal communications and the physical person. The scope also needs to extend to initiatives such as the drafting of legislation.

The document needs to be clear about the geo-political area within which it is intended that it be applied. For example, it may be applicable:

Clarity is also important regarding the categories of organisation that the document is intended for. This may be, for example:

(4) Responsibility for the PIA

A PIA guidance document needs to make clear that the responsibility for the conduct of a PIA rests with organisations that sponsor, propose or perform projects that have the potential to negatively impact privacy.

In many cases, external expertise will be acquired under contract, because few organisations would find it appropriate to invest in full-time employees who already had, and could sustain, up-to-date knowledge in such a specialised area. In addition, an appropriate consultant can provide access to external perspectives that would otherwise be difficult for the organisation to appreciate.

The guidance document needs to motivate organisational control over the PIA process, in order to ensure that:

The document needs to make clear that the commissioning of an assessment by an external organisation, such as a brand-name consultancy or legal firm, is of the nature of an external audit or a public education or marketing exercise. It does not represent a PIA because the organisation does not take ownership, and is unable to learn from the process and assimilate the findings. Instead, whether by intention or accident, performance by an external consultancy or agency builds a barrier between the sponsoring organisation and the people affected by it.

More specifically, the guidance document needs to make clear that an assessment undertaken by a regulatory or oversight agency is not a PIA, but rather is a form of accountability and external control. A PIA enables the sponsoring organisation to appreciate privacy concerns and avoid or mitigate them. To the extent that effective external review occurs (which is the case in very few jurisdictions), it may result in impositions on the organisation, but not in a reasoned, privacy-sensitive design.

(5) Timing of the PIA

The document needs to stimulate sufficiently early commencement of the PIA that information arising from it is fed forward into the design process. If that is not the case, then there is a considerable risk that the design will have undue negative privacy impacts, and that re-work and feature retro-fitting will be necessary. This gives rise to delays and to much higher costs than is the case where an in-depth understanding of privacy concerns is factored into the design process from the outset.

The document also needs to communicate that, where a project is large or long, the PIA process needs to be multi-phased, commencing at project initiation or at least during the requirements analysis phase, and running in parallel and inter-leaved with design, implementation and deployment.

(6) Scope of the PIA

The guidance needs to communicate that a PIA process has to evidence sufficient scope. Three aspects are particularly crucial to a successful undertaking.

(a) The Dimensions of Privacy

A PIA process must not be limited to data/information privacy, i.e. the protection of personal data. Other categories of importance are privacy of the physical person, privacy of personal behaviour, and privacy of personal communications. These dimensions are discussed in Clarke (2006).

(b) Stakeholders

The guidance document needs to make clear that:

(c) Reference-Points

A PIA process must of course take into account laws relevant to privacy. The guidance needs to draw to attention not only any primary privacy or data protection statute that may apply, but also the many other pieces of legislation that provide incidental protections or establish privacy-relevant regulatory requirements, and, in common law jurisdictions, torts (such as confidentiality) and case law. In the case of government agencies, their own enabling and/or governing legislation generally also contains privacy-relevant requirements.

However, the reference-points used in identifying negative privacy impacts need to be much broader than just the applicable laws. There are many public needs, expectations and concerns that are felt by individuals, categories of individuals and communities that may not be (or may not yet be) reflected in law. A PIA process that overlooks these aspects will result in a design that earns opprobrium from advocacy organisations and the affected public. Hence, despite being legally compliant, schemes will encounter resistance, and be the subject of complaints and negative media coverage.

Organisations therefore need a clear explanation of the fact that, for a PIA to be an effective risk management method, its scope needs to be much broader than the law.

(7) Stakeholder Engagement

The document needs to convey that the PIA process must include meaningful engagement by the sponsoring organisation with all stakeholders. For meaningful engagement to be achieved, all of the following are necessary:

Some organisations may be concerned about the exposure of information of commercial or competitive value or security-sensitivity, and others about the disclosure of information that is subject to constraints because no Cabinet decision has yet been made. The document accordingly needs to include advice on how to reconcile the need for meaningful engagement with the affected public with security and confidentiality limitations.

(8) Orientation

There are two important ways in which the guidance document can lead organisations into appropriate orientation of the PIA process.

(a) Process cf. Product

The PIA needs to be clearly and consistently depicted as being primarily about process. The guidance document needs to show how benefits accrue from the insights gained by the organisation's staff, and changes in understanding, design and behaviour within the organisation.

Organisations need to be guided away from perceiving a PIA as being merely a formal procedure that produces a PIA Report. Although the guidance document may well provide a sample structure and even boilerplate for the Report, and perhaps pointers to exemplars, the reader must not infer from the guidance document that the quality of the PIA can be properly gauged merely on the basis of the Report.

(b) Solutions cf. Problems

Consistently with the notion of 'risk management', the description of the PIA process must avoid limitations to 'problems', 'issues' and 'concerns'. It needs to focus on 'solutions', and specifically on means of avoiding and mitigating negative privacy impacts.

(9) The PIA Process

A PIA guidance document needs to describe the preliminary privacy issues analysis process, whereby projects can be screened, and threshhold tests applied, in order to determine whether a PIA is necessary, and, if so, what the scope of the assessment should be.

A structure needs to be provided for the PIA process as a whole. Phases needs to be outlined, such as a preliminary, preparatory, performance, documentation and review.

The document should draw to attention the considerable benefits that can be gained from integration of the PIA process into relevant corporate mechanisms, such as project funding, project approval, risk management, project management and internal review mechanisms.

The document needs to provide sufficient detail about the activities that are needed within each phase. The descriptions need to balance specific advice against discussion of the factors that should be considered in devising a plan appropriate to the particular project. Because of the diversity of circumstances, the emphasis needs to be on the intellectual challenges involved, and the temptation needs to be resisted of merely providing formulaic recipes and checklists.

It is inevitable that the corporate style of some organisations will make it difficult for them to embrace stakeholder analysis and conduct effective consultations. It is therefore important that the guidance document provide specific advice in those areas, and re-assert that the scope of the 'stakeholder' notion includes the people affected by the project and their representatives and advocates.

The guidance needs to lead the organisation into articulation of the outcomes forward through the design and implementation phases. Without some form of impetus to ensure the performance of the commitments that have been made, the PIA Report is at risk of becoming an endpoint, and the process well-meaning but ineffectual.

Although the focus needs to be primarily on process, guidance is also needed in relation to the contents of the PIA Report. The emphasis needs to be on the nature of the project, its potential negative impacts on privacy, and measures developed to avoid or at least mitigate those impacts. Supporting materials such as a description of the consultation process need to be specified.

(10) The Role of the Oversight Agency

In almost all jurisdictions outside the USA, an oversight agency exists such as a Data Protection or Privacy Commissioner. In the absence of such an agency, the responsibilities for performing the functions discussed in this sub-section fall variously on central agencies, industry regulators, industry associations, complaints-handling bodies such as ombudsmen, and auditors.

Oversight agencies have vital roles to play in ensuring quality in PIA processes. Specific activities include:

Oversight agencies should desirably be sufficiently resourced that they are able to monitor the performance of PIAs, at least through review of the resulting PIA Reports. The number of projects that require PIAs is likely to be so large, however, that oversight activities are likely to be limited to sampling or even only to occasional targeted investigations.

It is vital that oversight agencies not perform, nor become involved in the performance of, PIA processes. The advice provided to organisations must remain sufficiently abstract, and the relationship between them sufficiently arms-length, that the agency's independence and oversight functions are not impaired.

4. Application of the Criteria

A few early guidance documents were published by consultants, e.g. Clarke (1998a, 1998b), Karol (2001) and Marcella & Stucki (2003, pp. 332-348). The majority have been published by either privacy oversight agencies (such as Data Protection Commissioners) or central agencies. Some have been prepared by government agencies for use by their organisational sub-units. A comprehensive list of government guidance documents known to have been current in mid-2007 is in ICO (2007b), Appendix I.

The criterion list above has been applied to a range of government documents, in order to gain a broad picture of the extent to which they satisfy these quality tests. Many fall a long way short. Most commonly, they fail on criteria 6(a), 6(b), 6(c), and (7). Many do not require the assessment to look beyond the narrow confines of local data protection law, and not only fail to require consultation with the affected public but even to encourage it and in some cases even to mention the possibility of it.

A summary of the findings is presented below. The organisations responsible for the documents were approached for comment on the draft paper, and the feedback received was reflected in the revised paper.

The dozen guidance documents that were examined have been divided into three groups. The first group are so inadequate that the use of the term 'PIA' for them is called into serious question. The second group are of moderate quality but have significant deficiencies. The third group are of high quality, although in most cases still with some shortfalls.

Group 1. The following documents specify Data Protection Law Compliance Assessments, but inappropriately refer to them as PIAs:

Group 2. The following PIA Guidance documents are of moderate to good quality, but with material shortfalls against the evaluation critieria:

Group 3. A number of PIA Guidance documents are of high quality when evaluated against the critierion list proposed in this paper. They are listed below in the chronological order of their publication:

5. Conclusions

PIAs play a vital role in achieving both privacy protections for individuals and risk management for organisations. If PIAs are not done well, there will continue to be frequent and critical media coverage of privacy intrusions by organisations, resistance by customers and citizens to ill-conceived projects and schemes, project failures and loss of investment by shareholders and taxpayers alike.

PIA guidance documents are a primary means by which organisations can be encouraged to devise quality PIA processes and avoid harm to privacy and to investments. It is therefore important that PIA guidance documents be available and readily discoverable, be comprehensive and clear, and embody best practice.

This paper has presented a list of criteria for evaluating PIA guidance documents, which was developed on the basis of the relevant literature and the author's extensive knowledge of practices in government and business. The list of criteria presented in this paper is intended firstly as a basis for evaluating the quality of guidance documents. It also has applicability to the upgrading of guidance documents when they are revised, and to the development of new guidance documents. Organisations that are conducting PIAs can use the list as a supplement to existing guidance documents, in order to avoid overlooking important quality factors in the design of their own PIA process.

The list of criteria was applied to a dozen guidance documents published by government agencies in ten jurisdictions. Some of the them were found to be seriously wanting. All US documents were assigned a 'fail' grade, as were all but two of the current guides published in Canadian jurisdictions. The most significant inadequacies are the prescription of merely checks of compliance with the law, and the failure to convey the importance of engagement with the affected public. The documents of Australia, New Zealand and Hong Kong occupy the middle ground, in that they combine many good features with some significant weaknesses. The four jurisdictions whose guidance documents the analysis shows to stand out as best practice publications are (in chronological order by original publication-date) those of Ontario (1999/2001 and 2005), Alberta (2005/2009), the UK (2007) and Victoria (2009).


BC (2009) 'Privacy Impact Assessment Process (PIA)' Office of the CIO of British Columbia, January 2009, at

Clarke R. (1998a) 'Privacy Impact Assessments', Xamax Consultancy Pty Ltd, February 1998, at

Clarke R. (1998b) 'Privacy Impact Assessments', Xamax Consultancy Pty Ltd, February 1998, at

Clarke R. (2005) 'Privacy Statement Template' Xamax Consultancy Pty Ltd, December 2005, at

Clarke R. (2006) 'What's 'Privacy'?' Xamax Consultancy Pty Ltd, August 2006, at

Clarke R. (2009) 'Privacy Impact Assessment: Its Origins and Development' Computer Law & Security Review 25, 2 (April 2009) 123-135, PrePrint at

ICO (2007a) 'Privacy Impact Assessment Handbook' Information Commissioner's Office, Wilmslow, I.K., December 2007, at, mirrored here

ICO (2007b) 'Privacy Impact Assessments: International Study of their Application and Effects' Information Commissioner's Office, Wilmslow, I.K., December 2007, body of the Report at, mirrored here, and consolidated Appendices, here, with individual Appendices as follows:

IPCO (2005) 'Privacy Impact Assessment Guidelines for the Ontario Personal Health Information Protection Act' Information and Privacy Commissioner of Ontario, October 2005, at, mirrored here

Karol T.J. (2001) 'Cross-Border Privacy Impact Assessments:  An Introduction', Information Systems Control Journal, 3 (March 2001), at

Marcella A.J. & Stucki C. (2003) 'Privacy Handbook: Guidelines, Exposures, Policy Implementation, and International Issues' Wiley, 2003

MBS (2001) 'Privacy Impact Assessment Guidelines' 1999, revised 2001, Management Board Secretariat, Government of Ontario, at, mirrored here. The document is now administered by the Ministry of Government Services

NZPC (2002) 'Privacy Impact Assessment Handbook' Office of the New Zealand Privacy Commissioner, March 2002, reprinted June 2007, at, mirrored here

OAPC (2010) 'Privacy Impact Assessment Guide' Office of the Australian Privacy Commissioner, August 2006, rev. May 2010, at, mirrored here

OMB (2003) 'OMB Guidance for Implementing the Privacy Provisions of the E-Government Act of 2002' M-03-22, Office of Management and Budget, Washington DC, September 2003, at, mirrored here

OVPC (2009a) 'Privacy Impact Assessments - A Guide', Office of the Victorian Privacy Commissioner, August 2004, rev. May 2009, at$FILE/OVPC%20Privacy%20Impact%20Assessment%20Guide%20Edition%202%20May%202009.pdf, mirrored here

OVPC (2009b) 'Privacy Impact Assessments Report Template', Office of the Victorian Privacy Commissioner, May 2009, at$FILE/OVPC%20PIA%20Template%20Report%20May%202009.doc, mirrored here

OVPC (2009c) 'Privacy Impact Assessments - Accompanying Guide', Office of the Victorian Privacy Commissioner, May 2009, at$FILE/Accompanying%20Guide%20to%20OVPC%20PIA%20Template%20Report%20May%202009.pdf, mirrored here

PCPD (2010) 'Information Leaflet: Privacy Impact Assessments (PIA)' Office of the Privacy Commissioner for Personal Data, Hong Kong, July 2010, at, mirrored here

SA (2005) 'Privacy Compliance: Privacy Impact Assessments' Chapter 9.3 (pp.328-333) of 'Guidelines and Practices', Service Alberta, 2005 rev. 2009, at, mirrored here

TBC (2002a) 'Privacy Impact Assessment Policy' Treasury Board of Canada Secretariat, 2002, at, mirrored here

TBC (2002b) 'Privacy Impact Assessment Guidelines: A Framework to Manage Privacy Risks' Treasury Board of Canada Secretariat, 2002, at, mirrored here

TBC (2003) 'Privacy Impact Assessment (PIA) e-learning tool' Treasury Board of Canada Secretariat, Ottawa, October 2003, at

TBC (2010) 'Directive on Privacy Impact Assessment' Treasury Board of Canada Secretariat, Ottawa, April 2010, at, mirrored here

UKCO (2008) 'Data Handling Procedures in Government: Final Report' Cabinet Office, London, June 2008, available from, at

UKMOJ (2010) 'Undertaking Privacy Impact Assessments: The Data Protection Act 1998', August 2010, at

USDHS (2007) 'Privacy Impact Assessments: Official Guidance' Department of Human Services, Washington DC, May 2007, at, mirrored here

USDOI (2002) 'Privacy Impact Assessment and Guide' Department of the Interior, July 2002, at, mirrored here

USDOJ (2000) 'Privacy Impact Assessment for Justice Information Systems' US Department of Justice, Working Paper, August 2000, at [Link active in 2004, but broken in 2008]

USDOJ (2010) 'Privacy Impact Assessments - Official Guidance' US Department of Justice, August 2010, at mirrored here

Warren A., Bayley R., Bennett C., Charlesworth A.J., Clarke R. & Oppenheim C. (2008) 'Privacy Impact Assessments: International experience as a basis for UK Guidance' Computer Law & Security Report 24, 3 (April-June 2008) 233-242


The author greatly appreciates the assistance of his colleagues in the project-team that undertook an international study of laws, policies and practices relating to PIAs around the world (ICO 2007a), and prepared the UK Information Commissioner's PIA Handbook (ICO 2007b). That applies in particular to Robin Bayley, of Linden Consulting Inc, Victoria BC, and to Adam Warren (Project Manager) and Prof. Charles Oppenheim (Project Director), both of Loughborough University, Prof. Colin Bennett of the University of Victoria BC, and Andrew Charlesworth of the University of Bristol. Valuable feedback was also received from several Privacy Commissioners, and from Robert Gellman of Washington DC. I have benefited from many interactions with consultancy colleagues in Australia, in particular Nigel Waters, Anna Johnston and Chris Connolly. All evaluative comments are the responsibility of the author alone.

Author Affiliations

Roger Clarke is Principal of Xamax Consultancy Pty Ltd, Canberra. Because it is relevant to the research reported in this paper, it is necessary to declare the company's list of privacy clients, which is at

He is also a Visiting Professor in the Cyberspace Law & Policy Centre at the University of N.S.W., a Visiting Professor in the Department of Computer Science at the Australian National University, a longstanding member of the Board of the Australian Privacy Foundation and a longstanding member of the Advisory Board of Privacy International.

xamaxsmall.gif missing
The content and infrastructure for these community service pages are provided by Roger Clarke through his consultancy company, Xamax.

From the site's beginnings in August 1994 until February 2009, the infrastructure was provided by the Australian National University. During that time, the site accumulated close to 30 million hits. It passed 50 million in early 2015.

Sponsored by Bunhybee Grasslands, the extended Clarke Family, Knights of the Spatchcock and their drummer
Xamax Consultancy Pty Ltd
ACN: 002 360 456
78 Sidaway St, Chapman ACT 2611 AUSTRALIA
Tel: +61 2 6288 6916

Created: 14 September 2009 - Last Amended: 3 November 2010 by Roger Clarke - Site Last Verified: 15 February 2009
This document is at
Mail to Webmaster   -    © Xamax Consultancy Pty Ltd, 1995-2013   -    Privacy Policy