Roger Clarke's Web-Site
© Xamax Consultancy Pty Ltd, 1995-2024
Infrastructure
& Privacy
Matilda
Roger Clarke's Web-Site© Xamax Consultancy Pty Ltd, 1995-2024 |
|
|||||
| HOME | eBusiness |
Information Infrastructure |
Dataveillance & Privacy |
Identity Matters | Other Topics | |
| What's New |
Waltzing Matilda | Advanced Site-Search | ||||
Working Paper
Version of 28 September 2022
© Xamax Consultancy Pty Ltd, 2022
Available under an AEShareNet 
licence or a Creative
Commons 
licence.
This document is at http://rogerclarke.com/EC/MSRA.html
When organisations sponsor interventions into economic and social systems, it is normal for them to conduct an evaluation of the risks that the intervention entails. Conventional, standardised approaches to Risk Assessment are heavily committed to the perspective of the sponsoring organisation. The interests of other stakeholders may be considered, at least to the extent that they are perceived as representing constraints on the achievability of the sponsor's objectives. This narrows the focus to only those stakeholders that are perceived to have sufficient power, and overlooks legitimate stakeholders.
Many interventions have substantial impacts, variously by design, and in the form of side-effects and collateral damage. Most interventions embody application of information technologies, and the technologies that are being deployed now extend beyond data processing, into automated inferencing from data, automated decision-making, and automated action. Further concerns arise from the increasing opaqueness inherent in many technologies, whereby inferencing is fuzzy, decision-making is empirically-based and hence a-rational, actions are unexplainable and unauditable, redress is unachievable, and accountability is destroyed.
This article seeks a practicable mechanism whereby the interests of relevant players can be reflected in the assessment of interventions. Given that Risk Assessment (RA) is mainstream within organisations, the article investigates how RA can be augmented, with the intention of leveraging that familiarity and easing the absorption of external perspectives into each organisation's internal evaluations. The proposed Multi-Stakeholder Risk Assessment (MSRA) technique is described, exemplars of processes with some of the technique's characteristics are identified, and an illustrative case study is used to demonstrate its potential efficacy.
In performing their functions, organisations initiate new interventions into existing social and economic processes. Interventions can be of many kinds, including new or amended legislation, regulatory measures of some other kind, adaptation of institutional or sectoral infrastructure, particularly due to shifts in market power, changes in business processes, and new forms and new applications of transformative or disruptive technologies.
During the seven decades since computing began to be applied to data processing, applications of information technology (IT) have migrated far beyond the conversion of data into information, and now draw inferences from clusters of information, support and even make decisions based on information and inferences, and support and even directly implement actions driven by those decisions. Systems have burst out beyond organisational boundaries, linking pairs, chains and networks of organisations, and extending out to individuals. From the outset, it has been apparent that IT-based systems are capable of substantial impacts and implications, ranging from the highly beneficial to the very harmful, sometimes foreseen and managed, but in many cases unanticipated or ignored. The growth in scale and scope has been accompanied by a much greater growth in potential value to some participants, but also considerable increases in the likelihood and seriousness of harm to others.
To cope with the expanding scale and scope of IT-based projects, more rapid development techniques have been deployed. These feature even less quality assurance than was applied to the simpler systems of the past. The result has been an ongoing, poor record of plannability, delivery to time and budget, and performance reliability, maintainability and adaptability. Academics have spent a great deal of time studying the reasons underlying project failure, but failures persist, and the quality of service of many ongoing systems remains low.
Because of the high costs involved, the high risk of inadequate performance and failure, and the collateral damage arising from misconceived, poorly designed and poorly implemented interventions, the impacts and implications of interventions are recognised as requiring evaluation, preferably at several checkpoints along the way. A wide variety of evaluation techniques exists. Mainstream techniques applied by business enterprises include business case development and risk assessment. Techniques with broader scope include cost/benefit analysis and technology assessment.
Most evaluation techniques have a very tight focus on the interests of a single stakeholder, whereas other approaches are more amenable to consideration of the concerns of multiple players. The analysis reported in this paper is concerned with the interests of stakeholders of all kinds. The paper's purpose is to seek a practicable mechanism whereby the interests of relevant players can be reflected in the assessment of interventions.
The paper commences by summarising key insights from stakeholder theory. This enables consideration of the extent to which each of the variety of evaluation techniques is capable of reflecting the interests of multiple users. An alternative with promise is argued to be an adapted form of the conventional and well-documented business approach of Risk Assessment (RA), referred to as Multi-Stakeholder Risk Assessment (MSRA). As a preliminary test of the proposition, some exemplars of such an approach are identified. A summary is provided of an illustrative case study, of a large-scale government intervention popularly referred to as 'Robodebt'. The case is then used as a basis for showing how MSRA could have been applied in order to avoid the serious harm that the project gave rise to.
The term 'stakeholders' was coined as a counterpoint to 'shareholders', in order to bring into focus the interests of parties other than the corporation's owners (Freeman & Reed 1983). In IT contexts, users of information systems have long been recognised as stakeholders, commencing in the 1970s with employees. With the emergence of inter-organisational systems (Barrett & Konsynski 1982), and then extra-organisational systems (Clarke 1992), IT services extended beyond organisations' boundaries, and hence many suppliers and customers, both corporate and individual, became 'users' as well.
The notion of 'stakeholders' is broader than just users, however. It comprises not only participants in information systems but also "any other individuals, groups or organizations whose actions can influence or be influenced by the development and use of the system whether directly or indirectly" (Pouloudi & Whitley 1997, p.3). The term 'usees' is usefully descriptive of such once-removed stakeholders (Berleur & Drumm 1991 p.388, Clarke 1992, Fischer-Huebner & Lindskog 2001, Wahlstrom & Quirchmayr 2008, Baumer 2015).
The stakeholder notion has been subjected to further analysis and discussion during the almost four decades since it emerged. One analysis distinguishes stakeholders on the basis of power, legitimacy and urgency (Mitchell et al. 1997). Much as an approach based on ethics might dictate that 'legitimacy' has primacy, a strong tendency has been evident, in industry and government practice alike, to pay attention to the interests of only those stakeholders that are capable of significantly affecting the success of the project - as perceived by the project sponsor. The interests of that organisation are treated as paramount, the interests of powerful participants and usees are relegated to the role of constraints on the achievement of the primary organisation's objectives, and the interests of all other participants and usees are commonly marginalised or ignored (Achterkamp & Vos 2008).
A wide range of evaluation techniques exists, with widely varying approaches and foci. This analysis excludes consideration of activities that are merely concerned with checking an organisation's compliance with 'soft' regulatory instruments (such as codes of ethics and industry standards), or with a particular statute, delegated legislation such as a formal Code, or a broad body of law (such as safety, data privacy or consumer rights).
Mainstream techniques within organisations include Business Case Development (BCD, Schmidt 2005), Discounted Cash Flow Analysis (DCF, SoW 2013) and Net Present Value Analysis (NPV, Dikov 2020), financial sensitivity analysis and financial risk assessment. All of these depend on quantification, and in particular on the expression of costs and benefts in financial terms. The analysis is generally undertaken from the perspective of a single legal entity or government agency. In the case of interventions in which technology plays a major role, that organisation is typically the system sponsor.
Some single-organisation assessment techniques extend to factors that cannot be readily reduced to financial values, whose representations are referred to as 'non-quantifiable' or 'qualitative' data. These approaches include internal Cost-Benefit Analysis (CBA, Stobierski 2019) and Risk Assessment (RA), the second of which is further discussed below. All of these techniques are highly organisation-centric, and only reflect interests of some other party if that stakeholder is perceived by the organisation to be sufficiently powerful to be able to affect the organisation's capacity to achieve its purposes.
Some other techniques have a much broader frame of reference. Technology Assessment (TA) is concerned with the evaluation of potential impacts and implications of a particular technical capability (OTA 1977, Garcia 1991). Environmental Impact Assessment (EIA, Morgan 2012) is an approach that evaluates the effects on the physical environment (air, land and water) of a development project such as a mine, a dam, transport infrastructure, a manufacturing facility, a built-up area or even a single large building. Privacy Impact Assessment (PIA) considers effects on individuals' privacy interests - although often only the privacy interests involved in personal data (Clarke 2009, Wright & de Hert 2012). Social impact assessment (Becker & Vanclay 2003) has a broader remit, and surveillance impact assessment (Wright & Raab 2012, Wright et al. 2015) combines technological with psychological and social impact assessment.
It is feasible to apply some such form of impact assessment approach in order to address the concerns of particular stakeholders. However, the approach is seldom consistent with any particular organisation's perceived needs, and is in direct conflict with the legal responsibility of Board directors to serve the interests of shareholders. Moreover, it demands considerable depth and breadth of analysis and hence considerable resources with specialist expertise.
The purpose of this article is to seek a practicable mechanism whereby the interests of relevant players can be reflected in the assessment of interventions. None of the mainstream evaluation techniques outlined in the previous section fulfils that need. However, one particular technique that is in common use for evaluating proposals from the perspective of an individual organisation may be capable of adaptation to achieve the purpose. Risk Assessment (RA) has been in use sufficiently long to become the subject of industry standardisation (ISO 27005:2011, NIST 2012, EC 2016).
The assessment of risk needs to be based on a model of security, and on a set of terms with sufficiently clear definitions. In the sub-area of data security, a substantial literature exists, and this provides a useful framework that can be drawn on in other contexts. A depiction of the conventional security model is in Figure 1. Briefly, threatening events are conceived as impinging on vulnerabilities, resulting in security incidents, which may in turn give rise to harm to assets. Safeguards provide protections against threatening events, vulnerabilities and harm. Security is a condition in which harm is in part prevented, and in part mitigated, because threats and vulnerabilities are subject to safeguards that withstand countermeasures. The terms are defined in Appendix 1.
Extract from Clarke (2015, p.547-549)
The conventional Risk Assessment process uses a short sequence of steps to apply the concepts in Exhibit 1 in order to identify and understand residual, inadequately-addressed risks, and hence lay the foundation for Risk Management (RM) activities to address them. These are embodied in an extended process model presented in the following section.
The RA technique was conceived to serve the interests of the organisation that conducts the evaluation, or that commissions the work to be performed. Standards documents and most guidelines for performing the technique reflect those origins. On the other hand, the author argued in Clarke (2019, p.413), in the specific context of Artificial Intelligence (AI), that:
In this paper, the author argues that those assertions apply not only to applications of AI, and to technologically-based interventions, but to interventions generally. Further, it is contended that RA is capable of adaptation to accommodate stakeholders additional to the system sponsor, in a way depicted in Figure 2, and referred to here as Multi-Stakeholder Risk Assessment (MSRA):
Multi-Stakeholder Risk Assessment (MSRA) extends the conventional, organisation-internal RA process to encompass multiple, parallel assessments, all of which draw on a common base of information, but each of which is undertaken from the perspective of a particular stakeholder. The results of the various assessments are then integrated into a consolidated multi-perspective report. That integrated body of understanding is then applied during the Risk Management phases of the process.
The intention is that MSRA be minimally disruptive to whatever flavour of RA each organisation currently utilises internally. An outline of the conventional RA process is provided by the sequence of steps down the left-hand side of the diagram. This depicts RA as the first phase as a larger process, involving a series of nine steps. This 'analysis' phase is followed by a 'design' phase that prepares a Risk Management (RM) plan in sufficient detail to be executable, and an 'implementation' phase that brings the plan to fruition.
The primary extension involved in transitioning from RA to MSRA is the conduct of at least one, but possibly multiple, additional instances of steps 1.4 to 1.9. This may be done by stakeholders themselves (which is appropriate where the stakeholders are organisations with access to sufficient resources and expertise), by advocacy organisations on behalf of stakeholders (which is more suitable where stakeholders are individuals or small organisations), or by proxies for stakeholders (such as consultants, or, in the weakest form, staff-members of the sponsoring organisation, playing a role by adopting an external perspective).
To facilitate the performance of multiple, parallel risk assessments undertaken from different perspectives, it is necessary to make a small adaptation to step 1.3. A document needs to be produced and distributed that describes the intervention at a sufficient level of detail to enable each stakeholder group to undertake its own assessment. It needs to be introduced to participants through a briefing, and articulated by discussion.
To provide effective feedforward to the RM phase, the results of the various risk assessments need to be integrated into a consolidated report (step 2.1). This needs to be introduced by a briefing and articulated by discussion, this time for the internal project team. The adequacy of the resulting Risk Management plan needs to be evaluated against this consolidation of the multi-perspective risk assessment (step 2.3). To ensure that the stakeholders gain the intended benefits, and that the sponsoring organisation enjoys public support from the stakeholders, the loop is best closed by means of some form of stakeholder participation in the Risk Management implementation phase.
This a further development from Clarke (2019a, p.414)
The term 'Multi-Stakeholder Risk Assessment' (MSRA) was adopted by the author in Clarke (2019). Although it is a straightforward descriptor, searches of previous usage in the refereed literature have found only a few casual uses of the expression, of which Molarius et al. (2016) appears to be the most relevant, and Karobaga (2022) the most recent. The term 'Multi-Stakeholder Risk Management' (MSRM), applicable to the subsequent phases in which the consolidated risk assessment report is applied, has also put in a small number of appearances in the literature, notably in Young & Jordan (2002) and Shackelford & Russell (2016).
MSRA can serve the interests of stakeholders additional to the system sponsor in at least two ways, which are outlined in the following two sub-sections.
One approach to applying MSRA is for a risk assessment process to be performed by each relevant category of stakeholders, or by collaborations, collectives, advocacy organisations or proxies representing the interests of the each category. However, each of the parallel activities requires information, and adequate human resources with the relevant expertise.
The asymmetry of information, resources and power, and the degree of difference in world-views among stakeholder groups, is commonly so pronounced that the results of such independent activities are difficult to communicate to other players. There is accordingly doubt about whether the results will be effectively assimilated by the system sponsor, and integrated into the organisation's ways of working. In the terms used in stakeholder theory, only those stakeholders with sufficient market power are likely to be able to force the system sponsor to recognise the legitimacy of their claims. The stakeholders are readily played off by the system sponsor, who can portray them as being in competition with one another, rather than having substantial commonality of interest.
The chances of the promise of MSRA bearing fruit may be better where each of the parties perceives advantages to themselves in gaining an understanding of the various perspectives and, to the extent feasible, accommodating the interests of other parties. The technique is appropriate where the system sponsor recognises that it is advantageous to drive the parallel studies and engage directly with the relevant parties. The organisation can gain sufficiently deep understanding, and can reflect stakeholders' needs in the project design criteria and features. Moreover, it may be able to do so without bearing disproportionate cost, and with harm to its own interests kept within manageable bounds.
A second circumstance in which a multi-pronged risk assessment approach is feasible is where the system sponsor adopts it for 'business ethics', 'corporate social responsibility' or 'public policy' reasons. This appears most likely in circumstances where the institutional context reinforces collaborative and human-wellbeing values. For example, the system sponsor may be a public agency whose mission is to deliver particular economic and/or social value to particular population segments. Alternatively, the system sponsor may be working under a government grant or contract, or may be a joint venture in which at least some of the participating organisations recognise a social-responsibility imperative.
A third possibility exists. A requirement could be imposed on a system sponsor to reflect the interests of multiple stakeholders, with a sufficiently credible threat of enforcement action to motivate compliance. Apart from formal statutory authority, such a requirement might derive from the institutional environment, such as licensing conditions, or 'moral suasion' by a powerful regulatory agency, or perhaps just cultural conventions within the industry sector or country.
The argument presented above has been almost entirely theoretical, with little evidence of practical application, and limited attention to the necessary economic and social drivers to motivate adoption. This section tests whether the proposition has potential, by seeking out circumstances in which at least some of the characteristics of MSRA are apparent.
In many countries, large-scale interventions into the landscape are subject to substantial, formal and cumbersome procedures. Where the intervention is less dramatic, but is impactful, or is perceived or is feared to be so, the sponsor may find it necessary, advisable or advantageous to conduct an MSRA in order to identify and understand the concerns of the various interest groups. An example might be a facility that handles chemicals, such as a fire station, being developed or extended adjacent to, or upstream from, a lagoon that harbours frogs and whose surrounds are used for children's leisure activities.
Where a mining company wants to extract ore from an area that is in a reserve, or is subject to laws conferring some form of native title, it may need to negotiate with indigenous groups. If the company wants to get to grips with the possibly diverse, complex, contested and/or vague concerns of such groups, the MSRA approach may be an attractive tool, fulfilling informational needs, and evidencing to the public, the media and regulators that the organisation has taken due care, and sought to act responsibly.
During a period when decentralisation was in vogue, a government agency may have established an operational facility in a regional city or town. It may have been, or gradually become, a significant employer in the district. The time may come to close the facility, particularly if it performs functions that have been overtaken by technological innovation, such as bulk printing-and-posting or the scanning of manual forms arriving in the post. The agency may find it politically necessary, reputationally advisable and/or morally responsible, to conduct an assessment of the impact of closure, taking into account the circumstances of many segments of society, and both economic and social effects. A large corporation for which public image is important may also find it valuable to adopt a process of this nature. MSRA is devised such that the procedure is a variant of an existing business process, rather than a foreign, externally-imposed measure.
Medical implants may provide good exemplars of MSRA and multi-stakeholder risk management. The industry involves designers undertaking carefully-designed pilot studies, with active participation by multiple health care professionals, patients, and patient advocacy organisations.
A government might seek to break open a monopoly marketspace, perhaps through legislation or moral suasion. Such an intervention may be designed to restructure the institution and change its business processes, probably taking advantage of technology, in order to balance the interests of the parties involved in it, and the parties dependent on it. An MSRA might be an effective means of exposing the impacts of the current arrangements, gaining public support, and shaming the current beneficiaries into accepting the need for change. An example in the livestock industry was the hog auction market in Singapore (Neo 1992).
In the 1980s, Australian livestock producers, distant from major population-centres, faced high costs to get their stock to saleyards, and low prices offered for their product by well-informed agents for supermarket chains, in so-called 'farm-gate private treaties'. The problem-definition process reflected the principles intrinsic to MSRA. The producers' association then developed an early online auction scheme, the computer-aided livestock marketing (CALM) system. Supported by a newly-created class of certified stock inspectors, the scheme published the prices paid for various stock-categories nationwide. The result was reduced information asymmetry in the marketspace, and a better balance among the market-participants' interests (Clarke & Jenkins 1993).
Industry sectors such as health, international trade, and major infrastructure installations such as sea-ports and airports, are not organised as linear supply-chains, nor in a hub-and-spoke or star configuration. They feature many specialised enterprises, and many inter-linkages and flows among those enterprises, overlaid by considerable regulatory interference to satisfy such public-interest needs as public safety, hygiene, service quality and tax-collection. Changes to architecture, infrastructure, data flows and business processes need to take into account the interests of a wide range of participants and beneficiaries. MSRA has attributes that satisfy some of those needs (Clarke 1994, Cameron 2009).
A particular form of sectoral transformation is commonly referred to as the 'technology platform' model. Disruptors such as eBay (since 1995), booking.com (1996), Expedia (1996), Tripadvisor (2000), Mechanical Turk (2005), YouTube (2005), Airbnb (2008), Freelancer.com (2009), Pinterest (2009) and Uber (2009) have taken advantage of new technology and the start-up's lack of any legacy technology or labour force, by deploying new services at a speed that large, established, and in many cases highly-regulated corporations simply cannot replicate.
The profiles of some of these platforms include two additional features. One is long-term under-cutting of existing markets, linked with loss-leadership, and supported by low-paid piece-workers and ever-growing investment based on the assumption of future super-profits. The other is a deep disregard for the laws that apply to the sector that the innovator is determined to redefine. Case studies have highlighted abject failure by regulatory agencies to enforce those laws, in many jurisdictions (Wyman 2017, Clarke 2022). The dislocation and financial losses suffered by investors and employees, and loss of amenity by communities, would have been mitigated had an MSRA been undertaken and a transition strategy designed to enable progress without such dramatic negative consequences.
The preceding section identified multiple circumstances in which some aspects of the MSRA are evident, or the potential benefits of applying MSRA are readily detected. This section adopts a complementary approach, by seeking the greater depth that is afforded by a case study. Characteristics of a relevant case study include an impactful intervention, richness and diversity of stakeholders, scale, recency, and accessibility of reliable information. Given that this article proposes a new form of evaluation tool, no case is available that applies it. The profile most likely to offer insights is an unsuccessful intervention that includes an IT component. There are limited obligations on corporations to disclose information about their problematic initiatives, whereas public sector organisations operate in a somewhat more open environment. The case that was selected was a transformative IT project run during the period 2015-20 by a large Australian government agency.
Australia is a nation of 25 million people, with a substantial 'safety-net' and well-developed welfare-benefits administration mechanisms. The portfolio agency responsible for operations is the Department of Human Services (DHS), whose Centrelink Division runs a vast database, fortnightly payment processes that transfer over AUD150 billion p.a. to millions of clients, and client-interaction facilities in the forms of web-sites, interactive voice response (IVR), call-centres and shop-fronts.
The agency's operations have a strong focus on management of the enormous scope for fraud, error and waste. An aspect of particular significance is the checking of the fortnightly statements of income by clients of some of the programs, which affect the amount of the next fortnight's payments. The very long-running processes to detect and recover overpayments have included data matching programs for over three decades.
On 2015, DHS was lured by the hype about 'disruptive IT' into proposing a further level of automation of overpayment detection, debt recognition, and debt recovery. The project was officially referred to as the Online Compliance Intervention (OCI), but was soon dubbed by the media 'Robodebt'. It has been the subject of a number of official reviews, and published case studies are now appearing in the refereed literature. This section is underpinned by Clarke et al. (2022), which cites many official, refereed and where appropriate media sources.
DHS initiated the project in 2015, with initial roll-out in July 2016, and full-scale roll-out from September 2016. The key features of the design are listed in Table 1. Briefly, suspects were generated by matching data acquired from two different agencies, necessarily making some assumptions about the apportionment of each welfare client's earnings in particular time-periods. Apparent anomalies resulted in automatic notification to suspect clients of the need to provide documentation relating to their earnings in particular fortnights some time between 6 months and 7 years earlier. If a client failed to respond and satisfy the demand for evidence, a debt was automatically declared to exist, and collection processes were instigated, through deductions from later welfare payments in the case of ongoing clients, and by passing the details to a debt collector if the person was not longer a client.
| DHS Action | Observations | |
| 1 | Data matching of client income data as declared to the taxation authority (mostly annually and in arrears) with client income data declared directly to DHS (on a fortnightly basis and in arrears) | A new matching scheme was used, which avoided use of the purpose-designed 'Tax File Number' identifier |
| 2 | Inference of apparent overpayment if a material difference was found between the apportioned income from the taxation authority and the declared fortnightly income in DHS's own files | This was acknowledged to be an inherently risk-prone inference, which had previously only been used internally, in order to generate suspect transactions and clients for further investigation |
| 3 | Automated demands to clients, requiring the production of evidence in the form of payslips from their (then) employer(s) | This step had previously been undertaken by DHS, under authority of law. Outsourcing this work to clients inverted the onus of proof, imposing it on people who lack the power to issue their employers with demands for copies of documents |
| 4 | Clients were required to establish an account with an intermediary agency, use that to login to a new DHS site with a highly-inflexible user interface, and use it to upload information and scanned documents | Almost all aspects of which were new features of the system, and many clients were seriously challenged to understand the instructions and interface, let alone perform the required tasks |
| 5 | In the absence of any response, or if documentation received was deemed inadequate (e.g. copies of bank statements rather than payslips), the raising of a debt, and either auto-deductions from future payments, or the use of debt-collection agents | Because so little information was provided, most clients had no basis for understanding or challenging the debt |
Within weeks of the launch, the number of debt notices issued skyrocketed from 20,000 per annum to a peak of 20,000 per week, with a total of about 125,000 issued in the last 3 months of 2016. All client channels to DHS were hopelessly overloaded, and during some periods broke down completely. By December 2016, media coverage had commenced, documenting the harrowing experiences of many of the clients subjected to the scheme. Between mid-December 2016 and mid-January 2016, a series of media articles documented multiple logical and process flaws, with a strong focus on the automated nature of 'Robo-debt'.
DHA and the responsible Minister stonewalled. The Minister breached the law by publishing personal data of one client whose case had been described in media reports. Whistleblowers emerged from among the very large base of employees and contractors. The Ombudsman commenced an investigation of a few, narrow aspects of the process. A Senate Committee, with a majority of non-government members (reflecting the balance of party numbers in the upper House), launched a longer, slower, but broader Inquiry.
In April 2017, the Ombudsman's Report, limited in scope and agency-friendly, nonetheless identified some significant problems, and wrung some minor concessions out of DHS. In May 2017, the Senate Committee's first (of multiple) reports was damning, identifying far more inadequacies, and documenting considerable harm to individuals, arising from material inferencing errors, in many instances so seriously wrong that no debt existed at all. The government's response, delayed until September 2017, rejected everything the Senate Committee had said.
Despite ample evidence that the foundations on which the scheme was built were rotten, DHS refused to do anything constructive to address the problems the scheme was giving rise to. Further, because the agency did not provide clients with copies of the data on which the accusation was based, a large proportion of the instances were incapable of effective investigation by the client, by a lawyer or anyone else assisting the client, or by any authority that considered a complaint. There appear to be no avenues available under Australian law whereby even such serious breaches of due process and procedural fairness can be pursued.
As late as February 2019, the government continued to declare that the automatic debt notice process was reasonable, lawful and fair. Remarkably, it took total of 2-1/2 years, until November 2019, before the government accepted defeat, and ceased raising debts where the only information it was relying on was the averaging of taxation data. This appears to have been forced by an imminent case that DHS had inadvertantly let slip through to the courtroom, despite its previous assiduousness in avoiding test-cases.
It then took a further 6 months, until May 2020, for the Attorney-General to concede that all Centrelink debts raised using the 'income averaging' method were unlawful. It appears that only in about 40% of the cases that it commenced did the department pursue the debts in full, with about 20% of debts reduced, about 20% of debts deemed not worth pursuing, and about 20% withdrawn because they were clearly wrong. The government later accepted that even the 40% that it pursued in full, and the 20% in part, had to be repaid to the clients, because there was no legal basis for raising the purported debts.
Meanwhile, a class action had been mounted. DHS, in breach of its legal obligations under a Legal Services Direction, acted as anything but a 'model litigant', and fought the case all the way to the court door. (This is a standard tactic in some jurisdictions, to force the opponent to agree to a low settlement figure, for fear of the judge taking umbrage at a refusal to settle out of court). The Senate Committee's final report, and the judge's comments when approving the agreement to settle the class action, were scathing of the agency's behaviour.
Many of the individuals materially affected by the scheme were vulnerable in some sense of the word, variously financially, as a major source of money within a family or household, educationally, in terms of physical disability, and/or in terms of mental health. The financial dimension of the fiasco totalled over AUD 2 billion. However, unlike The Netherlands, where a much smaller scandal in 2020 resulted in an entire government resigning (Erdbrink 2021), the principle of Ministerial responsibility no longer exists in Australia, and none of the four revolving-door Ministers suffered any consequences. Meanwhile, the senior executive of DHS moved onto another appointment, and, 2 years after it became publicly obvious that she had presided over a disaster, was given an Order of Australia (AM), one of the nation's highest honours.
IT systems in the public sector are confronted by a substantial set of challenges, including scale, complexity, a multiplicity of stakeholders, diversity and conflict among the interests of the stakeholders, legal constraints, and rapidly-changing political priorities. The systems are developed, are operated, and are adapted, by or for conservative bureaucracies, which are subject to constraints in terms of financial resources and the quality and morale of available staff. Both the senior executives of government agencies and their political masters tend to be risk-averse, although their perceptions of the objectives, constraints and risks may be very different. A great many IT interventions in the public sector perform poorly, some to the point of being dysfunctional but nonetheless ongoing, and some are outright failures. A rational position for executives responsible for such systems to adopt is to look for approaches that can, if feasible, ensure success, if not then achieve some degree of success, and at least avoid having to acknowledge failure.
This section considers four such approaches, at the levels of strategy, contingency planning, reaction and proaction. In each case, the approach is outlined, and illustrated using elements of the Robodebt case study. That case is then used as a basis for imaginary scenarios and vignettes that identify ways in which the elements of MSRA could have been constructively applied in that particular real-world context.
Government agency executives understand that the expression of the agency's policies, and the design of its business processes and supporting systems, will always be perceived quite differently by diverse stakeholder groups. They may therefore recognise the benefits of internalising an appreciation of those groups and their perspectives, and institutionalising channels of communication between them and the agency. A means for doing this is a 'reference group', comprising enough advocacy organisations to adequately encompass all key interests, but small enough that coherent discussions can occur.
An establishment event is needed, to achieve sufficient mutual understanding and commitment. The nature and functions of the group may need to be reflected in a memorandum of understanding or code of practice. This enables periodic meetings to be held in which agency staff and members of the reference group interact. Some events can address matters that relate to agency-wide functions (such as reflection of ethnic and lingual diversity, and appeals and redress processes). Other events can relate to particular interventions that are being considered, designed, or adapted.
Conversations need to be seeded by advance disclosure and briefings by the agency. This needs to be followed by verbal by commentaries and conversation among reference group members, supplemented by written submissions. Some representatives are likely to need travel support, and some advocacy organisations may lack the resources to contribute effectively without funding support for analysis, discussions within the advocacy organisation, and preparation of a submission. To convey to contributors that the agency has a commitment to listening, and to reflecting the messages provided to them, the agency needs to provide meaningful reports back to reference group members shortly after each event, and to create and maintain a list of 'open issues'.
In the Robodebt context, the agency had comprehensive responsibilities in relation to the operational aspects of all national welfare benefits schemes. The primary user and usee segments, and the primary advocacy organisations active in each segment, should have been very familiar to relevant senior staff within the agency. In any case, each major Inquiry initiated by government, by regulatory agencies and by parliament provides evidence of active players on behalf of welfare recipients. The 62 such organisations that submitted to the Senate Committee Inquiry into Robodebt can be readily categorised as shown in Table 2. The richness of the set suggests that a super-agency may need to operate multiple reference groups, or maintain a 'long-list' of recognised organisations, and then invite a relevant sub-set of them for each event or event-series.
______________
No evidence has been found of the agency responsible for the Robodebt project operating a reference group of welfare advocacy organisations. [ ASK A FEW PEOPLE: INCL. ACOSS, CPSU? ] However, the scale and significance of the project was such that it could have formed one specifically for the purpose. The MSRA process in Figure 1 represents guidance for the conduct of an event-series relating to risk assessment of a particular intervention. The intersections between the Strategic Approach and MSRA are most marked in the identification of stakeholders (1.2), the parallel paths in which multiple stakeholders perform the central steps (1.4 to 1.9), and the activities intended to assimilate the findings into the design work (2.1 to 2.3).
It is unlikely that the process could have performed magic: the conflicts between the agency's values, perceptions and interests, and those of welfare receipients and their advocates, may have been so great that mutual understanding may have been unattainable. Use of MSRA by the agency within a strategic approach based on a reference-group would have nonetheless offered the opportunity for the agency's executives and project staff to understand both other perspectives and the intensity of feeling of users and usees. Interactions may have drawn attention to specific features of the intended scheme that appeared especially problematical, and identified client-segments likely to be particularly badly affected, and contingencies that had not been considered. Armed with that information, the agency may have been able to implement mitigation measures, reduce the temperature among advocacy organisations, and avoid some of the most damaging aspects of subsequent media exposure.
An agency may have concerns about the strategic approach, such as that it 'invites the barbarian inside the gates', and discloses information that may be harmful to the public purse, or to the interests of the agency. An alternative approach an agency can adopt is to find a proxy that can provide it with insights into the views of stakeholders. In particular, a consultancy with expertise in the area can be commissioned to identify issues and risks. The consultancy might be limited to a desktop analysis, or might utilise the agency's own segmentation studies and lists of advocacy organisations, or might run its own studies of segments and interviews with advocates, keeping them at arm's length from the agency.
From the agency's viewpoint, a key deliverable from such an assignment is clarity about the key issues perceived by those stakeholders that have sufficient power to interfere with the agency's ability to achieve its aims. The agency can then devise ways to manage the risks that confront it. For example, scenarios can be developed, and a bank of public relations plans and media releases drafted, ready for deployment in the event that particular contingencies arise.
In the Robodebt case, some evidence exists of public relations plans, because Ministerial statements were promptly issued, conveying that the government was tackling welfare cheating. In addition, successive statements 'stayed on message', and denied there were problems, even in the face of evidence that errors were proliferating. On the other hand, there is limited evidence that the public relations plans were preceded by any work to deliver insights into the key concerns of relevant client segments, or of the power of advocacy organisations, the media, and the parts of the legal profession that specialise in welfare recipient's rights.
Even without the involvement of stakeholders, MSRA would have been capable of offering value to the contingency planning approach. A consultancy could have been commissioned to perform parallel studies from the viewpoints of particular stakeholder groups (applying steps 1.4-1.9, and 2.1-2.3). That would have provided the agency with insights into concerns of the more powerful stakeholders, and the opportunity to adapt the design or the depictions of design features, to implement or prepare mitigation measures, and to prepare media briefings to pre-counter advocacy organisation complaints.
Even where an agency has failed to do any preparatory work of the kinds described in the two previous Approaches, action can be taken in the event that major issues emerge. The steps must be performed very promptly, and necessarily in an abbreviated manner, but the key features remain the same.
In the Robodebt case, the main evidence of adjustments by the agency was some changes made around the time of the Ombudsman's investigation about 3-6 months after commencement. However, the measures were minor, were not based on any interaction with stakeholders, and appear to have been designed to gain supportive comments from the oversight agency that could be used as a defence against subsequent rounds of criticism.
The agency could have harnessed MSR, by combining elements of it into a 'charm offensive'. This has its focus on steps 2.2 (alternative designs, additional safeguards and mitigation measures) and 2.4 (refinement of the design). One form this could take is an invitation to key advocacy organisations and/or media or other players, into the Ministerial office, for a 'heart-to-heart' discussion. This could be accompanied by public statements of assurance that the messages have been heard and are being acted upon. A trickle of announcements of minor adjustments could be provided, aimed at being attractive to the more powerful or noisier stakeholder groups, and dressed up for the media, for example by showcasing individuals who have benefited from the project.
MSRA could have been applied in a more positive fashion even at this late stage, by using steps 2.2 and 2.4 to devise adaptations to the system that have significant impacts on its problematic features. It is likely that a re-visit would have been necessary to the main phases of risk assessment (steps 1.4 to 1.9), in order to reflect the now-apparent perspectives of the more powerful among the affected users and usees. This might have been achievable by re-working the original risk assessment report; but that assumes that the document was of an adequate standard in the first place, and it would risk the contents of the original document becoming publicly known.
A variant would have been a rapid but open ab initio risk assessment, actively involving advocacy organisations, and hence compressing the parallel phases 1.4 to 1.9 into a single, joint activity and articulating it directly into 2.1 (integration into a risk assessment report that reflects multiple viewpoints). Because of the challenges of scale and complexity, particularly in making and testing changes to software that may well be poorly-written and poorly-documented, this may have required a pause in some aspects of the program.
The rationale underlying the Mult-Stakeholder Risk Assessment process is that its core is well-known to public sector agencies and the consultancies that support them, and is already used by them, in some cases perfunctorily, but in many circumstances to good effect. Conventional risk assessment is conducted from the perspective of the agency sponsoring an intervention. The MSRA process grafts additional, parallel analyses onto the main stream of assessment, then draws the findings of each of the stakeholder analyses back into the main stream, integrating them into a consolidated view. The result is a risk management plan founded on insights from all perspectives. The agency gains because its projects are far less likely to encounter serious turbulence, and far less likely to fail. Users and usees gain because collateral damage is greatly reduced, and mitigation measures are included in the scheme to deal with the instances that do arise.
In the case of Robodebt, early detection that the propositions were misconceived did not occur. It is challenging to identify any indication of any aspect of MSRA having been even considered, let alone applied. During preliminary steps 1.1 to 1.3, an overview of the requirements and conceptual design could have been exposed to a few key stakeholders. If there was a desire to avoid exposure to particular advocacy organisations, it would have been feasible to sample some of the others. Such organisations typically vary in their origins, motivations, experience in the field, the backgrounds of their key players, in their resource-base, in their media-savvy and in their degree of activism. Some may have been subject to a degree of capture by the agency, or may have been able to be encouraged to moderate their behavour in return for information and an increased chance of achieving influence. An agency is most likely to be prepared to invite less powerful, and less activist or noisy advocacy organisations, or perhaps proxies, such as consultancies with links to advocacy organisations.
Committed authoritarians and technocrats in the agency and the Minister's office are unlikely to have taken any notice of the information provided; but some of it would have caused pause among the more considered executives and Ministerial staff, and would have empowered staff who are aware of the real-world risks, enabling them to table their concerns in the form of clarifications of statements made by outsiders. However, the Robodebt case is quite extreme. It was so ill-founded, in logic and in law, that these first steps would have demonstrated quite clearly that the project needed to be scrapped, or at least its proponents told to go back to the starting-point and re-think their proposal.
If the project had been persisted with, the main body of the risk assessment activity (steps 1.4 to 1.9) would have followed. The agency may have preferred to avoid too much information leakage, in which case they could have internalised the process, by using stakeholder representatives or proxies in a devil's advocacy role within or adjacent to the agency's project team. Employees and even consultants performing that role need to play their cards delicately and avoid appearing to be 'courageous'; but some indications of the large number of defects in the emergent design would have been available to all but the most closed of minds within the design team and among the relevant senior executives.
The agency's primary function is the conduct and management of interactions with welfare recipients. This involves very large volumes of payments transactions, which are subject to complex rules even within programs let alone between programs, and frequent changes to rules. The agency is also responsible for interactions with welfare recipients themselves, with a strong preference for the less expensive channels - in order, self-service via a web-site, interactive voice-response (IVR), call-centre operators, and physical 'shopfronts'. However, the agency has little or no responsibility for policy aspects, and is accordingly distanced from the interests of the many different categories of welfare recipients. Given that distance, the vast volume of funds being distributed, the great scope for waste, and the existence of opportunities to defraud the public purse, the agency's orientation is strongly towards waste and fraud controls, and tends towards 'Old Testament'-style guilt and retribution assumptions about welfare recipients.
The government during the relevant period was strongly neo-liberal in flavour, which perceived electoral advantage in demonstrating strength, including against welfare-cheats. The confluence of attitudes between agency executives and Ministers appears to have been a key factor in the hostility towards meaningful interactions with advocacy organisations, who appear to have been perceived as 'enemy' rather than stakeholder representatives.
The single-mindedness of the senior executives and the project team went even further, in that they ignored government stakeholders and 'partner' agencies. The source of the data used to generate suspects, the taxation agency, was kept at arm's length (and it appears that the agency concerned may have been very happy to avoid association with the project). The central government agency that had soft-regulatory or oversight responsibilities in relation to major systems was politically weak, and dubiously competent (as confirmed by an audit, ANAO 2022), and it was completely sidelined. Evidence also emerged that no advantage was taken of the experience of the agency's own specialist compliance and IT systems staff, who were forced to administer a scheme that they recognised from the beginning as being deeply flawed. At the very least, the enlistment of internal expertise as a proxy source of stakeholders' views would have represented a valuable element of risk assessment and management.
Full implementation of an open scheme description, parallel risk assessments, and reflection of stakeholder views in the risk management plan is only one way in which MSRA could have contributed. The discussion above has identified multiple ways to apply MSRA that are less intrusive into what agency executives might perceive to be constraints on their freedom to perform their functions they see fit. MSRA is intended to benefit all parties; but even in compromised form, it can assist agencies to avoid disasters.
Interventions have short-term impacts and medium-term implications. Large-scale and otherwise impactful interventions need to be evaluated not just implemented. Many assessment techniques have a tight focus on the interests of the system sponsor. There are few drivers for multi-stakeholder assessment. To the extent that other stakeholders' interests are reflected, it is because those particular parties have sufficient power.
Inevitably, stakeholders that merely have legitimacy and/or urgency suffer depradations from interventions, because the interventions' sponsors are not attuned to their interests. In many cases, harm to their interests can be avoided, or at least mitigated, and even some benefits delivered, with only limited compromise to the sponsor's objectives, if their needs are factored into the design. If their needs are understood at an early stage, the costs of addressing them are minimised.
Even if the will exists to reflect stakeholders' legitimacy as well as their power, there are challenges in finding suitable evaluation techniques to apply. Business Case Development is driven by the prospects of profit. Most of the variants of Impact Assessment concern themselves with particular categories of effect that interventions may have. Technology Assessment, concerned as it is with a broad technology with many capabilities, which may give rise to a variety of interventions into economic and social systems, tends to be highly informative, but its influence is dissipated across many contexts.
The choice may seem strange, but the well-developed technique of Risk Assessment - a creature of rational enterprise management - may harbour the best prospects for reflecting the interests of multiple stakeholders. Preliminary testing of the Multi-Stakeholder Risk Assessment (MSRA) technique, by searching for contexts in which some of its hallmarks are evident, identified a variety of exemplars. This desktop evaluation was complemented by considering the technique's potential application to a large-scale government initiative whose misconceptions gave rise to considerable cost to the public purse, and seriously harmed many individuals. On the basis of these preliminary evaluations, it appears appropriate to seek exposure, experimentation and trialling, in order to establish whether MSRA can be a practicable mechanism to achieve the reflection of interests of the relevant players in the assessment of interventions.
Achterkamp M.C. & Vos J.F.J. (2008) 'Investigating the use of the stakeholder notion in project management literature, a meta-analysis' Int'l J. Project Management 26, 7 (October 2008) 749-757
ANAO (2022) ''Digital Transformation Agency's Procurement of ICT-Related Services' Australian National Audit Office, 21 September 2022, at https://www.anao.gov.au/work/performance-audit/digital-transformation-agency-procurement-ict-related-services
Barrett S. & Konsynski B. (1982) 'Inter-Organization Information Sharing Systems' MIS Quarterly 6, 4 (December 1982) 93-105
Baumer E.P.S. (2015) 'Usees' Proc. 33rd Annual ACM Conf. on Human Factors in Computing Systems (CHI'15), April 2015, at https://dl.acm.org/doi/pdf/10.1145/2702123.2702147
Becker H. & Vanclay F. (2003) 'The International Handbook of Social Impact Assessment' Cheltenham: Edward Elgar, 2003
Berleur J. & Drumm J. (Eds.) (1991) 'Information Technology Assessment' Proc. 4th IFIP-TC9 International Conference on Human Choice and Computers, Dublin, July 8-12, 1990, Elsevier Science Publishers (North-Holland), 1991
Cameron J. (2009) 'An integrated framework for managing eBusiness collaborative projects' PhD Thesis, UNSW, September 2009, at https://unsworks.unsw.edu.au/entities/publication/13ce09cb-ee14-456c-87a3-5e7547a4a2e4/full
Clarke R. (1992) 'Extra-Organisational Systems: A Challenge to the Software Engineering Paradigm' Proc. IFIP World Congress, Madrid, September 1992, PrePrint at http://www.rogerclarke.com/SOS/PaperExtraOrgSys.html
Clarke R. (1994) 'EDI in Australian International Trade and Transportation' Proc. 7th EDI-IOS Conference, Bled, Slovenia, 6-8 June 1994, PrePrint at http://www.rogerclarke.com/EC/Bled94.html
Clarke R. (2009) 'Privacy Impact Assessment: Its Origins and Development' Computer Law & Security Review 25, 2 (April 2009) 123-135, at http://www.rogerclarke.com/DV/PIAHist-08.html
Clarke R. (2015) 'The Prospects of Easier Security for SMEs and Consumers' Computer Law & Security Review 31, 4 (August 2015) 538-552, PrePrint at http://www.rogerclarke.com/EC/SSACS.html#App1
Clarke R. (2019a) 'Principles and Business Processes for Responsible AI' Computer Law & Security Review 35, 4 (2019) 410-422, PrePrint at http://www.rogerclarke.com/EC/AIP.html#MRM
Clarke R. (2019b) 'Risks Inherent in the Digital Surveillance Economy: A Research Agenda' Journal of Information Technology 34,1 (Mar 2019) 59-80, PrePrint at http://www.rogerclarke.com/EC/DSE.html
Clarke R. (2022a) 'Research Opportunities in the Regulatory Aspects of Electronic Markets' Electronic Markets 32, 1 (Jan-Mar 2022) 179-200, PrePrint at http://rogerclarke.com/EC/RAEM.html
Clarke R. (2022b) 'Evaluating the Impact of Digital Interventions into Social Systems: How to Balance Stakeholder Interests' Working Paper, Xamax Consultancy Pty Ltd, May 2022, at http://rogerclarke.com/DV/MSRA-VIE.html
Clarke R. & Jenkins M. (1993) 'The strategic intent of on-line trading systems: a case study in national livestock marketing' Journal of Strategic Information Systems 2, 1 (March 1993) 57-76, PrePrint at http://www.rogerclarke.com/EC/CALM.html
Clarke R., Michael K. & Abbas R. (2022) 'RoboDebt: An Exemplary Case Study of Public Sector Irresponsibility' Working Paper, Xamax Consultancy Pty Ltd, September 2022, at http://rogerclarke.com/EC/RDCS.html
Dikov D. (2020) 'Using the Net Present Value (NPV) in Financial Analysis' Magnimetrics, 2020, at https://magnimetrics.com/net-present-value-npv-in-financial-analysis/
EC (2016) 'EU general risk assessment methodology' European Commission, 2656912, June 2016, at https://ec.europa.eu/docsroom/documents/17107/attachments/1/translations/en/renditions/native
Erdbrink T. (2021) 'Government in Netherlands Resigns After Benefit Scandal', The New York Times, 15 Jan 2021, at https://www.nytimes.com/2021/01/15/world/europe/dutch-government-resignation-rutte-netherlands.html
Fischer-Huebner S. & Lindskog H. (2001) 'Teaching Privacy-Enhancing Technologies' Proc. IFIP WG 11.8 2nd World Conference on Information Security Education, Perth, Australia, 2001
Freeman R.E. & Reed D.L. (1983) 'Stockholders and Stakeholders: A New Perspective on Corporate Governance' California Management Review 25, 3 (1983) 88-106
Garcia L. (1991) 'The U.S. Office of Technology Assessment' Chapter in Berleur J. & Drumm J. (eds.) 'Information Technology Assessment' North-Holland, 1991, at pp.177-180
ISO 27005:2011 'Information technology--Security techniques--Information security risk management' International Standards Organisation, 2011, especially pp. 7-17 and 33-49
Karaboga M. (2022) 'Datenschutzrechtliche Gestaltungsmöglichkeiten jenseits der Ermächtigung des Individuums: Die Multi-Stakeholder-Datenschutz-Folgenabschätzung' In Friedewald M., Kreutzer M. & Hansen M. (eds) 'Selbstbestimmung, Privatheit und Datenschutz', DuD-Fachbeiträge, Springer Vieweg, April 2022, at https://link.springer.com/chapter/10.1007/978-3-658-33306-5_14
Mitchell R.K., Agle B.R. & Wood D.J. (1997) 'Toward a Theory of Stakeholder Identification and Salience: Defining the Principle of Who and What Really Counts' Academy of Management Review 22, 4 (1997) 853-886
Molarius R., Raeikkoenen M., Forssen K. & Maeki K. (2016) 'Enhancing the resilience of electricity networks by multi-stakeholder risk assessment: The case study of adverse winter weather in Finland' Journal of Extreme Events 3, 4 (2016)
Morgan R.K. (2012) 'Environmental impact assessment: the state of the art' Impact Assessment and Project Appraisal 30, 1 (2012) 5-14, at https://doi.org/10.1080/14615517.2012.661557
Neo B.S. (1992) 'The implementation of an electronic market for pig trading in Singapore' Journal of Strategic Information Systems 1, 5 (December 1992) 278-288
NIST (2012) 'Guide for Conducting Risk Assessments' US National Institute for Standards and Technology, SP 800-30 Rev. 1 Sept. 2012, pp. 23-36, at https://csrc.nist.gov/publications/detail/sp/800-30/rev-1/final
OTA (1977) 'Technology Assessment in Business and Government' Office of Technology Assessment, NTIS order #PB-273164', January 1977, at http://www.princeton.edu/~ota/disk3/1977/7711_n.html
Pouloudi A. & Whitley E.A. (1997) 'Stakeholder Identification in Inter-Organizational Systems: Gaining Insights for Drug Use Management Systems' Euro. J. of Information Systems 6, 1 (1997) 1-14
Shackelford S.J. & Russell S. (2016) 'Operationalizing Cybersecurity Due Diligence: A Transatlantic Case Study' South Carolina Law Review 67, 3 (Spring 2016) 7, at https://scholarcommons.sc.edu/cgi/viewcontent.cgi?article=4182&context=sclr
Schmidt M.J. (2005) 'Business Case Essentials: A Guide to Structure and Content' Solution Matrix , 2005, at http://www.solutionmatrix.de/downloads/Business_Case_Essentials.pdf
SoW (2013) 'Discounted Cash Flow Analysis' Street of Walls, 2013, at https://www.streetofwalls.com/finance-training-courses/investment-banking-technical-training/discounted-cash-flow-analysis/
Stobierski T. (2019) 'How To Do a Cost-Benefit Analysis & Why It's Important' Harvard Business School Online, September 2019, at https://online.hbs.edu/blog/post/cost-benefit-analysis
Wahlstrom K. & Quirchmayr G. (2008) 'A Privacy-Enhancing Architecture for Databases' Journal of Research and Practice in Information Technology 40, 3 (August 2008) 151-162, at http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.454.3815&rep=rep1&type=pdf
Wright D. & De Hert P. (eds) (2012) 'Privacy Impact Assessments' Springer, 2012
Wright D., Friedewald M. & Gellert R. (2015) 'Developing and testing a surveillance impact assessment methodology' International Data Privacy Law 5, 1 (2015) 40-53, at https://www.dhi.ac.uk/san/waysofbeing/data/data-crone-wright-2015a.pdf
Wright D. & Raab C.D. (2012) 'Constructing a surveillance impact assessment' Computer Law & Security Review 28, 6 (December 2012) 613-626, at https://www.dhi.ac.uk/san/waysofbeing/data/data-crone-wright-2012a.pdf
Wyman K.M. (2017) 'Taxi Regulation in the Age of Uber' N.Y.U. J. Legislation & Public Policy 20, 1 (April 2017) 1-100, at https://www.nyujlpp.org/wp-content/uploads/2017/04/Wyman-Taxi-Regulation-in-the-Age-of-Uber-20nyujlpp1.pdf
Young R.C. & Jordan E. (2002) 'IT Governance and Risk Management: an integrated multi-stakeholder framework' Asia Pacific Decision Sciences Institute, Bangkok, 2002, at http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.454.3272&rep=rep1&type=pdf
Adapted version of Clarke (2015, p.547-549)
This paper draws on and extends previous work undertaken on the topic, in the context of applications of AI, and previous work on digitalisation, including the digital surveillance economy and digital platforms. These are reported on in Clarke (2019a), Clarke (2019b) and Clarke (2022a). A working paper that proposed the idea of Multi-Stakeholder Risk Assessment was prepared for, and presented at the International Digital Security Forum (IDSF22) in Vienna on 1 June 2022 (Clarke 2022b).
Roger Clarke is Principal of Xamax Consultancy Pty Ltd, Canberra. He is also a Visiting Professor associated with the Allens Hub for Technology, Law and Innovation in UNSW Law, and a Visiting Professor in the Research School of Computer Science at the Australian National University.
| Personalia |
Photographs Presentations Videos |
Access Statistics |
 |
The content and infrastructure for these community service pages are provided by Roger Clarke through his consultancy company, Xamax. From the site's beginnings in August 1994 until February 2009, the infrastructure was provided by the Australian National University. During that time, the site accumulated close to 30 million hits. It passed 65 million in early 2021. Sponsored by the Gallery, Bunhybee Grasslands, the extended Clarke Family, Knights of the Spatchcock and their drummer |
Xamax Consultancy Pty Ltd ACN: 002 360 456 78 Sidaway St, Chapman ACT 2611 AUSTRALIA Tel: +61 2 6288 6916 |
Created: 8 January 2022 - Last Amended: 28 September 2022 by Roger Clarke - Site Last Verified: 15 February 2009
This document is at www.rogerclarke.com/EC/MSRA.html
Mail to Webmaster - © Xamax Consultancy Pty Ltd, 1995-2022 - Privacy Policy