Roger Clarke's Web-Site

 

© Xamax Consultancy Pty Ltd,  1995-2022


Roger Clarke's 'Case Study of Robodebt'

Robodebt: An Exemplary Case Study of
a Failed Transformative IT Project in the Public Sector

Review Version of 8 November 2022

Roger Clarke, Katina Michael and Roba Abbas **

© Xamax Consultancy Pty Ltd, 2022

Available under an AEShareNet Free
for Education licence or a Creative Commons 'Some
Rights Reserved' licence.

This document is at http://rogerclarke.com/DV/RDCS.html


Abstract

Large-scale systems to administer social welfare payments face considerable challenges. This case study reports on an Australian Government project officially called the Online Compliance Intervention (OCI) system, but which was widely referred to as 'Robodebt'. The scheme's purpose was to save labour costs and increase recovery of overpayments. The scheme extended the agency's reliance on the digital persona the agency maintains for each client, and featured automated inferencing, automated decision-making and automated action.

Intended digital transformation turned into large-scale failure, due to poor-quality data-matching, faulty inferencing and inadequate legal foundations. Welfare recipients were required to prosecute their innocence by producing documentation for income earned 2-7 years earlier, failing which the agency took action to recover what it asserted to be overpayments.

Within months of the launch, ample evidence had accumulated that the scheme was misconceived. Nonetheless, the agency persisted with it for a further 3 years. Enormous harm was caused to many thousands of vulnerable people. When the smoke cleared, affected clients received refunds and damages. The outcomes included hundreds of thousands of anxious clients, demoralised staff, and a bill totalling $A3 billion. Lessons are drawn for large-scale transformational IT projects featuring advanced technologies and automation.


Contents


1. Introduction

Applications of information technology (IT) in business and government began with the automation of existing systems (retronymed 'manual' systems). They quickly graduated to systems that rationalised existing processes, in order to take advantage of new possibilities that technologies had created. Over time, the emphasis shifted to business process reengineering (Hammer 1990), and then to organisational transformation (Venkatraman 1994). Most recently, the capacity of IT to achieve organisational disruption (Lyytinen & Rose 2003) has been lauded.

In parallel, the marriage of computing and telecommunications enabled the scope of IT-supported systems to broaden from organisational sub-units, via cross-organisational functions, to entire organisations. It then leapt into inter-organisational systems (1-to-1), and to multi-organisational systems in star, chain and network configurations. An often-overlooked development was the emergence of extra-organisational systems (Clarke 1992b), which extended beyond a particular organisation's boundaries not to one or more other organisations, but to individuals. At first, individuals had to be provided with access to network-connected devices. During the last decades of the 20th century, however, increasing numbers of people acquired desktop and laptop devices that could be harnessed into organisation-managed systems, and the first decades of the 21st century have seen the proliferation of handheld mobile phones and tablets, and their exploitation as an extension of organisations' own information infrastructure.

A concomitant development throughout the growth of such systems has been the digitisation of data, initially through purposeful capture of data specifically relevant to a particular purpose, and progressively as a by-product of other actions. Data capture has also been increasingly outsourced, so that organisations benefit from data-input by unpaid participants in systems, referred to in Abbas et al. (2022, p.78) as ''customer work". By about the turn of the current century, the point had been reached where a large proportion of the data desired by organisations was 'born digital' rather than having to be converted from other paper-based formats.

The availability of vast quantities of data has enabled organisations to achieve efficiencies by no longer dealing directly with other entities, and particularly not with those most time-wasting of entities, people. Organisations find it far more economical to transact with, and impose surveillance on, a digital persona in preference to a person (Clarke 1994, 2014). The term 'datafication' was coined by Cukier & Mayer-Schönberger (2013) as part of those authors' inculcation of breathless excitement about 'big data' as the harbinger of a post-rationalist world. See also Lycett (2014) and Newell & Marabelli (2015).

The more useful term 'digitalisation' has been applied to the shift from the interpretation and management of the world by means of human perception and cognition of phenomena, to processes that are almost entirely dependent on digital data (Brennen & Kreis 2016). Where the entities being interpreted and managed are people, the term carries with it overtones of 'objectification', as the human aspects of interactions are abandoned.

For-profit business enterprises are largely unfettered by public policy considerations in their application of digitalisation. Public sector organisations, on the other hand, operate in an environment in which there is an expectation that greater care will be taken about the direct impacts of transformative and disruptive technology-based interventions, and about their indirect implications or 'collateral damage'. Those impacts and implications are commonly felt by 'users', that is to say the people, inside and outside the organisation, who are direct participants in the relevant system. Too often overlooked are those who are indirectly but materially affected, usefully referred to as 'usees' (Berleur & Drumm 1991 p.388, Clarke 1992b, Fischer-Huebner & Lindskog 2001, Baumer 2015). Examples of usees include people who have entries in criminal intelligence and credit reporting databases, or who are dependents of government employees or welfare recipients.

In this paper, we contribute to understanding of the challenges involved in digitalisation in the public sector. We does this by reporting the results of a case study of a large-scale exercise in transformative IT applied to the administration of social welfare in Australia. Section 2 provides an overview of the context in which the intervention was imposed. Section 3 outlines the research method, and Sections 4 provides a narrative description of the project. This lays the foundation for a thematic analysis of the salient design and process aspects of what transpired to be a disaster of epic proportions, presented in Section 5. We discuss broader issues in public sector digitalisation in Section 6, but with the degree of care essential with any single-case study. Our focus is on public sector projects in the area of transformative digitalisation involving relatively simple inferencing, but with a view also to more ambitious data-analytic techniques such as machine learning.


2. The Context of the Case

Digitalisation depends on data not only being in digital form, but also being reliably associated with particular entities or identities. A considerable amount of public sector activity is accordingly concerned with the administration of relationships between government agencies and people, with governments striving to establish and maintain workable identification schemes (Michael & Michael 2006). Some countries have general-purpose identifiers, others use identifiers across a cluster of related functions, while others depend primarily on agency-level or program-level identifiers. In Australia, successive attempts to establish a national identification scheme have failed (Clarke 1987, Michael 2003). However, a multi-purpose identifier called a Tax File Number (TFN) was established in 1988 (Clarke 1992a) and is in ongoing use in the public sector for a wide range of financial programs.

One use is for data-matching schemes, which is the comparison of machine-readable records from separate sources, which contain personal data that is purported to relate to the same person. Data-matching is a longstanding dataveillance technique much-used by government agencies in various countries since the mid-1980s, particularly to help in addressing financial waste and fraud (Clarke 1994b). In Australia, many such schemes are largely unregulated, and subject only to a non-binding guideline published by the data protection oversight agency. One series of very large-scale programs caused considerable public concern, however, and is subject to a modestly formalised regulatory environment, the Data-Matching Program (Assistance and Tax) Act 1990.

In Australia, there has been a strong trend since about 1990 towards monolithic government, with programs and agencies now concentrated within super- or mega- 'portfolio agencies'. The Australian Taxation Office (ATO) is within the Treasury portfolio. In the social welfare area, policy in relation to the more than 100 support programs continues to be with specialist agencies in a variety of portfolios. The policy agency for unemployment benefits (referred to using various terms, including Newstart and Jobseeker), the age pension, disability support and crisis payments in relation to fires, floods, the COVID pandemic, etc. is the Department of Social Services. Military service pensions policy rests with the Department of Veterans Affairs, and childcare subsidy was at the time with the Department of Employment and Workplace Relations, whereas student support (Austudy and Abstudy) rested with the Departments of Education and of Social Service, and farm household allowance policy with the Department of Agriculture, Water and the Environment (SA 2022). Social welfare transfer payments represent about 20 percent of the Australian Government's Budget. The major schemes assist 2.6 million people of working age (13 percent of those age-groups and 10 percent of the population) and 2.5 million people of Age Pension age (a further 10 percent of the population). "For the poorest 20% of Australian households, social security payments provide more than 70% of their income" (Whiteford 2022).

Although policy responsibility vests in specialist agencies, the administration of social security is funnelled through a specialist agency called Centrelink, formed in 1997. It is a services-, data- and identifier-hub, handling payments to welfare recipients, all interactions with them, and associated surveillance activities. In 2011, Centrelink and the national health insurance service, Medicare, were combined into a new super-agency, Department of Human Services (DHS), also branded since 2019 as Services Australia. An enormous diversity exists, of programs, needs, clientele, and personal and regional circumstances. DHS delivers critical financial support to Australia's most vulnerable populations, extending to such categories as people with significant physical disabilities or mental health conditions, and single parents. Some people are in multiple such categories. Those living in remote and rural Australia have limited access to local services.

The scores of complex, longstanding and frequently-modified welfare schemes are designed to protect the public purse perhaps as much as to address economic needs. The operation of Centrelink's distributed and now-outsourced call-centres, and its remaining physical service-centres, are beset with enormous challenges. Staff are expected to master an array of disparate details and interact with large numbers of needy and sometimes disturbed individuals. Diseconomies of scale and scope are in play. The complexities, ambiguities, inconsistencies, conflicts and discretions within written policies, and the diversity of circumstances, combine to give rise to misunderstandings and errors, and inevitably to recriminations, requests for review and appeals.

Broad scope exists for fraud, error and inefficiencies. Considerable effort is invested in discovering where overpayments have occurred, and in seeking restitution. Official estimates of the scale of overpayments are in the range $1.5-4.0 billion p.a. (i.e. 1.0-2.5 percent of c.$160bn p.a. in payments). Successive generations of technology have been harnessed in support of Centrelink's work, and social services agencies have long operated one of the nation's largest IT facilities. This case study examines a project referred to internally as Online Compliance Intervention (OCI), although also referred to in some media reports in late 2016 as the Welfare Payment Infrastructure Transportation (WPIT) Program. It was intended by the agency as a form of transformative digitalisation. The study spans the period 2015-2022, from the pilot phase, via the launch in 2016 and its operation 2016-20, through to a series of official autopsies 2017-22. The relevant government agency is referred to in the remainder of this document as the Department of Human Services (DHS).


3. The Research Method

The previous sections have evidenced the complexity of the system operated by DHS. To gain insights into the conduct of a major digitalisation project in the area, it is necessary to examine events in their natural setting. The scale of activities and their dispersion across organisations and locations are such that fieldwork using observation-based methods would require enormous resources. In any case, the government agency involved was hostile to proposals that it should disclose information about its activities. For example, the final Report of a Parliamentary Committee on the matter concluded that the Government compounded its many felonies in relation to the program itself by "the deliberate withholding of information" (SCA 2022, p.17). The likelihood of observers being permitted inside the organisation appeared throughout to be very low.

The relevant research method to adopt was accordingly a case study based on published sources, seeking a holistic view of the design, conduct and impacts of the project over time. Although quantitative data needs to be used to the extent practicable, research of this nature is dependent largely on qualitative data. The research was not undertaken in the scientistic or positivist traditions (Benbasat et al. 1987, Yin 2017). Nor was a critical realism framework applied, whereby "a causal explanation for a given phenomenon is inferred by explicitly identifying the means by which structural entities and contextual conditions interact to generate a given set of events" (Wynn & Williams 2012). The approach adopted is interpretivist, because many stakeholders exist, with distinct points-of-view and conflicting values and interests, and an analysis can only achieve relevance if it reflects those differences (Walsham 1995).

Reflecting the motivations discussed above, the objective of the study was defined as:

To seek understanding of the factors that resulted in a public sector digitalisation project failing to achieve its objectives and causing enormous collateral damage.

Access to DHS records was not feasible, because its operations were not transparent at the time, and it strenuously resisted requests for access to information by its clients, by lawyers acting on their behalf, and by a Parliamentary Committee, aided by the dysfunctionality of the country's Freedom of Information mechanism. The study accordingly drew on multiple secondary sources of evidence. Preference was given to authoritative sources, including government reports, Parliamentary Committee Reports and a judgment by a senior court. News media have been cited where no more authoritative source could be found.

The first published report was on investigations by the Ombudsman (Omb 2017). It was limited to the purported debts raised, and excluded any consideration of any of the policy rationale, the debt raising and recovery program, procedural fairness, and the use of external debt collection agencies. The Ombudsman's Report was expressed in a manner very sympathetic to the Department and the project, despite the litany of problems that it uncovered - as was evidenced by the delight apparent in the DHS response (pp.45-46), and its use of the document as a major part of its defence against the subsequent Senate Committee Report (GR 2017). Nonetheless, the Ombudsman's Report does provide some information that had previously been suppressed, and it did result in corrections to some of the many deficiencies.

The second authoritative source is a Senate Committee Report (SCA 2017), which elicited an entirely dismissive response from the Government (GR 2017). The Senate Committee had a much broader Terms of Reference than the Ombudsman, and encompassed the scheme's error rates, compliance with legal, regulatory and public policy requirements, and impacts. In Australia, the Government is formed in the lower House of Representatives, whereas the Committee reflected the party representation in the Senate, with a mix of Government and opposition members and a chair from the cross-bench Greens Party (whose policies, other than in relation to environmental matters, can be depicted as being in the social democratic tradition). Subsequent sources, of less value for the present purposes, include a series of reports from a second Inquiry by the same Senate Committee, in 2020-22, including the final report (SCA 2022). A Royal Commission has been established, but its report will arrive 7-9 years after the events, and its focus is likely to be less on policy and practice and more on politics.

The third key source is the judgment of a senior court in a subsequent court-case (Pry 2021). In addition, some publicly-available sources from the agency itself were used, together with media reports that provided information that assists in understanding the business process inherent in OCI, and its outcomes. Such peer-reviewed articles as have appeared were also checked for information and analysis. As the present article was being completed, a formal case study appeared in the refereed literature (Rinta-Kahila et al. 2022). That article adopts algorithmic decision-making (ADM) as a theoretical frame, and uses open coding followed by inductive coding. That contrasts with the present work, which situates the project within a frame of the application of transformative digitalisation in a government agency, and adopts a narrative approach reflecting a blend of academic research and professional and consultancy practice expertise, followed by a thematic analysis and a discussion of the broader implications.

The following section presents the relevant aspects chronologically, from 2015 to 2022, with the majority of the inferences and interpretations held over to a later section of the article. The narrative approach allows an anticipatory tone, thick description in explanation, and the scope to delve more deeply into particular elements of the story that shed light on the events. As the project unravels, the hopeful application of technology to help the Government improve its promise to Australian taxpayers gives way firstly to questioning how fundamental flaws went unnoticed and/or unaddressed for so long, followed by appreciation of the staff toil and the human toll, and then widespread bemusement about the complete absence of accountability for the fiasco.


4. Case Study of the OCI Project

This section traces the history of the Online Compliance Intervention (OCI) project. An outline of the timeline is in Table 1. The section commences by summarising relevant predecessor activities. It then outlines the project's purposes and the design features intended to fulfil them. The implementation and operational phases are described, followed by evidence of the scheme's impacts, and - despite the Government's sustained denial of the scheme's misconception, illegality and harmful effects - its eventual demise and the subsequent repercussions.

Table 1: Timeline of the Robodebt Project

2015 Budget Appropriation in May
Two-Stage Pilot Scheme, from July
2016 Initial implementation, in July
Full roll-out, from September
Eruption of media coverage, from December
2017 Ongoing media coverage and public anger
Continued illegal and harmful operation
2018 Continued illegal and harmful operation
2019 Continued illegal and harmful operation
Withdrawal of the scheme, in November
2020 Commencement of repayments, in July
2021 Withdrawal of the defence against a class action at the court door,
accepting a consent judgment with pitiful redress to aggrieved parties
2022 Change of Government, in May
Finalisation of the court case, in June
Commencement of the Royal Commission, in August
Completion of repayments to clients, in September
Cancellation of the outstanding 200,000 debts, in October

4.1 Prologue: Data Matching - 1990 to 2015

Data is gathered by DHS from its clients. DHS also has substantial powers to demand information from other sources, such as organisations for which welfare recipients perform work and from which they receive payments. In addition, using specific authority enacted in 1990, "DHS began data matching activities in 1991" (Omb 2017, p.5, fn7), and it "has conducted PAYG data matching activities with the [Australian Taxation Office] ATO since 2004 ..." (p.5).

"Income support payments are subject to an income test which means that a recipient's fortnightly payment may be reduced once their income reaches a specific threshold" (SCA 2017, p.13). The purpose of data matching with ATO is to check data provided to DHS by clients against that available through the taxation system: "payment recipients must report their income fortnightly ... Where it is found that a recipient has incorrectly reported their fortnightly income, and the correct amount would have affected their entitlement to a payment, a ... debt may be raised" (SCA 2017, p.14). Discrepancies may indicate that overpayments have occurred, many of them minor. Many are the result of errors, variously by the client and by DHS, and some are fraudulent. Prior to 2016, investigative work by DHS had been giving rise to over 20,000 debt notices annually.

Although the matching itself is automated, the process in use until 2015 continued to involve a considerable amount of manual work. In the spirit of digitalisation, DHS initiated a project with the intention of reducing the manual effort through further application of technology. An appropriate term for the organisation that is the primary driver and primary beneficiary of an IT-based information systems project is 'system sponsor' (Clarke & Davidson 2020, p.489). In the Government's Budget of May 2015, funding for a project was allocated, announced in the Media Release as follows:

The Government has zero tolerance for rorting of our welfare system. We will put a strong welfare cop on the beat focusing on deterrence, detection, investigation and prosecution to track down suspected welfare fraud and non-compliance. From 1 July 2015, the Government will increase DHS's capability to detect, investigate and deter suspected welfare fraud and non-compliance. ... It will achieve net savings of around $1.5 billion. (MRR, 2015)

Evidence to the Royal Commission 7 years later showed that the legal branch of the policy agency, the Department of Social Services, provided advice in 2014 that the use of income averaging in the manner proposed may be unlawful; but the department was pressured by the Minister to adopt the policy ahead of the 2015 budget (Bannister 2022).

What that Media Release meant in concrete terms only slowly became publicly apparent. According to the Ombudsman's Report, the main efficiencies were to be gained by:

A DHS Deputy Secretary stated in evidence that "we are trying to eliminate the need for anybody [on DHS staff] to [go to the employer] so that it is easier [for DHS]" (SCA 2017, p.16). The efficiencies were argued to encompass both reductions in labour costs significantly greater than the ongoing technology costs combined with substantial improvements in the identification and recovery of overpayments. Together, these resulted in "forecast ... $3.7billion worth of [net] savings ... over four years from 2016-17" (SCA 2017, p.4).

On the other hand, the interpretation of the Community and Public Sector Union was that: "[OCI] is a fairly obvious consequence of a department that no longer has the resources to provide effective services. The decision to replace the human oversight of debt recovery with automated data matching was absolutely based on a desire and an imperative to save money" (SCA 2017, p.19).


4.2 The OCI Business Process - 2015-16

A succinct description of the core of the Online Compliance Intervention scheme was provided in the Ombudsman's Report:

[After] the OCI matches the earnings recorded on a customer's Centrelink record with historical pay as you go (PAYG) income data from the Australian Taxation Office (ATO) ... customers are asked to confirm or update their income using the online system. If there is a discrepancy in the data match, this can result in a debt the customer must repay. Parts of the debt raising process previously done manually by compliance officers within DHS are now done using this automated process. (Omb 2017, p.4)

More detailed descriptions are provided of the manual system (Omb 2017, p.31), the pilot (p.32), the interim business process from 2015 (p.33), the live process July 2016 to January 2017 (pp.33-35), and the changes made in February 2017 (pp.35-38). However, none of the authoritative sources appear to make clear whether OCI applied to every means-tested benefits scheme administered by DHS (as implied by the expression "income support payments" (Omb 2017, p.33), or only some of those schemes (as implied by Government statements in January 2017 that the scheme was to be later extended to age and disability pensions). A simplified model of the primary process is in Figure 1.

The actions required of recipients involved considerable assimilation of information, searches for evidence, and identification of and contacts with previous employers. Recipients were also required to register with an additional intermediary website called myGov (Omb 2017, p.34). No telephone contact-point was provided. The notification included the statement that DHS "will update [their] details using the enclosed employment income information" (p.34), which represented an implied threat to use that data to compute a liability by the client to DHS.

Figure 1: Simplified Model of the OCI Process


4.3 Implementation and Operation - 2016-19

DHS rolled out the measure in three stages:

"The initial group ... included those who received income support payments in the 2010 to 2014 financial years" (Omb 2017, p.33). The launch was at the beginning of the 2017 financial year, so the demands related to periods 2-7 years in the past. No information has been found showing the distribution over those 5 years. DHS was aware that many of the targets "were no longer Centrelink customers" (p.33). It is unclear whether DHS estimated the proportion whose contact-details were no longer current and who would therefore quite probably not receive the letter, whether they monitored for evidence of non-receipt, and whether they handled those cases any differently from current clients.

The recipients were forced to think back to the relevant period(s), search for documentation relating to those periods, and contact employers who they may have had no contact with for quite some time, and request (because they very probably had no standing to demand) copies of documents, whose names vary depending on the employer. The minimum period they had to go back was 2 years, but the longest look-back required was 7 years, to July 2009 (i.e. the beginning of the 2010 financial year). It is unclear whether DHS had any evidence about, or did any studies using, for example, its own staff, or randomly-selected members of the public, let alone welfare recipients, in order to establish the extent to which the scheme may have demanded a level of intellectual capacity, online expertise and assiduousness in record-keeping that many benefit recipients do not have.

These notices were now being sent in an automated manner, without human (i.e. government employee or contractor) intervention. The number of debt notices skyrocketed from 20,000 per annum to 20,000 per week (Sarah 2016). Rather than causing alarm to the Government, it was trumpeted as a success that, after just four months of the scheme, in January 2017, 169,000 debt notices had been sent to some of Australia's most vulnerable individuals. It was credited by the Government as having already "identified close to $300 million in overpayments to welfare recipients" (Belot & McGhee 2017) - although the Minister at the time referred to that figure, apparently erroneously, as having been recovered rather than merely identified (Belot 2017a). It was also announced that the Government was considering expanding the OCI beyond Jobseeker unemployment benefits to age and disability pensions (Belot 2017b).

Many recipients were unable to get through to the overloaded call-centres, with many reporting delays of 3-6 hours before a call was answered, and during some periods the automated telephone system was overwhelmed, and calls did not even reach the automated-response system. The 28 percent of the population who live in rural and remote areas, representing about one-third of welfare-recipients, have to travel long distances to Centrelink offices. In regional and suburban offices alike, people had to wait in long queues to talk to under-resourced and under-informed counter-staff. Where the alleged debtor was no longer a benefits recipient, DHS used private-sector debt collectors. The contracted firms included one that was soon afterwards successfully prosecuted for coercion, false representations and unconscionable conduct when dealing with a consumer (Bajkowski 2022). It appears that almost half of all debts raised under the OCI scheme may have been in this category.

"In the 2016 calendar year, 126,571 [individuals] ... were sent debt notices under the OCI" (Omb, p.4, fn5), with the vast majority apparently sent in the 4 months from September to December. However, as early as the beginning of December, only 2 months of the main rollout, complaints about the scheme were rife (TT 2016), and by late December it was public knowledge that the basis on which OCI was built was grossly flawed and the error-rate very high (Knaus 2016). Despite the annual summer slow-down from early December 2016 to late January 2017, media pressure built rapidly. The term 'Robodebt' was coined, and widely adopted by no later than mid-January 2017. The Ombudsman felt it necessary to commence a study, DHS felt the need for some token changes, and a Senate Committee lumbered into existence.

Some changes were made in late 2016 and early 2017, but they were limited to corrections of obvious blunders in the original detailed design, and not corrected until 6-9 months after rollout commenced. These related to (Omb 2017, pp.35-38) the absence of the phone-number from the letter, exception-handling where DHS was aware that the letter has not reached the target-person, extension of the 21-day response-time to 28 days, acceptance of bank statements in lieu of payslips, fixing of errors in the UI/UX design, the pausing of recovery action while a matter is under internal review (not previously implemented, in breach of fair process), and easing of the circumstances in which a 10 percent debt-recovery fee was automatically applied. The correction of greatest substance was the belated recognition that complex cases are unsuitable for automation and need to "fall out for manual intervention" (p.36).

The Government drew further public approbation when, in an endeavour to discredit a complainant who had gone public, the Minister at the time released personal data about the individual from DHS records. It was quite probably a breach of the law, but no enforcement action was ever taken (Knaus 2017).


4.4 Collapse - 2017-22

By April 2017, the scheme's problems were well-understood and could be clearly explained to the Senate Committee (Barbaschow 2017). That Senate Committee's report was scathing: "Communication problems included letters not being received, trouble contacting the department via phone, difficulty in receiving intelligible income data used to calculate purported debts, hard to navigate online communication portals, difficult to understand correspondence and a lack of material translated into other languages" (SCA 2017, p.41, with specifics documented on pp.43-48 and 56-59). "The committee notes it is clear there is a significant communication problem when 65,000 from 300,000 people do not respond to requests from the department to engage" (p.48). Inadequacies under the heading 'Communication barriers' were listed as Vulnerability flags, Literacy, English as a secondary language, Disability-related communication barriers, Geographic barriers, and difficulties people faced with all three communication channels (pp.48-55 and 59-69).

The Senate Committee concluded that:

The key concern with the OCI process, is the outsourcing of the income checking process to individuals. With this comes an inherent reversal of the burden of proof - the department claims an income discrepancy and requires an individual to seek the information required to prove the discrepancy does not exist. If the individual fails, they will owe a debt of potentially many thousands of dollars to the department. The two fundamental resources a person needs to undertake this process is a method of communicating, and once that communication channel is opened, the receipt of information that is both comprehensive and comprehendible [sic]. The department is clearly failing to provide those two necessary tools to allow people to challenge the income discrepancy, and is reaping the benefit through debt payments. (SCA 2017, p.69)

Inadequacies were also noted in the process to clarify or review a purported debt, the policy on handling queries and its impacts, the challenges posed by reversing the onus of proof, the impact on community legal centres, and even in the subsequent process improvements made by the department (SCA 2017, pp.71-89). The Senate Committee also noted the absence of any obligation on DHS to comply with any guidance in relation to the use of debt collectors, or even to take any responsibility for malpractice by them (SCA 2017, pp.29-32).

Remarkably, it took a further 2-1/2 years, until November 2019, before the Government accepted defeat, and ceased raising debts "where the only information we are relying on is our own averaging of Australia Taxation Office income data" (Farrell 2019). This appears to have been forced by an imminent case that DHS had inadvertently let slip through to the courtroom (Karp 2019), despite its previous assiduousness in avoiding test-cases. It then took a further 6 months, until May 2020, for the Attorney-General to concede that "all Centrelink debts raised using the 'income averaging' method were unlawful" (Henriques-Gomes 2020).

A class action was mounted. In breach of the agency's formal public policy obligation to be a 'model litigant', DHS resisted the action until the court door, whereupon it conceded the illegality of its actions, and all claims - although for the pitifully inadequate sum of $A100m, representing $A250 per member of the class action. The department agreed to settle the class action only in November 2020, with judicial approval further delayed until June 2021 (Pry 2021, SCA 2022 p.6), and repayments and payments not completed until the end of September 2022.

The Senate Community Affairs Reference Committee had commenced a second review of OCI in July 2019. It published a succession of five interim reports from September 2020 onwards and a final report in May 2022. The last (SCA 2022) was entitled 'Accountability and justice: Why we need a Royal Commission into Robodebt', and used the headline 'A massive failure of public administration'.

The Coalition Government was defeated in the election of May 2022, and the Labor opposition Opposition quickly moved to establish a Royal Commission, in August 2022.

It had been widely understood that the then Coalition Government had wiped all OCI-generated debts in mid-2020. However, it was reported in the media in mid-October 2022 that the new Labor Minister had announced the cancellation of about 124,000 cases still under review and another 73,000 where potential debts had been identified but the clients had not been informed (Brookes 2022).


4.5 Impacts

DHS had projected that the number of overpayment reviews processed each year would leap almost forty-fold from 20,000 to 783,000 (Omb 2017, p.5). Yet the agency appears not to have realised that this would give rise to very large increases in the numbers of transactions conducted not only in the online system, but also at the call-centre and physical service-centres. It might have been expected that additional staff would have been put in place, particularly for the inevitably fraught transitional phase. In fact, it appears that the agency was forced to comply with Government-imposed reductions in headcounts, such that call-centre and front-counter staffing was lower than it had been prior to the new system being launched. If human resources had been involved in cross-checking notices, the volume would have been internally obvious, and precautionary reviews undertaken.

The staff union drew attention to the "classic false economy" and the "costly reverse workflows where staff are taken offline to deal with complex and difficult disputes over incorrectly raised automated debts." (SCA 2017, p.19). Staff dissatisfaction, embarrassment, and stress were widespread, as a result of having to deal with ill-informed, alarmed, depressed, disturbed and rude clients (Belot 2017a). The union submitted that "[OCI] has been an absolute disaster for many Centrelink use[r]s and also for the workers charged with implementing a system they know to be deeply flawed and unfair" (SCA 2017, p.18).

Both the Ombudsman's and Senate Committee's Reports detailed many personal accounts of the stress and distress caused to the benefits-recipients subjected to it, with repeated mentions of "feelings of anxiety, fear and humiliation". "Individuals had spent hours finding the required pay slips and bank statements, some dating back to 2010-11, often for Centrelink to find that no debt was owed" (SCA 2017, p.38).

On the basis of smatterings of data extracted from DHS at various times during 2017, it appears that only in about 40 percent of the cases that it commenced did the department pursue the debts in full, with about 20 percent of debts reduced, about 20 percent of debts deemed not worth pursuing, and about 20 percent withdrawn because they were clearly wrong. It later accepted that even the 60 percent that it pursued had to be repaid, because there was no legal basis for raising the purported debts.

The Senate Committee's Report noted that over $A1.7 billion in illegitimate debts were imposed on 433,000 people, of whom 381,000 individuals were pursued, many through private debt collection agencies, to repay over $750m to the Commonwealth. The debts that had been raised were later withdrawn, the payments made in the meantime were eventually refunded, and a further $112m was paid in damages.

The admitted costs of implementation were $600m to mid-2019 alone (Burgess 2019), but many elements of the total cost remain shrouded because the Government refused to disclose such information. "The Income Compliance Program impacted hundreds of thousands of people and, for many, resulted in devastating emotional and psychological harm. It has undermined many people's financial security as well as their willingness to engage with and trust government services" (SCA 2022, p.1). See also (Braithwaite 2020).

The Federal Court judgement included statements of a kind not commonly seen in such documents:

One thing ... that stands out from the objections is the financial hardship, anxiety and distress, including suicidal ideation and in some cases suicide, that people or their loved ones say was suffered as a result of the Robodebt system, and that many say they felt shame and hurt at being wrongly branded Òwelfare cheatsÓ. Some of the objections were heart- wrenching and one could not help but be touched by them. (Pry 2021, p.7)

The cost in human life has been the subject of speculation, with one observation being that:

More than 2030 people died after receiving a [OCI] debt notice ... Of those, 429 - roughly one-fifth - were aged under 35. The figures cover a period from July 2016 to October 2018 ... there were 3139 deaths of people aged between 15 and 35 in 2016 overall. (Medhora 2019)

5. Problematic Features

This section builds on the narrative presentation above, by examining themes that emerged in relation to the OCI's prospects of success, public policy issues and/or impacts on people. The first group of themes has to do with the design of the scheme, and the second group with its conduct.


5.1 Design Aspects

(a) Avoidance of the Data Matching Regulatory Mechanism

On the one hand, "[t]he department provides the ATO with the identity information of Centrelink recipients which the ATO matches against their records. In order for the ATO to provide income information to the department, the ATO must identify a high confidence match between the identity information provided and the ATO's records" (SCA 2017, p.14). On the other, "the department has chosen not to use TFNs under the OCI program. This has meant that, unlike previous data-matching processes, the OCI program is not legally bound by the provisions of the Data-matching Act ..." (SCA 2017, p.28).

The Tax File Number (TFN) scheme was introduced in 1991, and has operated since then, for the express purpose of enabling efficient and reliable identification of fund flows relevant to income taxation processes. Since that time, the ATO and other agencies, in particular DHS, have used the TFN in multiple, ongoing data-matching programs. It is very difficult to see any reason for the design decision to not use it in the OCI scheme other than to avoid having to comply with the statutory provisions that apply to TFN-based data-matching under the Data-matching Program (Assistance and Tax) Act 1990, and instead pretend to comply with a voluntary and weak set of guidelines issued by the data protection oversight agency (OAIC 2014).

"[DHS] sends approximately 80 million identities to the ATO and receives approximately six million matches back from the ATO" (Omb 2017, p.34). That seemingly low match-rate raises questions about the accuracy and reliability of the expedient adopted by DHS, and whether it is significantly inferior to the match-rate that would have been achieved if the TFN were used.

The Senate Committee's Recommendation 4 was that:

[A]ll data-matching guidelines and protocols be adhered to, including the Data-matching Program (Assistance and Tax) Act 1990, regardless of whether the department is using tax file numbers. This will require the department to halt the Online Compliance Intervention process while steps are taken to ensure compliance ... . (SCA 2017, p.109)

The Government did not accept the proposition (GR 2017, p.10). In the absence of any coherent explanation from DHS or the Government, it appears that the reason was that the program may not have been able to be performed effectively if it had to comply with that law, and may not even have been able to be performed at all.

(b) Non-Compliance with Data Protection Law

The data protection oversight agency (OAIC) identified multiple breaches of Privacy Act requirements in relation to data quality and steps to correct poor quality data (SCA 2017, p.25). The Australian Privacy Foundation argued that there were far more breaches involved than those identified by OAIC (SCA 2017, pp.28-29). Data quality failures noted by the Senate Committee included:

(c) The Fatal Assumption Inherent in the Inferencing Technique

The Ombudsman's Report provided the following explanation:

`Averaging' refers to the practice of treating income as if it was earned at a consistent rate over a total period of employment rather than applying the precise amounts against the fortnights in which the income was actually earned. DHS currently applies averaging where a person accepts the PAYG data or does not enter data for all fortnights. (Omb 2017, p.7, fn13)

It is unclear over what period each client's income from each employer was averaged, because the only information that appears to be publicly available is that it was "over the period the employer told the ATO the customer worked for them" (Omb 2017, p.34). However, " ... employers must only provide [to ATO] an annual figure paid during that financial year ... [and only about half] of records [from employers] were for a full year employment" (SCA 2017, p.16). So it appears very likely that, in a great many cases, erroneous periods were used as a basis for the averaging calculation.

The Senate Committee noted multiple aspects of the problem:

A further aspect came to light in the court-case that finally brought DHS to it senses. "A Freedom of Information request revealed that [the client] had actually been underpaid by $480 for the period concerned. If the government were serious about `the right payment', it would have sent the client a cheque, not a debt" (Whiteford 2021, p.354).

The view from a regional social services advocacy organisation (ACTCOSS) was that:

[This is a] labour market in which people are trying to work and comply with their Centrelink requirements, ... a market in which people get bits and pieces of work; work irregular hours and often spend periods of time across a financial year out of the workforce. This leads to it being way more complicated and extremely onerous to comply with a ... system that assumes that people either have or do not have a job across a financial year. (SCA 2017, p.35)

It appears that inferring the income for a period by averaging will be wrong about half the time, and the data from ATO is inadequate to determine which category each case falls into. The averaging approach is in no way a suitable basis for any administrative decision-making, let alone where those decisions impose punitive measures, let alone in relation to vulnerable populations, let alone by automated means unchecked by any appreciation of the reality and by 'common sense'. Evidence to the Royal Commission in 2022 showed that the ATO had, in 2017, asked DHS to Òcease and desistÓ using taxpayer data for the robodebt scheme, but was ignored (Wong 2022).

Importantly, the DHS's own 'Operational Blueprint' acknowledged "difficulties", including "If employment is for a part of a year only" and "If income varied greatly during the year" (Omb 2017, p.42). The agency also tacitly acknowledged that the "difficulties" were severe, in that, during the 30 years of use since the early 1980s, its use had been "limited to last resort situations" (Omb 2017, p.42). Given that the agency was well aware that averaging was suitable only for "last resort situations", it is reasonable to infer culpability on the part of Ministers and DHS executives in adopting the technique.

The Federal Court judgment was scathing:

In the course of the proceeding the Commonwealth admitted that it did not have a proper legal basis to raise, demand or recover asserted debts which were based on income averaging from ATO data. The evidence shows that the Commonwealth unlawfully asserted such debts, totalling at least $1.763 billion against approximately 433,000 Australians. Then, including through private debt collection agencies, the Commonwealth pursued people to repay these wrongly asserted debts, and recovered approximately $751 million from about 381,000 of them. (Pry 2021, p.1)

Further:

[I]t is fundamental that before the state asserts that its citizens have a legal obligation to pay a debt to it, and before it recovers those debts, the debts have a proper basis in law. ... Having regard to that, and the profound asymmetry in resources, capacity and information that existed between them and the Commonwealth, it is self-evident that before the Commonwealth raised, demanded and recovered asserted social security debts, it ought to have ensured that it had a proper legal basis to do so. The proceeding revealed that the Commonwealth completely failed in fulfilling that obligation. Its failure was particularly acute given that many people who faced demands for repayment of unlawfully asserted debts could ill afford to repay those amounts. (Pry 2021, pp.3-4)

The judgment continued that the Commonwealth will:

(d) Outsourcing to the Individual of the Checking Process Against Other Sources

"Previously, where a discrepancy was identified between [the data held by DHS and ATO], the department manually checked the information for accuracy and contacted the recipient and/or their employer to clarify the information" (SCA 2017, p.15). "Under manual data-matching arrangements in 2009-2010, approximately 25.5 per cent of identified discrepancies were resolved as the recipient or employer was able to provide information which confirmed the recipient had received the correct Centrelink payment" (SCA 2017, p.17).

However:

[T]he process of checking the ATO lump sum income records against the department's fortnightly income records, a time-consuming process previously undertaken by departmental personnel, has [as part of OCI] been outsourced to the individual income payment support recipients. (SCA 2017, p.2)

"Recipients are [now] directed to an online portal to check the information and provide supporting evidence of their fortnightly income, dating back to 2010 for some people" (SCA 2017, p.17). The result was that "the significant reduction in workload for the department by this outsourcing, has allowed for a huge increase in the number of income discrepancy investigations that the department initiates" (SCA 2017, p.17). In short, the agency abandoned its obligations to perform data quality assurance in order to increase its enforcement activities, and it did so without recognition of or concern that a large proportion of the additional enforcement actions were unjustified, because the putative debt was illusory.

Staff have powers that they can exercise, have sophisticated technological facilities at their disposal, know who to contact, have training, and perform the tasks frequently. Individual benefits recipients have none of those advantages. Some are incapable of working out what needs to be done, many lack the skills to execute the tasks, and many are cowed by DHS and/or by their employers or ex-employers.

The Senate Committee was emphatic that this was unjustifiable, with its Recommendation 6 being that "the department resume full responsibility for calculating verifiable debts (including manual checking) relating to income support overpayments, which are based on actual fortnightly earnings and not an assumed average" (SCA 2017, p.109). The recommendation was ignored in mid-2017. The agency inflicted ongoing harm to further hundreds of thousands of individuals until it was forced to abandon the scheme over 3 years after the mid-2016 launch, in November 2019.

(e) Reversal of the Onus of Proof

Previously, DHS checked that the information being used for debt collection was accurate. Under OCI, the welfare recipient was forced to present counter-evidence to a vague debt collection notice. Many citizens lacked the capacity to respond adequately, if at all. The Senate Committee expressed concern about:

[T]he shift in the onus from the department to the individual recipient to verify whether or not a purported debt exists. The committee is particularly concerned that individuals do not have access to the same resources and coercive powers as the department to access historical employment income information. (SCA 2017, p.19)

Despite the enormous challenges to individuals created by the reversal of the onus of proof, in 20 percent of cases "the individual has been able to provide clarifying information and this has resulted in no debt being owed" (p.33). Given the difficulties faced by benefits recipients, rehearsed throughout the Senate Committee Report, the 20 percent was clearly the tip of a much larger iceberg of instances in which the inference DHS had drawn was a serious error.

In particular in relation to the reversal of the onus of proof, the Committee noted what it referred to as psychological, financial and time impacts; the change in the time when checking records held by the department is done; and the absence of a basis in law (SCA 2017, pp.79-84):

The committee accepts that challenging these purported debts has taken considerable effort on behalf of those individuals. The committee notes that no other party is entitled in law to assert that a debt exists and require the other party to disprove it (SCA 2017, p.84)

Further:

[W]here a purported debt is raised against an individual, the committee considers that the department should be forthcoming with the calculations that demonstrate how the debt was arrived at. The committee considers that these factors contributed to the increased number of requests for assistance received by community legal centres. Funding for these legal centres remains a substantial issue. (SCA 2017, p.89)

(f) Automation beyond Inferencing to Decision and Action

In automating the decision and taking action without human review or even the capacity to intervene, OCI appears to have relied on s.6A(2) of the Social Security (Administration) Act 1999 (Cth), which states that "a decision made by the operation of a computer program under an arrangement made under subsection (1) is taken to be a decision made by the Secretary".

Although that provision was inserted into the Act in April 2001, it appears that this project may have been one of the first occasions on which the automation of administrative decision-making has been conducted by an Australian government agency, at least on a large scale (Carney 2019). People's lives were placed at the direct mercy of a computational system, without human consideration of the matter prior to action being taken that was harmful to the people concerned. This suggests blind faith in technology, and excessive trust in the digital persona as an adequate proxy for the person, in digitalisation, and in transformative IT.

The algorithm used to automatically identify putative instances of overpayment was seriously flawed, and in many cases that gave rise to a further automated decision that a debt existed, which it appears could in turn give rise to an automated referral to a debt collection agency. The Senate Committee rejected the proposition that such substantial automation could be acceptable: "The committee considers that it is important that calculation of debts is based on complete and accurate information, and that the fluctuations in recipient's income, particularly if they are employed on a casual or part-time basis, should be closely reviewed before issuing a debt notice" (SCA 2017, p.36).

In addition, the lack of human intervention, coupled with the apparent absence of managerial supervision, meant decision-making was not only delegated to a machine, but the delegation remained in place for an extended period. Clarke (2019, p.426-427) identifies 7 degrees of autonomy. Three are of the nature of decision support systems, with human decision-makers remaining in control. The other three involve automated decision-making, in which the system respectively notifies a human supervisor of an impending action (enabling override or veto), acts and informs the human it has acted (enabling interruption, suspension or cancellation of the action), or acts without any scope for human intervention. Great care is essential in granting such delegations.

Errors are also made by engaged humans trained to deal with delicate cases pertaining to human welfare; but humans have the capacity to notice anomalies, to apply common sense, to feel discomfort, and to draw their superiors' attention to problems. The absence of a human-in-the-loop was fatal to the project, and to some of the agency's clients.

(g) A Recapitulation

Basic standards need to be enunciated and then respected. In a public statement issued early in the project, in January 2017, entitled 'The Imminent Threat of Automated Government', the Australian Privacy Foundation laid out the reasonable public expectation that government agencies and business enterprises alike be under legal obligations to:

5.2 Process Aspects

(a) External Transparency, Consultation, Engagement

"The ATO ... noted that it had not been consulted on the design or implementation of the system" (Omb 2017, p.22). This was despite the Australian Taxation Office being the agency that performed the matching process and that provided data from its files on which the entire scheme depended. In addition, "DHS advised our office it did not consult with the [then Digital Transformation Office] DTO in the development and testing of the OCI" (Omb 2017, p.24, fn.54). DTO, shortly afterwards re-invented as the Digital Transformation Agency (DTA), was the agency with nominal oversight on any government project that automated processes that were previously conducted by human government staff. The later involvement of DTA during 2017 was relatively minor.

Nor did DHS consult with the Australian Council of Social Service (ACOSS), "despite ACOSS's clear articulation of the desire for engagement in the stakeholder process" (SCA 2017, p.22). Neither was any association engaged that represents pensioners generally. Nor was the Community and Public Sector Union (CPSU), consulted, despite it representing the DHS's staff (SCA 2017, p.23). Advocacy organisations that sought to convey concerns to the agency were not provided with any opportunity to do so, and were left with no other option than to pursue the matter through the media, and before such formal inquiries as were conducted.

From a public policy perspective, this is reprehensible behaviour, in clear breach of the norms applicable to impactful projects planned by government agencies. From an executive perspective, it is irresponsible behaviour that should be almost unknown in large government agencies, failing as it does first principles relating to stakeholder analysis, and risk identification and assessment.

Even the agency-friendly Ombudsman felt constrained to use firm expression

The project management team failed to ensure that key external stakeholders were effectively consulted during key planning stages. It also failed to effectively communicate with stakeholders after the full rollout of the OCI in September 2016, resulting in confusion and inaccuracy in public statements made by key non-government organisation (NGO) stakeholders, journalists and individuals (Omb 2017, p.23)

Further:

[A] key lesson for government agencies and policy makers when proposing to rollout large scale measures which require people to engage in a new way with new digital channels, is for agencies to engage with stakeholders ... . (Omb 2017, p.24)

DHS has had ample opportunity over many decades to familiarise itself with relevant advocacy organisations, establish a register, and run periodic conferences and as-required meetings on specific matters. Most such organisations make representations both directly to DHS and to frequent hearings of parliamentary committees. In the case of the Senate Committee in 2017, many representative and advocacy organisations gave evidence, providing a firm basis for such a register within DHS.

(b) Internal Engagement

Limited information is on the public record concerning the extent to which DHS took advantage of its own corporate knowledge in designing the scheme. However, the three sources of relevant information that have been located are sufficient to suggest that the executives responsible for the project may be guilty of professional incompetence in this area as well:

It remains remarkably unusual for Australian government agencies to recognise that many people are affected by powerful government agencies applying impactful information technologies, particularly when they embody automated inferencing, decision or action, and that the interests of non-participant usees may need to be considered. In the context of ICO, usees included financial dependents of welfare recipients, non-dependant family members (particularly the parents of despondent clients), and more generally the households in which DHS clients live. Many further categories are relevant, however, such as clients and ex-clients who did not receive the notification or lacked the means to respond and hence did not engage with the new sub-system, legal support centres and the counselling services that felt a surge of demand. It was necessary for a Senate Committee and a Federal Court judge to draw to attention that families are economically dependent on DHS clients, and that mothers become distraught when an adult child is driven by public service ineptitude to take their own life.

Remarkably, it also appears to have been beyond the relevant executives' understanding, or perhaps just outside their frame of reference, that the scheme's success depended to a considerable extent on the capacity of the intended users to actually use the poorly-conceived and poorly-designed facilities. It still appears to be beyond the understanding of those executives that agencies can benefit from the use of participative or co-analysis and -design of complex socio-technical systems. But, at the very least, it has to be expected that user interface and user experience design and testing is undertaken, and that it directly involves members of the targeted client segments. Within the organisation's boundaries meanwhile, it simply beggars belief that people employed as 21st century executives responsible for such systems would, during the analysis and design process of a compliance scheme, fail to take advantage of the expertise of the agency's own specialist compliance staff.

(c) Denial

It is commonplace to attribute greater harm to 'government cover-ups' than to the original actions of government that gave rise to the problem. The extent to which this maxim was ignored is remarkable. The earliest protestation found, to the effect that there was no problem, was less than 3 months after launch, in mid-December 2016: Ò[DHS is] confident in the online compliance system and associated checking process with customersÓ (Knaus 2016). In January 2017, it was reported that "[Minister] Tudge had rejected assertions people were being issued with unfounded debt notices or having difficulties in updating their details" (Anderson & Belot 2017), despite ample evidence of both. The Government's September 2017 response to the Senate Committee was dismissive: "the Government rejects the central conclusions and recommendations of the Chair's Report, especially the conclusion that the online system lacked procedural fairness" (GR 2017, p.5).

As late as February 2019, a spokesperson for the then Minister, Keenan, declared that the automatic debt notice process is "reasonable, lawful and fair" (Medhora 2019). In all, it took more than 2 years before the tone of Ministerial statements changed - in the period between May 2019, when a court-case brought by Deanna Amato was heard, and November 2019, when the judgment was handed down. This concluded that it was Ònot open on the material before the decision-makerÓ to decide that the client had received social security benefits she was not entitled to, because there was Òno probative materialÓ that the average reflected Amato's actual income (Karp 2019).

The reluctance to accept the baselessness of the scheme continued far longer, however. "Government departments must at all times act with 'best practice', and in legal issues must also act as a 'model litigant'" (SCA 2017, p.108. See also Appendix B of LSD 2017). In breach of that legal instruction, DHS was anything but a model, using multiple methods to delay proceedings, and to increase litigants' costs, capitulating only at the last possible moment in a conventional but immoral negotiating tactic to force the litigant to accept a substantially lower settlement amount. No action was taken to sanction DHS for its breach of the Legal Service Direction.


6. Broader Issues in Public Sector Digitalisation

The previous sections have provided a description of the OCI project, and identified the problems that gave rise to a great deal of harm to users and usees, and, despite a resolute project sponsor, project failure. This section considers what can be learnt from the debacle that is of relevance to digitalisation projects in the public sector generally. Considerable care is needed, because of the many dimensions across which contexts and projects vary, including cultural factors at organisational and societal levels, the nature of the government agency and its IT applications, and the purposes of systems sponsors. On the other hand, some commonalities exist as well, such as the technologies, the promotional techniques of technology providers, uncritical enthusiasm for technological innovation, and the categories of needy clients.

(a) Recognition and Management of the Sources of Error

Different approaches to error management are appropriate depending on the means used to generate inferences from data. The OCI used what is usefully referred to as a 'genuinely algorithmic' inferencing technique - the simplistic algorithmic being, in many cases, division by 26 of what DHS presumed to be an annual figure, to generate a putative fortnightly average. Rule-based expert systems applies sets of rules to the facts of cases. The Machine Learning branch of Artificial Intelligence (AI/ML) inferencing techniques, in particular artificial neural networks, use 'training-sets' to produce a set of weightings which are then applied to data representing a particular instance.

Attempts to produce a general set of guidelines for responsible inferencing that applies across all three of those categories are confronted by considerable challenges. One categorisation of ways in which "automated assisted decision-making may well be productive of deficient reasoning" is provided by Hogan-Doran (2017, pp.14-15), where the inferencing process:

For 'genuinely algorithmic' techniques, analyses of sources of error exist, as do guidelines on how to avoid those errors (e.g. Clarke 2017). Such tools, in applying procedures to data in order to draw inferences, express a solution and at least imply a problem-definition. These approaches to inferencing therefore support the important feature of explainability. Rule-based expert systems, on the other hand, merely define a problem-domain. Inferences are drawn for individual circumstances, but there is no explicated problem or question. Whereas algorithmic schemes inherently embody a rationale, and enable expression of a humanly-understandable explanation, rule-based systems require investment to deliver explanations and thereby enable decision review.

The AI/ML field, contrary to popular expression, is not algorithmic in nature. It is purely empirical, creating weightings on factors on the basis of whatever prior instances are fed into it. ML inferencing is a-rational, and does not provide a basis for explanations of how inferences were drawn. Variants of 'The established weightings were applied to the data, resulting in the output' are not usefully different from 'The computer says "no''' (Clarke 2019, pp.425-426). The challenges involved in recognising error in AI/ML-based inferencing are accordingly far greater.

In discussions of errors in AI/ML applications, the misconceived term 'algorithmic bias' is much-used. Terms such as 'model bias', 'data bias' and 'socio-cultural bias' are also used as loose classifiers; but no comprehensive categorisation of problems appears to have been settled upon. See, however, AHRC (2020). Banks & London (2017), for whom algorithmic bias exists when a data process is "not merely a neutral transformer of data or extractor of information" (p.1), distinguish the following elements:

Despite the remaining fog, comprehensive guidance on responsible application of AI/ML is available from a variety of sources, notably Zeng et al. (2019), Clarke (2019b) and Jobin et al. (2019).

No evidence was found that DHS applied the necessary professionalism to ensure that sources of error were examined and understood, and design features incorporated to deal with them. It is essential that organisations undertaking projects with substantial impacts on and implications for users and usees take advantage of the ample guidance available on error management.

(b) Due Process / Procedural Fairness

Depending on the jurisdiction, there may be legal or merely moral constraints on the conduct of processes by government agencies that may have detrimental effects on people. Terms such as 'due process' and 'procedural fairness' are applied to constraints of this nature (Citron 2008). Key aspects of procedural fairness are symmetry between the parties in relation to information, power and resources; participation in the process; submission of evidence and argument by the parties; opportunity to respond to evidence and argument by the parties; decision-making based on evidence; and impartiality of the decision-maker. The OCI process failed many of these tests. For example:

I sat with my daughter--and I have worked for the public sector for many years--and attempted to go through the questions which were in this form with her. Some of the questions were nonsensical. I had no idea what was being asked. You cannot progress unless you answer the question, so people are making a guess. They are putting in whatever information in order to get the form completed. They do not understand. (SCA 2017, p.41)

The Senate Committee concluded that OCI embodied:

A fundamental lack of procedural fairness ... in every stage [from] the drafting of the policy [through] the testing phase ... in the failure to carry out a risk assessment before the process started. In sending letters without checking addresses and taking a lack of response as a refusal to engage. In the averaging of income data, which invents a fortnightly income-earned sum for the purposes of then charging people with a debt knowing full well it is going to be wrong. In the millions of calls that went unanswered ... In the lack of information released to individuals which they required in order to challenge a debt. ... This lack of procedural fairness disempowered people, causing emotional trauma, stress and shame (SCA 2017, p.107)

In the hands of a skilled (and hence expensive, and hence perhaps inaccessible) lawyer, this alone may have been sufficient to sink the OCI program beneath the thin ice on which it had been constructed and over which it skated for more than 3 years.

Even if system sponsors are not subject to legal or public policy constraints in relation to procedural fairness, or are willing and able to ignore them, they are well-advised to consider the possibility that apparently weak stakeholders may in fact wield considerable power, or have it wielded by others on their behalf, in such a way as to deny the benefits that the agency seeks, or destroy the project completely. In this case study, the volume of individuals who were obviously being subjected to unreasonable behaviour reached critical mass, such that media reports and the efforts of community legal centres eventually defeated one of the country's largest organisations.

(c) Explanation of the Decision Rationale

One particularly critical element of procedural fairness is the provision of clear information about the reasoning and the data underlying decisions. The Senate Committee asserted that "it is important recipients ... are given adequate information as to how their purported debt has been calculated" (SCA 2017, p.39). One welfare recipient gave evidence that:

That figure of my debt of $3,154.11 was remarkably precise, but if there was any kind of detailed computation behind it I have never seen it. I have even pulled an FOI on my case and I cannot make head or tail of how that figure was arrived at. They came up with this figure, but they provided no accounting for it and they provided no explanation, initially, as to how it arose. They just said, 'Here's your debt; pay it or prove you don't owe it'. (SCA 2017, p.71)

This had broader implications, with community legal services which were "unable to understand how their client's purported debt has been raised and have resorted to Freedom of Information (FOI) requests in an attempt to gather information relating to their client's purported debt" (SCA 2017, p.39).

It is not common for legal rights to exist requiring explanations for decisions, in either the private or the public sector. As datafication takes hold, calls are being made for such rights to be created, in order to enable the identification of administrative blunders, and to provide people subjected to them with some ability to fight back against poor systems design and poor decisions (Wachter & Mittelstadt 2019).

As discussed earlier in this section, explainability is a feature of 'genuinely algorithmic' inferencing techniques. The approach used in this project was explainable - and had the explanation been publicly exposed, it would have been entirely clear that the approach was seriously inadequate. It is more challenging to deliver explanations when using rule-based systems, and it is essentially infeasible to do so where AI/ML techniques such as artificial neural networks are applied. Their inherent a-rationality leads to opaqueness instead of transparency. Use of AI/ML inferencing therefore undermines the essential qualities of procedural fairness in administrative schemes, of replicability, of auditability, of correctability, and hence of accountability (Clarke 2019a, Michael et al. 2021).

(d) Responsibility in the Exercise of Power

Once again, the Senate Committee's Report was crystal-clear on the matter:

As the provider of social security in Australia, the department holds a position of power in its recipients' minds, and the power imbalance this creates cannot be underestimated when considering individuals' reactions to the OCI system. (SCA 2017, p.36) There is a significant power imbalance between income payment recipients and the department, and communication therefore does not take place on a level playing field ... In discussing the impacts of this power imbalance, where individuals tend to assume 'the department is right', organisations pointed to a number of adverse outcomes for individuals, such as people paying purported debt notices without question and people accepting pre-filled income data that averages their income without checking it for accuracy, leading to incorrect debt calculations. (SCA 2017, p.41) [The Committee has] concerns that the OCI system has created a climate of fear where recipients of letters feel they cannot challenge the information provided by the department or risk losing the financial safety net which the department provides. (SCA 2017, p.34)

It appears unlikely that many jurisdictions make illegal the irresponsible exercise of government power. On the other hand, such behaviour may be inconsistent with public policy, and it may be harmful to the standing of government agencies with the public, and of the Government with the electorate. Many government agencies in many countries may be impervious to the concerns of stakeholders, or of public opinion generally. On the other hand, some agencies may well find that the morality of the behaviour of members of the public in their dealings with the agency declines, perhaps precipitously, and perhaps uncontrollably, if the agency is seen to be arrogant and entirely unconcerned with its stakeholders' interests.

At the level of politics, authoritarian governments have little reason to fear electoral displeasure, and some democratic governments may attract or sustain electoral support if they are seen to be acting against the interests of some segments of the population, such as organised crime, but also demonised clients, such as 'dole-bludgers', 'welfare cheats' and 'deadbeat dads'. One interpretation of the then Australian Government's refusal to change course when OCI's many deficiencies were exposed is that it calculated that it would gain at the ballot-box at the forthcoming election because of the appeal to voters of recovering overpayments to 'dole-bludgers' and 'welfare cheats'. In fact, by the time of the election, far more electors than just the 'soft left' appear to have been seriously unimpressed with the debacle that OCI had become, and in any case the remnant positive impact on some electors had been swamped by other perceived weaknesses in the then Government's performance. The general consensus among pundits was that the Government lost the election rather than the opposition winning it. The outcomes of elections generally depend on many factors, but it may be that highly publicised project disasters, headlined by the waste of more than $A2 billion, may be damaging to any governments' prospects of re-election. It is also possible that irresponsibility in dealing with vulnerable population segments may exacerbate negative impacts on the credibility of government agencies and of governments.

The OCI system's targets were people who depend on social security payments for housing and basic necessities like food and paying energy bills and who by definition lacked assets, lacked steady incomes, and worked casually, reliant on sporadic work often at short notice, sourced in many cases from multiple employers, resulting in unpredictability in earnings. Even the agency-sympathetic Ombudsman's Report drew attention to "the risk of over-recovering debts from social security recipients and the potential impact this may have on this relatively vulnerable group of people" (Omb 2017, p.8). The Senate Committee was "particularly concerned about the impact that debt repayments are having on income support recipients who are on very low incomes and former recipients who may be on very low wages" (SCA 2017, p.97). In countries where large proportions of the population are dependent on social welfare, the scope exists for populist parties to attract a lot of votes by highlighting the Government's irresponsible behaviour.

(e) Risk Assessment and Management

Risk assessment and risk management (RA/RM) are well-established processes (e.g. NIST 2012, ENISA 2016, IEC 31010:2019). Their purpose is to ensure that difficulties that might beset an organisation are anticipated, and variously deterred, prevented, detected and mitigated. One of the key responsibilities of governing boards and senior executives is to ensure the effective use of these techniques. The senior executives of DHS failed miserably in that regard.

For public sector organisations, however, RA/RM may not be enough, because RA/RM is concerned solely with the interests of the organisation itself. The interests of other stakeholders are relevant in RA/RM only to the extent that those other stakeholders have sufficient power that they represent a threat to fulfilment of the organisation's purposes and hence need to be managed as a risk (Mitchell et al. 1997, Achterkamp & Vos 2008). Particularly for government agencies that recognise as an objective the social and/or economic welfare of disadvantaged people, RA/RM may therefore be a too single-minded technique.

Various approaches can be adopted to reflecting the interests of users and usees, including technology assessment, and environmental, privacy and social impact assessment. However, these are commonly resisted by organisations. An alternative approach that offers promise is expansion of the scope of RA/RM, such that the organisation's understanding is complemented by insights from the perspectives of all stakeholders with material interests in a project's outcomes, not only if they have power, but also if they lack power but their interests nonetheless have legitimacy and urgency. An outline of such an alternative approach, Multi-Stakeholder Risk Assessment (MSRA) is in Clarke (2022).

(f) The Need for Socio-Technical Design

DHS operates a still somewhat `high-touch' operation, with Interactive Voice Response (IVR) and call-centres complemented by a moderate intensity of footprints and counter-services. Under ongoing pressure to reduce staff and costs, DHS grasped at the promise of automation based on assumptions about datafication and digitalisation. In doing so, it left behind such of the basic tenets of human-centred design as it had previously respected (Norman 2013). The agency reverted to a merely technical vision of efficient-but-intolerant IT-delivered inferencing, decision, and action. As a consequence, it lost sight of the need to recognise stakeholders and appreciate their interests.

Socio-technical design builds on open systems thinking (von Bertalanffy 1950), and acknowledges that systems comprise technical working in combination with social or human elements (Emery 1959, Abbas & Michael 2022). It offers a principled approach to the implementation of socio-technical interventions by seeking balance between humanistic values and technological capabilities (Cherns 1976, Cherns 1987, Trist 1981, Mumford 2000, Mumford 2006). Developers of public sector information systems need to adopt design methods that reflect socio-technical insights (Abbas, Michael et al. 2021; Abbas, Pitt et al. 2021).

(g) Transparency, Engagement, Participation, Co-Design

A parallel, and somewhat complementary, path has been traced to that of socio-technical design. Participative analysis and design techniques (Hirschheim 1985), rediscovered and renamed 'co-design' in recent years, seek better results for system sponsors and other stakeholders alike, through direct involvement of affected parties sin the process of developing new systems and adapting existing systems (Sanders & Stappers 2008, 2014).

Even if this is too 1970s avant garde for conservative government agencies to contemplate, an organisation can use a slimmed-down approach to gain insights into the perspectives of users and usees. This involves providing sufficient information to advocacy organisations to enable them to engage with the agency on the aspects of greatest concern to the individuals on whose behalf the advocates speak. The fact that it appears necessary to convey this point suggests that the benefits of transparency and engagement have to be communicated anew to each generation of government executives.

In the case of DHS and OCI, insights were needed in relation to the interests and the capabilities of the relevant categories of social welfare recipients. To some extent this requires direct access to samples of individuals in each segment, in focus groups, and in laboratory- and beta-testing of UI/UX designs. Generally, the broader questions will be better addressed through engagement with advocacy organisations. Even if an agency currently lacks a reference group of such organisations, a suitable pool is very simply developed. A review of the functions of the 62 such organisations that submitted to the Senate Committee's first hearings (SCA 2017) delivers this categorisation:

The failure of DHS executives in this regard appears likely to have played a major part in both its original blunders in conception and design, and its 2-3 years of denial of what was by then blindingly obvious to the rest of the country. Even minimalist attention to stakeholders' views would have saved a great deal of trauma for individuals, and billions of wasted taxpayer funds.

(h) Organisational and Individual Accountability

"DHS' project planning did not ensure all relevant external stakeholders were consulted ... " (Omb 2017, p.3). Further:

DHS' planning ... fell short [in relation to] testing, ... rollout ... [and] communication to staff and stakeholders ... The risk management plan recognised the need for a strong stakeholder strategy to mitigate the risk that insufficient communication with appropriate key stakeholders may result in failure to realise expected program savings. (Omb 2017, p.23, fn.48)

On the other hand, there is no evidence any such mitigation measures having been designed-in or even retro-fitted. Hence any Risk Assessment element of OCI that may have been performed appears to have been no more than a token effort or box-ticking exercise. The Senate Committee concluded that the "procedural fairness flaws" alone were so serious that its Recommendation 1 was that the OCI scheme be suspended (SCA 2017, p.108). DHS and the Government ignored that call (along with all of the other Recommendations) and continued the scheme for a further 18 months, prolonging the agony for hundreds of thousands of benefits recipients and thousands of staff-members.

The Tasmanian Council of Social Service asked:

Where is good government, good decision-making and leadership when a system is failing? Where is the leadership that is bold enough to say: 'We got this wrong. We will pull it back. We will rework it. We will review it. We will talk to the stakeholders who know best to try and get it right.' Where is good government in understanding and taking seriously its duty of care to its citizens to protect the most vulnerable and not cause vulnerability or harm its own citizens? (p.107)

The Federal Court judgement, 4 years after the event, was expressed very directly:

The proceeding has exposed a shameful chapter in the administration of the Commonwealth social security system and a massive failure of public administration. It should have been obvious to the senior public servants and to the responsible Minister(s) at different points who designed and were charged with overseeing the Robodebt system that many social security recipients do not earn a stable or constant income, and any employment they obtain may be casual, part-time, sessional, or intermittent and may not continue throughout the year. (Pry 2021, p.2)

Moreover:

For those perpetual critics of the ... class action regime, the present case is one more example where the regime has provided real, practical access to justice. It has enabled approximately 394,000 people, many of whom are marginalised or vulnerable, to recover compensation from the Commonwealth in relation to conduct which it belatedly admitted was unlawful (p.8)

At the political level, a social welfare scandal in The Netherlands involved only 26,000 individuals, 6 percent of the number in the OCI debacle. Yet, when the Report was tabled in January 2021, the principle of governmental accountability was respected, and the Prime Minister and the entire Cabinet of The Netherlands resigned (Erdbrink 2021).

In Australia, on the other hand, the notion of Ministerial responsibility has all but disappeared. Evidence presented to the Royal Commission in late 2022 suggests that the proposal originated in the policy agency, the Department of Social Security, or perhaps with its Minister at the time. Internal legal advice was ignored, and no external legal advice was acquired, enabling funding to be approved in the 2015 Budget for implementation by the operational agency, DHS (Bannister 2022, Morton 2022). The Minister for Social Security at that time was Scott Morrison. Despite having overridden legal advice and instigated a scheme that caused substantial harm to welfare-recipients, and reputational and financial disaster for the agency and the Government, Morrison's career was clearly not hampered. He moved to the more senior position of Treasurer at the end of 2015, and was subsequently Prime Minister 2018-22.

Meanwhile, during the period 2015-22, the position of Minister for Human Services was recognised as carrying with it considerable negative publicity, and 5 Government front-benchers were shuttled into and out of the position a total of 6 times during the 7 years, for an average term of 14 months each. The Ministers were Marisa Payne (during the project's conception and approval), Stuart Robert (nominally with IT industry background, but with a track-record of parliamentary misbehaviour, for 4 months in the preparation period, and again for 1 year 10 months during the period of denial and collapse), Alan Tudge (also with a record of misbehaviour, for 1 year 10 months, early operational period), Michael Keenan (1 year 5 months, late operational period, who retired from parliament, handing the chalice back to Stuart Robert), and Linda Reynolds (for the last 14 months of the dying Coalition Government). Each had ample opportunity to investigate, appreciate the problematic nature of the project, and take action to stop it. None did. All were culpable; none suffered any consequences.

[ This ascerbic cartoon by master-cartoonist Kathy Wilcox appeared in the national press the same day as the review version of this case study was published, on 8 November 2022.
Click on the thumbnail to enlarge it:

Within DHS, the Deputy Secretary with the most direct responsibility, Malisa Golightly, already held a Public Service Medal, from 2010. It is unclear from the public record when she left the role, but she passed away in late December 2021. Another Deputy Secretary who gave evidence to the Senate Committee, Jonathan Hutson, appears to have retired in July 2018. Another witness, Jason McNamara, continues as a General Manager in similar roles to that which he held in 2017. The overall responsibility for the relevant activities, however, for the entire period 2011-21, was Kathryn Campbell. Her career, rather than merely surviving, flourished. She had also sustained a part-time military career, and was promoted in 2018 to command the entire Army Reserve. In January 2019, Campbell was awarded one of the nation's highest honours, the Order of Australia (AO), almost exactly 2 years after the disastrous nature of Robodebt had become apparent to everyone outside DHS - although still 2 years before the Government accepted it was a failure and closed it down. In mid-2021, she was promoted to one of the most highly-ranked roles in the public service, Foreign Affairs and Trade, rejoining her Minister during 2013-15, Marise Payne. She was Secretary there for 12 months, until shortly after the Labor Government won office, when she was moved to the unattached list.

What is sometimes termed 'algorithmic fallout' afflicts people who are subjected to automated processes imposed by ill-advised transformative digitalisation projects. Noble (2018) argues that 'algorithms of oppression' give rise to societal burden (undue feelings of anxiety and distress), distrust in the effectiveness and operationalization of AI-based systems (such as automated surveillance-based welfare systems), attacks on personal character (causing personal shame, hurt and anguish), and suicidal ideation and suicide. The contrasting impacts on careers in The Netherlands and Australian cases raises the question as to whether other countries' Ministers of State, and government agencies and their senior executives, will suffer reputational harm from their association with mis-conceived, harmful and failed projects, or enjoy a protected existence like those in Australia, where incompetence is being tolerated and even celebrated.


7. Conclusions

One of the largest government agencies, in a large first-world country, with long and deep experience in the application of information technology to social welfare administration, was so beguiled by the promise of transformative digitalisation that it abandoned its responsibilities, and reverted to mindless, technologically-determined design, causing massive dislocation and harm to the most disadvantaged members of society. The agency then refused for 2-3 years to accept that the project was disastrous in conception, design, execution and effect, prolonging the agony. The deficiencies were so numerous that this case study provides a catalogue of reminders to public sector agencies everywhere of the reasons why socio-technical rather than merely technical design, risk assessment and risk management, transparency and engagement, and professionalism, are so essential to agency activities.

More generally, the increasing power of technology has not been matched by advances in quality of data and quality of inferencing. Instead, both quality and rationality have been seriously reduced, even where humanly-understandable, genuinely algorithmic data analytics are used. The much-touted, but opaque and a-rational magic of empirical techniques such as AI/ML threaten to further lower data and inferencing quality, and to undermine replicability, auditability, correctability, and hence accountability. Where government projects are permitted to be conducted in the manner in which an Australian agency irresponsibly ran the Online Compliance Intervention project, without direct accountability by the relevant Ministers, agencies and senior executives, public confidence in both agencies and technology are likely to be seriously harmed, and public scepticism, sullen non-compliance and active opposition to initiatives to become engrained.


Reference List

Abbas, R. & Michael, K. (2022) 'Socio-Technical Theory: A Review' In S. Papagiannidis (Ed) 'TheoryHub Book', at: https://open.ncl.ac.uk/theories/9/socio-technical-theory/

Abbas R., Michael K., Sargent J.P. & Scornavacca E. (2021) 'Anticipating Techno-Economic Fallout: Purpose-Driven Socio-Technical Innovation' IEEE Transactions on Technology and Society 2,3 (September 2021) 111-113, at https://ieeexplore.ieee.org/iel7/8566059/9500078/09500093.pdf

Abbas R., Michael K., Michael M.G., Perakslis C. and Pitt J. (2022) 'Machine Learning, Convergence Digitalization, and the Concentration of Power: Enslavement by Design Using Techno-Biological Behavior' IEEE Transactions on Technology and Society 3, 2 (June 2022) 76-88, doi: 10.1109/TTS.2022.3179756

Abbas R., Pitt J., & Michael K. (2021) 'Socio-technical design for public interest technology' IEEE Transactions on Technology and Society 2,2 (2021) 55-61

Achterkamp M.C. & Vos J.F.J. (2008) 'Investigating the Use of the Stakeholder Notion in Project Management Literature: A Meta-Analysis' International Journal of Project Management 26 (2008) 749-757

AHRC (2020) 'Using artificial intelligence to make decisions: Addressing the problem of algorithmic bias Technical Paper' Human Rights Commission, Australia, 2020, at https://tech.humanrights.gov.au/sites/default/files/2021-05/AHRC_RightsTech_2020algorithmic_bias.pdf

Anderson S. & Belot H. (2017) 'Centrelink's debt recovery system working, Human Services Minister Alan Tudge says' ABC News, 11 Jan 2017, at https://www.abc.net.au/news/2017-01-11/centrelinks-debt-recovery-system-to-remai-government-says/8174644

APF (2017) 'The Imminent Threat of Automated Government' Australian Privacy Foundation, 18 January 2017, at https://privacy.org.au/2017/01/18/the-imminent-threat-of-automated-government/

Banks D. & London A.J. (2017) 'Algorithmic Bias in Autonomous Systems' Proc. 26th Int'l Joint Conf. on Artificial Intelligence (IJCAI 2017), at https://www.cmu.edu/dietrich/philosophy/docs/london/IJCAI17-AlgorithmicBias-Distib.pdf

Bannister M. (2022) 'Commissioner appalled by robodebt evidence' AAP/The Canberra Times, 2 November 2022, at https://www.canberratimes.com.au/story/7965606/commissioner-appalled-by-robodebtevidence/

Barbaschow A. (2017) 'Privacy Foundation calls for human intervention in Centrelink debt collection' zdNet, 2 April 2017, at https://www.zdnet.com/article/privacy-foundation-calls-for-human-involvement-in-entrelink-automated-debt-collection-process/

Baumer E.P.S. (2015) 'Usees' Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI'15), April 2015

Belot H. (2017a) 'Centrelink debt recovery: Government knew of potential problems with automated program' ABC News, 12 Jan 2017, at https://www.abc.net.au/news/2017-01-12/government-knew-of-potential-problems-wit-centrelink-system/8177988

Belot H. (2017b) 'Centrelink's controversial data matching program to target pensioners and disabled, Labor calls for suspension' ABC News, 17 Jan 2017, at https://www.abc.net.au/news/2017-01-17/labor-calls-for-suspension-of-centrelink-ebt-recovery-program/8187934

Belot H. & McGhee A. (2017) 'Centrelink debt recovery program to undergo changes following public criticism' ABC News, 16 Jan 2017, at https://www.abc.net.au/news/2017-01-16/government-to-change-centrelink-debt-recoery-program/8185366

Benbasat I., Goldstein D. & Mead M. (1987) 'The Case Study Research Strategy in Studies of Information Systems' MIS Qtly 11,3 (September 1987) 369-386

Berleur J. & Drumm J. (Eds.) (1991) 'Information Technology Assessment' Proc. 4th IFIP-TC9 International Conference on Human Choice and Computers, Dublin, July 8-12, 1990, Elsevier Science Publishers (North-Holland), 1991

Braithwaite V. (2020) 'Beyond the bubble that is Robodebt: How governments that lose integrity threaten democracy' Aust J Soc Issues 55,3 (September 2020) 242-259, at https://static1.squarespace.com/static/5c05f8595cfd7901fc57139d/t/5f8a808b459ade60acd28dc/1602912397885/Beyond+the+bubble+that+is+Robodebt+How+governments+that+lose+integrity+threaten+democracy.pdf

Brennen S. & Kreiss D. (2016) 'Digitalization and Digitization' International Encyclopedia of Communication Theory and Philosophy, October 2016, PrePrint at http://culturedigitally.org/2014/09/digitalization-and-digitization/

Brookes J. (2022) '197,000 notices wiped from `shameful' robodebt scheme' Innovation Australia, 12 October 2022, at https://www.innovationaus.com/197000-notices-wiped-from-shameful-robodebt-scheme

Burgess K. (2019) 'Robodebt Centrelink compliance program cost $600m to recoup $785m so far' The Canberra Times, October 28 2019, at https://www.canberratimes.com.au/story/6407984/four-of-five-robodebt-reviews-hitwith-a-debt-as-cost-revealed/

Carney T. (2019) 'Robo-debt illegality: The seven veils of failed guarantees of the rule of law?' Alternative Law Journal 44,1 (2019) 4-10, at https://journals.sagepub.com/doi/pdf/10.1177/1037969X18815913

Cherns A. (1976) 'The principles of sociotechnical design' Human Relations 29,8 (1976) 783-792

Cherns A. (1987) 'Principles of sociotechnical design revisited. Human Relations, 40,3 (1987) 153-161

Citron D.K. (2008) 'Technological Due Process' Washington University Law Review 85 (2008 1249, at https://core.ac.uk/download/pdf/229123628.pdf

Clarke R. (1987) 'Just Another Piece of Plastic for your Wallet: The 'Australia Card' Scheme' Computers & Society 18,1 (January 1988), Addendum in Computers & Society 18,3 (July 1988), PrePrint at http://www.rogerclarke.com/DV/OzCard.html

Clarke R. (1992a) 'The Resistible Rise of the National Personal Data System' Software Law Journal 5,1 (January 1992), PrePrint at http://www.rogerclarke.com/DV/SLJ.html

Clarke R. (1992b) 'Extra-Organisational Systems: A Challenge to the Software Engineering Paradigm' Proc. IFIP World Congress, Madrid, September 1992, PrePrint at http://www.rogerclarke.com/SOS/PaperExtraOrgSys.html

Clarke R. (1994a) 'The Digital Persona and its Application to Data Surveillance' The Information Society 10,2 (June 1994), PrePrint at http://www.rogerclarke.com/DV/DigPersona.html

Clarke R. (1994b) 'Dataveillance by Governments: The Technique of Computer Matching' Information Technology & People 7,2 (December 1994) 46-85, PrePrint at http://www.rogerclarke.com/DV/MatchIntro.html

Clarke R. (2014) 'Promise Unfulfilled: The Digital Persona Concept, Two Decades Later' Information Technology & People 27, 2 (Jun 2014) 182 - 207, PrePrint at http://www.rogerclarke.com/ID/DP12.html

Clarke R. (2017) 'Guidelines for the Responsible Application of Data Analytics' Computer Law & Security Review 34, 3 (May-Jun 2018) 467- 476, PrePrint at http://www.rogerclarke.com/EC/GDA.html

Clarke R. (2019a) 'Why the World Wants Controls over Artificial Intelligence' Computer Law & Security Review 35, 4 (2019) 423-433, PrePrint at http://www.rogerclarke.com/EC/AII.html

Clarke R. (2019b) 'Principles and Business Processes for Responsible AI' Computer Law & Security Review 35, 4 (2019) 410-422, PrePrint at http://www.rogerclarke.com/EC/AIP.html

Clarke R. (2022) 'Evaluating the Impact of Digital Interventions into Social Systems: How to Balance Stakeholder Interests' Proc. Int'l Digital Security Forum (IDSF22), Vienna, 1 June 2022, PrePrint at http://rogerclarke.com/DV/MSRA-VIE.html

Clarke R. & Davison R.M. (2020) 'Through Whose Eyes? The Critical Concept of Researcher Perspective' J. Assoc. Infor. Syst. 21, 2 (March-April 2020) 483-501, PrePrint at http://rogerclarke.com/SOS/RP.html

Cukier K. & Mayer-Schönberger V. (2013) 'The Rise of Big Data' Foreign Affairs (May/June 2014) 35

Emery F. (1959) 'Characteristics of socio-technical systems' In 'The Social Engagement of Social Science: A Tavistock Anthology' Volume 2. pp. 157-186), University of Pennsylvania Press

ENISA (2016) 'Risk Management:Implementation principles and Inventories for Risk Management/Risk Assessment methods and tools' European Union Agency for Network and Information Security, June 2016, at https://www.enisa.europa.eu/publications/risk-management-principles-and-inventores-for-risk-management-risk-assessment-methods-and-tools

Erdbrink T. (2021) 'Government in Netherlands Resigns After Benefit Scandal', The New York Times, 15 Jan 2021, at https://www.nytimes.com/2021/01/15/world/europe/dutch-government-resignation-rute-netherlands.html

Farrell P. (2019) 'Government halting key part of robodebt scheme, will freeze debts for some welfare recipients' ABC News, 19 November 2019, at https://www.abc.net.au/news/2019-11-19/robodebt-scheme-human-services-departmenthalts-existing-debts/11717188

Fischer-Huebner S. & Lindskog H. (2001) 'Teaching Privacy-Enhancing Technologies' Proceedings of the IFIP WG 11.8 2nd World Conference on Information Security Education, Perth, Australia

GR (2017) 'Government Response to the Senate Committee Report', Australian Government, September 2017, at https://www.aph.gov.au/Parliamentary_Business/Committees/Senate/Community_Affair/SocialWelfareSystem/Additional_Documents?docType=Government%20Response

Hammer M. (1990) 'Reengineering Work: Don't Automate, Obliterate' Harvard Business Review (July-August 1990) 104-112

Henriques-Gomes L. (2020) 'All Centrelink debts raised using income averaging unlawful, Christian Porter concedes' The Guardian, 31 May 2020, at https://www.theguardian.com/australia-news/2020/may/31/all-centrelink-debts-raisd-using-income-averaging-unlawful-christian-porter-concedes

Hirschheim R.A. (1985) 'User Experience with and Assessment of Participative Systems Design' MIS Quarterly (December 1985), at https://www.academia.edu/download/51009081/viewcontent.pdf

Hogan-Doran D. (2017) 'Computer says "no": automation, algorithms and artificial intelligence in Government decision-making' The Judicial Review 13 (2017) 1-39, at https://www.academia.edu/38540521/Computer_says_no_automation_algorithms_and_artficial_intelligence_in_Government_decision_making?email_work_card=view-paper

IEC 31010:2019 'Risk management - Risk assessment techniques' International Standards Organisation, 2019

Jobin A., Ienca M. & Vayena E. (2019) 'The global landscape of AI ethics guidelines' Nature Machine Intelligence 1 (September 2019) 389-399, at https://doi.org/10.1038/s42256-019-0088-2

Karp P. (2019) 'Government admits robodebt was unlawful as it settles legal challenge' The Guardian, 27 November 2019, at https://www.theguardian.com/australia-news/2019/nov/27/government-admits-robodeb-was-unlawful-as-it-settles-legal-challenge

Knaus C. (2016) 'Centrelink officer says only a fraction of debts in welfare crackdown are genuine' The Guardian, 23 December 2016, at https://www.theguardian.com/australia-news/2016/dec/23/centrelink-officer-says-oly-a-fraction-of-debts-in-welfare-crackdown-are-genuine

Knaus C. (2017) ''Reasonably clear' Alan Tudge's office broke law in Centrelink case, says Labor legal advice' The Guardian, 3 Apr 2017, at https://www.theguardian.com/australia-news/2017/apr/03/reasonably-clear-alan-tudes-office-broke-law-in-centrelink-case-says-labor-legal-advice

Lee A.S. (1989) 'A Scientific Methodology for MIS Case Studies' MIS Qtly 13,1 (March 1989) 33-50

LSD (2017) 'Legal Services Directions 2017', 2 July 2018, at https://www.legislation.gov.au/Details/F2018C00409

Lycett, M. (2014), Datafication: Making Sense of (Big) Data in a Complex World, European Journal of Information Systems, 22, 4 (December 2014) 381-386, at http://v-scheiner.brunel.ac.uk/bitstream/2438/8110/2/Fulltext.pdf

Lyytinen K. & Rose G.M. (2003) 'The Disruptive Nature of Information Technology Innovations: The Case of Internet Computing in Systems Development Organizations' MIS Quarterly 27, 4 (December 2003) 557-595

Medhora S. (2019) 'Over 2000 people died after receiving Centrelink robo-debt notice, figures reveal' ABC TripleJ, 18 Feb 2019, at https://www.abc.net.au/triplej/programs/hack/2030-people-have-died-after-receivig-centrelink-robodebt-notice/10821272

Michael, K. (2003) 'The technological trajectory of the automatic identification industry: the application of the systems of innovation (SI) framework for the characterization and prediction of the auto-ID industry' PhD thesis, School of Information Technology and Computer Science, University of Wollongong, 2003, at http://ro.uow.edu/theses/309. See in particular 'Section 4.3 The Evolution of the Citizen ID Number'

Michael K., Abbas R. & Pitt J. (2021) 'Maintaining Control over AI' Issues in Science and Technology: Forum XXXVII, 3 (Spring 2021)

Michael K. & Michael M.G. (2006) 'Historical lessons on ID technology and the consequences of an unchecked trajectory' Prometheus 24,4 (2006) 365-377

Mitchell R.K., Agle B.R. & Wood D.J. (1997) 'Toward a Theory of Stakeholder Identification and Salience: Defining the Principle of Who and What Really Counts' Academy of Management Review 22, 4, 853-886

Morton R. (2022) 'Robo-debt: Liberals knew it was illegal before it started' The Saturday Paper, 5 November 2022, at https://www.thesaturdaypaper.com.au/news/politics/2022/11/05/robo-debt-liberals-new-it-was-illegal-before-it-started#mtr

MRR (20125) 'Welfare Integrity, Fairness and Sustainability for all Australians' Minister for Social Security, Scott Morrison, Budget Media Release, 12 May 2015, at https://formerministers.dss.gov.au/15850/welfare-integrity-fairness-and-sustainaility-for-all-australians/

Mumford E. (2000) 'A socio-technical approach to systems design' Requirements Engineering 5,2 (2000) 125-133

Mumford E. (2006) 'The story of socio-technical design: reflections on its successes, failures and potential' Info Systems J 16 (2006) 317-342, at https://executiveinsight.typepad.com/files/the-story-of-socio-technical-design.pf

Newell S. & Marabelli M. (2015) 'Strategic Opportunities (and Challenges) of Algorithmic Decision-Making: A Call for Action on the Long-Term Societal Effects of 'Datification'' The Journal of Strategic Information Systems 24, 1 (2015) 3-14, at http://marcomarabelli.com/Newell-Marabelli-JSIS-2015.pdf

NIST (2012) 'Guide for Conducting Risk Assessments' National Institute of Standards and Technology, Special Publication SP 800-30 Rev. 1, September 2012, at http://csrc.nist.gov/publications/nistpubs/800-30-rev1/sp800_30_r1.pdf

Noble S.U. (2018) 'Algorithms of Oppression: How Search Engines Reinforce Racism' New York University Press, 2018

OAIC (2014) 'Guidelines on data matching in Australian Government administration' Office of the Australian Information Commissioner, rev. 18 June 2014, at https://www.oaic.gov.au/privacy/guidance-and-advice/guidelines-on-data-matching-n-australian-government-administration

Omb (2017) 'Centrelink's automated debt raising and recovery system: A Report about the Department of Human Services' Online Compliance Intervention System for Debt Raising and Recovery' Australian Ombudsman, April 2017, at https://www.ombudsman.gov.au/__data/assets/pdf_file/0022/43528/Report-Centrelink-automated-debt-raising-and-recovery-system-April-2017.pdf

Pry (2021) Prygodicz v Commonwealth of Australia (No 2) [2021] FCA 634 (11 June 2021) - the eventual judgment (see Prelims below), at http://www.austlii.edu.au/cgi-bin/viewdoc/au/cases/cth/FCA/2021/634.html?context1;query=Prygodicz;mask_path=

Redden J. (2018) 'The Harm That Data Do' Scientific American, 1 November 2018, https://www.scientificamerican.com/article/the-harm-that-data-do/

Rinta-Kahila T., Someh I., Gillespie N., Indulska M. & Gregor S. (2022) 'Algorithmic decision-making and system destructiveness: A case of automatic debt recovery' European Journal of Information Systems 31,3 (2022) 313-338, at https://www.tandfonline.com/doi/pdf/10.1080/0960085X.2021.1960905?needAccess=tru

SA (2021) 'Individuals' Services Australia, 24 June 2021, at https://www.servicesaustralia.gov.au/individuals/centrelink

SA (2022) 'A guide to Australian Government payments' Services Australia, at https://www.servicesaustralia.gov.au/sites/default/files/co029-2207.pdf

Sanders E.B.-N. & Stappers P.J. (2008) 'Co-creation and the new landscapes of design' CoDesign 4,1 (2008) 5-18, at https://www.tandfonline.com/doi/pdf/10.1080/15710880701875068

Sanders E.B.-N. & Stappers P.J. (2014) 'Probes, toolkits and prototypes: three approaches to making in codesigning' CoDesign 10,1 (2014) 5-14, at https://www.researchgate.net/publication/262851486_Probes_toolkits_and_prototype_Three_approaches_to_making_in_codesigning

Sarah M. (2016) 'Welfare debt squad hunts for $4bn' The Australian, 5 December 2016, at http://www.theaustralian.com.au/national-affairs/welfare-debt-squad-hunts-for-4b-in-overpayments/news-story/e19c5b0d4a39aa07364a41269fdc11c9

SCA (2017) 'Design, scope, cost-benefit analysis, contracts awarded and implementation associated with the Better Management of the Social Welfare System initiative' The Senate Community Affairs References Committee, Australian Parliament House, June 2017, at https://www.aph.gov.au/Parliamentary_Business/Committees/Senate/Community_Affair/SocialWelfareSystem/Report

SCA (2022) 'Accountability and justice: Why we need a Royal Commission into Robodebt' Senate Community Affairs References Committee, May 2022, at https://parlinfo.aph.gov.au/parlInfo/download/committees/reportsen/024846/toc_pd/AccountabilityandjusticeWhyweneedaRoyalCommissionintoRobodebt.pdf;fileType=application%2Fpdf

Trist E.L. (1981) 'The evolution of socio-technical systems' Vol. 2, Toronto: Ontario Quality of Working Life Centre, 1981

TT (2016) 'Guilty until proven innocent ... Centrelink gone rogue ...' Tasmanian Times, 5 December 2016, at https://tasmaniantimes.com/2016/12/guilty-until-proven-innocent-centrelink-gone-ogue/

Venkatraman N. (1994) 'IT-Enabled Business Transformation: From Automation to Business Scope Redefinition' Sloan Management Review 35,2 (Winter 1994) 73-87

Von Bertalanffy L. (1950) 'The theory of open systems in physics and biology' Science, 111,2872 (1950) 23-29

Wachter S. & Mittelstadt B. (2019) 'A Right to Reasonable Inferences: Re-Thinking Data Protection Law in the Age of Big Data and AI' Forthcoming, Colum. Bus. L. Rev. (2019), at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3248829

Walsham G. (1995) 'Interpretive Case Studies in IS Research: Nature and Method' European Journal of Information Systems 4, 2 (May 1995) 74-81

Whiteford P. (2021) 'Debt by design: The anatomy of a social policy fiasco - Or was it something worse? Aust. J of Public Admin 80,2 (June 2021) 340-360, at https://onlinelibrary.wiley.com/doi/epdf/10.1111/1467-8500.12479

Whiteford P. (2022) 'For the poorest 20% of Australian households, social security payments provide more than 70% of their income' John Menadue's Public Policy Journal, 20 May 2022, at https://johnmenadue.com/social-security/

Wong K. (2022) 'Robodebt scheme told to `cease and desist' AAP, 4 November 2022, at https://www.aap.com.au/news/debt-collectors-to-front-royal-commission/

Wynn D. & Williams C.K. (2012) 'Principles for Conducting Critical Realist Case Study Research in Information Systems' MIS Quarterly 36, 3 (September 2012) 787-810, at https://ecommons.udayton.edu/cgi/viewcontent.cgi?article=1076&context=mis_fa_pub

Yin R.K. (2017) 'Case Study Research and Applications: Design and Methods' 6th edition, Sage, 2017

Zeng Y., Lu E. & Huangfu C. (2019) 'Linking Artificial Intelligence Principles' Proc. AAAI Workshop on Artificial Intelligence Safety (AAAI-Safe AI 2019), 27 January 2019, at https://arxiv.org/abs/1812.04814


The Primary Case Sources

The Investigation Reports

Omb (2017) 'Centrelink's automated debt raising and recovery system: A Report about the Department of Human Services' Online Compliance Intervention System for Debt Raising and Recovery' Australian Ombudsman, April 2017, at https://www.ombudsman.gov.au/__data/assets/pdf_file/0022/43528/Report-Centrelink-automated-debt-raising-and-recovery-system-April-2017.pdf

SCA (2017) 'Design, scope, cost-benefit analysis, contracts awarded and implementation associated with the Better Management of the Social Welfare System initiative' The Senate Community Affairs References Committee, Australian Parliament House, June 2017, at https://www.aph.gov.au/Parliamentary_Business/Committees/Senate/Community_Affair/SocialWelfareSystem/Report

GR (2017) 'Government Response to the Senate Committee Report', Australian Government, September 2017, at https://www.aph.gov.au/Parliamentary_Business/Committees/Senate/Community_Affair/SocialWelfareSystem/Additional_Documents?docType=Government%20Response

Pry (2021) Prygodicz v Commonwealth of Australia (No 2) [2021] FCA 634 (11 June 2021), at http://www.austlii.edu.au/cgi-bin/viewdoc/au/cases/cth/FCA/2021/634.html?context1;query=Prygodicz;mask_path=

SCA (2022) 'Accountability and justice: Why we need a Royal Commission into Robodebt' Senate Community Affairs References Committee, May 2022, at https://parlinfo.aph.gov.au/parlInfo/download/committees/reportsen/024846/toc_pd/AccountabilityandjusticeWhyweneedaRoyalCommissionintoRobodebt.pdf;fileType=application%2Fpdf

Other Relevant Official Documents

DMP (1994) 'Guidelines for the Conduct of the Data-Matching Program' Delegated Legislated under the Data Matching Program (Assistance and Tax) Act 1990, October 1994, nominally enforceable, but applicable only if the matching process uses the Tax File Number (TFN), at https://www.legislation.gov.au/Details/F2009B00268

OAIC (2014) 'Guidelines on Data Matching in Australian Government Administration' Privacy Commissioner, June 2014, unenforceable, https://www.oaic.gov.au/agencies-and-organisations/advisory-guidelines/data-matcing-guidelines-2014 [broken link on 6 Sep 2022]

Centrelink (2016) 'Data Matching Protocol', undated, but apparently of mid-2016, suppressed by Centrelink for a year, and only disclosed incidentally by another agency as an attachment to a letter of 16 May 2017 by the Privacy Commissioner responding to a Senator's questions, at http://www.aph.gov.au/DocumentStore.ashx?id=4f473c96-f345-4314-ad5d-a2c2eecfba47p> OAIC (2017) Vacuous statements by the Privacy Commissioner, January-March 2017, at https://www.oaic.gov.au/media-and-speeches/statements/centrelink-debt-recovery-sstem [broken link on 6 Sep 2022]

DHS (2017) 'If you owe us money, you'll need to pay us back' Department of Human Services, 2016-17, https://www.servicesaustralia.gov.au/owing-money-for-centrelink-debt [but continually changing]

Amato v The Commonwealth of Australia (Federal Court VID611/2019), at https://www.comcourts.gov.au/file/Federal/P/VID611/2019/3859485/event/30114114/dcument/1513665

Prygodicz v Commonwealth of Australia [2020] FCA 1454 (17 September 2020) http://www.austlii.edu.au/cgi-bin/viewdoc/au/cases/cth/FCA/2020/1454.html?contex=1;query=Prygodicz;mask_path

Commonwealth of Australia v Prygodicz [2020] FCA 1516 (14 October 2020) http://www.austlii.edu.au/cgi-bin/viewdoc/au/cases/cth/FCA/2020/1516.html?contex=1;query=Prygodicz;mask_path

Prygodicz v Commonwealth of Australia (No 2) [2021] FCA 634 (11 June 2021) - the eventual judgment, at http://www.austlii.edu.au/cgi-bin/viewdoc/au/cases/cth/FCA/2021/634.html?context1;query=Prygodicz;mask_path

Prygodicz v Commonwealth of Australia (No 3) [2022] FCA 826 (23 March 2022), at http://www.austlii.edu.au/cgi-bin/viewdoc/au/cases/cth/FCA/2022/826.html?context1;query=Prygodicz;mask_path

Royal Commission into the Robodebt Scheme:


Author Affiliations

Roger Clarke is Principal of Xamax Consultancy Pty Ltd, Canberra. He is also a Visiting Professor associated with the Allens Hub for Technology, Law and Innovation in UNSW Law, and a Visiting Professor in the Research School of Computer Science at the Australian National University. He is also a longstanding Board-member of the Australian Privacy Foundation.

Katina Michael is a Professor at Arizona State University, a Senior Global Futures Scientist in the Global Futures Laboratory and has a joint appointment in the School for the Future of Innovation in Society and School of Computing and Augmented Intelligence. She is the director of the Society Policy Engineering Collective (SPEC) and the Founding Editor-in-Chief of the IEEE Transactions on Technology and Society. She is also a longstanding Board-member of the Australian Privacy Foundation.

Roba Abbas is a Senior Lecturer and Academic Program Director in the School of Business, at the University of Wollongong in Australia. Her work is focused on methodology approaches to complex socio-technical systems design. She is a Co-Editor of the IEEE Transactions on Technology and Society.



xamaxsmall.gif missing
The content and infrastructure for these community service pages are provided by Roger Clarke through his consultancy company, Xamax.

From the site's beginnings in August 1994 until February 2009, the infrastructure was provided by the Australian National University. During that time, the site accumulated close to 30 million hits. It passed 65 million in early 2021.

Sponsored by the Gallery, Bunhybee Grasslands, the extended Clarke Family, Knights of the Spatchcock and their drummer
Xamax Consultancy Pty Ltd
ACN: 002 360 456
78 Sidaway St, Chapman ACT 2611 AUSTRALIA
Tel: +61 2 6288 6916

Created: 9 September 2022 - Last Amended: 8 November 2022 by Roger Clarke - Site Last Verified: 15 February 2009
This document is at www.rogerclarke.com/DV/RDCS.html
Mail to Webmaster   -    © Xamax Consultancy Pty Ltd, 1995-2022   -    Privacy Policy