Algorithmic decision-making and system destructiveness: A case of automatic debt recovery

As the pace of our technology-saturated world continues to increase, so too does the need for governments and organisations to rely on artificial intelligence and automation to deliver services to the general public. But what happens when algorithmic decision-making (ADM) produces lasting devastation on citizens and society?

Recent information systems literature has raised concerns regarding ADM’s negative unintended consequences - such as widespread discrimination - which, in extreme cases, can be destructive to society. The extant empirical literature, however, has not sufficiently examined the destructive effects of governmental ADM.

In a recent paper, Dr Tapani Rinta-Kahlia's reports on a case study of the Australian Government’s “Robodebt” programme that was designed to automatically calculate and collect welfare overpayment debts from citizens but ended up causing severe distress to citizens and welfare agency staff. Employing perspectives from systems thinking and organisational limits, we develop a research model that explains how a socially destructive government ADM programme was initiated, sustained, and delegitimised. The model offers a set of generalisable mechanisms that can benefit investigations of ADM’s consequences. Our findings contribute to the literature of unintended consequences of ADM and demonstrate to practitioners the importance of setting up robust governance infrastructures for ADM programmes.

Keynote speaker:

Tapani Rinta-Kahila is a Lecturer at The University of Queensland. His research addresses current issues with socio-technical change in organisations. Focus areas include the organisational implementation of artificial intelligence and automation, the identification and prevention of technologies’ negative effects on individuals and organisations, as well as the discontinuance of incumbent information technologies.