Confused about AI and worried about what it means for your future and the future of the world? You’re not alone. AI is everywhere―and few things are surrounded by so much hype, misinformation, and misunderstanding.
In AI Snake Oil, computer scientists Arvind Narayanan and Sayash Kapoor cut through the confusion to give you an essential understanding of how AI works and why it often doesn’t, where it might be useful or harmful, and when you should suspect that companies are using AI hype to sell AI snake oil―products that don’t work, and probably never will.
While acknowledging the potential of some AI, such as ChatGPT, AI Snake Oil uncovers rampant misleading claims about the capabilities of AI and describes the serious harms AI is already causing in how it’s being built, marketed, and used in areas such as education, medicine, hiring, banking, insurance, and criminal justice.
The Rise of AI as a Consumer Product
ChatGPT burst onto the scene in late 2022, going viral overnight as people shared amusing examples of its capabilities. Within two months, it reportedly had over 100 million users. This sparked a wave of AI integration across industries, from legal work to creative fields. However, the rapid adoption also led to misuse and errors, such as news websites publishing AI-generated stories with factual mistakes.
The Double-Edged Sword of Predictive AI
While generative AI shows promise, predictive AI often falls short of its claims. Companies tout the ability to predict outcomes like job performance or criminal behavior, but evidence suggests these tools are frequently inaccurate and can exacerbate inequalities. For instance, a healthcare AI tool meant to predict patient needs actually reinforced racial biases in care. The authors argue that many predictive AI applications are “snake oil” – products that don’t work as advertised.
The Need for AI Literacy
The book aims to provide readers with the tools to critically evaluate AI claims and identify “snake oil.” The authors argue that understanding AI is crucial for navigating its growing influence in society.
“We think most knowledge industries can benefit from chatbots in some way. We use them ourselves for research assistance, for tasks ranging from mundane ones such as formatting citations correctly, to things we wouldn’t otherwise be able to do such as understanding a jargon-filled paper in a research area we aren’t familiar with.”
How Predictive AI Goes Wrong
The False Promise of Predictive Accuracy
Many companies claim their predictive AI tools can accurately forecast outcomes like job performance or criminal behavior. However, these claims often fall apart under scrutiny. The authors cite examples like COMPAS, a tool used in criminal justice that claims to predict recidivism but performs only slightly better than random guessing. They argue that the complexity of human behavior and social contexts makes accurate prediction extremely difficult, if not impossible, in many cases.
The Dangers of Automated Decision-Making
When predictive AI is used to make consequential decisions about people’s lives, the risks of harm increase dramatically. The authors describe cases where AI systems have denied healthcare, incorrectly flagged individuals for welfare fraud, or unfairly assessed job candidates.
They warn: “Predictive AI is quickly gaining in popularity. Hospitals, employers, insurance providers, and many other types of organizations use it. A major selling point is that it allows them to reuse existing datasets that have already been collected for other purposes, such as for bureaucratic reasons and record keeping, to make automated decisions.”
Reinforcing and Amplifying Biases
Predictive AI often perpetuates and exacerbates existing societal biases. The authors discuss how these systems can disproportionately harm marginalized groups. For example, a healthcare AI tool meant to identify patients needing extra care actually recommended lower levels of care for Black patients compared to white patients with similar health needs. This occurred because the AI was trained on historical data that reflected existing disparities in healthcare spending.
The Illusion of Objectivity
Many organizations adopt predictive AI with the belief that it will make decision-making more objective and fair. However, the authors argue that this is often an illusion. They explain that AI systems inherit the biases present in their training data and in the societies that produce that data.
Additionally, the opacity of many AI systems can make it difficult to identify and address these biases. The authors caution against the “automation bias” – the tendency to over-rely on automated systems even when they make errors.
Why Can’t AI Predict the Future?
The Limits of Computational Prediction
Despite advances in computing power and data collection, accurately predicting complex social outcomes remains elusive. The authors explore historical attempts at prediction, from weather forecasting to social simulations, highlighting the inherent challenges. They argue that while some phenomena can be predicted with reasonable accuracy, many social and individual outcomes are fundamentally unpredictable due to their complexity and the role of chance events.
The Fragile Families Challenge: A Case Study in Prediction Failure
The authors discuss a large-scale study called the Fragile Families Challenge, where researchers attempted to predict children’s life outcomes using a vast amount of data. Despite having access to thousands of data points and employing advanced AI techniques, the predictions were only slightly better than random guessing.
..none of the models performed very well—the best models were only slightly better than a coin flip. And complex AI models showed no substantial improvement compared to the baseline model consisting of just four features.
The Role of Randomness and Complexity in Human Life
The chapter emphasizes how random events and complex interactions can dramatically shape individual lives in ways that are impossible to predict. The authors argue that many life outcomes are influenced by small initial advantages that compound over time, as well as unpredictable “shocks” like accidents or unexpected opportunities. They suggest that this fundamental unpredictability challenges the very premise of many predictive AI applications.
The Challenges of Predicting Aggregate Outcomes
While predicting individual outcomes is extremely difficult, the authors also explore the challenges of predicting aggregate social phenomena like economic trends or disease outbreaks. They discuss examples like the COVID-19 pandemic, where even short-term predictions were highly unreliable due to the complex interplay of biological, social, and political factors. The authors argue that many predictive failures stem from underestimating the role of rare, high-impact events that can dramatically alter the course of social systems.
The Long Road to Generative AI
From Image Processing to Language Models
The chapter explores how techniques developed for image processing were adapted for natural language tasks, leading to the creation of powerful language models. The authors describe the key innovations that enabled this transition, including the development of attention mechanisms and the transformer architecture. They explain how these advancements ultimately led to the creation of large language models capable of generating human-like text.
The Ethical Challenges of Generative AI
While celebrating the technical achievements, the authors also highlight the ethical concerns raised by generative AI. They discuss issues such as the exploitation of low-wage workers for data annotation, the appropriation of artists’ work without compensation, and the potential for AI-generated disinformation. The authors argue that these challenges require urgent attention and potential regulatory responses.
Debunking AI Doomsday Scenarios
The authors challenge popular narratives about the existential risks posed by advanced AI. They argue that many of these scenarios, such as the idea of a superintelligent AI taking over the world, are based on flawed assumptions and misunderstandings of AI technology.
“We think AGI is a long-term prospect, and that society already has the tools to address its risks calmly. We shouldn’t let the bugbear of existential risk distract us from the more immediate harms of AI snake oil.”
The Ladder of Generality
Instead of viewing AI development as a sudden leap to artificial general intelligence (AGI), the authors propose the concept of a “ladder of generality.” This framework describes AI progress as a series of incremental steps, each increasing the flexibility and capability of AI systems. They argue that this perspective provides a more realistic and nuanced understanding of AI development, countering alarmist narratives about exponential or runaway AI growth.
The Limitations of AI Safety Research
The chapter critiques some approaches to AI safety research, particularly those focused on long-term existential risks. The authors argue that many proposed solutions, such as trying to “align” AI with human values, are based on speculative scenarios and may not address more immediate and concrete AI risks.
They suggest that resources might be better spent addressing current AI harms and developing practical safeguards.
Focusing on Realistic AI Governance
Rather than trying to prevent the development of advanced AI altogether, the authors advocate for practical approaches to AI governance. They emphasize the importance of addressing specific, well-defined risks associated with AI technologies, such as cybersecurity vulnerabilities or the potential for AI-enhanced bioweapons. The authors argue for strengthening existing regulatory frameworks and institutions to handle these challenges.
Why Can’t AI Fix Social Media?
The Complexities of Content Moderation
The authors explore the challenges of using AI for content moderation on social media platforms. They argue that despite advances in AI technology, fully automated content moderation remains an elusive goal. The chapter details how context, cultural nuances, and the ever-changing nature of online discourse make it difficult for AI systems to consistently make accurate moderation decisions.
The Human Cost of AI-Assisted Moderation
While AI is increasingly used in content moderation, the authors highlight the continued reliance on human moderators, often working in difficult conditions. They discuss the psychological toll of this work and the ethical concerns raised by outsourcing moderation to low-wage workers in developing countries. The authors note: “It’s trauma-inducing work, done by hundreds of thousands of invisible, low-wage workers, mostly in less-affluent countries, working for third-party outsourcing firms rather than directly for platform companies.”
The Limitations of AI in Understanding Context
The chapter explores numerous examples where AI-based content moderation systems have failed due to their inability to understand context. From misinterpreting art and political satire to struggling with cultural differences, these failures highlight the gap between AI’s pattern recognition capabilities and human-level understanding of content.
The Need for Human Oversight and Policy Decisions
The authors argue that many content moderation challenges are fundamentally about policy and values, not just technology. They emphasize the need for human oversight in setting and interpreting content policies, and discuss the complexities of applying these policies globally. The chapter suggests that improving content moderation requires not just better AI, but also more transparent and accountable decision-making processes by social media companies.
The AI Hype Cycle
The authors examine why exaggerated claims and myths about AI capabilities continue to circulate. They discuss the concept of the “hype cycle” and how AI repeatedly goes through phases of inflated expectations followed by disappointment. The chapter explores how various stakeholders, including companies, researchers, and the media, contribute to this cycle of hype.
The Role of Commercial Interests
The chapter delves into how companies often overhype their AI products for commercial gain. The authors use the example of a healthcare AI tool that claimed to predict sepsis but was found to be ineffective when independently evaluated. They write: “Epic was not shy about showing off its adoption rates. Hundreds of hospitals had adopted the system, and Epic claimed its model reduced mortality rates due to sepsis in these hospitals. The model was hailed for allowing clinicians to spend more time with patients.”
Academic Hype and the Reproducibility Crisis
The authors discuss how academic research can contribute to AI hype, often due to flawed methodologies or overstated conclusions. They highlight the “reproducibility crisis” in AI research, where many published results cannot be replicated or verified. This section emphasizes the need for more rigorous standards and transparency in AI research.
Media Amplification of AI Myths
The chapter examines how media coverage often amplifies and perpetuates myths about AI. The authors critique sensationalist reporting and the tendency to uncritically repeat company claims about AI capabilities. They argue for more skeptical and nuanced reporting on AI developments.
Where Do We Go from Here?
Rethinking AI Regulation
The authors advocate for a nuanced approach to AI regulation that goes beyond simplistic calls for either strict control or complete freedom. They discuss the potential for existing regulatory frameworks to be adapted for AI governance and emphasize the need for regulators to have sufficient technical expertise. The chapter explores ideas like partial lotteries for decision-making as alternatives to predictive AI in some contexts.
Addressing Labor and Inequality Concerns
The authors discuss the potential impacts of AI on the job market and inequality. They argue for proactive measures to address labor displacement and ensure that the benefits of AI are more equitably distributed. The chapter explores ideas like universal basic income and stronger labor protections as potential responses to AI-driven economic changes.
