Developing a Turing test for ethical AI

AI is ready for a new challenge: Can we train the machine behind the curtain to incorporate humanistic ethics into its processes?

Developing a Turing test for ethical AI

Artificial intelligence developers have always had a “Wizard of Oz” air about them. Behind a magisterial curtain, they perform amazing feats that seem to bestow algorithmic brains on the computerized scarecrows of this world.

AI’s Turing test focused on the wizardry needed to trick us into thinking that scarecrows might be flesh-and-blood humans (if we ignore the stray straws bursting out of their britches). However, I agree with the argument recently expressed by Rohit Prasad, Amazon’s head scientist for Alexa, who argues that Alan Turing’s “imitation game” framework is no longer relevant as a grand challenge for AI professionals.

Creating a new Turing test for ethical AI

Prasad points out that impersonating natural-language dialogues is no longer an unattainable objective. The Turing test was an important conceptual breakthrough in the early 20th century, when what we now call cognitive computing and natural language processing were as futuristic as traveling to the moon. But it was never intended to be a technical benchmark, simply a thought experiment to illustrate how an abstract machine might emulate cognitive skills.

Prasad argues that the AI’s value resides in advanced capabilities that go far beyond impersonating natural-language conversations. He points to AI’s well-established capabilities of querying and digesting vast amounts of information much faster than any human could possibly manage unassisted. AI can process video, audio, image, sensor, and other types of data beyond text-based exchanges. It can take automated actions in line with inferred or prespecified user intentions, rather than through back-and-forth dialogues.

We can conceivably envelop all of these AI faculties into a broader framework focused on ethical AI. Ethical decision-making is of keen interest to anybody concerned with how AI systems can be programmed to avoid inadvertently invading privacy or taking other actions that transgress core normative principles. Ethical AI also intrigues science-fiction aficionados who have long debated whether Isaac Asimov’s intrinsically ethical laws of robotics can ever be programmed effectively into actual robots (physical or virtual).

If we expect AI-driven bots to be what philosophers call “moral agents,” then we need a new Turing test. An ethics-focused imitation game would hinge on how well an AI-driven device, bot, or application can convince a human that its verbal responses and other behavior might be produced by an actual moral human being in the same circumstances.

Building ethical AI frameworks for the robotics age

From a practical standpoint, this new Turing test should challenge AI wizards not only to bestow on their robotic “scarecrows” their algorithmic intelligence, but also to equip “tin men” with the artificial empathy needed to engage humans in ethically framed contexts, and render to “cowardly lions” the artificial efficacy necessary for accomplishing ethical outcomes in the real world.

Ethics is a tricky behavioral attribute around which to develop concrete AI performance metrics. It’s clear that even today’s most comprehensive set of technical benchmarks—such as MLPerf—would be an inadequate yardstick to measure whether AI systems can convincingly imitate a moral human being.

People’s ethical faculties are a mysterious blend of intuition, experience, circumstance, and culture, plus situational variables that guide individuals over the course of their lives. Under a new, ethics-focused Turing test, broad AI development practices fall into the following categories:

Baking ethical AI practices into the ML devops pipeline

Ethics isn’t something that one can program in any straightforward way into AI or any other application. That explains, in part, why we see a growing range of AI solution providers and consultancies offering assistance to enterprises that are trying to reform their devops pipelines to ensure that more AI initiatives produce ethics-infused end products.

To a great degree, building AI that can pass a next-generation Turing test would require that these apps be built and trained within devops pipelines that have been designed to ensure the following ethical practices:

  • Stakeholder review: Ethics-relevant feedback from subject matter experts and stakeholders is integrated into the collaboration, testing, and evaluation processes surrounding iterative development of AI applications.
  • Algorithmic transparency: Procedures ensure the explainability in plain language of every AI devops task, intermediate work product, and deliverable app in terms of its adherence to the relevant ethical constraints or objectives.
  • Quality assurance: Quality control checkpoints appear throughout the AI devops process. Further reviews and vetting verify that no hidden vulnerabilities remain—such as biased second-order feature correlations—that might undermine the ethical objectives being sought.
  • Risk mitigation: Developers consider the downstream risks of relying on specific AI algorithms or models—such as facial recognition—whose intended benign use (such as authenticating user log-ins) could also be vulnerable to abuse in dual-use scenarios (such as targeting specific demographics).
  • Access controls: A full range of regulatory-compliant controls are incorporated on access, use, and modeling of personally identifiable information in AI applications.
  • Operational auditing: AI devops processes create an immutable audit log to ensure visibility into every data element, model variable, development task, and operational process that was used to build, train, deploy, and administer ethically aligned apps.

Trusting the ethical AI bot in our lives

The ultimate test of ethical AI bots is whether real people actually trust them enough to adopt them into their lives.

Natural-language text is a good place to start looking for ethical principles that can be built into machine learning programs, but the biases of these data sets are well known. It’s safe to assume that most people don’t behave ethically all the time, and they don’t always express ethical sentiments in every channel and context. You wouldn’t want to build suspect ethical principles into your AI bots just because the vast majority of humans may (hypocritically or not) espouse them.

Nevertheless, some AI researchers have built machine learning models, based on NLP, to infer behavioral patterns associated with human ethical decision-making. These projects are grounded in AI professionals’ faith that they can identify within textual data sets the statistical patterns of ethical behavior across societal aggregates. In theory, it should be possible to supplement these text-derived principles with behavioral principles inferred through deep learning on video, audio, or other media data sets.

In building training data for ethical AI algorithms, developers need robust labeling and curation provided by individuals who can be trusted with this responsibility. Though it can be difficult to measure such ethical qualities as prudence, empathy, compassion, and forbearance, we all know what they are when we see them. If asked, we could probably tag any specific instance of human behavior as either exemplifying or lacking them.

It may be possible for an AI program that was trained from these curated data sets to fool a human evaluator into thinking a bot is a bonafide homo sapiens with a conscience. But even then, users may never completely trust that the AI bot will take the most ethical actions in all real-world circumstances. If nothing else, there may not have been enough valid historical data records of real-world instances to train ethical AI models in unusual or anomalous scenarios.

Just as significant, even a well-trained ethical AI algorithm may not be able to pass a multilevel Turing test where evaluators consider the following contingent scenarios:

  • What happens when diverse ethical AI algorithms, each authoritative in its own domain, interact in unforeseen ways and produce ethically dubious results in a larger context?
  • What if these ethically assured AI algorithms conflict? How do they make trade-offs among equally valid values in order to resolve the situation?
  • What if none of the conflicting AI algorithms, each of which is ethically assured in its own domain, is competent to resolve the conflict?
  • What if we build ethically assured AI algorithms to deal with these higher-order trade-offs, but two or more of these higher-order algorithms come into conflict?

These complex scenarios may be a snap for a moral human—a religious leader, legal scholar, or your mom—to answer authoritatively. But they may trip up an AI bot that’s been specifically built and trained for a narrow range of scenarios. Consequently, ethical decision-making may always need to keep a human in the loop, at least until that glorious (or dreaded) day when we can trust AI to do everything and anything in our lives.

For the foreseeable future, AI algorithms can only be trusted within specific decision domains, and only if their development and maintenance is overseen by humans who are competent in the underlying values being encoded. Regardless, the AI community should consider developing a new ethically focused imitation game to guide R&D during the next 50 to 60 years. That’s about how long it took the world to do justice to Alan Turing’s original thought experiment.

Copyright © 2021 IDG Communications, Inc.