Imagine a couple of caffeine-addled biochemistry majors late at night in their dorm kitchen cooking up a new medicine that proves remarkably effective at soothing colds but inadvertently causes permanent behavioral changes. Those who ingest it become radically politicized and shout uncontrollably in casual conversation. Still, the concoction sells to billions of people. This sounds preposterous, because the FDA would never let such a drug reach the market.
Yet this madness is happening everywhere online. Every day, we view streams of content custom-selected by simple software algorithms, some created in dorms, based on a technique called adaptive reinforcement learning. With each click, the algorithms learn to personalize the feed to their users’ tastes, thereby reaping profits for their owners. But the designers made a simple mistake: They assumed that human tastes are fixed. In reality, algorithms applied to malleable humans can have drastically different and pernicious side effects on a global scale. They modify our tastes to make us ever more predictable, edge us toward extremes, and ultimately erode civility and trust in society. It is time we stop blithely allowing this and create the digital equivalent of drug trials.
Intelligent systems at scale need regulation because they are an unprecedented force multiplier for the promotion of the interests of an individual or a group. For the first time in history, a single person can customize a message for billions and share it with them within a matter of days. A software engineer can create an army of AI-powered bots, each pretending to be a different person, promoting content on behalf of political or commercial interests. Unlike broadcast propaganda or direct marketing, this approach also uses the self-reinforcing qualities of the algorithm to learn what works best to persuade and nudge each individual.
Manipulating user preferences and using bot armies to leverage widespread deceit has disrupted societal cohesion and democratic processes. To protect the cognitive autonomy of individuals and the political health of society at large, we need to make the function and application of algorithms transparent, and the FDA provides a useful model.
The US Food and Drug Administration requires controlled testing on animals to establish safety, and then more testing on small populations to establish efficacy. Only then can a company offer a new drug to the masses. Software, by contrast, is typically subjected only to “unit tests,” to assure new lines of code perform as expected, and “integration tests,” to assure the updates don’t degrade the system’s performance. This is like checking a drug for contaminants without testing the effects of its active ingredients.
Of course, we cannot assemble a traditional FDA-style review panel to weed out deliberately false content in every article online. But we do already have tools for both platforms and users to detect falsehood and screen out dubious sources, including reputation systems, third-party raters, notary-like institutions, traceability, and, as is now the case in California, a regulation that requires bots to self-identify as such and makes it illegal to deploy bots that knowingly deceive others in an attempt to encourage a purchase or influence a vote.
But this is not enough to safeguard society. Facebook and its click maximizer reconfigured society without apparent intent to deceive. A more comprehensive digital testing process would have to mitigate the potential harm to people and society regardless of developers’ motives.
What might these digital clinical trials look like? An oversight body would need to carry the authority of a government agency like the FDA, but also employ the depth of technical know-how found at existing technology-focused governing bodies like ICANN. It would need to house a rich diversity of expertise to grasp the breadth of society, seating psychologists and sociologists alongside programmers and economists. Because not every piece of code needs tight oversight, it would need distinct trigger points on when to review and at what level of scrutiny, similar to the ways the FDA’s powers stretch or recede for pharmaceuticals versus nutritional supplements.
It would need to employ system-level testing that not only observed how the software works but also modeled the second- and third-order effects it could have across society, such as amplifying aggressive resentment against other groups and facilitating addiction to social games. This technology would resemble the methods that political scientists and intelligence agencies use to build computational models of societies undergoing perturbations like food crises and epidemics. And with tools that test for both resilience and ripple effects in place, we might then implement a stepwise release process similar to the stages of FDA testing—trials moving from these “sandbox” simulations to a small group of users, and then on to the market as a whole.
This is only the starting point for an oversight body that could eventually facilitate a freer flow of beneficial innovation, much like how the rules of the road streamline traffic. The structure, processes, and menu of testing tools it uses would have to remain flexible enough to keep pace with continuous technological change. Regardless of the final design, we need to start immediately. While the drug approval process can be flawed or corrupted, as the ongoing opioid nightmare attests, its model is far better than the minuscule defenses we have today to protect us from weapons of mass societal destruction.
It will take time to create a mechanism that protects individuals and society without blocking the paths to promising new technologies. We can expect to endure more societal disruption in the interim, as commercial and political incentives continue to lead the private sector away from the types of proactive protections we need. We will be nudged further toward extremes, and hopes for an open, fruitful, and diverse discourse in the digital town square will wither. That is not good for our society nor for entrepreneurial innovation. It is in our collective interest to avoid that fate.
WIRED Opinion publishes pieces written by outside contributors and represents a wide range of viewpoints. Read more opinions here. Submit an op-ed at opinion@wired.com.
- 3 years of misery inside Google, the happiest place in tech
- The weird, dark history of 8chan and its founder
- 8 ways overseas drug manufacturers dupe the FDA
- A Boeing code leak exposes security flaws deep in a 787
- The terrible anxiety of location sharing apps
- 🏃🏽♀️ Want the best tools to get healthy? Check out our Gear team’s picks for the best fitness trackers, running gear (including shoes and socks), and best headphones.
- 📩 Get even more of our inside scoops with our weekly Backchannel newsletter
