The potential reach of the state into our individual biology and genetic makeup is expanding at an unprecedented rate. Global responses to the COVID-19 pandemic have crystallized just how quickly and readily machines, algorithms, and computing power can be combined with biological data and used in technologies that subjugate bodies and control populations.

As the Chinese city of Wuhan went into lockdown, the authorities carried out large-scale remote temperature measurements of households in apartment complexes through drones equipped with infrared cameras. Drones were also used to patrol public places, tracking whether people were travelling outside without face masks or violating other quarantine rules. Chinese police forces debuted augmented reality (AR) smart glasses powered by artificial intelligence (AI) capabilities, designed to recognize individuals with potential COVID-19 symptoms. The glasses have facial recognition capability to identify and profile individuals in real-time and can also record photos and videos. As Wuhan started to open up again, the authorities introduced “Health Code,” an app people were required to use when entering and exiting residential areas, supermarkets, subways, and taxis, among other spaces. The app stores your personal information, including your ID number, where you live, whether you have been with people carrying the virus, and what symptoms they had. As you touch in or out on entering or exiting, the app gives you a colour: green means you can go anywhere, yellow means you have to quarantine for 7 days, red for 14 days. The app also surreptitiously collects—and shares with the police—your location data.

And this type of surveillance wasn’t used just in China. A range of countries have adopted intrusive and coercive forms of surveillance and use of personal and biological data reminiscent of dystopian novels like Nineteen Eighty-Four and Brave New World. As other countries went into lockdown, surveillance cameras with facial recognition tracked quarantine evaders or gauged elevated temperatures of potentially infected individuals in crowds. Fine-grained location data transmitted from mobile phones determined how many people were obeying lockdown orders, fever-detecting cameras screened travellers arriving at airports, and algorithms monitored social media posts for signs of COVID-19’s spread. Contact-tracing apps, centrally storing user interactions, provide “social graphs” of who you have physically met over a period of time. “Immunity passports” or “risk free certificates” combine facial recognition technology with COVID-19 testing and medical records.

As genomic technologies develop and converge with AI, machine learning, automation, affective computing, and robotics, an ever more refined record of our biometrics, emotions, and behaviors will be captured and analysed. Governments and, increasingly, private companies will be able to sort, categorize, trade, and use biological data far more precisely than ever before, creating unprecedented possibilities for social and biological control. Some even argue that the new geopolitical order will be based on the commodification of a new resource emerging from the convergence of the artificial intelligence and biotech industries: our biological and genomic data.

These game-changing developments will deeply impact how we view health and treat disease, how long we live, and how we consider our place on the biological continuum. They will also radically transform the dual-use nature of biological research, medicine, and health care and create the possibility of novel biological weapons that target particular groups of people and even individuals. In the coming decade, managing the fast and broad technological advances now under way will require new governance structures that draw on individuals and groups with cross-sectoral expertise—from business and academia to politics and defense—to identify emerging security risks and make recommendations for dealing with them.

Adding computing power to bioinformatics

Genomic technologies are driving a vast expansion in genomic data, from gene sequences and entire genomes to data that links genes to specific functions and other types of metadata for humans, other animals, plants, and microbes. This data is becoming increasingly digitized, and computational power is significantly changing how genomic data is analysed. The integration of AI computation into biology opens up new possibilities for understanding how genetic differences shape the development of living organisms, including ourselves, and how these differences make us and the rest of the living world susceptible to diseases and disorders, and responsive to drugs and treatments.

Advanced pattern recognition and the abstraction of statistical relationships from data—the hallmarks of machine and deep learning—have shown significant potential to help researchers make sense of complex genomic data sets and extract clinically relevant findings. Take two prominent examples: functional genomics and tailored drug discovery.

The ability of machine learning to link, correlate, and analyse data is particularly useful for interpreting gene functions and identifying genetic markers responsible for certain diseases, as one expert report highlights. Known as functional genomics, this field of research makes it possible to predict how likely someone is to develop diseases such as type 1 diabetes or breast cancer or to develop certain traits and capabilities, such as height or resistance to specific pathogens, that result from complex genetic influences. Deep learning also enables computer-based experimentation for functional genomics, and work is underway to predict how genetic sequences might function before they are assembled—even if the combination has not been seen in nature. Computational power has also helped researchers understand the evolving relationship between our genotypes, phenotypes (i.e. physical characteristics), and microbiomes (the bacteria and viruses that live on and inside the human body), and to improve our genotype-phenotype functional knowledge of pathogens.

Private industry has been instrumental in developing data mining techniques. Google’s genomic AI platform, DeepVariant, for example, has been at the forefront of developing an automated, deep learning approach to identifying genetic variants in an individual genome from billions of short sequences.

Adding computational power to drug development facilitates “parallel read operations of 10 billion nucleic acid molecules in a single experiment,” according to another expert report, and can increase experimental precision to single molecule manipulations. Emphasizing biological modelling, the report notes that the use of deep learning to develop new drug candidates has overcome many limitations of physics-based models, enabling models to be built from simple representations of chemical and biological entities and automating suggestions of synthesizable structures with improved properties. It also highlights how the convergence of automation with evolutionary algorithms vastly expands the number of materials that can be synthesized, tested, and optimized.

In developing new drug candidates, a robot can reportedly screen over 10,000 compounds per day through conventional “brute-force” tactics. However, while simple to automate, this approach is still relatively slow and wasteful; every compound in the library needs to be tested. The first AI robot to automate early-stage drug design came on stream in 2015. Called “Eve,” it was developed by researchers at the Universities of Aberystwyth and Cambridge, who earlier had developed “Adam,” a machine to independently discover scientific knowledge. To make drug candidate screening processes intelligent, Eve randomly selects a subset of compounds from a library, carries out various tests on them, and, based on the compounds that pass the tests, uses statistics and machine learning to predict new structures that might score even better.

Private companies also play a significant contributing role in developing machine learning for drug development. The pharmaceutical giant Novartis, for example, used computational power to develop a vaccine in less than three months from the first reported human infections of H7N9 influenza virus. In another example, Deep Genomics uses its AI platform to map pathological genetic pathways to inform drug development.

Advances in function genomics and drug discovery, as well as in other areas, offer the possibility of developing bespoke, or personalized, treatments using machine learning analysis of genomic and health data. “Precision public health” aims to deliver the right intervention to the right population at the right time, and it is already beginning to deliver genomic-based interventions for health and health care. The Centers for Disease Control and Prevention promotes its wide use of artificial intelligence and machine learning to improve public health surveillance (forecasting of influenza for example) and disease detection, mitigation, and elimination. While still in its early days, precision medicine—spanning personalized vaccines and antibodies, personalized treatment relying on virology and microbe research, personalized cancer treatments, and treatments involving in vivo gene editing—is also starting to become a reality.

A number of private companies, such as Tempus, IBM, and Pfizer, are actively exploring possibilities, though these efforts are still mostly focused on understanding how machine learning could help identify genetic markers or patients that should or could be candidates for personalized treatments. Experts emphasize that there is “still pervasive uncertainty about how accurate deep machine-learning will be in drawing useful interferences between the different datasets that make our biology.”

Rising security concerns

Various risk assessment frameworks have been used to get a sense of the potential security risks arising from the mix of artificial intelligence and biotechnology. But balancing the generality needed to capture a broad scope of converging technologies in the life sciences with the need to maintain enough specificity to capture nuances has proved difficult. The main security concerns boil down to worries that, if the intent were there, the convergence of emerging technologies could be used to speed up the identification of harmful genes or DNA sequences. More specifically, there are concerns that adding advanced pattern recognition to genomic data could significantly facilitate: the enhancement of pathogens to make them more dangerous; the modification of low-risk pathogens to become high-impact; the engineering of entirely new pathogens; or even the re-creation of extinct, high-impact pathogens like the variola virus that causes smallpox. These possibilities are coming at a time when new delivery mechanisms for transporting pathogens into human bodies are also being developed. In addition to the bombs, missiles, cluster bombs, sprayers, and injection devices of past biowarfare programs, it could now also be possible to use drones, nano-robots, even insects.

Added to these pathogen-specific risks are traditional cyber risks and “cyber-biosecurity” risks focused particularly on the bioeconomy. Cyber-biosecurity risks include waging adversarial attacks on automated bio-computing systems, biotech supply chains, or strategic cyber-biosecurity infrastructure. Malicious actors could, for example, use AI malware to co-opt networks of sensors and impact control decisions on biotech supply chains with the intent to damage, destroy, or contaminate vital stocks of vaccines, antibiotics, cell, or immune therapies. In another scenario, AI malware could be used to automate data manipulation with the intent to falsify, erase, or steal intelligence within large curations of genomics data. Such data poisoning could affect how pathogens are detected and analysed. It could also affect biointelligence on complex diseases in subpopulations collected over many years

The merger of the biological data revolution with computing power has created another serious security concern: ultra-targeted biological warfare. In past biowarfare programs, weapons targeted their intended victims through geographic location. Advances in biotechnology open up the possibility that malicious actors could deploy a biological weapon over a broad geographic area but only affect targeted groups of people, or even individuals.

The possibility of such “genetic weapons” was first discussed in the biological arms control community in the 1990s, as the Human Genome Project to map the full complement of human genes got underway. The UK government said “it cannot be ruled out that information from such genetic research could be considered for the design of weapons targeted against specific ethnic or racial groups.” The British Medical Association cautioned that “the differential susceptibility of different populations to various diseases” had been considered in the past, and that “whilst we should hope that genetic weapons are never developed, it would be a great mistake to assume that they never can be, and therefore that we can safely afford to ignore them as a future possibility.” A report from the Stockholm International Peace Research Institute (SIPRI) spoke of the potential for “future development of weapons of mass extermination which could be used for genocide.”

Developments in genomic technologies and other emerging technologies, especially machine and deep learning, have spurred renewed concerns. “Access to millions of human genomes—often with directly associated clinical data—means that bioinformaticists can begin to map infection susceptibilities in specific populations,” a recent report from the United Nations Institute for Disarmament Research warned. A United Nations University report, meanwhile, asserts that “deep learning may lead to the identification of ‘precision maladies,’ which are the genetic functions that code for vulnerabilities and interconnections between the immune system and microbiome. Using this form of bio-intelligence, malicious actors could engineer pathogens that are tailored to target mechanisms critical in the immune system or the microbiome of specific subpopulations.” A 2018 National Academies of Sciences report suggests “[a]ctors may consider designing a bioweapon to target particular subpopulations based on their genes or prior exposure to vaccines, or even seek to suppress the immune system of victims to ‘prime’ a population for a subsequent attack. These capabilities, which were feared decades ago but never reached any plausible capability, may be made increasingly feasible by the widespread availability of health and genomic data.”

It is important to note that there are barriers limiting access to targeted biological weapons. The technical base, expertise, and funding required for the design of a targeted biological weapon suggest that only a significantly resourceful and motivated actor would be likely to explore this possibility.

Re-envisioning biological disarmament

Experts at SIPRI have suggested that, because of the complexity required to create them, ultra-targeted biological weapons are relatively unlikely to be used: “If the purpose is to harm a specific individual or group, most malevolent actors would surely resort to more low-tech or direct methods, such as firearms or poison.” This suggestion may be accurate, but it is not, unfortunately, a sufficient basis for biological arms control in the 21^st century. As one of the great champions of biological disarmament, Matthew Meselson, professor of molecular biology at Harvard University, reflected in 2000 as he contemplated the century ahead in an essay on averting the hostile exploitation of biotechnology: “[A]s our ability to modify fundamental life processes continues its rapid advance, we will be able not only to devise additional ways to destroy life but will also become able to manipulate it—including the processes of cognition, development, reproduction and inheritance… Therein could lie unprecedented opportunities for violence, coercion, repression, or subjugation.”

The current disarmament regime, the Biological Weapons Convention, has been in force since 1975. The treaty comprehensively prohibits biological weapons, understood as biological agents used for harmful purposes, and countries that are party to the treaty agree that it unequivocally covers all microbial or other biological agents or toxins, naturally or artificially created or altered, as well as their components, whatever their origin or method of production.

On the whole, this covers the pathogen-specific risks and risks of ultra-targeted biological weapons. Indeed, the UK government, which first raised the issue of genetic weapons as a possibility in the mid 1990s, specifically stated that genetic weapons would be a “clear contravention” of the treaty. Cyber-biosecurity risks are not covered by the BWC, but the BWC and arms control treaties more generally are not appropriate instruments to address these sorts of risks.

Where there might be some uncertainty around the coverage of the BWC is where the harms do not involve biological agents. Developments in science and technology are making novel biological weapons conceivable that, instead of using bacteria or viruses to make us sick, directly target the immune, nervous or endocrine systems, the microbiome, or even the genome by interfering with, or manipulating, biological processes. This could be achieved, for example, by using a construct based on synthetic structures created or inspired by DNA or RNA, but not qualifying as DNA, RNA, or any other known, naturally occurring nucleic acid. In this sort of case, the coverage of the BWC is less clear, but the intent of the treaty to prohibit such harm is beyond doubt.

The real challenge for the treaty, however, is not in its coverage but in ensuring that countries comply with it and live up to their obligations. This oversight is particularly difficult because the relevant materials, equipment, and technical know-how are diffused across multiple and varied scientific disciplines and sectors—and they are increasingly in private, rather than public, hands. Moreover, biological agents themselves exist in nature and are living organisms generally capable of natural reproduction and replication.

The dual-use nature of biology and the challenges it poses for compliance assessment was recognized in the early phase of the negotiations on the treaty. In a 1968 statement to the predecessor of the Conference on Disarmament, the United Kingdom noted, for instance, that no verification is possible in the sense of the term as we normally use it in disarmament discussions. In other words, it was not considered possible to verify the BWC with the same level of accuracy and reliability as the verification of nuclear treaties like the Treaty on the Non-Proliferation of Nuclear Weapons (NPT), which was negotiated immediately prior to the BWC. Consequently, Article I of the BWC—through which states “agree to never under any circumstances acquire or retain biological weapons”—is therefore vague in demarcating the borders of prohibited and legitimate activities. Article I merely refers to biological agents “of types and in quantities that have no justification for prophylactic, protective, or other peaceful purposes.”

The main responsibility for compliance assessment in the BWC falls on the countries that are party to it, unlike in its sister-conventions, the Chemical Weapons Convention and the NPT, where compliance assessment is tasked to the Organisation for the Prohibition of Chemical Weapons (OPCW) and the International Atomic Energy Agency (IAEA). In the BWC, each country relies on its own resources to assess other countries’ compliance. The United Nations Security Council acts as the final arbitrator on allegations of compliance breaches—though it has not to date been requested to investigate any allegations. The role of the treaty’s Implementation Support Unit is purely to support countries in their efforts to implement the BWC.

The “general purpose criterion” of the BWC means the treaty permits almost any kind of biological research for defensive or protective purposes. Some such work is justifiable. Other research edges closer to the blurred line between defensive and offensive work. Distinguishing permitted biodefense projects from those that are prohibited is difficult; one cannot just assess the facilities, equipment, material, and activities involved, but must also examine and interpret the purpose, or intent, of those activities.

A series of significant and accelerating advances in abilities to manipulate genes and biological systems have made Cold War-era tools of compliance assessment increasingly outdated. Among those new abilities are:

• The addition of machine and deep learning to bioinformatics.
• The coupling of those developments with “cloud labs,” third-party firms that are centralizing and scaling the wet work of genetic engineering.
• Research on deliberately releasing viruses into the environment.
• The rise in biodefense programs and the build-up in those programs’ capacities.

To establish the intent of biological research, it is not enough to simply count fermenters, measure the sizes of autoclaves, and limit amounts of growth media. A growing number of countries recognize that biology, to a large extent, defies material accountancy-type verification methodologies. The United Kingdom, for instance, recently noted that BWC compliance is “much more one of transparency, insight, and candour, rather than material balances or counting discrete objects such as fermenters.”

Somewhat ironically in our ever-increasing digital world, the last few years has seen a move away from strictly quantitative approaches and binary models of compliance assessment in biological arms control toward more qualitative methods. Leading countries are exploring means of demonstrating good practices and responsible science through new voluntary initiatives that enable them to demonstrate transparency and build trust—initiatives such as peer review, implementation review, and transparency visits. These information-sharing initiatives emphasize interaction and flexibility, expert-level exchanges of best practices rather than just on-site monitoring, and a broad conception of relevant laboratories and facilities—and they have been deemed to add real value to compliance judgements by participating states.

Similarly, as a way to complement laws and regulations around biosecurity, civil society groups have led the development of norm-building controls like codes of conduct, prizes, awards, competitions, and other incentives for good behavior. The flip-side—leveraging reputational risks, corporate shaming, and social pressures for poor behavior—is also beginning to be explored. It has become abundantly clear that, in the Fourth Industrial Revolution, compliance with the Biological Weapons Convention needs to be less about verifying a binary state—being “in compliance” or “not in compliance”—and more about analyzing justifications provided for the activities in question and managing dual use potential.

For biological arms control and the UN more generally, the broader challenges involve extending the management regime to stakeholders other than countries, particularly to private industry and civil society groups but also others, and maintaining contemporary relevance as the global forum for security debates around emerging technologies. Technologies, like biotechnologies, that have traditionally been siloed in elite institutions and national labs and monitored by national governments are now increasingly accessible to and even controlled by private tech platforms and research communities around the world. In the era of AI, limiting access to intangible knowledge and computerized tools involving dual use research will only become more difficult.

There is a narrowing window of opportunity to structurally evolve biological arms control. One way to do this—and to link the biological field with other emerging technologies—is to actively encourage collaborations across the fields of AI, cyber, and biotechnologies to develop responsible security practices through which scientists learn about each converging field and its impact on dual use research. Yet, on its own, this type of collaboration is not enough to protect the world from misuse of powerful and converging technologies.

One idea for improving the management of broad and fast technological advances involves a World Economic Forum-like “network of influence,” composed of exceptional individuals from business, academia, politics, defense, civil society, and international organizations, to act as a global Board of Trustees to oversee developments relevant to biological threats in science, business, defense, and politics and to decide on concerted cross-sector actions. A similar idea—not limited to the biological field but cutting across emerging technologies—would develop a “Global Foresight Observatory” comprising a constellation of key public and private sector stakeholders convened by a strategic foresight team within the UN. The Board of Trustees or the Global Foresight Observatory could be supplemented by a secondary oversight layer that enrolls individuals and select institutions in key positions to act as “sentinels.” These sentinels could have dual functions: first, to actively promote responsible science and innovation, and second, to identify security risks for consideration by the Board or the Observatory.

These new governance structures could be supplemented by political initiatives—AI and bioinformatics groups, for example—to establish a new type of transparency, confidence-building, and BWC compliance assessment, and to support the prevention of biological weapons development and the management of dual use biological research. The colossal challenges of converging technologies will require bold ideas like these to re-envision future biological arms control.