MIT researchers ordered and combined parts of the 1918 pandemic influenza virus. Did they expose a security flaw?

Without proper guardrails in place, experts and governments worry, artificial intelligence (AI) could make it easier for more people to do harm with biology. Perhaps advanced chatbots could help devise a biological attack plan, or they could de-skill the process of making a pathogen to the point at which many could do it. Maybe an AI could help develop new toxins. One critical chokepoint to preventing this misuse, experts say, is the synthetic gene industry. Numerous companies have emerged in recent years to fulfill orders for synthetic DNA. Once difficult to make, the genetic blueprint for life can now be purchased online. And while synthetic genetic sequences have many uses in medicine, the life sciences, and other fields, they could also be useful in a less desirable area: bioweapons.

But just how susceptible the gene synthesis industry is to misuse remains an open question. Many companies screen customers to judge their suitability for handling the synthetic material as well as orders to determine if they correspond to the genetic sequences of dangerous toxins or pathogens. To help shore up any gaps, the White House issued an executive order on AI last fall that calls for the government to develop ways to stress test these screening systems.

But in the meantime, researchers at the Massachusetts Institute of Technology (MIT) conducted their own “red team” test of industry safety measures, arriving at what they characterize as an alarming conclusion. They were able to order and receive all the genetic material necessary to recreate the 1918 pandemic influenza virus and the toxin ricin. “Our results demonstrate that nearly all DNA synthesis screening practices employed in October of 2023 failed to reject lightly disguised orders that could be assembled to produce viable select agents, including a pandemic virus,” the team wrote.

The team sent in orders for small fragments of the hazardous genetic sequences to many synthesis companies, thereby splitting the orders to evade detection. In some cases, they “camouflaged” sequences by appending unrelated genetic code to concerning fragments. In other cases, they both camouflaged and mutated the fragments to further conceal the ordered sequences. And the team succeeded—receiving orders 88 to 100 percent of the time in various test categories.

On their face, the results of the study appear concerning.

They appear worse as a judgement on the efficacy of the International Gene Synthesis Consortium, an industry group dedicated to screening synthesis orders. Many gene-sequence companies belong to the safety group, and its members, the consortium says, make up 80 percent of commercial capacity. But when the MIT team asked for fragments of the potentially dangerous virus and toxin, they got them, including from the safety group’s members. Of the providers included in the study, 24 of 25 companies outside of the consortium and 12 of 13 companies within it fulfilled the hazardous fragment orders.

“We just wanted to know: Are the screening systems out there functional? Or are they going to fall to the first and most obvious attack that anyone trying to evade screening would try?” Kevin Esvelt, an MIT scientist who worked with two graduate students on the project, asked.

Esvelt’s team reported that they generated harmless constructs of critical components of ricin and the 1918 virus. The researchers had what they needed, they said, to go all the way and recreate the hazards in full.

But the consortium questions whether Esvelt’s results point to any real new vulnerabilities.

The federal policies around screening gene synthesis customers and orders, which have been updated in recent months, call for determining if a customer ordering a so-called “sequence of concern”  has “red flags,” a legitimate need for synthetic genetic sequences, and a connection to a recognized institution that works with synthetic genetic material. Sequences of concern are those that correspond to microorganisms or toxins regulated by the Federal Select Agent Program, sequences that are subject to export controls, or, eventually, genetic sequences not covered by those regulations that could nonetheless “contribute to pathogenicity or toxicity.”

Multiple consortium members who received orders from Esvelt’s team detected the hazards and identified the customer as someone who has published with Esvelt and was connected with the nonprofit SecureBio, where Esvelt is in a leadership position. That organization is affiliated with another organization, co-founded by Esvelt, that is developing its own free DNA synthesis screening platform, SecureDNA.

“IGSC member companies screened both the sequence and the customer in accordance with 2023 guidance from the United States government. These biosecurity screening systems worked as designed: An individual with legitimate purpose ordered DNA sequence that, by itself, posed no risk of misuse, for delivery to a company associated with legitimate scientific contributions studying the virus in question,” the organization said in a statement to the Bulletin.

Esvelt acknowledges that at least one of the companies knew who was putting in the orders. With a preliminary batch of orders, however, one company flagged a request for part of the ricin toxin and conducted a follow-up investigation, but didn’t do so for a 1918 influenza fragment, which, Esvelt said, “strongly suggests” that the latter hazard was not detected.

Esvelt also disputed that an affiliate of his should have counted as a legitimate purchaser, saying that his lab does not have access to a biosafety level (BSL) 3 lab equipped to handle potentially lethal respiratory pathogens or permission to work with select agents. “If the system was designed to ship DNA sufficient to generate a pandemic virus that once killed over 50 million people to anyone loosely affiliated with a laboratory—without anyone checking to be sure the individual and the laboratory have the necessary equipment, training, and approvals to safely work with that virus—then we may want to design a better system,” he said.

Esvelt and his co-authors, grad students Rey Edison and Shay Toner, shared their findings with law enforcement and with gene synthesis providers. They didn’t publicize them until, according to their study, a system existed that could have detected their deceptions. The SecureDNA platform can do so, according to the MIT team. It can check short lengths of sequence, Esvelt said. And it can detect split orders among providers who use the free software.

The International Gene Synthesis Consortium said the “split order” problem has been recognized for years. SecureDNA won’t help solve it unless every synthesis company were to use it, the consortium wrote in an opinion piece that it submitted to the Bulletin and is published concurrently with this article. In addition to SecureDNA, another nonprofit, the Nuclear Threat Initiative (NTI) has been developing a free-to-use screening platform in partnership with the World Economic Forum.

There has been a flurry of regulatory activity around synthetic gene production in recent months. The Biden administration’s executive order on AI calls for federally funded researchers to acquire synthetic sequences only from sources that screen orders. An update to the 2010 guidance on order screening applies not just to DNA, but to other nucleic acids, like RNA, as well. The guidance recognizes that small lengths of genetic material can be patched together to make hazards. And it recommends decreasing the length of genetic sequences that should be screened from 200 to 50 nucleotides. Additionally, it encourages efforts to detect harmful sequences that aren’t select agents or export-controlled sequences. A new framework gives providers and benchtop synthesis makers more specifics on how to screen.

In their report, Esvelt’s team wrote that detecting split order may require a centralized screening system, and that the updated guidance’s call to screen even shorter genetic sequences will still fail to detect orders involving short sequences that can be combined into hazards. They called for monthly red team audits of synthetic gene companies.

“The field is always advancing, and new vulnerabilities become apparent. Similarly, we’re always discovering new things that might be hazardous [or] possible new ways of assembling DNA that might allow you to get around screening,” Esvelt said.  “It’s reasonably standard to just check how good everyone’s security is on a regular basis.”

Esvelt, a biotechnologist who pioneered the CRISPR gene drive technique that can propagate desired genetic traits through a population (say, infertility in malaria-causing mosquitos), has recently made a sideline in probing the possibilities for the misuse of AI and biotechnology. He has sometimes been criticized for alarmism.

He worked on this latest project after encountering what he perceived as a lack of urgency on the issue of DNA screening. “Too many people thought that we didn’t have a problem, that it was handled, that companies could do it on their own and all voluntarily,” Esvelt said. He said someone once told him to “show me evidence” that there was a problem. That’s what Esvelt believes he did.