Scientists handpicked by the Trump administration to write a critical review of climate science used various LLMs to review their work, only to find that AI told them they were cherry-picking their facts and using “outdated or narrow sourcing.”
Earlier this year, a judge ruled that the Energy Department and Secretary Chris Wright violated the law when he convened a group of climate experts to write a critical review of climate science and the impacts of greenhouse gas emissions. Specifically, the judge determined the agency violated the provisions of the Federal Advisory Committee Act that stipulate advisory groups must be open, transparent, and balanced.
The judge ordered emails and other documents related to the Climate Working Group report released to the environmental advocacy groups that sued the agency, and in March, the Environmental Defense Fund and the Union of Concerned Scientists released those documents to the public—all 4,167 500-page PDFs.
Among other things, the documents reveal how the five experts in the Climate Working Group incorporated AI into their workflow—as research assistant, editor, pseudo-peer reviewer, and imaginary critic. They used the feedback to tweak their language to sound more neutral, anticipate possible objections, and even asked it to compose poems criticizing a congressionally mandated climate report.
“The period when we worked on the CWG report coincided with everybody just starting to try and figure out AI and LLMs,” Canadian economist Ross McKitrick wrote in an email to the Bulletin.“Like everyone we were curious about it, but we didn’t rely on it. We certainly did not delegate the writing to Copilot or any other AI platform.”
It was Steven Koonin’s idea to upload the group’s work to see what an LLM would make of it, according to McKitrick. “You couldn’t help but be impressed at what these models were already capable of, but we did our own research and writing,” he wrote.
The other members of the group did not respond to requests for comment.
Although there’s been much discussion about how AI-assisted writing and research has become more common in all fields, including scientific research and publishing, it’s rare to get an unfiltered look at exactly how people are using it behind-the-scenes.
Most of the time, the Climate Working Group turned to LLMs for prosaic tasks.
For example, while drafting the report, Steven Koonin—a former chief scientist at BP who worked in Obama’s Energy Department and is now a fellow at Stanford’s Hoover Institution—asked Microsoft Copilot to “distill the current executive summary into 1 page of text.”
“Not too bad, but with some flubs,” he wrote the group. “I’ll try some other LLMs and send around a more serious draft by Friday.”

Koonin also used an LLM to rework a section to avoid self-plagiarism. “Because the current climate modeling box in Chapter 5 reproduces verbatim most of what I’d published in a popular article, I reworded it by editing the results of [sic] running it through an LLM,” he wrote.
More significantly, Koonin asked Copilot to review their report for “scientific accuracy” and “potential bias.”
Copilot replied with observations like, “Some of the cited studies, while valid, don’t reflect broader scientific consensus” and “[e]vidence is often cherry-picked to support a CO2-positive view.”
It also said, “Words like ‘exaggerated,’ ‘one-sided,’ ‘alarmist,’ and ‘science fiction’ appear frequently. This rhetorical tone may compromise the objectivity typically expected in a scientific review.” And it pointed out the repeated use of “skeptical phrases” like “naïve to assume” and “misleading metric.”

The feedback prompted the group to try to edit out “skeptic talk” and “red flag words,” although most of the latter, climate scientist Judith Curry noted, were in Secretary Wright’s foreword.
But in general, they were quite happy with the AI’s review. “This is amazing, far better than what we would get from ‘real’ scientists,” Curry wrote.
“Yes, I’m also amazed at how good it is,” Koonin replied. “Perhaps Al really will eliminate most current white-collar jobs!”

As the group was finalizing the report, Koonin asked Copilot to “craft a rebuttal OpEd” to anticipate what their critics might say. When he shared the draft with the group, he added, “Perhaps worth some pre-emptive wording in our final.”
The group continued to lean on AI after the report was published and as they worked more on their follow-up project: a critique of the Fifth National Climate Assessment, the congressionally mandated report on climate change impacts, risks, and responses by the US Global Change Research Program. This second report was never published, because the Climate Working Group was disbanded after the Environmental Defense Fund and the Union of Concerned Scientists filed the lawsuit alleging violations of the Federal Advisory Committee Act, or FACA.
Shortly after that, Curry asked Grok, “is the DOE Climate Working Group a FACA? Provide arguments for and against.”
After “thinking” for 49 seconds, Grok did as it was told, replying, for example, that the Climate Working Group may be subject to FACA because it was “explicitly assembled as a group to produce a unified report advising the DOE (and indirectly the EPA) on climate science and policy implications, fitting this criterion per DOE’s own internal guidance.” On the other hand, it countered, “FACA typically applies to chartered groups with structured operations, and the CWG lacks such designation, potentially classifying it as ad hoc individual consultations rather than a collective advisory body.”
At multiple points, the experts seemed to demonstrate a growing frustration with LLMs, as the AI responses continued to show a greater preference for “consensus” sources rather than contrarians like themselves.
“Copilot doesn’t seem to see the difference between self-citation and data transparency,” McKitrick complained, after Koonin sent around Copilot’s review of their Fifth National Climate Assessment critique.
A day later, he added that LLM critiques can be “helpful in anticipating lines of attack, but they are overly algorithmic” and often conclude that the Climate Working Group authors “make a valid point” but since it’s not the consensus “it’s probably wrong.”
“It seems to me that if CoPilot swims in a pool with lots of chaff and little wheat, it will be convinced that the chaff must be considered valid and therefore is unable to discern the correctness of arguments against the chaff,” climate scientist John Christy concurred in a different thread.

Meteorologist and climate scientist Roy Spencer suggested pushing back if the AI tools weren’t providing satisfactory responses. “Yes, LLMs mostly summarize what’s on on [sic] the InterWebs,” he wrote, after Christy complained they were “skewed toward volume.”
“If the interwebs agree we are evil, that’s what you will get back,” he continued. “BUT.. Grok is pretty good about noting dissenting voices. Also, you might need to tell it to ‘dig deeper’. Some persistent folks can even get it to change its mind.”

After their report was published, the group fielded a query from the New Yorker about AI use, which claimed “unnamed scientists” had said their work had been cited incorrectly, and wondered if that was because the authors had used AI.
“Were any AI tools used?” asked Andrea Woods, a deputy director in the Energy Department Office of Public Affairs who has since returned to work in communications at the American Petroleum Institute.
“At one point, I used an LLM analysis to ‘de-snark’ the nearly final text,” Koonin replied. “But, to my knowledge, there was no AI use for substantive content or citations.”
“Agreed,” Christy said. “AI was not used to influence the content.”
“Correct,” McKitrick added. “AI did not determine the content.”
There does not appear to be any evidence that the Climate Working Group members used AI to generate text or citations wholesale, although some scientists have pointed out citations that mix-up real authors, journals, and titles in a way that suggests AI may have been used to generate full citations from partial notes or links.
By far the most surprising and creative use of AI was when Koonin asked Copilot to write a limerick criticizing the Fifth National Climate Assessment.

Copilot seemed to assume that the main criticism to be leveled at the National Climate Assessment was that it didn’t call for climate action strongly enough.
Koonin said this “didn’t have the sense I’d expected” and asked for a redo, focusing on how misleading the report was.
The second limerick was apparently more to Koonin’s liking, because he suggested using it as their preface.
This story was updated to include details about potentially AI-generated citations.

Leave a Reply