A Reddit user recently found out that the A.I. agent their company had been using for analytics assistance since November had been making up numbers the entire time. The company’s board was shown a deck “with fake insights,” and a V.P. of sales “made territory decisions based on data that didn’t exist.” The Redditor continued: “The A.I. was just inventing plausible sounding percentages. I only caught it by accident when someone asked me to double check something. I started digging, and holy shit, it’s bad.”

And now for the main event…

Dario Amodei, the co-founder and C.E.O. of Anthropic, spoke bluntly last week when he was asked whether the company’s A.I. models could be used for authoritarian purposes—telling The New York Times that he worries about autonomous weapons and mass surveillance making a “mockery” of the First and Fourth amendments. “Constitutional rights and liberty along many different dimensions can be undermined by A.I. if we don’t update these protections appropriately,” he warned.

It was a somewhat extraordinary statement from an executive whose technology has already been deeply embedded into the U.S. military. Last summer, the Department of Defense handed out sizable contracts to Anthropic, xAI, OpenAI, and Google. More recently, the department announced the launch of GenAI.mil, a platform intended to give department personnel secure access to A.I. models and facilitate their rapid deployment. “A.I. is America’s next Manifest Destiny,” said Pentagon C.T.O. Emil Michael, the former chief business officer of Uber, at the time. “And we’re ensuring that we dominate this new frontier.” Indeed, the Pentagon has been pushing for A.I. models to be quickly accessible across classified networks, as well. In the intervening months, the frontier labs on the receiving end of those contracts have been working through the mandate. In December, Google released a vague statement suggesting that its models would be available on unclassified military applications. A few weeks later, xAI noted that Grok would support both “enterprise A.I. and critical mission use cases” but pointed to future support for classified workloads. Most recently, OpenAI announced that it was bringing a specially designed version of ChatGPT to GenAI.mil for use, at least initially, on unclassified work. (The model’s safeguards, OpenAI said, were intended to support “all lawful uses” of its technology.) Anthropic, whose models are the only ones currently approved for classified military use, has yet to make a similar announcement. For months, Anthropic has been locked in negotiations with the Department of Defense (which the administration calls the Department of War) over concerns that the military might use its models to power fully autonomous weapons systems or enable the mass surveilling of U.S. citizens. But the Pentagon’s patience is apparently at an end: On Monday, Axios reported that Defense Secretary Pete Hegseth is on the verge of severing the department’s relationship with Anthropic, and might go so far as to designate the company a “supply chain risk”—a veritable economic death sentence. The designation, usually reserved for foreign actors, would prevent federal contractors from using the company's models on DoD systems, and would certainly seem to undercut its recent $380 billion valuation. It’s difficult to tell how big of a blow this would be. “The Department of War’s relationship with Anthropic is being reviewed,” Pentagon spokesman Sean Parnell told me in an emailed statement. “Our nation requires that our partners be willing to help our warfighters win in any fight. Ultimately, this is about our troops and the safety of the American people.” According to Axios, Anthropic is willing to relax its usage restrictions—but doesn’t seem interested in giving ground on the two major sticking points concerning autonomous weapons and domestic surveillance. The Pentagon, meanwhile, is insisting that its A.I. partners allow the military to leverage their tech for “all lawful purposes.” And so now we have a far more consequential and financially severe version of ABC’s George Stephanopoulos bribe or CBS’s 60 Minutes settlement—an example, perhaps ham-handed, of the administration reminding a defiant private enterprise that there’s only one boss these days. Can Amodei, who left OpenAI to found Anthropic partly out of his moral concerns about the former’s direction, preserve the immense value of his super-unicorn even if it means adopting the moral dexterity required of large-cap C.E.O.s in the Trump II era? Well, after this brushback pitch, it sounds like he’s getting there. An Anthropic spokesperson told me that the company remains “committed to using frontier A.I. in support of U.S. national security. That’s why we were the first frontier A.I. company to put our models on classified networks and the first to provide customized models for national security customers.” Claude, the spokesperson continued, “is used for a wide variety of intelligence-related use cases across the government, including the DoW, in line with our Usage Policy. We are having productive conversations, in good faith, with DoW on how to continue that work and get these complex issues right.”

I recently discussed this policy battle with Elke Schwarz, a professor of political theory at the Queen Mary University of London and an expert on military A.I. use. She noted that “the bar Anthropic has set is already quite low—the mass surveillance of Americans and fully autonomous weapons (rather than, say, mass surveillance more generally, and the use of Claude for A.I.-weapon systems).” But to her, the “real story here is that other companies—OpenAI, Google in particular—have agreed to lower their guardrails.”

Indeed, the past few years have seen restrictions around military use of A.I. gradually falling away. Google, famous in the early 2000s for its since-abandonned “Don’t be evil” motto, had a long-running pledge that it would not pursue A.I. applications in weaponry or surveillance and would generally avoid enabling “technologies that cause or are likely to cause overall harm.” But those promises faded as the industry has evolved and expanded—and as the political climate has changed. Those fighting the wars in Gaza and Ukraine took advantage of A.I. technologies from major companies to direct drones and select bombing targets, among other things. Today, despite mentions of “responsible” deployment and development, Google’s A.I. Principles make no mention of off-limits applications, harms, or autonomous weaponry. Similarly, OpenAI edited its own usage policies page in 2024, deleting an earlier rule that would have prohibited the use of ChatGPT for “military and warfare” applications. (Neither OpenAI nor Google replied to requests for comment.) Earlier this week, the Journal reported that Claude had been used in the U.S. operation to capture former Venezuelan President Nicolás Maduro. The push to drive the rapid adoption and proliferation of commercial L.L.M.s across the military comes despite expert concerns over bias, reliability, security, and, ultimately, accountability for military decision-making. In keeping with the Trump administration’s approach to A.I. and policymaking in general, the DoD’s interpretation of “responsible A.I.” is focused on ideology, rather than safety or efficacy. “We must not employ A.I. models which incorporate ideological ‘tuning’ that interferes with their ability to provide objectively truthful responses to user prompts,” Hegseth wrote in a January memo. The memo further stated that the DoD “must also utilize models free from usage policy constraints that may limit lawful military applications.”

This isn’t the first time Anthropic has been caught in Trumpian crosshairs. Several months ago, White House A.I. czar David Sacks accused Anthropic of having an “agenda to backdoor Woke A.I. and other A.I. regulations through Blue states like California.” The backlash at the time was so intense that Amodei felt the need to publish a fresh blog, reasserting Anthropic’s “commitment to American A.I. leadership.” Amodei, a Democratic donor, has attempted not to talk much about politics since Trump returned to the White House. But it hasn’t gone unnoticed that neither Amodei nor Anthropic has pledged money to the administration. (Google, Microsoft, Meta, Palantir, Apple, Amazon, and Nvidia, meanwhile, are all helping to build the ballroom.)

But this latest threat from the DoD puts Anthropic in a bit of a tough spot. If they bend to the Pentagon’s wishes, they’ll surely minimize risk to their business—which Anthropic has previously said comes predominantly from enterprise customers. As Amodei looks to ensure a pragmatic approach to scaling compute alongside revenue growth, a major loss that could result from getting blackballed by the DoD certainly wouldn’t help. And yet, Anthropic has a reputation to uphold as a safety-oriented, human-centric A.I. lab—a reputation that might be tarnished if it were to compromise its usage policies like its peers. “For Anthropic,” Schwarz told me, “this is a real test as to how firm the lines are and where they might see the benefit (financially) in holding those lines (or the need to loosen them).” At the moment, it’s hard to see either side giving in, but the financial power of a department with a near-trillion-dollar budget has a way of focusing the mind. The most likely outcome is that Anthropic will fold, perhaps while insisting that its core principles haven’t changed, or that it’s morally preferable for the Pentagon to deploy its tech than that of a less scrupulous competitor. But it’s hard to know what kind of long-term brand damage Anthropic might incur, beyond the opportunity cost of a $200 million contract. Anthropic’s outlook and approach to A.I. safety is its core differentiator from OpenAI, and indeed, the very reason it was founded in the first place. To capitulate here would seem to undermine the company’s founding mission; but money has a logic of its own.

The recent release of some A.I. tools for legal and financial software is seemingly driving an ongoing sell-off across stock markets. But when it comes to actually automating the law, it is likely not as easy as releasing a chatbot—advanced, agentic, or otherwise. [Lawfare]

It’s been more than three years since ChatGPT burst onto the scene, amid promises of radical disruption and economic transformation. But the impact of A.I., despite the trillions that have been poured into it, has yet to show up in clear economic data. “It looks like A.I. will likely be labor enhancing in some sectors rather than labor replacing in all sectors,” economist Torsten Slok said. [Apollo] Software engineering is more than code generation; much of it involves the maintenance of codebases. A recent controlled trial sought to study the difference between A.I.-enabled code generation and A.I.-enabled code maintenance. And it found… pretty much no impact. [Modern Software Engineering] At least two police departments have purchased access to an A.I. tool called GeoSpy, which, as the name suggests, analyzes a range of inputs to geolocate photos. [404 Media]

That’s all for today. I’ll see you on Thursday.

Ian

Need help? Review our FAQ page or contact us for assistance. For brand partnerships, email ads@puck.news.

You received this email because you signed up to receive emails from Puck, or as part of your Puck account associated with {{customer.email}}. To stop receiving this newsletter and/or manage all your email preferences, click here.

Puck is published by Heat Media LLC. 107 Greenwich St., New York, NY 10006