“I do not fear the rise of superintelligence; I do, however, fear the rise of rapacious and selfish individuals, organizations, and governments who seek to use computing in any form to extend their power and control over others.” —Grady Booch, a storied IBM fellow and critical researcher, on X, explaining that his “prediction of doom,” otherwise known in the industry as “p(doom),” has less to do with A.I.’s technological capabilities than with the people developing it.

A recent survey of teens by Common Sense Media found that A.I. companionship is starting to become mainstream. The organization found that 72 percent of surveyed teens have used an A.I. companion at least once, and more than half are “regular” users. Still, the survey found that most users approach the bots “pragmatically, rather than as substitutes for human relationships.” Half are distrustful of A.I.-generated advice, 80 percent of those who use A.I. companions still spend more time with their real friends, and 67 percent of respondents find conversations with actual humans more satisfying. (Yes, I found that a bit low, too...)

As the technology underpinning generative A.I. became more advanced, perhaps it was inevitable that chatbots would be leveraged to sell artificial companionship. The chatbot company Replika, for instance, sells “the A.I. companion who cares,” one that’s “always here to listen and talk.” Character A.I.’s tagline used to be “A.I. that feels alive,” before eventually evolving into “interactive entertainment.” (The company, as you recall, has been the target of litigation—including a lawsuit related to the tragic suicide of a 14-year-old boy who was allegedly influenced by his interactions with characters on the app. At the time, Character said it was “heartbroken by the tragic loss of one of our users,” and detailed the addition of a number of new safety features.)

Michael Robb, Common Sense’s head of research, told me he was surprised by the survey results—namely, that so many teens are seeking out this form of companionship this early in the A.I. era. “My concern is always about displacement or replacement of human relationships,” he told me. “Teens need high-quality human interaction with peers, with trusted adults, with parents, in order to have healthy development. If they decide they want to spend more time with A.I. companions than humans, that kind of displacement could result in some pretty bad outcomes.” He added that he’s particularly concerned about kids who are already isolated. By seeking out A.I. companions, they might inadvertently isolate themselves further.

And now, on to the main event…

Last week, a relatively small A.I. safety research nonprofit called Metr published the results of a study exploring one of the more highly touted use cases for artificial intelligence: coding assistants, and particularly how they perform “in the wild.” The results didn’t necessarily break the internet, but they did garner millions of views and thousands of reposts on X. That’s because the main takeaway was that, among the software developers who participated—nearly all of whom expected A.I. coding tools to speed up their workflow by 20 percent—the tools in fact slowed them down by 19 percent.

In benchmark tests, of course, A.I. coding tools tend to perform incredibly well. And while there seems to be widespread real-world adoption and plenty of hype around the potential for these tools to improve coders’ workflow, there’s been little empirical evidence to support it. Meanwhile, the A.I. software development market—worth a modest $674 million in 2024, according to Grand View Research—continues to be a point of heavy investment, and is expected to grow at a 42 percent annual rate until 2033, when it will potentially be valued at around $15 billion. This year, both Google and Microsoft indicated that around a third of their respective codebases are A.I.-generated. In short, the excitement around A.I. coding companies is far from fading.

When I caught up with Joel Becker and Nate Rush, two of the study’s principal architects, they told me that they were as surprised by the results as anybody else. Metr had recruited a group of 16 experienced software developers, who were asked to complete a total of 246 tasks that, on average, took about two hours each. Each task, at random, either allowed or disallowed the use of an A.I. coding tool. (Becker noted that they used early-2025 coding tools that have since become more capable, which gives you a sense of how quickly this technology is progressing.)

In their paper, the researchers laid out 20 factors that might have contributed to the slowdown in completion rate when using A.I. tools. And though Becker and Rush said it’s essentially impossible to determine the weight of one factor over another, two of the more likely ones involved over-optimism around usability—that the coders placed too much faith in the tool to help them complete the task, and therefore took a backseat—and the low reliability of the tools, themselves. One developer also said that some of his speed gains were eaten up by time spent circling the social media black hole in between prompts. Another, who experienced a 38 percent slowdown when using A.I., said, “We like to say that L.L.M.s are tools, but treat them more like a magic bullet.”

Rush and Becker, who plan to conduct additional studies on this same subject in the future, were quick to say that these results are not necessarily generalizable beyond this particular test setting—experienced software developers working with early-2025 tools in code bases with which they were already familiar. “We certainly don’t think that this applies everywhere,” Rush said. “We see it as a demonstration that self-reports are not particularly reliable.”

The study, according to Dr. Armando Solar-Lezama, a distinguished professor at the MIT Schwarzman College of Computing, has been “really valuable” in advancing our understanding of A.I. coding tools beyond simple benchmark tests. Still, he noted a number of caveats, including the small scale of the study, and the fact that the models they were using were likely trained on that code. (The developers worked on open-source projects.) “So [the models] have a level of expertise on that code that they don’t have on internal, proprietary company code, where a lot of commercial development is taking place,” he told me.

As an aside, Solar-Lezama noted that the findings aligned with his own experience of using A.I. coding tools, primarily Cursor. At first, he explained, the tool “really gets you hooked. But then once you start using it, you find that, well, actually, not so much.” He continued: “Sometimes you get a big win, and sometimes you realize you wasted a lot of time reading through code that is not really what you actually wanted to do.”

His observations touched on the perennial issue of A.I. reliability, and underscored the fact that, as Solar-Lezama told me, the “need for people to gain deep expertise in the topics they work on is not going away.” In the study, for example, the developers accepted less than 44 percent of A.I.-generated code, meaning they were able to identify bad code before implementing it. Solar-Lezama envisioned a scenario in which less-experienced developers implement bad code, then take quite a while to find, understand, and rectify the errors.

Perhaps there is some future in which a C.E.O., for instance, could prompt a model to generate a fully functional, ready-to-deploy app. But Solar-Lezama noted that there are still a “lot of roadblocks” before we get to that point. “I don’t think that’s going to happen in my lifetime,” he said, chuckling. “We’ll see.”

I’ll see you next week.
Ian

Need help? Review our FAQ page or contact us for assistance. For brand partnerships, email ads@puck.news.

You received this email because you signed up to receive emails from Puck, or as part of your Puck account associated with {{customer.email}}. To stop receiving this newsletter and/or manage all your email preferences, click here.

Puck is published by Heat Media LLC. 107 Greenwich St, New York, NY 10006