Jokes and jokes are infinitely more complex and subjective, but Google DeepMind researchers found a consensus among professional comedians: “AI is really bad at this.
It's one of many opinions gathered from a survey of twenty professional comedians and performers at the Fringe Workshops at the Edinburgh Festival last August 2023 and online. The findings show that large-scale linguistic models (LLMs) accessed by chatbots present significant challenges and raise ethical concerns in generating humor about AI.
The study included a three-hour workshop where comedians participated in comedy writing with popular LLMs like ChatGPT and Bard. It also assessed the quality of results through a human-computer interaction questionnaire based on the Ten-Year Innovation Support Index (CSI), which measures how well a tool supports innovation.
The participants also discussed the motivations, processes and ethical issues of using AI in a humorous way in a focus group.
The researchers asked comedians to write AAAAAP comedy routines and then had them rate the results and share their thoughts. And the results… weren't good.
One participant described the AI-generated material as “so dull, so boring—I stopped reading. It was really bad.” Another said the score was “a throwback draft that I knew I had to redo and improve on.”
“And I don't want to live in a better world,” said another.
The study found that LLMs were able to generate long lists and snippets of routines, but they lacked the unique human elements that make something funny. When asked to generate the abstract structure, the models “created a scene that provided a lot of structure,” but when it came to details, “the LMLs failed as a creative support tool.”
Among the reasons, according to the authors, is that the tools used in the study are based on all the accumulated material, covering every subject area, and thus “the LMS. This also introduced a form of bias, which the comedians pointed out.
The participants suggested that existing mediating strategies used by censored and regulated LLMs reinforce hegemonic views by eradicating minority groups and their perspectives, and qualify this as censorship.
Popular LLMs are prohibited, the researchers said, citing the so-called “HHH criteria,” calling for honest, harmless, and helpful results—”encapsulating most of what users want from integrated AI.”
“A cruise ship from the 1950s is funny, but a little racist,” the article was described by one commenter.
Another participant said, “The wider the appeal of something, the less great it can be.” “If you make something that suits everybody, it might end up being nobody's favorite.”
The researchers emphasized the importance of keeping in mind the subtle differences between hurtful rhetoric and offensive words used in protest and sarcasm. The comedians, meanwhile, complained that AI failed because it didn't understand jokes like sarcasm, dark humor or comedy.
“A lot of my stuff can have a dark part in it, and then I don't write any dark stuff, because I think I'm going to kill myself,” reported one participant. “So he stopped giving me anything.”
The fact that the chatbots are text-based doesn't help, the study found.
“Given that currently widely available LLMs are accessed primarily through a text-based chat interface, they felt that the utility of these tools was limited to the set of domains required to produce a complete comic product,” the researchers said.
“Any script can be an okay script, but a great actor makes it really interesting,” said Participant.
The research suggests that AI's limitations in humor writing extend beyond simple content generation. The comedians emphasized that perspective and perspective are uniquely human characteristics, one comedian said, adding that humans “add a lot more emotion and emotion and subtlety” because of their life experience and relationship with the material.
Many described the centrality of personal experience in good comedy, which allowed them to construct authentic and engaging narratives of memories, friendships and beliefs. Moreover, comedians emphasized the importance of understanding cultural context and audience.
“The kind of comedy I do in India will be very different from the kind of comedy I can do in England because my social status will change,” said one of the participants.
Thomas Winters, one of the researchers cited in the study, explains why this is so difficult for AI to combat.
“Humor's frame-shifting premise shows how difficult it is to find a machine,” he said. “This heavy reliance on human reasoning—memory, language skills for meaning synthesis, and world knowledge cues—has often led researchers to assume that humor is an AI-whole problem.
OpenIC's Sito Mira Murathi recently said that “some creative jobs may be lost, but maybe they shouldn't have been there in the first place.” Given the current technological capabilities, however, it seems that comedians can breathe a sigh of relief.
Edited by Ryan Ozawa.
Daily Debrief Newspaper
Start every day with top news stories, plus original features, podcasts, videos and more.
