AI-generated news is harder to understand, study shows

Traditionally-crafted news articles are more comprehensible than articles produced with automation. This was the finding of a Ludwig Maximilian University of Munich (LMU) study that was recently published in the journal Journalism: Theory, Practice, and Criticism.

The research team at the Department of Media and Communication (IfKW) surveyed more than 3,000 online news consumers in the UK. Each of the respondents rated one of 24 texts, half of which had been produced with the help of automation and half of which had been manually written by journalists.

“Overall, readers found the 12 automated articles significantly less comprehensible,” says lead author Sina Thäsler-Kordonouri. This was despite the fact that the AI-generated articles had been sub-edited by journalists prior to publication.

Worse handling of numbers and word choice

According to the survey, one of the reasons for reader dissatisfaction was the word choice used in the AI texts. Readers complained that the AI-produced articles contained too many inappropriate, difficult, or unusual words and phrases. Furthermore, readers were significantly less satisfied with the way the automated articles handled numbers and data.

The deficiencies readers perceived in the automated articles’ handling of numbers and word choice partly explain why they were harder to understand, the researchers say. However, readers were equally satisfied with the automated and manually-written articles in terms of the character of the writing and its narrative structure and flow.

More human sub-editing required

Professor Neil Thurman, who led the project, suggests that “when creating and/or sub-editing automated news articles, journalists and technologists should aim to reduce the quantity of numbers, better explain words that readers are unlikely to understand, and increase the amount of language that helps readers picture what the story is about.”

This study is the first to investigate both the relative comprehensibility of manual and automated news articles and explore why a difference exists.

“Our results indicate the importance not only of maintaining human involvement in the automated production of data-driven news content, but of refining it,” says Thäsler-Kordonouri.