FIR #490: What Does AI Read?

Studies purport to identify the sources of information that generative AI models like ChatGPT, Gemini, and Claude draw on to provide overviews in response to search prompts. The information seems compelling, but different studies produce different results. Complicating matters is the fact that the kinds of sources AI uses one month aren’t necessarily the same the next month. In this short midweek episode, Neville and Shel look at a couple of these reports and the challenges communicators face relying on them to help guide their content marketing placements. Links from this episode: Webinar: What is AI Reading? (Muck Rack)AI Search Volatility: Why AI Search Results Keep ChangingStudy finds nearly two-thirds of AI-generated citations are fabricated or contain errorsMajor AI conference flooded with peer reviews written fully by AI The next monthly, long-form episode of FIR will drop on Monday, December 29. We host a Communicators Zoom Chat most Thursdays at 1 p.m. ET. To obtain the credentials needed to participate, contact Shel or Neville directly, request them in our Facebook group, or email [e-mail]. Special thanks to Jay Moonah for the opening and closing music. You can find the stories from which Shel’s FIR content is selected at Shel’s Link Blog. You can catch up with both co-hosts on Neville’s blog and Shel’s blog. Disclaimer: The opinions expressed in this podcast are Shel’s and Neville’s and do not reflect the views of their employers and/or clients. Raw Transcript: Shel Holtz Hi everybody, and welcome to episode number 490 of For Immediate Release. I’m Shel Holtz. Neville Hobson And I’m Neville Hobson. One of the big questions behind generative AI is also one of the simplest: What is it actually reading? What are these systems drawing on when they answer our questions, summarize a story, or tell us something about our own industry? A new report from Muckrec in October offers one of the clearest snapshots we’ve seen so far. They analyzed more than a million links cited by leading AI tools and discovered something striking. When you switch citations on, the model doesn’t just add footnotes, it changes the answer itself. The sources it chooses shape the narrative, the tone, and even the conclusion. We’ll dive into this next. Those sources are overwhelmingly from earned media. Almost all the links AI sites come from non-paid content, and journalism plays a huge role, especially when the query suggests something recent. In fact, the most commonly cited day for an article is yesterday. It’s a very different ecosystem from SEO, where you can sometimes pay your way to the top. Here, visibility depends much more on what is credible, current, and genuinely covered. So that gives us one part of the picture. AI relies heavily on what is most available and most visible in the public domain. But that leads to another question, a more unsettling one raised by a separate study published in the JMIR Mental Health in November. Researchers examined how well GPT-4.0 performs when asked to generate proper academic citations. And the answer is not well at all. Nearly two thirds of the citations were either wrong or entirely made up. The less familiar the topic, the worse the accuracy became. In other words, when AI doesn’t have enough real sources to draw from, it fills the gaps confidently. When you put these two pieces of research side by side, a bigger story emerges. On the one hand, AI tools are clearly drawing on a recognizable media ecosystem: journalism, corporate blogs, and earned content. On the other hand, when those sources are thin, or when the task shifts from conversational answers to something more formal, like scientific referencing, the system becomes much less reliable. It starts inventing the citations it thinks should exist. We end up with a very modern paradox. AI is reading more than any of us ever could, but not always reliably. It’s influenced by what is published, recent, and visible, yet still perfectly capable of fabricating material when the trail runs cold. There’s another angle to this that’s worth noting. Nature reported last week that more than 20% of peer reviews for a major AI conference were entirely written by AI, many containing hallucinated citations and vague or irrelevant analysis. So if you think about that in the context of the Muckrec findings in particular, it becomes part of a much bigger story. AI tools are reading the public record, but increasing parts of that public record are now being generated by AI itself. The oversight layer that you use to catch errors is starting to automate as well. And that creates a feedback loop where flawed material can slip into the system and later be treated as legitimate source material. For communicators, that’s a reminder that the integrity of what AI reads is just as important as the visibility of what we publish. All this raises fundamental questions. How much has earned media ...