Countering Disinformation in a Post-ChatGPT World

Countering disinformation is about to get much harder: In the near term, ChatGPT and similar chatbots powered by large language models, or LLMs, will let threat actors master a range of malicious activities, including manufacturing more believable lies at scale. As National Security Agency (NSA) Director of Cybersecurity Rob Joyce has said, ChatGPT will enable malicious foreign actors to craft very convincing, native-language, English text for phishing schemes, false backstories, and even malign influence operations.

ChatGPT and similar LLM-powered tools use artificial intelligence (AI) to ingest massive quantities of data from the internet, identify patterns in language, and answer user questions with text that is tricky to distinguish from human writing. Disinformation threats created by the abuse of these tools would fall under the umbrella of "information to human (I2H)" attacks, as opposed to cyberattacks aimed at stealing information.

Even before the advent of ChatGPT, disinformation posed a serious risk to election integrity and public safety. The Department of Homeland Security (DHS) is responsible for countering the triple-headed threat of misinformation, disinformation, and malinformation (MDM). The Cybersecurity and Infrastructure Security Agency (CISA), an operational component of DHS, has raised awareness about MDM threats to critical infrastructure.

Threat actors are investing heavily in spreading falsehoods. In 2022, the Russian state budget for disinformation, propaganda, and other influence operations was estimated to be over $1.5 billion, triple what it had been just one year prior. Now, LLMs have the potential to significantly increase the “bang for the buck” in disinformation campaigns by making it quicker and simpler to create vastly larger quantities of inauthentic content and distribute it to ever-wider audiences.

To understand how LLMs might scale up adversarial falsehood factories, let’s begin with a disinformation example from 2016: The Russian GRU hacked and leaked damaging material about the Clinton campaign and the Democratic Party, most infamously in the release of internal communications of the Democratic National Committee (DNC) and the emails of the Clinton campaign chair, John Podesta. While initial efforts to launder a mix of legitimate and altered material stalled under DCLeaks and Guccifer 2.0, the operation later got public traction when the information was filtered through the Wikileaks platform. Russia’s Internet Research Agency was then able to use its trolls and bots to further spread falsehoods, which in turn led to media coverage that created an appearance of legitimacy.

Now, a hypothetical LLM-enabled scenario would likely involve more realistic false narratives. A tool like ChatGPT with an LLM trained on public data would likely be sufficient for threat actors to achieve this new level of realism, although LLMs could also potentially be trained with specific datasets to increase the odds of fooling select audiences. If the disinformation campaign involved both real and false information, distinguishing between the two might be harder than ever before. Also, LLM-enabled disinformation might be more difficult to detect immediately on social media because adversaries could more easily vary their language rather than repeating the same phrases again and again in large volumes. Just as adversaries will rely more on automation to create more compelling disinformation narratives and “deep fake” imagery, so too will security teams need to apply automation in new ways to counter emerging threats.

LLMs have shortcomings that can be exploited by cybersecurity professionals, threat hunters, and counterintelligence teams. For starters, LLMs are limited by the amount and type of information they can ingest. This has led to ChatGPT answering many questions with incorrect or simply made-up responses. In one particularly bizarre case, the chatbot seemed to insult a user who correctly pointed out that the year was 2023 and not 2022, as the program continued to insist. Edward Tian, a 22-year-old college student studying GPT-3, a predecessor of ChatGPT, took only three days to create a tool to detect AI-generated content, which he called GPTZero. While teams are working to improve the quality of responses provided by LLMs, the structural limitations of the model will continue to have vulnerabilities that can be exploited.

GPTZero essentially uses ChatGPT’s machine learning model against it by scanning text for indicators consistent with AI-generated content. While attempts to test the efficacy of GPTZero found mixed results, at least one study found it reasonably reliable in identifying longer pieces while struggling with shorter texts (such as tweets).

Another effort to combat disinformation is the DISARM Foundation, an organization established to build a common framework similar to the MITRE ATT&CK framework for cybersecurity. The DISARM Framework is an attempt to use an understanding of adversarial tactics, techniques, and procedures for crafting and executing disinformation campaigns to find ways to detect and mitigate or disrupt them.

In an interview on The Cyber5 Podcast, Jon Brewer, executive director of the DISARM Foundation, stated, “Long-term success looks like us having a tool that is really nice and simple, could be on a mobile, and could actually help people to spot a piece of dis/misinformation and call it out. I mean, we’ve had this with email, so people can report spam with a click of a button, and that’s had a big impact.” Similar but more robust detection tools utilizing machine learning (ML)/AI technology should be developed to guide cognitive security operations centers (SOC) to identify inauthentic content associated with information operations and enable rapid response and even the prebuttal, rather than rebuttal, of disinformation campaigns.

The successful prebuttal of Russian narratives ahead of Russia’s 2022 invasion of Ukraine demonstrated the value of proactively detecting and countering disinformation. In contrast to the traditional practice of keeping intelligence information secret to protect sources and methods, the U.S. government publicly stated what it knew about Russia’s plans before the invasion. This helped prevent the fracturing of key alliances—historically a primary goal of Russian disinformation campaigns—and maintain unity of support for Ukraine’s defense. Now, additional effort is needed to stay ahead of tomorrow's MDM threats.

At the national level, organizations should establish counter-MDM plans that focus on establishing capabilities to prevent or limit the spread of disinformation within their authorities. Recommendations on countering disinformation released in August 2022 by the DHS inspector general and the Homeland Security Advisory Council provide DHS and the government at large with a roadmap for developing a more strategic approach to disinformation. In addition, public-private partnerships will be increasingly important for debunking disinformation before it spreads.

Security teams and leaders at many critical infrastructure businesses responsible for providing national critical functions should proactively implement CISA's guidance to industry on countering MDM, to include monitoring for any shifts in online activity related to their business or sector.

In addition, innovators should seize the opportunity to unlock the potential of new technical solutions: Frameworks like DISARM and tools like GPTZero can and should be integrated into cognitive SOCs. Multidisciplinary teams skilled in AI, cognitive security, threat hunting, counterintelligence, and other fields can find creative ways to use these tools to detect MDM campaigns. Building such capabilities now would empower policymakers, private-sector entities, and society at large to resist and counter MDM more effectively in the future.