Duke U. pits fact-checking apps against Trump’s State of the Union speech

During President Trump’s second State of the Union speech Tuesday night, a research team at Duke University tested two home-brewed fact-checking programs to measure the bounds of the technology. One crashed and the other exceeded expectations, but both are helping the students building them apply cutting-edge technologies to real-world challenges.

The tools were developed by the Duke Tech & Check Cooperative, a $1.2 million program launched in 2017 through the Duke University Reporters’ Lab designed to bring together research teams from universities and the Internet Archive to support the advancement of live, automated fact-checking. Together, researchers say the tests on Tuesday demonstrated that live fact-checking technology holds promise, but still needs some work, as evidenced by the sometimes confusing results returned by the software.

The first app, called FactStream, crashed ten minutes into the speech following an unexpected surge of traffic, according to an announcement from Bill Adair, the project’s director. FactStream is supported by human fact-checkers working behind the scenes. The Reporter’s Lab says it’s a crucial first step toward the “holy grail” of automated fact-checking.

The other tool, a fully automated fact-checking software code-named “Squash,” is not yet a publicly available app, but during internal testing during the address, Adair said it outperformed expectations.

“We went into last night’s speech with very modest expectations,” he said. “I said before the speech I’d be happy if the speech simply triggered fact-checks to pop up, even if it was a poor match.”

During Trump’s speech, Adair said Squash generated 20 pop-ups, and six of them were on the right track.

Built off of FactStream, Squash automatically converts speech to text and then searches fact-check databases from the Post, FactCheck.org and PolitiFact. When it finds a match, a summary of the fact-check pops onto the screen of the device running the program.

The tool matched a claim made during the speech with a similar claim made by the president last month that “El Paso … went from being one of the most dangerous cities in the country to one of the safest cities in the country overnight” after “a wall was put up” along the Mexico border. According to FactCheck.org, this is false.

Other matches found by the software were less helpful.

Adair said Trump’s mention of human progress — “We also celebrate 50 years since brave young pilots flew a quarter of a million miles through space to plant the American flag on the face of the moon” — triggered a fact-check about the long delay it takes to get a permit to build a road.

Complementing the classroom

Jun Yang, the team’s faculty co-lead and a professor of computer science, told EdScoop that the automated pipeline that converts speech to text and evaluates a claim’s truthfulness is made up of different algorithmic components that feed the output into the next stage of the automation. As a result, there is potential for error at every junction.

“The accuracy of one component may affect others,” Yang said.

Overall, he said the two apps were able to produce several instances of correct fact-checking, but Tuesday’s test helped the team of students spot several areas of improvement. The students will now be able to make improvements to the products based off of the test’s feedback.

The team of Duke students has been able to apply state-of-the-art processes to their fact-checking apps, Yang said, and by tackling the challenge of automated fact-checking, they are forced to explore the latest technology.

“It complements what we can teach in the classroom a lot, ” Yang said.

He said the project allows students to see how learned concepts and techniques are applied in practice. By working on this real-world challenge, Yang said students are also able to think about the applications of computer science more creatively, and can discover new solutions to complex challenges. Yang also said he was very impressed to watch the team apply cutting-edge technology to a concrete problem.

However, the work is not over. Both FactStream and Squash have some bugs to work out. Adair said he hopes the team will be able to release a public version of the automated app, Squash, later this year — just in time for presidential debate season.

As for FactStream, the app is back online and now displays fact-checks on its home screen, including several errors caught both in Trump’s speech and in the Democrat’s rebuttal given by former Georgia gubernatorial candidate Stacey Abrams.