The current Iffy Index of Unreliable News Sources had a predecessor. In 2017, I needed a list of U.S. fake-news sites for a research project. The existing lists all sucked. Their data was old, incomplete, and full of 404s.
So I merged the major fake-news site lists then purged the inactive sites. That’s when I realized my computer now housed the most up-to-date index of unreliable news sources. Poynter agreed and published it… for about three days.
Here’s what happened: Originally, Columbia Journalism Review was going to publish the index. A grad student did some vetting. We made a beta version. But they never quite got around to setting a publish date, always too busy with their other projects. After five months, I asked and they let me shop it elsewhere.
A few days later Poynter picked it up and within weeks put it online. The response was immediate. Readers sent us heartfelt appreciations:
You and your ilk are mentally ill and possessed by satan. Oh, and you suck too. Burn in hell.
A few publishers questioned their inclusion, politely and professionally:
We take accuracy and fairness extremely seriously at The Daily Signal. We correct mistakes, in a transparent, public way. We make sure that we have reached out to people mentioned in articles before we publish, giving them a chance to respond or deny/confirm.
—The Daily Signal
Others took a different approach:
Thank you SO MUCH for putting us on your list! You do realize that most Americans are now aware that CNN, the Washington Post, NY Times are simply GLOBALIST PROPAGANDA,
—All News Pipeline
CNN was a favorite caps-worthy topic: “It constantly spews forth FAKE NEWS,” “anti-Truth,” and “liberal POS Fake news media.” Even the global-propagandist liberals at CNN emailed us, to defend a conservative news site:
I don’t think the Free Beacon belongs on that list. I think there’s a real difference between sites that lean a certain way ideologically and ones that knowingly provide false and/or bigoted information. The Free Beacon often has excellent reporting. Thanks.
—Jake Tapper, CNN
We reviewed the submissions and delisted about 2% of the index’s 500+ sites. (We took off The Daily Signal, rated mostly factual by Media Bias/Fact Check (MBFC). But kept the Free Beacon, due to its frequent failed fact checks, and All News Pipeline, for its low MBFC factual-reporting level.)
The crowd-sourced flood of feedback quickly strengthened the index. Until Poynter pulled the plug, replacing the index with an editor’s-letter apology questioning the methodology.
They were right: The methodology was worth questioning. As I wrote, “Our index compiles existing site lists, curated by academics and journalists. For now, we depend on their expertise for accuracy.”
In other words, I relied on the professionals who curated the existing lists. The same lists researchers regularly relied on for their studies. The same studies Poynter regularly published (even praising those lists). Mis/disinformation research had a circular dependence on flawed data, as did its widely disseminated, often-unquestioned, results.
My goal was to give researchers (including myself) a more reliable index of unreliable sources. My data was more complete, current, and accurate than any previous attempt to list sites that spread disinformation. So, in that, I succeeded. But that’s no excuse: Before publishing, I (and Poynter) should have better documented the justification for publicly labeling each site on the list as unreliable.
Mistakes made, lessons learned
The weaknesses of the old index are strengths in the new one. The Iffy index now:
- Ignores political bias, only bullshit gets a site into the index.
- Relies on professional site reviewers, who respond to re-review requests.
- Uses well-defined, defensible ratings (MBFC’s factual-reporting level).
- Links to evidence documenting the reasons for a low rating (failed fact-checks).
- Emails the publishers in the Iffy.news index (in process).
So the site vetting is done by independent professionals, who document the reasons for each rating, continually rate new sites, and reconsider past ratings. The index only lists sites given a low or very-low MBFC factual-reporting rating. MBFC bases its ratings on fact-checks by verified signatories of (Poynter’s) International Fact-Checking Network. (I’d hoped to include NewsGuard ratings too, but they prohibit others from publishing their data.)
And the winner is…
One last bit of data. In its brief time at Poynter, the index offered readers a form to submit sites for review. We got 100+ responses. Several suggested sites they felt we should remove but the majority submitted sites to add to the index. By far, the publication cited most as unreliable was… poynter.org.
People weren’t shy in telling us why:
What kind of censorious, officious, humorless mountebanks are you???? Have you nothing better to do than to play Internet Hall Monitor all day??? This is the worst sort of neo-Liberal thought-policing.
I’m hoping the extra care I’ve devoted to the current index version will lead to more civil discourse. Yeah, right:
You are the enemy of the American people and we’re going to kill you fascist f@%%*†s.
