Republican presidential candidates Mitt Romney, left, Newt Gingrich… (Michael Reynolds / EPA )
Politico headlined a story last week "Mitt, Paul winning Facebook primary." About the same time, the Washington Post reported "Romney with the momentum in S. Carolina," that conclusion based on its new Twitter-tracking app, @MentionMachine.
One of the most striking innovations of campaign 2012 media coverage has been the attempt by news outlets to harness Twitter and Facebook, not just for a spot check on individual voters' feelings but to take the temperature of the electorate in a broader way.
The vast trove of messages and status updates embedded in Facebook, in particular, has created what technology journalist-blogger Marshall Kirkpatrick called "the biggest, most dynamic census of human opinion and interaction in history." But the initial Facebook/Politico analysis belied how fraught the nascent science of "sentiment analysis" is, producing "total bunk," said Micah L. Sifry, the creator of the website Techpresident, which examines the nexus of political practitioners and technologists.
The attempt to analyze the data should come as no surprise in a season when social media have assumed an ever-larger profile, with regular input from such media in televised debates and even a session in which Republican hopefuls answered tweeted questions in 140-character Twitter bursts.
But experts in polling and computer science caution against inexact and overblown conclusions when converting far-flung messages into hard data. They said media outlets should recognize that computer programs are still in their infancy when it comes to distilling human feelings from digitized text.
Imagine a computer trying to parse, for instance, the exact intention of someone who posts "I love having Ron Paul in this race!" Is this hypothetical tweeter a) a stalwart supporter of the Texas congressman b) someone who likes Paul's candor and consistency, but would never vote for him c) a President Obama supporter who enjoys seeing Paul muddle the Republican field d) an observer employing a bit of irony or e) none of the above.
Marc A. Smith, a sociologist who studies online communities and founded the Silicon Valley-based Social Media Research Foundation, said "we are in the Model T Ford era of information systems" and analyzing their content.
Scott Keeter, the president of the American Assn. of Public Opinion Research, said that members of the professional organization and journalists should "proceed with a degree of humility" in deciding what social media can tell us about political campaigns. "Until we have more experience with real world outcomes, it's hard to know the meaning of what we have captured from social media," said Keeter, director of survey research at the Washington-based Pew Research Center for the People & the Press.
Much of the debate followed a Jan. 12 article by Politico, the online news site, which reported that it had partnered with Facebook to examine all "posting, sharing and linking about candidates" from Dec. 12 to Jan. 10. The arrangement was a first not only in that Facebook delved into both public and private messages but also used computer analysis to "identify positive and negative emotion in text." (The company stressed that while computers draw an aggregate view of user sentiment, human beings do not monitor individual messages.)
Facebook said it employed a "well-validated software tool used frequently in social psychological research." But Smith said he was "highly skeptical" of some of the precise findings in the Facebook analysis. He added that the intellectual disciplines focused on deciphering texts — natural language processing and computational linguistics — "are very deep and can do remarkable things, but they don't necessarily have the ability to predict the next president of the United States of America."
Politico's report didn't actually predict an outcome, reporting instead that a surge in Facebook mentions for Mitt Romney effectively "predicted" (though after the vote) a strong finish for Romney in the New Hampshire primary.
What the story did not say is that the summary of the Facebook chatter was not nearly as accurate when it came to the Iowa caucuses. Politico's charts show Romney in a fairly distant third in Facebook mentions leading up to the Iowa event. In the actual voting the candidate finished in a virtual tie for first.
Keeter said those examining the data should realize that Facebook users represent only a portion of the U.S. population. About 65% of adults who go online told Pew researchers that they use at least one social networking site. The portion of those online who utilize social networks is much lower for those 50 to 64 years old (51%) and those 65 and older (just 33%.)