Hulu has been hailed as the future of television, but its reputation has been based in part on measurement techniques rooted in the past.
The highflying online TV site frequently is touted as among the most popular video sites in the U.S., based on data from the industry-leading ComScore measurement service. A recent overhaul of ComScore's methods, however, brought Hulu's numbers crashing to Earth.
Hulu's estimated viewership shriveled to 24 million in June, from 43.5 million in May, under ComScore's new methodology. It was the most dramatic decline of any of the top Internet video sites. Now Hulu, long proud of its ranking as the Internet's No. 2 most popular site, is barely hanging on in the top 10, according to ComScore.
The site's statistical fall from grace dramatically underscores one of the main impediments to the Internet reaching its full financial potential as an entertainment medium: reliably measuring the size of online audiences.
Wildly divergent audience estimates and erratic month-to-month fluctuations have bedeviled websites since the days of dial-up Internet access. As the Internet matured and attracted more advertisers, media and technology companies began demanding a trusty umpire to make the calls about which sites are on first, second and so on down the line.
The stakes for getting it right have never been higher. Advertisers are expected to spend $25.1 billion this year in online advertising in the U.S. alone, according to researcher EMarketer.
"The inconsistencies across methodologies and venders and the cacophony of numbers in the marketplace are clearly confusing," said Sherrill Mane, a senior vice president of the Interactive Advertising Bureau, a trade group that has been advocating for greater accountability in online measurement. "They're truly hampering the growth of the medium."
The three dominant measurement firms — ComScore, Nielsen and Quantcast — have been working since 2007 with an independent media auditing group to make improvements so the Web data they report don't have a fun-house quality, in which the same site's traffic can look emaciated or bulging, depending on the viewer's angle.
"It's maddening," said Tim Hanlon, a Chicago digital media strategist. "You would think 15 years on, we would be in a better place. But we're still talking about fundamental discrepancies in things like page counts."
Getting an accurate read of online audiences is important to media planners, who need a realistic picture of the digital landscape to determine where to buy advertising and how to allocate a budget. Traffic numbers also serve as a barometer in identifying which sites are on the rise and which are in decline.
"In the digital space, we're always looking for the next big thing. We're trying to predict where audiences are going," said Michael Hayes, managing director in charge of digital operations for Initiative, a marketing firm. "Young audiences shift fairly quickly, so we're looking for where the audiences have been shifting to, to put our advertisers in front."
Failing to crack the top 10 or 15 in a particular ranking can have a profound financial consequences for websites as well, especially digital start-ups that are looking to attract advertisers or seeking funding from potential investors.
"The first few years of our business, we were so dramatically underrepresented and misrepresented by a bunch of these different systems it delayed our ability to really create a sales organization," said Break Media Chief Executive Keith Richman, whose Break.com site hosts original and user-created videos targeted at 18- to 34-year-old men.
"If you're a small one- or two-man shop and you're underrepresented, you're kind of DOA," Richman said.
Contradictory data from the competing measurement firms can make it hard to present coherent results. Take, for example, Justin.TV, a site that lets users produce and watch live streaming video. ComScore said its global audience rose to 15.6 million in June from 12 million in March. Quantcast, however, shows the site's global viewership plunging over that same period, dropping to 17.2 million in June from 26 million three months earlier.
"We have a spreadsheet internally that we keep updated with third-party numbers. I am as surprised as the next guy when we update it," said Evan Solomon, Justin.TV's vice president of marketing. "It's like opening a box when you don't know what's in it."
Other forms of media — radio, television and newspapers — don't suffer from this problem. Broadcast and print have long been dominated by single sources of audience information agreed upon by all parties: Arbitron Inc. for radio listeners, Nielsen for TV viewers and the Audit Bureau of Circulations for print subscribers.