Chartbeat Shines A Light On Dark Social, Finds Missing Mobile Referrers

Chartbeat appears to have solved a piece of the dark social puzzle.

Turns out a significant percentage of unattributed traffic to publishers is coming from mobile apps and last night at 6 p.m. Eastern the analytics company flipped a switch on its real-time dashboard to place that traffic in its proper bucket.

It’s still early, so Chartbeat is being cautious about over-promising, but chief data scientist Josh Schwartz told Marketing Land that sites are already seeing a 10 to 20% decrease in dark traffic, meaning they have a clearer picture of the source of that traffic.

Not surprisingly, there’s a high dose of mobile Facebook. “We saw mobile Facebook traffic increase by about 40% on sites with big Facebook presences,” Schwartz said. “On sites that are very, very heavy on mobile facebook we saw even larger increases. I saw some sites with 3 or 4x increases on mobile Facebook. But that’s a rarity, not the norm.”

Chartbeat’s adjustment came after it discovered that a number of highly used mobile apps have added the app identity to the user agent setting, which typically is used to identify the browser and operating system of web pages. Now the source of mobile app traffic from Buzzfeed, Pinterest, Flipboard, Chinese social network Tencent QQ and Chinese search engine Baidu, among others, is being properly attributed in Chartbeat.

Schwartz said he believes Chartbeat is the first analytics provider to activate such a feature.

This is great news for publishers who have been struggling to pinpoint the source of traffic without referring information. Up to now, most of the attention has been paid to the assumption that most such traffic comes from people sharing content via email, instant messaging and other private means, the supposition made by Alexis Madrigal when he coined the term dark social in an Atlantic.com article in 2012. That theory was reiterated this week in a study by RadiumOne, which we covered here: “Dark Social” Is Still Nearly 70 Percent Of Sharing Activity.

Chartbeat’s new findings don’t change the fact that such organic peer-to-peer sharing is common, but they add depth to the discussion. Chartbeat, which provides real-time insight for most major publishers, has been studying this issue for years; it helped Madrigal with his original research, finding then that about 25% of all referral traffic — and 69% of social referrals — was unattributed.

[pullquote]”If you take the time to look at each major referrer and the time series of dark social you’ll probably see a smoking gun or a couple of smoking guns.”[/pullquote]

Since 2012 much has changed on the Internet. Many sites have switched over to secure https which often masks referral information. And, of course, mobile use has soared, especially on mobile apps. Chartbeat decided to take another close look at the issue. This time it found that the dark social percentage has increased from one quarter to about a third, sometimes 35-38%, and even higher on mobile, in the 40-50% range.

“That’s a big deal,” Schwartz said. “Obviously, every editorial team in the world has people who spend all their time thinking about ‘Do we spend our time on SEO? Do we spend our time doing Facebook promotion, Twitter promotion?’ And if you are a mobile-first site and you can’t attribute where half your traffic came from, it’s a problem. Are you looking at the right ROI in terms of where you are investing your time and energy?”

So Chartbeat set up a test site to analyze the issue and was able to eliminate some possible dark social sources. Google and Google News, for instance, and all non-https sites reliably produced good referral data. Others were less consistent. Facebook doesn’t set a referrer when a desktop user opens a link in a new tab. And Facebook’s mobile app doesn’t always set a referrer. Reddit on the desktop and the mobile web is trackable, but the top mobile apps — which drive the majority of Reddit traffic — are not.

Among the other findings during Chartbeat’s study is that dark social traffic can be properly identified by correlating its timing with known referral traffic on individual stories. On about 75% of stories, they are able to see where traffic is actually coming from by comparing traffic surges.

“So for instance,” Schwartz said, “if you see that at 12:01:30 there’s a giant spike on mobile Facebook and at exactly the same second there’s a giant spike on dark social, you can pretty safety assume that actually the dark social is misattributed Facebook traffic.

“We can’t in general say OK dark social is actually Facebook in disguise or Reddit in disguise or anything else in disguise because it’s a hodgepodge of a bunch of stuff. But in general if you are interested on a particular story in figuring out what the dark social is with high probabity, you’ll be able to do it. With high probability if you take the time to look at each major referrer and the time series of dark social you’ll probably see a smoking gun or a couple of smoking guns.”

You can read more about Chartbeat’s findings and Madrigal’s latest thinking on the topic at his new employer, Fusion: Dark social traffic in the mobile app era