Deep Jive Interests

The Jason Calacanis Experiment (Not to be confused with Netscape, ‘natch) last week (Nov 24) involved a call out to his faithful readers to prove once and for all that Alexa is indeed the old-and-busted of publicly available traffic-sensing webware. He asked everyone to install the alexa tool bar, if they hadn’t done so already, and was out to prove that by gaming the Alexa system, its traffic results were flawed (the roots of this go back to a geek fight on Valleywag, as Nick Denton stated that Netscape’s numbers had dropped like a stone since Calacanis started his first experiment — but using Alexa’s traffic data).

Now its only been about six days, which is only about half the time that Jason said he would be tracking (he gave it 10 days or so), so the results might be a bit premature. But I thought it would be interesting to examine the traffic at this point, because with the buzz the Rojas-Winer-Calacanis open source iPod killer is getting, it will be hard to interpret the results in the next few days. Is the bump in traffic — if it happens — because of the experiment, or because there are other reasons to visit the blog?

Let’s look at the traffic.

,

So, when you look at the traffic graph (courtesy of Alexaholic), you can see that there was indeed a bump in traffic, although it looks modest.  This occurred over what looks like 48h, and then dropped down.  And it continues to drop as of yesterday to below its previous valley.  In relative terms it doesn’t look like much, but when you look at the Y-axis, the bump is in fact a reach of 100-150 million above where it was previously, strictly based on the Alexa traffic — one presumes.

On the other hand, the Alexa experiment was notable in and of itself — which probably drew genuine traffic (and unique visitors) towards Calacanis.com to see what the hubaloo was all about.  Therefore, it is hard to say how significant this bump in traffic really is (although perhaps not — see below).
But, let’s put this bump in traffic – even if we presume that ALL of it is due to the Alexa experiment (readers downloading the bar and/or reloading like mad) – in context.  How does this bump do relative to other traffic patterns over time?  Is it much bigger or much smaller?  Does it, for example, stand out as an exceptionally large traffic bump?

When you look at the longer term graph over, say 6 months, it really doesn’t look like it.  In fact, it almost looks like part of the normal ebb and flow of the usual traffic.  It almost looks like this bump in traffic is not “significant”, as there wasn’t a significantly huge bump in traffic for the short period of the experiment (and it looks like traffic is on the down hill slide since that time … although I expect it to perk back up because of the open source iPod thing).
But, what if we compared Alexa’s traffic patterns with another non-Alexa product?  How about a “control” … like Quantcast?Quantcast computes their traffic data based on a “panel” of internet surfers (they just follow a whole whack of them and where they go), as well as “pixel measurement”, to create a “Mass inference” of where internet traffic is going.  I’m no traffic expert, but if you’re interested in the full explanation go their website.  But clearly they measure it differently than Alexa.

Yikes.  It looks like there is a similar, modest, bump in traffic, but about 1 day later (peaks on the 27th, whereas on Alexa it peaks on the 26th) which drops back down again.  What does this mean?

If they have similar traffic patterns, that might mean that the influx of traffic because the experiment made the “news” was actually real, and that Alexa wasn’t gamed at all.  That is, if the gaming went off perfectly, we should see an inappropriately high spike in the traffic on the Alexa graph compared to a “control”, such as Quantcast … which we clearly don’t see.

Furthermore, when put into the longer term context, such a spike doesn’t even look significant in the context of previous traffic patterns.

Is Alexa “wrong?” I think the appropriate answer is that “its hard to know” because the experiment was flawed.  (btw, there were similar, if not identical patterns in “reach”, and “rank” — for the sake of brevity I decided not to include those graphs, but you’re welcome to check them out yourself).
Why?  Two reasons.   If many of his viewers are not using Internet Explorer (as I suspect many aren’t, but are using Safari or Firefox), perhaps Alexa didn’t “detect” all of the gaming attempts.  The second reason is that perhaps not enough people participated in the experiment (“I thought everyone else would do it!” vs. “this is a waste of my time”).

At the end of the day, I don’t know if there are any published experiments to “prove” Alexa’s broken; but if Mr. Calacanis is interested in pursuing another one, maybe he should a) do it a) in “private” and b) with definite (and defined) number of participants (and a way to prove they’re doing it).

Not that he will of course.  Heck — even I’m more interested in an iPod killing gadget than proving Alexa’s worthiness, right now. ;)