Update: Looks like the folks at Feedburner have resolved the issue quickly and amicably — see the comments below — and its just the way things ought to be fixed. Well done, Eric and the gents (and ladies) at Feedburner!
In the battle for news aggregator supremacy, Megite made a respectable name for itself alongside the current champion (in my mind), Techmeme, as an aggregator to contend with. I use it from time to time myself, as there are news tidbits that aren’t readily found in many other places. Kudos to Matt Chen and his crew for rolling out the service.
In spite of its growing popularity, however, it looks like Feedburner is no fan of Megite. I just got off the horn with Matthew, and it seems like Feedburner is blocking Megite’s feed crawlers. This is a problem, as feedburner serves thousands of blogger’s feeds, and its has a double-whammy effect.
First of all, Megite won’t be able to credit Bloggers with appropriate links to their site when a story comes up — because it can’t “find” their feed if its being served by Feedburner.
Secondly, because Megite can’t find them, it can’t list their story either underneath the appropriate story item. As it stands now, many of the news items are quite bare with no “discussion” item below them.
I wonder if it has anything to do with how often Megite’s crawlers “ping” the feedburner system, as I know when I’m getting picked up by Megite, this blog certainly does get contacted every few minutes on their behalf.
More to come as it develops.
6 Comments
Talked to Matt Chen a few hours back, this really is shitty of FB.
This sounds awful. I like the FB folks a whole lot though and reserve judgement until hearing their side of the story. Seems to me that Matt Chen is a good guy too, but gotta hear both sides of this story.
Marshall, we don’t say FB is not good, maybe it is their software that makes the decision. But since FB hosts other bloggers’ RSS feeds, “Currently feeding 328,504 publishers who’ve burned 554,266 feeds”, and most of famous bloggers, such as this blog, techcrunch, are using their service, FB should give a clear and reasonable guidelines on crawlers that are different from a normal site’s crawling policy because they host many different blogs’ feeds. Otherwise Megite like service sites can not get up to date or real time feeds if they say that they can only be crawled every 10 seconds. But anyhow, it would be nice that they can notify us and give us guidelines before blocking, Megite is willing to following their reasonable guidelines.
What is more, for some blogs’s feeds, they also change the link for each post inside the feeds. That means, Megite crawler needs to check FB first to get the original url for a post.
Update: FB guys works really fast and they removed the blocking of Megite. We still need to work out some details on how to crawl. Thanks all.
Hey all,
Eric from FeedBurner here. Just wanted to explain why Megite was blocked. We always look for a handful of things in each request/useragent we receive to determine how to handle it and whether or not we need to block it. The Megite requests has the following characteristics
1) The reverse DNS looked like it was coming from a home DSL user (netblock-66-245-250-218.dslextreme.com)
2) Megite is not using If-Modified-Since requests or compression
3) The identifying User-Agent (from our sampling) was a generic ‘libwww-perl/5.805′ rather than something that identified itself as Megite
Matt sent me a note this morning at 1am central time which I received when I got into the office this morning at 9am central. By 9:55am central time Joe Kottke, our Director of Network Operations, had unblocked Megite and sent an explanation to Matt including the info above.
So that’s the deal. No snubbing has taken place. :-) All is well in feed land now. If anyone has any questions about this (or anything that comes up in the future) please feel free to follow up with me any time (contact info below). Have a great superbowl weekend guys!
Cheers,
Eric
—-
Eric Olson
Publisher Services – FeedBurner
E-mail: erico@feedburner.com
Office: 312.756.0022 x2034
Mobile: 508.335.9221
Skype/AIM: EricJohnOlson
One Trackback
[...] Has Feedburner Snubbed Megite? Feedburner blocking Megite raises interesting questions. Isn’t their business to provide feeds? I depend on them for mine and would rather they don’t make such calls on my behalf. Hope they work it out. (tags: megite feedburner) [...]