What exactly is this crawler special about, and why didn;t this take off. Recently Wikia acquired this.
http://grub.org/

Sorry if this is a wrong thread to have posted

I think it is justs an add-ons... To access wiki...

Grub is a distributed web crawler. It maintains an index of url's including status information (changed/not-changed/unavailable). The problem is, it uses your cpu and bandwidth and they sell the information to other search engines.

Nice post JoejeF. That's a great tip for newbies.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.