I intend over the next few months to learnt Java with the purpose of building my own simple web crawler/spider. I have seen a few open source spiders but would like to build my own if possible.

What I would like to ask is how would I go about learning java and also would the building of a simple spider be very hard?

My requirements of the spider are as follows:

Go to the entered URL and gather all content from the site
Collect link structure

The app I am developing will need to be able to build a structured sitemap of the specified URL.

One final note is how would I go about building a browser add-on? What languages can they be built in and which browser is best/easiest to develop for?


Recommended Answers

All 12 Replies

I have built a java web crawler/spider before with a front end resembling google for a previous uni project and I would say it is a moderate program to try and do, not overly difficult but a definate challenge for a new java coder.

Some of the main bits you will need to learn to do this is iostreams to read the urls in and JDBC so that you can store the data(you could do it by reading into an array/vector but i wouldnt recommend it as it would eat memory).

There is loads on the web about spider methods and algorithms like word ranking etc but i am sure you have already read up about how they work.

It is probobly quite a good project as you could make it on the command line and then redo it with a gui later if you wanted to.

As for browser plugins I would probobly go for a firefox plugin but then again why stop at a search engine, why not build your own browser too. :mrgreen:

I think the java.sun site had a tutorial on creating one of these. This is actually the next project I want to take up!

hi black knight, i m also building a web crawler in java as a project work.can u giude me?i m new to java.

can any one help me out i want to build a web crawler

commented: Unnecessarily hijacking an old thread. -1
commented: 1. hijacking 's never fun 2. what's stopping you? -1

Only if you start a new thread for your request and demonstrate that you have made some effort on your own.

that link doesn't work...... and i can't find wizard solutions website, to try to join or whatever... how do you get to wizard solutions

The post is over four years old. Not everything on the web persists forever.

actually, i wasn't interested in that topic anymore

hi i am akash & i am designing my web crawler in JAVA, can you please send me the code....my id is <EMAIL SNIPPED>....thanks

hi i am akash & i am designing my web crawler in JAVA, can you please send me the code....my id is <EMAIL SNIPPED>....thanks

Here is the code

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.