Web Crawler

[maverick_79in]

Hi,

Well i am new to these forum as well as IT field.But i was looking for a right form to post up my add for help.

I am an Aerospace student from scotland dont knwo much about IT.
I have been assigend the IT Project from the university to Design a SIMPLE WEB CRAWLER Using JAVA to get some scientific Earth data from the net and filter them according to the parameter they have given and for this these fuckers have gave me only 30 days.

I need a help from experts like u.Please i have got few question if any one can help me i would be thankful to you.
1) What's the basic for this i should read cause i havent done much proramming and if any one of u have got PDF can u forward me.
2) I tried to download abour crawler but could not get what sort of JAVA i have to learn wether all of it or part of it for my application.
3) Any online Tutorials or links to read.
4) Can any one can help/Teach me to write code ?
5) I have got 30 to max 45 Days to get it done.

Waiting for a positive feedback from any one in this forum.Please help me if u can.
You can email me on [e-mail]

Cheers,
Maverick.

[server_crash]

I wrote a very simple one not too long ago, actually, it was simpler than what your wanting. If you don't know much about programming, then you definitely have some work ahead of you.

As for the tutorials you'll need to read, there's a lot! You will want to read about Strings and String methods so that you can parse through all of the HTML code. Then, you got to worry about the Robot Exclusionary Standard, so you'll need to read about how to access files and all that good stuff. There's the java.net and java.io package you'll need to get familar with so you can create Streams, URls, and connections to the websites.
The java.sun.com site is the best place to look for those tutorials.
There's a lot more too it than that though. I really don't see how you can go into this not knowing much about the language and come out with something like that, but good luck. If you need anything else, let me know.

Here is some resources I looked at while creating mine, although in the end none of them turned out to be helpful..I just designed my own.
http://moguntia.ucd.ie/programming/webcrawler/
http://research.compaq.com/SRC/mercator/papers/www/paper.pdf
http://cis.poly.edu/suel/papers/crawl.pdf

[maverick_79in]

Hi,

Well sorry for the late reply cause i had my exams.
I have got the detail about the project.Its like i have to link the webcrawler to 2 websites and interact with this 2 sites.
The college have agreed to use any dummmy programme and make the necessary changes.

Can you help me with any ready dummy programme by which i can make the necessary changes and just provide them with basic module.

Regards,
Maverick.

I wrote a very simple one not too long ago, actually, it was simpler than what your wanting. If you don't know much about programming, then you definitely have some work ahead of you.

As for the tutorials you'll need to read, there's a lot! You will want to read about Strings and String methods so that you can parse through all of the HTML code. Then, you got to worry about the Robot Exclusionary Standard, so you'll need to read about how to access files and all that good stuff. There's the java.net and java.io package you'll need to get familar with so you can create Streams, URls, and connections to the websites.
The java.sun.com site is the best place to look for those tutorials.
There's a lot more too it than that though. I really don't see how you can go into this not knowing much about the language and come out with something like that, but good luck. If you need anything else, let me know.

Here is some resources I looked at while creating mine, although in the end none of them turned out to be helpful..I just designed my own.
http://moguntia.ucd.ie/programming/webcrawler/
http://research.compaq.com/SRC/mercator/papers/www/paper.pdf
http://cis.poly.edu/suel/papers/crawl.pdf

[server_crash]

That would be much easier, but I don't know of any starter programs like that. I'll search around and see what I can find.

[maverick_79in]

any news

Right now i am going through the JAVA BASICS

maverick

[freesoft_2000]

Hi everyone,

You can read the below threads, they comes with some sample codes and examples as well as detailed expalanations

http://java.sun.com/developer/technicalArticles/ThirdParty/WebCrawler/

http://www.devarticles.com/c/a/Java/Crawling-the-Web-with-Java/

I hope this helps you

Yours Sincerely

Richard West