 |  |  |  |  |  |  |  |
Web Crawler
 |
Hi,
Well i am new to these forum as well as IT field.But i was looking for a right form to post up my add for help.
I am an Aerospace student from scotland dont knwo much about IT.
I have been assigend the IT Project from the university to Design a SIMPLE WEB CRAWLER Using JAVA to get some scientific Earth data from the net and filter them according to the parameter they have given and for this these ****ers have gave me only 30 days.
I need a help from experts like u.Please i have got few question if any one can help me i would be thankful to you.
1) What's the basic for this i should read cause i havent done much proramming and if any one of u have got PDF can u forward me.
2) I tried to download abour crawler but could not get what sort of JAVA i have to learn wether all of it or part of it for my application.
3) Any online Tutorials or links to read.
4) Can any one can help/Teach me to write code ?
5) I have got 30 to max 45 Days to get it done.
Waiting for a positive feedback from any one in this forum.Please help me if u can.
You can email me on pranav162@gmail.com
Cheers,
Maverick.
Well i am new to these forum as well as IT field.But i was looking for a right form to post up my add for help.
I am an Aerospace student from scotland dont knwo much about IT.
I have been assigend the IT Project from the university to Design a SIMPLE WEB CRAWLER Using JAVA to get some scientific Earth data from the net and filter them according to the parameter they have given and for this these ****ers have gave me only 30 days.
I need a help from experts like u.Please i have got few question if any one can help me i would be thankful to you.
1) What's the basic for this i should read cause i havent done much proramming and if any one of u have got PDF can u forward me.
2) I tried to download abour crawler but could not get what sort of JAVA i have to learn wether all of it or part of it for my application.
3) Any online Tutorials or links to read.
4) Can any one can help/Teach me to write code ?
5) I have got 30 to max 45 Days to get it done.
Waiting for a positive feedback from any one in this forum.Please help me if u can.
You can email me on pranav162@gmail.com
Cheers,
Maverick.
I wrote a very simple one not too long ago, actually, it was simpler than what your wanting. If you don't know much about programming, then you definitely have some work ahead of you.
As for the tutorials you'll need to read, there's a lot! You will want to read about Strings and String methods so that you can parse through all of the HTML code. Then, you got to worry about the Robot Exclusionary Standard, so you'll need to read about how to access files and all that good stuff. There's the java.net and java.io package you'll need to get familar with so you can create Streams, URls, and connections to the websites.
The java.sun.com site is the best place to look for those tutorials.
There's a lot more too it than that though. I really don't see how you can go into this not knowing much about the language and come out with something like that, but good luck. If you need anything else, let me know.
Here is some resources I looked at while creating mine, although in the end none of them turned out to be helpful..I just designed my own.
http://moguntia.ucd.ie/programming/webcrawler/
http://research.compaq.com/SRC/merca.../www/paper.pdf
http://cis.poly.edu/suel/papers/crawl.pdf
As for the tutorials you'll need to read, there's a lot! You will want to read about Strings and String methods so that you can parse through all of the HTML code. Then, you got to worry about the Robot Exclusionary Standard, so you'll need to read about how to access files and all that good stuff. There's the java.net and java.io package you'll need to get familar with so you can create Streams, URls, and connections to the websites.
The java.sun.com site is the best place to look for those tutorials.
There's a lot more too it than that though. I really don't see how you can go into this not knowing much about the language and come out with something like that, but good luck. If you need anything else, let me know.
Here is some resources I looked at while creating mine, although in the end none of them turned out to be helpful..I just designed my own.
http://moguntia.ucd.ie/programming/webcrawler/
http://research.compaq.com/SRC/merca.../www/paper.pdf
http://cis.poly.edu/suel/papers/crawl.pdf
Hi,
Well sorry for the late reply cause i had my exams.
I have got the detail about the project.Its like i have to link the webcrawler to 2 websites and interact with this 2 sites.
The college have agreed to use any dummmy programme and make the necessary changes.
Can you help me with any ready dummy programme by which i can make the necessary changes and just provide them with basic module.
Regards,
Maverick.
Well sorry for the late reply cause i had my exams.
I have got the detail about the project.Its like i have to link the webcrawler to 2 websites and interact with this 2 sites.
The college have agreed to use any dummmy programme and make the necessary changes.
Can you help me with any ready dummy programme by which i can make the necessary changes and just provide them with basic module.
Regards,
Maverick.
•
•
•
•
Originally Posted by server_crash
I wrote a very simple one not too long ago, actually, it was simpler than what your wanting. If you don't know much about programming, then you definitely have some work ahead of you.
As for the tutorials you'll need to read, there's a lot! You will want to read about Strings and String methods so that you can parse through all of the HTML code. Then, you got to worry about the Robot Exclusionary Standard, so you'll need to read about how to access files and all that good stuff. There's the java.net and java.io package you'll need to get familar with so you can create Streams, URls, and connections to the websites.
The java.sun.com site is the best place to look for those tutorials.
There's a lot more too it than that though. I really don't see how you can go into this not knowing much about the language and come out with something like that, but good luck. If you need anything else, let me know.
Here is some resources I looked at while creating mine, although in the end none of them turned out to be helpful..I just designed my own.
http://moguntia.ucd.ie/programming/webcrawler/
http://research.compaq.com/SRC/merca.../www/paper.pdf
http://cis.poly.edu/suel/papers/crawl.pdf
That would be much easier, but I don't know of any starter programs like that. I'll search around and see what I can find.
•
•
Join Date: Jun 2004
Posts: 609
Reputation: 
Solved Threads: 7
Practically a Master Poster
Hi everyone,
You can read the below threads, they comes with some sample codes and examples as well as detailed expalanations
http://java.sun.com/developer/techni...ty/WebCrawler/
http://www.devarticles.com/c/a/Java/...Web-with-Java/
I hope this helps you
Yours Sincerely
Richard West
You can read the below threads, they comes with some sample codes and examples as well as detailed expalanations
http://java.sun.com/developer/techni...ty/WebCrawler/
http://www.devarticles.com/c/a/Java/...Web-with-Java/
I hope this helps you
Yours Sincerely
Richard West
Microsoft uses "One World, One Web, One Program" as a slogan.
Doesn’t that sound like "Ein Volk, Ein Reich, Ein Führer" to you, too?
— Eric S. Raymond
Tell me what type of software do you like and what would you pay for it
http://www.daniweb.com/techtalkforums/thread19660.html
Doesn’t that sound like "Ein Volk, Ein Reich, Ein Führer" to you, too?
— Eric S. Raymond
Tell me what type of software do you like and what would you pay for it
http://www.daniweb.com/techtalkforums/thread19660.html
|
Similar Threads
- Building a simple Java web crawler (Java)
- Search Engine quesiton (Site Layout and Usability)
- accessing redirected pages through java crawler (Java)
- making a web crawler (Java)
Other Threads in the Java Forum
- Previous Thread: Changing background images in frames
- Next Thread: finding ip thru a java application
| Thread Tools | Search this Thread |
You need to join our community in order to build your list of favorite forums. You can visit the forum index for a listing of all forums.
Related Forum Features
Latest Java Posts
- Today's Posts
- Unanswered Threads
6 @param actuate android api applet application arc array arrays automation balls binary bluetooth bold business byte c++ chat class client code codesnippet collections compare component coordinates database defaultmethod detection doctype dragging ebook eclipse educational error file fractal froglogic game givemetehcodez graphics gui guitesting helpwithhomework hql html ide ideas image ingres input integer internet intersect invokingapacheantprogrammatically j2me java javaexcel javaprojects jni jpanel jtextarea julia linux list map method methods mobile mysql netbeans newbie nextline parameter php pong problem program programming project recursion recursive scanner sell server set sms sort sql string sun swing swt terminal threads tree web websites windows
