Java communication with Internet

[joshSCH]

I've written many java apps; however, I have never created anything that communicated with the Internet, or more specifically, web pages. How exactly would you go about doing this.. For instance, say I wanted to write a program that would log me in Daniweb, and send messages or posts to all of its members..

Or, how would I write a program that would receive input from the user in the form of a user name, and then look up the username on a high scores webpage. The program would output the user's score, and subscore..

Am I making any sense at all? How difficult would it be to write something like this?

[joshSCH]

I suppose I would use URLConnection.. not too familiar with it though.

[Ezzaral]

First let me say that I haven't written any apps to do this myself, so perhaps there are specific libraries that make these things much easier, but basically you are just talking about communicating through a socket via http. In your login case, a POST would need to be sent with the appropriate data. In the web lookup case, you are merely requesting a page and parsing the html text that is returned.

For some basic examples of http through a socket, take a look at this example: http://www.exampledepot.com/egs/java.net/ReadFromURL.html?l=rel

There are few related examples listed at the bottom of the page - one of which involves sending a POST.

Anyway, that is pretty generic info but hopefully will give you a place to start.

[joshSCH]

Yea, thanks.. just looking for some basic information. If I have any question, I'll just post here.. you seem very knowledgeable :)

[Ezzaral]

Just ran across this interesting example on JavaWorld http://www.javaworld.com/javaworld/jw-11-2004/jw-1101-spider.html?page=2 "Create Intelligent Web Spiders" which demonstrates connecting to servers, parsing html, etc. It might provide some good techniques to get you started.

[joshSCH]

hmm.. Is there a way in which I can communicate with daniweb through java.. I've tried using an output stream writer to post for me, but it doesn't work.. The InputStreamReader can connect, and print the html associated with daniweb.. however, it does not actually read the posts b/c they are submitted through javascript (I think).. Am I making sense? I'm trying to discover how to read/write via embedded programs on websites..

[joshSCH]

hmm... you know what, I think I will have to examine the html source in order to discover what my program should interact with..

After examining the source for this site, I found

<form action="/search/index.php" method="post">

I think that if I am going to interact with posing, then I will have to use this method "post".. although I am not quite sure how to do this..

Am I assuming this correctly?

[edit] This might also be important:

<input type="hidden" name="loggedinuser" value="43776" /> <input type="submit" class="button" value="Post Quick Reply" accesskey="s" title="(Alt + S)"

[peter_budo]

hmm.. Is there a way in which I can communicate with daniweb through java.. I've tried using an output stream writer to post for me, but it doesn't work.. The InputStreamReader can connect, and print the html associated with daniweb.. however, it does not actually read the posts b/c they are submitted through javascript (I think).. Am I making sense? I'm trying to discover how to read/write via embedded programs on websites..

Did you actually considered that if you wish to do something like that you would need access to the database? Besides user-name and password you would have to know the structure of the datadase to be able to run queries to display you topic in selected section so then you can read them.

Also I'm little suprised that you have so limited knowledge of HTML

I'm not saying that thing is imposible but you would need detailed information about system as Dani and her team builded, you will need direct access to DB and even if you success what is the point? What would you achive with this?

[joshSCH]

Did you actually considered that if you wish to do something like that you would need access to the database? Besides user-name and password you would have to know the structure of the datadase to be able to run queries to display you topic in selected section so then you can read them.

Yes, as I said.. I figured that I would need to do something like that.. however, as I also said, I don't know how.

Also I'm little suprised that you have so limited knowledge of HTML

I'm surprised that you cannot spell 'surprised'. Anyway, I believe I found what I was looking for in the html.. Am I not correct?

I'm not saying that thing is imposible but you would need detailed information about system as Dani and her team builded, you will need direct access to DB and even if you success what is the point? What would you achive with this?

Would I need detailed information about the system? Not really.. wouldn't I just need to know what methods are specifically used to generate posts, and what specific calls are made to the server..

Why did you respond so rudely? I am simply asking how to do something.. if you don't know then don't post.. I always wanted to write a program that communicated with this online game highscore table.. and then me and twomers decided to race to see who could be the first to write an autoposting program for Daniweb (he in c++, and me in java) in the count down game.. So, I am trying to learn how I would go about communicating with a web server.. So, please try not to be so rude next time.. Thanks.

[peter_budo]

Sorry dude, but I do not see any rude word in my post just wondering how you want to achive something if you do not know technology you wish to use. I will ignore your attack on my spelling ;)

>Anyway, I believe I found what I was looking for in the html.. Am I not correct?
Unfortunately you found nothing as post method is general request to do something take an action. This method is usable by any programming or scripting language which is to work with URL request/response. You can learn more about post/get methods here.

>Would I need detailed information about the system? Not really.. wouldn't I just need to know what methods are specifically used to generate posts, and what specific calls are made to the server..
What about providing suporting informations like is a new post or answer to existing post, to which section your post will belong? That is the reason behind knowing the system you try to access.

>So, I am trying to learn how I would go about communicating with a web server..
You may start reading some tutorials on Sun website like Custom Networking and JDBC Database Access

[joshSCH]

Sorry dude, but I do not see any rude word in my post just wondering how you want to achive something if you do not know technology you wish to use.

The tone sounded rude to me..

Thanks for your advise.. It seems quite difficult, but I'll give it a try:-/

[ahmed.khalil]

Hi peter & joshsch

Reference to peter's comment simply peter if you don't know
Don't talk how hard is this?????

However joshsch just for you info u don't need to know the database
Diagrams and all what our friend descriped if this is true then how
Most of the network software communicate with google and how come the emails are sent from the applications without knowing the server database and alot more detailed examples however you don't need the database daigram since u are giving the input to the server so ur application can communicatin through input so for example for logging on you can use two ways the java auth api or by simply inputing directly in the textbox and submiting it by a post or get depending on the webpage so it's not underground hacking to do this task the computer can never tell if the one browse the webpage is a human or a stream they are all at the end requests and responses.

Really shame that i have to be so rude to peter in my first post

n.b: Excuse my english spealing joshsch :)

[peter_budo]

Not to worry I'm not so easily to be upset

However you would need to elaborate little more on subject, so far as I'm aware in order to make a post you have to be log in the webie, the post is send from a certain forum section and by this it inherite some identification. I do not think you can just blindly send some data and this will be add to DB.

So if you can please provide more explanation, there is always time to learn something new

[joshSCH]

What about when guests make posts? They don't log in, do they?

[peter_budo]

still it needs forum section identification plus you do not want to post as guest :)

[joshSCH]

That's true.. eh, this is definitely my hardest program yet..

[Fungus1487]

i have no clue of how to do this in JAVA but i am 99% certain you will not have to be logged in to do this you just use the userID value in the POST method which dani web send via a hidden input box i believe after looking at the html briefly daniweb uses the posthash input box to determine which thread to post in. just ismply firing this without telling it which thread it is in will not work. as if dani web have done their job correctly (and from the high quality of this site i presume so) they would have setup the databse to reject null entries in such fields.

[peter_budo]

I do recommend then you check activity behind website with this tool and tell me what did you actually seen.
If admin/moderator feel this should not be posted please remove the link/post