943,587 Members | Top Members by Rank

Ad:
  • Java Discussion Thread
  • Unsolved
  • Views: 3639
  • Java RSS
You are currently viewing page 1 of this multi-page discussion thread
Oct 7th, 2007
0

write in FORMs with Java

Expand Post »
I`m trying to write a web crawler but i `don`t know how to handle the situation when i have to login to acces a site. How to type in that login form from my java program? I`ve searched on the net but the results points to JSP. I haven`t worked with JSP before, isn`t there a simplier method to complete those user and password HTML fields ??
Similar Threads
Reputation Points: 10
Solved Threads: 1
Light Poster
claudiu_is is offline Offline
40 posts
since Jul 2007
Oct 7th, 2007
0

Re: write in FORMs with Java

search google for 'curl', that should get you started...
Reputation Points: 30
Solved Threads: 3
Light Poster
schoolsoluction is offline Offline
27 posts
since Sep 2007
Oct 7th, 2007
0

Re: write in FORMs with Java

You are talking about JavacURL right ? the cURL library for Java, well, I tried to use that but the documentation from there http://curl.haxx.se/libcurl/ is made for C. In the downloaded packets of JavacURL i found just some instructions for installing that library in the Eclipse editor and a javadoc with few classes and methods but with lots of fields; I don`t know what fields I should use in my program ....Isn`t there another way to resolv my problem except this stupid JavacURL ? I`ve lost hours tonight fiding a way to make use this.. If there isn`t another solution, I`ll try again in the morning, but I hope you guys will gave some usefull tips about JavacURL
Reputation Points: 10
Solved Threads: 1
Light Poster
claudiu_is is offline Offline
40 posts
since Jul 2007
Oct 8th, 2007
0

Re: write in FORMs with Java

You can call the .exe or shell script with some arguments...

Thats the way I do it... Curls saves the results in text files and then I analyse those files... followed by a new call to curl...
Reputation Points: 30
Solved Threads: 3
Light Poster
schoolsoluction is offline Offline
27 posts
since Sep 2007
Oct 8th, 2007
0

Re: write in FORMs with Java

Click to Expand / Collapse  Quote originally posted by claudiu_is ...
I`m trying to write a web crawler but i `don`t know how to handle the situation when i have to login to acces a site. How to type in that login form from my java program? I`ve searched on the net but the results points to JSP. I haven`t worked with JSP before, isn`t there a simplier method to complete those user and password HTML fields ??
HttpUrlConnection as a POST request. See http://coding.derkeiler.com/Archive/.../msg01212.html for an example (found in 20 seconds using Yahoo). It is for HttpsUrlConnection, rather than HttpUrlConnection, so strip out the SSL stuff, otherwise, it should be the same.

In summary for you, read the form, finding the different parameters (including the hidden ones), then, using those parameter names, create a POST response and connect again.

Hopefully, you are familiar with the forms already, as I would not count on being able to create some generic login form manipulator.

Edit: And, yes, the code is not that good, and he is complaining that it is not working over a proxy, but it is at least a starting point for you.
Last edited by masijade; Oct 8th, 2007 at 8:05 am.
Moderator
Reputation Points: 1471
Solved Threads: 490
Industrious Poster
masijade is offline Offline
4,043 posts
since Feb 2006
Oct 8th, 2007
0

Re: write in FORMs with Java

A nice handy help for doing network thingies from Java is Apache Commons Net (and/or Apache Commons HttpClient).
Team Colleague
Reputation Points: 1658
Solved Threads: 331
duckman
jwenting is offline Offline
7,719 posts
since Nov 2004
Oct 10th, 2007
0

Re: write in FORMs with Java

i`ve just found out that i need session authentication(this is the tupe of authentication for trackers), does that mean that i have to work with cookies ? uhhh, cookies are new for me
Last edited by claudiu_is; Oct 10th, 2007 at 6:20 am.
Reputation Points: 10
Solved Threads: 1
Light Poster
claudiu_is is offline Offline
40 posts
since Jul 2007
Oct 10th, 2007
0

Re: write in FORMs with Java

Yes indeed.

With curl for exemple, you login. Then you get a session coockie. That one you write to a file and pass it on to all your later curl calls...
Reputation Points: 30
Solved Threads: 3
Light Poster
schoolsoluction is offline Offline
27 posts
since Sep 2007
Jul 1st, 2009
0

Re: write in FORMs with Java

I am using URL Connection to login to the site http://sms.vn/send/login.jsp but it just returns the same login page again.
My program will do like this: it opens an url connection to get the login form from the site, fill in the user name and password, ask for user to type captcha and send the POST request to the site with those information. My code works for another site like:
http://www.1your.com/drupal/sample_login.php
I am so sure that my user name and password are right, you can try it, it is free and will not cause much harm for me
Java Syntax (Toggle Plain Text)
  1. import java.io.IOException;
  2. import java.io.PrintStream;
  3. import java.net.MalformedURLException;
  4. import java.net.URL;
  5. import java.net.URLConnection;
  6. import java.net.URLEncoder;
  7. import java.util.Scanner;
  8. import java.util.logging.Level;
  9. import java.util.logging.Logger;
  10.  
  11. /**
  12.  *
  13.  * @author Duy Khang
  14.  */
  15. public class NewClass2 {
  16.  
  17. private static String LOGIN_INPUT = "login";
  18. private static String LOGIN_VALUE = "LOGIN";
  19. private static String NICK_INPUT = "nick";
  20. private static String NICK_VALUE = "vodkhang";
  21. private static String PASS_INPUT = "pass";
  22. private static String PASS_VALUE = "vhnkozou";
  23. private static String CHECK_CODE_INPUT = "checkcode";
  24. private static String CHECK_CODE_VALUE = "";
  25.  
  26. public static void main(String[] args) {
  27. try {
  28. URL url = new URL("http://sms.vn/send/login.jsp");
  29. URLConnection connection = url.openConnection();
  30. connection.setDoOutput(true);
  31.  
  32. Scanner sc = new Scanner(connection.getInputStream());
  33. while (sc.hasNextLine()) {
  34. System.out.println(sc.nextLine());
  35. }
  36. Scanner sc2 = new Scanner(System.in);
  37. System.out.println("enter the check code value: ");
  38. CHECK_CODE_VALUE = sc2.nextLine();
  39.  
  40. String encodedLoginUserName = URLEncoder.encode(NICK_VALUE, "UTF-8");
  41. String encodedLoginPassword = URLEncoder.encode(PASS_VALUE, "UTF-8");
  42.  
  43.  
  44. URLConnection connection1 = url.openConnection();
  45. connection1.setDoOutput(true);
  46. PrintStream output = new PrintStream(connection1.getOutputStream());
  47.  
  48. StringBuilder request = new StringBuilder();
  49. request.append(LOGIN_INPUT);
  50. request.append("=");
  51. request.append(LOGIN_VALUE);
  52. request.append("&");
  53.  
  54. request.append(NICK_INPUT);
  55. request.append("=");
  56. request.append(encodedLoginUserName);
  57. request.append("&");
  58.  
  59. request.append(PASS_INPUT);
  60. request.append("=");
  61. request.append(encodedLoginPassword);
  62. request.append("&");
  63.  
  64. request.append(CHECK_CODE_INPUT);
  65. request.append("=");
  66. request.append(CHECK_CODE_VALUE);
  67. System.out.println("request: " + request);
  68.  
  69. output.println(request);
  70. System.out.println("----------------------------------------------------");
  71. Scanner sc3 = new Scanner(connection1.getInputStream());
  72. while (sc3.hasNextLine()) {
  73. System.out.println(sc3.nextLine());
  74. }
  75. } catch (MalformedURLException ex) {
  76. Logger.getLogger(NewClass1.class.getName()).log(Level.SEVERE, null, ex);
  77. } catch (IOException ex) {
  78. Logger.getLogger(NewClass1.class.getName()).log(Level.SEVERE, null, ex);
  79. }
  80. }
  81. }
Reputation Points: 10
Solved Threads: 0
Newbie Poster
dkhang is offline Offline
3 posts
since Apr 2008
Jul 1st, 2009
0

Re: write in FORMs with Java

I am using URL Connection to login to the site http://sms.vn/send/login.jsp but it just returns the same login page again.
My program will do like this: it opens an url connection to get the login form from the site, fill in the user name and password, ask for user to type captcha and send the POST request to the site with those information. My code works for another site like:
http://www.1your.com/drupal/sample_login.php
I am so sure that my user name and password are right, you can try it, it is free and will not cause much harm for me
Java Syntax (Toggle Plain Text)
  1.  
  2. import java.io.IOException;
  3. import java.io.PrintStream;
  4. import java.net.MalformedURLException;
  5. import java.net.URL;
  6. import java.net.URLConnection;
  7. import java.net.URLEncoder;
  8. import java.util.Scanner;
  9. import java.util.logging.Level;
  10. import java.util.logging.Logger;
  11.  
  12. /**
  13.  *
  14.  * @author Duy Khang
  15.  */
  16. public class NewClass2 {
  17.  
  18. private static String LOGIN_INPUT = "login";
  19. private static String LOGIN_VALUE = "LOGIN";
  20. private static String NICK_INPUT = "nick";
  21. private static String NICK_VALUE = "vodkhang";
  22. private static String PASS_INPUT = "pass";
  23. private static String PASS_VALUE = "xxxxx";
  24. private static String CHECK_CODE_INPUT = "checkcode";
  25. private static String CHECK_CODE_VALUE = "";
  26.  
  27. public static void main(String[] args) {
  28. try {
  29. URL url = new URL("http://sms.vn/send/login.jsp");
  30. URLConnection connection = url.openConnection();
  31. connection.setDoOutput(true);
  32.  
  33. Scanner sc = new Scanner(connection.getInputStream());
  34. while (sc.hasNextLine()) {
  35. System.out.println(sc.nextLine());
  36. }
  37. Scanner sc2 = new Scanner(System.in);
  38. System.out.println("enter the check code value: ");
  39. CHECK_CODE_VALUE = sc2.nextLine();
  40.  
  41. String encodedLoginUserName = URLEncoder.encode(NICK_VALUE, "UTF-8");
  42. String encodedLoginPassword = URLEncoder.encode(PASS_VALUE, "UTF-8");
  43.  
  44.  
  45. URLConnection connection1 = url.openConnection();
  46. connection1.setDoOutput(true);
  47. PrintStream output = new PrintStream(connection1.getOutputStream());
  48.  
  49. StringBuilder request = new StringBuilder();
  50. request.append(LOGIN_INPUT);
  51. request.append("=");
  52. request.append(LOGIN_VALUE);
  53. request.append("&");
  54.  
  55. request.append(NICK_INPUT);
  56. request.append("=");
  57. request.append(encodedLoginUserName);
  58. request.append("&");
  59.  
  60. request.append(PASS_INPUT);
  61. request.append("=");
  62. request.append(encodedLoginPassword);
  63. request.append("&");
  64.  
  65. request.append(CHECK_CODE_INPUT);
  66. request.append("=");
  67. request.append(CHECK_CODE_VALUE);
  68. System.out.println("request: " + request);
  69.  
  70. output.println(request);
  71. System.out.println("----------------------------------------------------");
  72. Scanner sc3 = new Scanner(connection1.getInputStream());
  73. while (sc3.hasNextLine()) {
  74. System.out.println(sc3.nextLine());
  75. }
  76. } catch (MalformedURLException ex) {
  77. Logger.getLogger(NewClass1.class.getName()).log(Level.SEVERE, null, ex);
  78. } catch (IOException ex) {
  79. Logger.getLogger(NewClass1.class.getName()).log(Level.SEVERE, null, ex);
  80. }
  81. }
  82. }
Reputation Points: 10
Solved Threads: 0
Newbie Poster
dkhang is offline Offline
3 posts
since Apr 2008

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.
Message:
Previous Thread in Java Forum Timeline: J2ME file system access
Next Thread in Java Forum Timeline: Simple question annoying the crap out of me





About Us | Contact Us | Advertise | Acceptable Use Policy
Forum Index | Build Custom RSS Feed


Follow us on Twitter


© 2011 DaniWeb® LLC