I was playing around a bit scraping quotes from Yahoo and found a URL that will give you the quote as CSV text. You might see if this bit of code below helps you out:
try {
// Create a URL for the desired symbol
String symbol = "FFIDX";
URL url = new URL("http://download.finance.yahoo.com/d/quotes.csv?s="+symbol+"&f=sl1d1t1c1ohgv&e=.csv");
// Read all the text returned by the server
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
String inputLine;
StringBuilder doc = new StringBuilder();
while ((inputLine = in.readLine()) != null){
doc.append(inputLine);
}
if (doc!=null){
String[] parsed = doc.toString().split(",");
System.out.println("Symbol: "+parsed[0]);
System.out.println("Price: "+parsed[1]);
}
in.close();
} catch (MalformedURLException e) {
} catch (IOException e) {
}
Ezzaral
Posting Genius
15,985 posts since May 2007
Reputation Points: 3,250
Solved Threads: 847
Wow.. yea, I was working with approximately that same code trying to view the data.. However, you managed to find a csv, and easily extract the same data I've been searching for :)
If only yahoo had streaming quotes... :-/
I'll see if I can manage to follow your example using my broker's website..
joshSCH
Industrious Poster
4,864 posts since Jul 2005
Reputation Points: 1,315
Solved Threads: 10
hmm.. I just ran across JScrape, a java screen scraping API. Perhaps I can use this to read in the information! :)
joshSCH
Industrious Poster
4,864 posts since Jul 2005
Reputation Points: 1,315
Solved Threads: 10
Ah, hell..
Anyone know how to connect to a https website? I've got a username and password, but I have not a clue in the world as to how to code for this.. I've checked online, but to no avail :(
Also, on that JScrape: http://www.apsquared.net/JScrape.html
The program doesn't work for me.. I think there is a problem with the custom libraries.. how do I load them so I can import them properly?
joshSCH
Industrious Poster
4,864 posts since Jul 2005
Reputation Points: 1,315
Solved Threads: 10
Ah, hell..
Anyone know how to connect to a https website? I've got a username and password, but I have not a clue in the world as to how to code for this.. I've checked online, but to no avail :(
This is a short example I found over in another forum (I've never used HTTPS myself):
import java.net.URL;
import java.io.*;
import javax.net.ssl.HttpsURLConnection;
public class Test
{
public static void main(String[] args) throws Exception
{
String httpsURL = "https://your.https.url.here/";
URL myurl = new URL(httpsURL);
HttpsURLConnection con = (HttpsURLConnection)myurl.openConnection();
InputStream ins = con.getInputStream();
InputStreamReader isr=new InputStreamReader(ins);
BufferedReader in =new BufferedReader(isr);
String inputLine;
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
in.close();
}
}
Also, on that JScrape: http://www.apsquared.net/JScrape.html
The program doesn't work for me.. I think there is a problem with the custom libraries.. how do I load them so I can import them properly?
I assume you downloaded the "Lite" version? All of the libraries they mention are freely available, however on the site there is a "JScrape Full" link to download a version that includes all of the third party libs.
Ezzaral
Posting Genius
15,985 posts since May 2007
Reputation Points: 3,250
Solved Threads: 847
This is a short example I found over in another forum (I've never used HTTPS myself):
import java.net.URL;
import java.io.*;
import javax.net.ssl.HttpsURLConnection;
public class Test
{
public static void main(String[] args) throws Exception
{
String httpsURL = "https://your.https.url.here/";
URL myurl = new URL(httpsURL);
HttpsURLConnection con = (HttpsURLConnection)myurl.openConnection();
InputStream ins = con.getInputStream();
InputStreamReader isr=new InputStreamReader(ins);
BufferedReader in =new BufferedReader(isr);
String inputLine;
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
in.close();
}
}
Hmm.. but where do I enter my username/password?I assume you downloaded the "Lite" version? All of the libraries they mention are freely available, however on the site there is a "JScrape Full" link to download a version that includes all of the third party libs.No, I downloaded the full version.. the YahooStock.java program still doesn't execute. Even with the libraries that were installed.. I think there is a problem with it finding the libraries..
joshSCH
Industrious Poster
4,864 posts since Jul 2005
Reputation Points: 1,315
Solved Threads: 10
Hmm.. but where do I enter my username/password?
You will probably have to send those as a POST request and store the cookie or session ID that is returned by the server. Take a look at the page source to see if they are using a form for login and what parameters are being sent. I'm just speculating here, as I've never played around with http and web requests all that much.
As far as the ssl connection, see the following Sun info on Secure Socket Extension: http://java.sun.com/j2se/1.4.2/docs/guide/security/jsse/JSSERefGuide.html
No, I downloaded the full version.. the YahooStock.java program still doesn't execute. Even with the libraries that were installed.. I think there is a problem with it finding the libraries.. What editor are you using? You will need to add them to the project classpath and how to do so is specific to the editor.
Ezzaral
Posting Genius
15,985 posts since May 2007
Reputation Points: 3,250
Solved Threads: 847
I use JCreator.
And sorry if I confused anyone..
But, I need to connect to my broker's website via https before I can access the applet. So, first I need to write a program that will connect, and sign me in my account. Then I need one that will communicate with the applet. (The applet uses socks port btw).
joshSCH
Industrious Poster
4,864 posts since Jul 2005
Reputation Points: 1,315
Solved Threads: 10
Okay, I have scrapped they idea of an applet to applet communication process as the code is to arduous to go through.
However, I can achieve the same results from logging in, and simply using the web server. I have acquired JScraper to screen scrape the data for me, and I have finally figured out how to make it work in JCreator. Okay, so all is good with that.. but, I do not know how to use it specifically with Scottrade.
Like Ezzaral said, I need to figure out the POST parameters to login, then code it into java.. I think I may know the parameters, but you html gurus may want to double check me:
https://trading.scottrade.com/default.aspx
I am searching for ways to connect to the https server and log in.. but if anyone can help me, it would be greatly appreciated.
joshSCH
Industrious Poster
4,864 posts since Jul 2005
Reputation Points: 1,315
Solved Threads: 10
Sorry, you're reaching a bit too far past my Java networking/internet skills for me to be of much additional help.
I glanced at the page source for the login and there are several hidden form fields in addition to the account number and password. No idea what you would need to do for those.
It would be easier to just pull the csv text from Yahoo! :)
Ezzaral
Posting Genius
15,985 posts since May 2007
Reputation Points: 3,250
Solved Threads: 847