Ad:

Java Discussion Thread
Unsolved
Views: 14491

Page 1
2
Next

You are currently viewing page 1 of this multi-page discussion thread

Feb 19th, 2006

parsing html

Expand Post »

The probably isn't the parsing actually, I can't even get to that part yet. The webpage uses a different character set, "windows-1252". But even after setting the reader to use that charset (which exists in the system), I still get the ChangedCharSetException.

Java Syntax (Toggle Plain Text)
String link = "myurl.com";
 
URL url = new URL(link);
			URLConnection conn = url.openConnection();
			Reader reader = new InputStreamReader(conn.getInputStream(),Charset.forName("windows-1252"));
 
			EditorKit kit = new HTMLEditorKit();
			HTMLDocument doc = (HTMLDocument)kit.createDefaultDocument();
			//throws error here while reading
			kit.read(reader, doc, 0);
String link = "myurl.com";
 
URL url = new URL(link);
			URLConnection conn = url.openConnection();
			Reader reader = new InputStreamReader(conn.getInputStream(),Charset.forName("windows-1252"));
			
			EditorKit kit = new HTMLEditorKit();
			HTMLDocument doc = (HTMLDocument)kit.createDefaultDocument();
			//throws error here while reading
			kit.read(reader, doc, 0);

Here's the first couple lines from the html file:

(Toggle Plain Text)

<html>
<head>
<meta http-equiv="Content-Language" content="en-us">
<meta http-equiv="Content-Type" content="text/html; charset=windows-1252">
<meta http-equiv="Pragma" content="no-cache">

Is there perhaps some way of reading the file but ignoring the meta data?

Similar Threads

Parsing html form. in PHP

Phaelax

Reputation Points: 92

Solved Threads: 51

Practically a Posting Shark

Offline

856 posts
since Mar 2004

Permalink

Feb 21st, 2006

Re: parsing html

Hello!

Look at Html Parse Demo example here:
http://javafaq.nu/java-example-code-656.html

If it does not work for you just search for "HTMLEditorKit" there
I found ~10 examples on how to handle different tags..
By the way, there are all examples are by API, package, class... So you can find everything you need fast yourself

allexx

allexx

Reputation Points: 10

Solved Threads: 0

Newbie Poster

Offline

1 posts
since Feb 2006

Permalink

Feb 21st, 2006

Re: parsing html

I read where the microsoft encoding name really isn't a valid encoding name! I haven't read anything about what to change it to or anything, though. You could also look into adding support for it via the charsetprovider. My guess is that charset is not supported, so try this:

Java Syntax (Toggle Plain Text)
boolean isSupported(String charsetName)
boolean isSupported(String charsetName)

and see if it is or not.

server_crash

Reputation Points: 113

Solved Threads: 19

Postaholic

Offline

2,108 posts
since Jun 2004

Permalink

Feb 22nd, 2006

Re: parsing html

I checked to see if it was supported, and it said it was.

Phaelax

Reputation Points: 92

Solved Threads: 51

Practically a Posting Shark

Offline

856 posts
since Mar 2004

Permalink

Feb 22nd, 2006

Re: parsing html

I took a look at the exception a little, and it seems a bit weird. It happens as the name implies, whenever the charset is changed.......

But when and why is it changed?? (I guess that would solve everything)

I'm only going to take a stab, but I think you need some decoding or something. The read method or the editor kit is converting to some kind of format that it likes, regardless of whether you specifiy otherwise. I don't know that format and I don't know how to find out. I just think you need to convert before you try to read...

Maybe I'm wrong, but it could be worth a try.

server_crash

Reputation Points: 113

Solved Threads: 19

Postaholic

Offline

2,108 posts
since Jun 2004

Permalink

Feb 23rd, 2006

Re: parsing html

This is driving me nuts.
Should this not be decoding?

Java Syntax (Toggle Plain Text)
Charset cs = Charset.forName("windows-1252");
Reader reader = new InputStreamReader(conn.getInputStream(),cs.newDecoder());
Charset cs = Charset.forName("windows-1252");
Reader reader = new InputStreamReader(conn.getInputStream(),cs.newDecoder());

Did some searching around bug forums on Sun website. Though not a bug, I found related problems and this seems to work ok. I haven't tried parsing anything yet, but its not throwing the error anymore.

Java Syntax (Toggle Plain Text)
HTMLDocument doc = (HTMLDocument)kit.createDefaultDocument();
//added this method call
doc.putProperty("IgnoreCharsetDirective", new Boolean(true));
 
kit.read(reader, doc, 0);
HTMLDocument doc = (HTMLDocument)kit.createDefaultDocument();
//added this method call
doc.putProperty("IgnoreCharsetDirective", new Boolean(true));

kit.read(reader, doc, 0);

Phaelax

Reputation Points: 92

Solved Threads: 51

Practically a Posting Shark

Offline

856 posts
since Mar 2004

Permalink

Feb 23rd, 2006

Re: parsing html

Quote originally posted by Phaelax ...

Reader reader = new InputStreamReader(conn.getInputStream(),cs.newDecoder());

cs.newDecoder() just creates a new decoder. The question is, will the reader automatically decode using your decoder? You'll have to answer that one because I don't know.

Quote ...

I haven't tried parsing anything yet, but its not throwing the error anymore.

So it's working now? Not sure why it wouldn't work before and work now. Only time I've seen such this is when a crappy IDE like bluj is used..doubt that's the problem.

server_crash

Reputation Points: 113

Solved Threads: 19

Postaholic

Offline

2,108 posts
since Jun 2004

Permalink

Feb 24th, 2006

Re: parsing html

heh, i was using bluej. i need to update my netbeans. But it started working after I set the property to ignore the charset, before I didn't have that set to ignore. But I got my document parsed and everything sorted the way I want it.

Phaelax

Reputation Points: 92

Solved Threads: 51

Practically a Posting Shark

Offline

856 posts
since Mar 2004

Permalink

Feb 26th, 2006

Re: parsing html

Got a new problem. I just can't seem to figure out how to get the value of the Anchor tag. Not the href attribute, but the value between the opening and closing tags.

Java Syntax (Toggle Plain Text)

<a href="http://something.com">i want this text</a>

Here's the full code for how I'm currently getting the attributes.

Java Syntax (Toggle Plain Text)
import java.net.*;
import java.io.*;
import javax.swing.text.html.*;
import javax.swing.text.EditorKit;
import javax.swing.text.*;
import java.nio.charset.Charset;
 
public class Nullsoft
{
 
	public Nullsoft()
	{
		/*
		 * iTunes radio station lists?
		 * <a rel="nofollow" href="http://pri.kts-af.net/" target="_blank">http://pri.kts-af.net/</a>
		 */
 
 
		String genre = "ambient";
		String link = "<A href="http://yp.shoutcast.com/directory/index.phtml?s="+genre">http://yp.shoutcast.com/directory/index.phtml?s="+genre;
 
		URL url = null;
		try
		{
			url = new URL(link);
			URLConnection conn = url.openConnection();
			Reader reader = new InputStreamReader(conn.getInputStream());
 
			EditorKit kit = new HTMLEditorKit();
			HTMLDocument doc = (HTMLDocument)kit.createDefaultDocument();
			doc.putProperty("IgnoreCharsetDirective", new Boolean(true));
			kit.read(reader, doc, 0);
 
			HTMLDocument.Iterator it = doc.getIterator(HTML.Tag.A);
 
			while(it.isValid())
			{
				SimpleAttributeSet s = (SimpleAttributeSet)it.getAttributes();
				String href = (String)s.getAttribute(HTML.Attribute.HREF);
				System.out.println(href);
				it.next();
			}
 
		}
		catch(ChangedCharSetException e){
			System.out.println(e.getCharSetSpec());
		}
		catch(Exception e){
			System.out.println(e);
		}
 
	}
 
	/**
	 *
	 */
	public static void main(String[] args)
	{
		Nullsoft ns = new Nullsoft();
	}
}
import java.net.*;
import java.io.*;
import javax.swing.text.html.*;
import javax.swing.text.EditorKit;
import javax.swing.text.*;
import java.nio.charset.Charset;
 
public class Nullsoft
{
	
	public Nullsoft()
	{
		/*
		 * iTunes radio station lists?
		 * <a rel="nofollow" href="http://pri.kts-af.net/" target="_blank">http://pri.kts-af.net/</a>
		 */
		
		
		String genre = "ambient";
		String link = "<A href="http://yp.shoutcast.com/directory/index.phtml?s="+genre">http://yp.shoutcast.com/directory/index.phtml?s="+genre;
		
		URL url = null;
		try
		{
			url = new URL(link);
			URLConnection conn = url.openConnection();
			Reader reader = new InputStreamReader(conn.getInputStream());
			
			EditorKit kit = new HTMLEditorKit();
			HTMLDocument doc = (HTMLDocument)kit.createDefaultDocument();
			doc.putProperty("IgnoreCharsetDirective", new Boolean(true));
			kit.read(reader, doc, 0);
			
			HTMLDocument.Iterator it = doc.getIterator(HTML.Tag.A);
			
			while(it.isValid())
			{
				SimpleAttributeSet s = (SimpleAttributeSet)it.getAttributes();
				String href = (String)s.getAttribute(HTML.Attribute.HREF);
				System.out.println(href);
				it.next();
			}
			
		}
		catch(ChangedCharSetException e){
			System.out.println(e.getCharSetSpec());
		}
		catch(Exception e){
			System.out.println(e);
		}
		
	}
	
	/**
	 *
	 */
	public static void main(String[] args)
	{
		Nullsoft ns = new Nullsoft();
	}
}

Phaelax

Reputation Points: 92

Solved Threads: 51

Practically a Posting Shark

Offline

856 posts
since Mar 2004

Permalink

Feb 26th, 2006

Re: parsing html

Figures, soon as I post this I got an idea. Since I can get the offsets of the tag within the document, why not just extract the text straight from the document myself?

Java Syntax (Toggle Plain Text)
int start = it.getStartOffset();
int end = it.getEndOffset();
String name = doc.getText(start, end-start);
int start = it.getStartOffset();
int end = it.getEndOffset();
String name = doc.getText(start, end-start);

I thought that might return the tags themselves, but its not. Gives me exactly what I wanted.

Phaelax

Reputation Points: 92

Solved Threads: 51

Practically a Posting Shark

Offline

856 posts
since Mar 2004

Page 1
2
Next

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.

Previous Thread in Java Forum Timeline: Drag and Drop headaches

Next Thread in Java Forum Timeline: Topic -gui

DaniWeb Message

Cancel Changes

User Name
Password