parsing html

Hello!

Look at Html Parse Demo example here:
http://javafaq.nu/java-example-code-656.html

If it does not work for you just search for "HTMLEditorKit" there
I found ~10 examples on how to handle different tags..
By the way, there are all examples are by API, package, class... So you can find everything you need fast yourself

allexx

I read where the microsoft encoding name really isn't a valid encoding name! I haven't read anything about what to change it to or anything, though. You could also look into adding support for it via the charsetprovider. My guess is that charset is not supported, so try this:

boolean isSupported(String charsetName)

and see if it is or not.

I checked to see if it was supported, and it said it was.

I took a look at the exception a little, and it seems a bit weird. It happens as the name implies, whenever the charset is changed.......

But when and why is it changed?? (I guess that would solve everything)

I'm only going to take a stab, but I think you need some decoding or something. The read method or the editor kit is converting to some kind of format that it likes, regardless of whether you specifiy otherwise. I don't know that format and I don't know how to find out. I just think you need to convert before you try to read...

Maybe I'm wrong, but it could be worth a try.

This is driving me nuts.
Should this not be decoding?

Charset cs = Charset.forName("windows-1252");
Reader reader = new InputStreamReader(conn.getInputStream(),cs.newDecoder());

Did some searching around bug forums on Sun website. Though not a bug, I found related problems and this seems to work ok. I haven't tried parsing anything yet, but its not throwing the error anymore.

HTMLDocument doc = (HTMLDocument)kit.createDefaultDocument();
//added this method call
doc.putProperty("IgnoreCharsetDirective", new Boolean(true));

kit.read(reader, doc, 0);

Reader reader = new InputStreamReader(conn.getInputStream(),cs.newDecoder());

cs.newDecoder() just creates a new decoder. The question is, will the reader automatically decode using your decoder? You'll have to answer that one because I don't know.

I haven't tried parsing anything yet, but its not throwing the error anymore.

So it's working now? Not sure why it wouldn't work before and work now. Only time I've seen such this is when a crappy IDE like bluj is used..doubt that's the problem.

heh, i was using bluej. i need to update my netbeans. But it started working after I set the property to ignore the charset, before I didn't have that set to ignore. But I got my document parsed and everything sorted the way I want it.

Got a new problem. I just can't seem to figure out how to get the value of the Anchor tag. Not the href attribute, but the value between the opening and closing tags.

<a href="http://something.com">i want this text</a>

Here's the full code for how I'm currently getting the attributes.

import java.net.*;
import java.io.*;
import javax.swing.text.html.*;
import javax.swing.text.EditorKit;
import javax.swing.text.*;
import java.nio.charset.Charset;
 
public class Nullsoft
{
	
	public Nullsoft()
	{
		/*
		 * iTunes radio station lists?
		 * http://pri.kts-af.net/
		 */
		
		
		String genre = "ambient";
		String link = "<A href="http://yp.shoutcast.com/directory/index.phtml?s="+genre">http://yp.shoutcast.com/directory/index.phtml?s="+genre;
		
		URL url = null;
		try
		{
			url = new URL(link);
			URLConnection conn = url.openConnection();
			Reader reader = new InputStreamReader(conn.getInputStream());
			
			EditorKit kit = new HTMLEditorKit();
			HTMLDocument doc = (HTMLDocument)kit.createDefaultDocument();
			doc.putProperty("IgnoreCharsetDirective", new Boolean(true));
			kit.read(reader, doc, 0);
			
			HTMLDocument.Iterator it = doc.getIterator(HTML.Tag.A);
			
			while(it.isValid())
			{
				SimpleAttributeSet s = (SimpleAttributeSet)it.getAttributes();
				String href = (String)s.getAttribute(HTML.Attribute.HREF);
				System.out.println(href);
				it.next();
			}
			
		}
		catch(ChangedCharSetException e){
			System.out.println(e.getCharSetSpec());
		}
		catch(Exception e){
			System.out.println(e);
		}
		
	}
	
	/**
	 *
	 */
	public static void main(String[] args)
	{
		Nullsoft ns = new Nullsoft();
	}
}

Figures, soon as I post this I got an idea. Since I can get the offsets of the tag within the document, why not just extract the text straight from the document myself?

int start = it.getStartOffset();
int end = it.getEndOffset();
String name = doc.getText(start, end-start);

I thought that might return the tags themselves, but its not. Gives me exactly what I wanted.

I think I did the exact same thing once writing a console web crawler. Took some tricky work with the indexOf() method.

Well, actually it was the link i was after.

Hi,

I am getting a javax.swing.text.ChangedCharSetException when I use the following code. Where do I set the prperty that you are talking about? ( I have a meta tag that is causing the exception.

try {
Reader r = new FileReader("PJMData.htm");
ParserDelegator parser = new ParserDelegator();
HTMLEditorKit.ParserCallback callback = new PJMParser();
parser.parse(r, callback, false);
} catch (IOException e) {
e.printStackTrace();
}

Let me know. Thanks.

I'm not sure in your case, since you're using the parser callback whereas I read the html into a document. The property I mentioned is set on the document itself.

doc.putProperty("IgnoreCharsetDirective", new Boolean(true));

Could you read your htm file into a document first then use the parser on it?