Member Avatar for tn2130

Hey everyone.

Im new to java and having some problems.

The main idea is to connect to a website and collect information off it and store it in an array.

What I want the program to do is to search the website find a key word, and store what comes after the key word..

on the front page of daniweb along the bottom of the website there is a section called "Tag Cloud" which is filled with tags / short words

Tag Cloud: "i want to store what is written here"

My idea is to first read in the html of the website and then search that file for the key word followed by the text using Scanner and StringTokenizer then store as a array.

is there a better way / easier?

where do you suggest i look for some examples

here is what i have so far.

import java.net.*;
import java.io.*;

public class URLReader {

    public static void main(String[] args) throws Exception {
        
        URL dweb = new URL("http://www.daniweb.com/");
        URLConnection dw = dweb.openConnection();
        BufferedReader in = new BufferedReader(new InputStreamReader(hc.getInputStream()));
        System.out.println("connected to daniweb");
        String inputLine;

        PrintStream out = new PrintStream(new FileOutputStream("OutFile.txt"));
        
        try {
        while ((inputLine = in.readLine()) != null)
            out.println(inputLine);

            //System.out.println(inputLine);
            //in.close();
        out.close();
        System.out.println("printed text to outfile");
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        }
                                       
        try {
            Scanner scan = new Scanner(OutFile.txt);
            String search = txtSearch.getText();
            while (scan.hasNextLine()) {
                line = scan.nextLine();
            //still working
                while (st.hasMoreTokens()) {
                    word = st.nextToken();
                    if (word == search) {
                   
                    } else {
                       
                    }
                }
            }
            scan.close();
            SearchWin.dispose();
        } catch (IOException iox) {
        }
    }

any help at all would be very much appreciated!

  • 2 Contributors
  • 1 Reply
  • 295 Views
  • 6 Hours Discussion Span
  • Latest Post Latest Post by NormR1

Dani AI

Generated

A concise plan that fits the thread: stream the page and parse its HTML with a real HTML parser (don’t roll your own tokenizing). is right to suggest processing the input as it is read rather than writing a full HTML dump to disk first; the Java URL/URLConnection examples show how to open a stream and read lines directly. (docs.oracle.com)

For HTML extraction, use jsoup (works well for Java, handles broken HTML, and lets you query with CSS selectors). Example workflow: fetch the Document, find the tag-cloud container with a selector, collect the text of each tag into a List, then if needed convert to an array. The jsoup cookbook documents selector patterns and extraction methods. (jsoup.org)

Code sketch (replace the selector with the actual container on the page):

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.util.*;

Document doc = Jsoup.connect("http://example.com").userAgent("Mozilla/5.0").get();
Elements items = doc.select(".tag-cloud a"); // adjust to actual selector
List<String> tags = new ArrayList<>();
for (Element e : items) tags.add(e.text());
String[] tagArray = tags.toArray(new String[0]);

A few practical notes: compare Java strings with equals() (not ==) when matching text, and prefer an ArrayList for collected results before converting to a fixed array. The Java tutorial documents string comparison details. (docs.oracle.com)

Avoid parsing HTML with ad hoc regexes or tokenizers—HTML nesting and real-world malformed markup make regex fragile; use a parser instead. Also follow the site’s crawling rules and terms (check robots.txt / the Robots Exclusion Protocol) before automated scraping. (stackoverflow.com)

This fills gaps in the original approach: stream instead of save, use an HTML parser (jsoup) for robust selection, and handle Java string equality and dynamic collection properly.

Member Avatar for NormR1

Why write the html page lines to a file? Scan them as they are read from the site.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.