Member Avatar

Hi all,

I recently made a web crawler as part of a Uni team project building a small search engine. I've managed to incorporate it into our GUI, there is a button which starts the crawling process by creating the crawler in a thread. (I did this because otherwise the button would freeze while the crawler worked)

However I would also like the crawler to update a JLabel on the GUI while it's working, i.e. how many it's done and how many it has to go.

How can I set this? I've tried a while loop in the button event, this doesn't work. Also tried setting it from the crawler but it cannot access from outside the object. How would you solve this?

Any pointers or advice in general is appreciated also. I enjoy programming and am eager to learn if I'm doing anything wrong or could improve any of my practices. Thank you :)

import javax.swing.*;
import java.awt.*;
import java.awt.event.*;
import java.lang.Thread;

public class CrawlAnalyseGUI extends JFrame
	private JButton crawlButton, searchButton;
	private JTextField searchField;
	private JLabel crawlerStatus;
	private JPanel searchPanel, crawlerPanel;
	public CrawlAnalyseGUI()
		final WebCrawler crawler = new WebCrawler();
		searchField = new JTextField("Enter Search Term", 10);
		searchButton = new JButton("Search");
		searchButton.addMouseListener(new MouseAdapter()
			public void mouseReleased(MouseEvent e)
				// do search!
		searchPanel = new JPanel();
		crawlerStatus = new JLabel("Waiting to start...");
		crawlButton = new JButton("Start Crawler");
		crawlButton.addMouseListener(new MouseAdapter()
			public void mouseReleased(MouseEvent e)
				String buttonText = crawlButton.getText();
				if(buttonText.equals("Start Crawler"))
					/* does not update label...

					while(crawler.Q.size() > 1)
						crawlerStatus.setText("Left in queue: "+ crawler.Q.size() +", and "+ crawler.FoundDocs.size() +"Pages crawled.");
					crawlButton.setText("Stop Crawler");
					// deprecated/unsafe. What should I do instead?
		crawlerPanel = new JPanel();
		this.setLayout(new GridLayout(2, 1, 5, 5));
	public static void main(String[] args)
		CrawlAnalyseGUI gui = new CrawlAnalyseGUI();

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;


import java.lang.Thread;
import java.util.ArrayList;

public class WebCrawler extends Thread
	ArrayList<CrawledDoc> FoundDocs = new ArrayList<CrawledDoc>();
	ArrayList<String> Q = new ArrayList<String>();
	ArrayList<String> backupQ = new ArrayList<String>();
	boolean correctProtocol = true;
	Document page = null;
	Element firstP;
	Elements links;
	int URLcount = 0, docID = 0;
	String[] initialURLs = {"", "", "", "", "", "", ""};
	String title, content, abslink = null, stemmed;
	TextAnalyser Analyse = new TextAnalyser();
	public void run()
		// add beginning URLs to Queue
		for(int counter = 0; counter < initialURLs.length; counter++)
		// overall loop which ends when all URLs have been crawled
		while(Q.size() > 1)
			// remove the front URL
			String L = Q.remove(1);
			System.out.println("Opening: "+ L);
			// connect to URL and get the HTML page
				page = Jsoup.connect(L).get();
			catch(SocketException e){}
			catch(IOException e){}
			catch(IllegalArgumentException e){}
			// retrieve all the links from the page, the first paragrah and the title.
			// (all taken from the retrieved HTML).
			links ="a[href]");
			firstP ="p").first();
			title = page.title();
			// check that content was found, replace if not, therefore avoiding NullPointerException
			if(firstP == null) content = title;
			else content = firstP.text();
			// loop to check every link found on the page which has been crawled
			// ensure its a link to a web-page and that we don't already have it.
			// then it can be added.
			for(Element link : links)
				abslink = link.attr("abs:href");
				if((abslink.startsWith("ftp:")) || (abslink.startsWith("mailto:"))) correctProtocol = false;
				if((!Q.contains(abslink)) && correctProtocol) 
					System.out.println(abslink + " has been added to the queue.");
        	// because a document has been opened, an object will be made
        	// this object needs a unique ID, so an integer keeps track and is used every time

			// Because we don't know how many tokens we may get from each page
			// we make a new ArrayList each time. The content is tokenized
			ArrayList<String> tokened = new ArrayList<String>();
			tokened = Analyse.tokenizer(content);
			// these tokens are then put through a stopper and porters algorithm.
			// The output of this is saved alongside the original content (for readability)
			String stopped = Analyse.stopper(tokened);
			Porter porter = new Porter(stopped);
			stemmed = porter.getStem();

			// Object is created saving all the information about this page.
			// The link is saved so that we can click on it and visit the full page
			// The documentID is kept so we can distinguish between them in the inverted file.
			// The title is saved so we can use it as a title for results.
			// The original content is saved for readability on the results page, and the stemmed to be searched.
			// Finally add this Object to the ArrayList.
			CrawledDoc tempDoc = new CrawledDoc(abslink, docID, title, content, stemmed);

			// I would like to update it here!
			//gui.crawlerStatus.setText("Left in queue: "+ Q.size() +", and "+ FoundDocs.size() +"Pages crawled.");
			// Visual representation of Object to demonstrate to group structure of it.
        	System.out.println("Q size: "+ Q.size());
			System.out.println("\nCrawledDoc["+docID+"] = Link: "+abslink+"\nDocID: "+docID+"\nTitle:"+title+"\nContent: "+content+" \nContentStemmed: "+stemmed+"\n");

Recommended Answers

There's a standard way to do this:
1. Change the WebCrawler constructor to take an instance of CrawlAnalyseGUI as a parameter - so line 15 will read
final WebCrawler crawler = new WebCrawler(this);
Now the webcrawler has a reference to the GUI.
2. Add a public method to CrawlAnalyseGUI that can …

Jump to Post

All 3 Replies

There's a standard way to do this:
1. Change the WebCrawler constructor to take an instance of CrawlAnalyseGUI as a parameter - so line 15 will read
final WebCrawler crawler = new WebCrawler(this);
Now the webcrawler has a reference to the GUI.
2. Add a public method to CrawlAnalyseGUI that can be used to update the GUI with progress from the crawler - mabe something like
public void updateProgress(int toDo, int done) {
// update JLabels etc

3. In the WebCrawler, whenever there is progress to report, call updateProgress on the GUI instance (as passed to its constructor).
4. Sit back, watch it work perfectly, feel smug.

Member Avatar

Hi James,

A working program and a sense of smugness? Sounds good to me! I will give it a go and report back if I encounter any problems. Thanks :)

edit: works like a dream, thank you for explaining that to me I feel that is going to come in very handy!

Be a part of the DaniWeb community

We're a friendly, industry-focused community of 1.20 million developers, IT pros, digital marketers, and technology enthusiasts learning and sharing knowledge.