Member Avatar for Dark Master

hi forum,
i m developing a simple web crawler in java.upon entering an URL, the crawler downloads the corresponding web page and continues this process.but i m having problem in accessing web pages which are redirected to a diferent URL.one such example is www.telegraphindia.com ,in which a new part gets added to the original URL. can anybody help.thanks in advance.

  • 2 Contributors
  • 3 Replies
  • 115 Views
  • 1 Week Discussion Span
  • Latest Post Latest Post by chrisbliss18

Dani AI

Generated

used URL.openStream() which only gives you the response body; it hides the HTTP status and headers you need to detect redirects. was right that the server returns a 3xx and a Location header. The practical approach is to use HttpURLConnection so you can read getResponseCode() and the Location header, resolve relative locations against the original URL, and then request the new URL (or let the connection follow redirects for you).

A simple manual-follow example (stops after a fixed number of redirects, resolves relative Location values):

URL url = new URL(startUrl);
int maxRedirects = 10;
for (int i = 0; i < maxRedirects; i++) {
    HttpURLConnection conn = (HttpURLConnection) url.openConnection();
    conn.setInstanceFollowRedirects(false); // inspect Location manually
    conn.setConnectTimeout(5000);
    conn.setReadTimeout(5000);
    int code = conn.getResponseCode();
    if (code >= 300 && code < 400) {
        String loc = conn.getHeaderField("Location");
        if (loc == null) throw new IOException("Redirect without Location");
        url = new URL(url, loc); // resolves relative redirects
        conn.disconnect();
        continue;
    }
    try (InputStream in = conn.getInputStream()) {
        // read page content from 'in'
    }
    break;
}

Practical tips: detect redirect loops and enforce a max redirect count; use new URL(baseUrl, location) or URI.resolve() to handle relative Location headers; use CookieManager if the site requires cookies across redirects; set a reasonable User-Agent to avoid being blocked; handle meta-refresh or JavaScript redirects by parsing the HTML if needed; be aware that some redirects change POST to GET (303) and automatic following may change behavior. See the Java HttpURLConnection API for details and MDN for HTTP redirect semantics: HttpURLConnection API and HTTP redirections.

Recommended Answers

All 3 Replies

Member Avatar for chrisbliss18

Let's look at the response when requesting this page:

HTTP/1.1 302 Object moved
Date: Tue, 13 Sep 2005 16:06:26 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Location: section/frontpage/index.asp
Content-Length: 148
Content-Type: text/html
Set-Cookie: ASPSESSIONIDACTBSRRB=FALIOKOCIJNCLJAPOONLFLCF; path=/
Cache-control: private

<head><title>Object moved</title></head>
<body><h1>Object Moved</h1>This object may be found <a HREF="section/frontpage/index.asp">here</a>.</body>

The header indicates a response code of 302. 302 responses include a "Location" directive that indicates where the actual response can be found at (if properly formatted that is). As you can see from the response, the Location is specified as "section/frontpage/index.asp". All you need to do is request that page from the same domain in order to get the information you want.

Member Avatar for Dark Master

thanks criss for ur reply, but i dont know how to implement ur suggetion.i hv created a inputstream object and used a url.openstream() method to access the contents of the page.can u suggest how i can capture the redirected portion of the URL.also how can i find out the http response codes that u showed.plz help.

Member Avatar for chrisbliss18

You should use the HttpURLConnection class for requesting pages through HTTP. This class has a setFollowRedirects method that allows you to tell the class to automatically follow redirects. This class has many methods that you will find very helpful since it gives you the ability to read response messages and header information from the response.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.