how to read content of html page and save in text file

Please support our C# advertiser: Programming Forums - DaniWeb Sister Site
Thread Solved

Join Date: Jun 2009
Posts: 17
Reputation: sandipan.rcciit is an unknown quantity at this point 
Solved Threads: 0
sandipan.rcciit sandipan.rcciit is offline Offline
Newbie Poster

how to read content of html page and save in text file

 
0
  #1
Jul 15th, 2009
hello friend
i have a text file name url.txt contain lots of url, one url in a line.

now i want to get the content of each an every page which are open on the basis of those urls,

here is my code:

using System;
using System.IO;
using System.Net;
using System.Text;

class WebFetch
{
	static void Main(string[] args)
	{
		// used to build entire input
		StringBuilder sb  = new StringBuilder();

		// used on each read operation
		byte[]        buf = new byte[8192];
        int counter = 0;
        string line;

        // Read the file and display it line by line.
        System.IO.StreamReader file =
           new System.IO.StreamReader("c:/wamp/www/isbn/url.txt");
        while ((line = file.ReadLine()) != null)
        {
            // prepare the web page we will be asking for
            HttpWebRequest request = (HttpWebRequest)
                WebRequest.Create(line);

            // execute the request
            HttpWebResponse response = (HttpWebResponse)
                request.GetResponse();

            // we will read data via the response stream
            Stream resStream = response.GetResponseStream();

            string tempString = null;
            int count = 0;
        
            do
            {
                // fill the buffer with data
                count = resStream.Read(buf, 0, buf.Length);

                // make sure we read some data
                if (count != 0)
                {
                    // translate from bytes to ASCII text
                    tempString = Encoding.ASCII.GetString(buf, 0, count);

                    // continue building the string
                    sb.Append(tempString);
                }
            }
            while (count > 0); // any more data to read?

            // print out page source
            
           Console.WriteLine(sb.ToString());
        }

        file.Close();


		
	}
}

its working and show the content of the file in console but the records are repeated.
that mean content of 1st then 1st and 2nd then 1st 2nd 3rd.

but i want the records are unique.
for example 5 url 5 records.

this code give me 15.

plz help me what is the problem in that code??

that is the content of url.txt:
  1. https://isbndb.com/api/books.xml?access_key=RPGYD5PC&index1=isbn&value1=3890071341
  2. https://isbndb.com/api/books.xml?access_key=RPGYD5PC&index1=isbn&value1=8831754750
  3. https://isbndb.com/api/books.xml?access_key=RPGYD5PC&index1=isbn&value1=0941419940
  4. https://isbndb.com/api/books.xml?access_key=RPGYD5PC&index1=isbn&value1=0941419711
  5. https://isbndb.com/api/books.xml?access_key=RPGYD5PC&index1=isbn&value1=3921029570
Reply With Quote Quick reply to this message  
Join Date: Oct 2008
Posts: 2,721
Reputation: adatapost has much to be proud of adatapost has much to be proud of adatapost has much to be proud of adatapost has much to be proud of adatapost has much to be proud of adatapost has much to be proud of adatapost has much to be proud of adatapost has much to be proud of adatapost has much to be proud of adatapost has much to be proud of 
Solved Threads: 501
Moderator
adatapost's Avatar
adatapost adatapost is offline Offline
Posting Maven

Re: how to read content of html page and save in text file

 
0
  #2
Jul 15th, 2009
sandipan.rcciit,
Remove the content of StringBuilder object after printing it on console window.
  1. sb.Length=0;
Reply With Quote Quick reply to this message  
Join Date: Jun 2009
Posts: 17
Reputation: sandipan.rcciit is an unknown quantity at this point 
Solved Threads: 0
sandipan.rcciit sandipan.rcciit is offline Offline
Newbie Poster

Re: how to read content of html page and save in text file

 
0
  #3
Jul 15th, 2009
thanks it is now working properly...
Reply With Quote Quick reply to this message  
Reply

This thread has been marked solved.
Perhaps start a new thread instead?
Message:




Views: 1288 | Replies: 2
Thread Tools Search this Thread



Tag cloud for C#
About Us | Contact Us | Advertise | DaniWeb | Acceptable Use Policy | RSS Feed

©2003 - 2009 DaniWeb® LLC