Need a bit of advice. Basically I am building a webcrawler and in order to do so
I have to extract the page source of a webpage which I can do so like this:
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click Dim request As System.Net.HttpWebRequest = System.Net.HttpWebRequest.Create(TextBox2.Text) Dim response As System.Net.HttpWebResponse = request.GetResponse() Dim sr As System.IO.StreamReader = New System.IO.StreamReader(response.GetResponseStream()) Dim sourcecode As String = sr.ReadToEnd TextBox1.Text = sourcecode End Sub
The above code works fine on most of the websites I have tried to extract page source from but for some reason it is failing to extract the html contents of few websites like these where the message posted by a user is nowhere to be seen? The webpage in question is this: http://www.vbforums.com/showthread.php?t=654378
Is there something I have missed or is it due to forum protection etc which is preventing the vb application from extracting the whole page source?