networkstream connection fails & httpwebrequest connection not reading fast

[maverick0612]

I am connecting to a data stream that pushes data continuously and when I tried to set up a network stream I just could not connect and kept getting the 'not found' error. I used the httpwebrequest with webrequest.keepalive = true and started to consume the data by reading into a buffer and then appending to a stringbuilder. Once the stringbuilder reaches max size I flush it to a text file. The problem is that I get disconnected from the data stream because I am not reading the data fast enough. I need to keep the connection open and read the data while ensuring data gets stored in text files. Here is what I have and would appreciate some advice on this. If I use method "CONNECT" it does not connect to the stream.

         While (True)
          webrequest = DirectCast(Net.WebRequest.Create(url), HttpWebRequest)
          webrequest.Credentials = New System.Net.NetworkCredential(username, password)
          webrequest.Method = "GET"
          webrequest.KeepAlive = True
          webrequest.Timeout = 300000
         webrequest.ContentType = "application/json"
          webrequest.Headers.Add("Accept-Encoding: Deflate, gzip")
          responseStream = webrequest.GetResponse().GetResponseStream
          responseStream = New GZipStream(responseStream, CompressionMode.Decompress)

       Dim bufferread(8100) As Byte
       While (True)
         Try
            numbytesread = responseStream.Read(bufferread, 0, BUFFER_SIZE)
            Dim dataStream As StreamReader = New StreamReader(responseStream)
            If numbytesread > 0 Then
                responseData = Encoding.UTF8.GetString(bufferread, 0, numbytesread)
                numbytesread = responseData.Length
                parsingTools.appendXMLtoFile(responseData, apiKey)
                timestamp = DateTime.Now
           ElseIf numbytesread = 0 Then
                Tools.forceFileclosing()
                Continue While
           End If
           If DateTime.Now.Subtract(timestamp).TotalSeconds > 30 Then
                Tools.forceFileclosing()
                webrequest.Abort()
                exit while
           End If

    Catch e As WebException
         If e.Status = WebExceptionStatus.ProtocolError Then
             response = DirectCast(e.Response, HttpWebResponse)
            response = e.Response
            Dim sreader As New StreamReader(response.GetResponseStream)
           Using sr As StreamReader = New StreamReader(response.GetResponseStream)
                            responseData = WebUtility.HtmlDecode(sr.ReadToEnd)
                            If responseData.Contains("Access Denied") Then
                                Exit Sub
                            End If
          End Using
           WebsRequest()
       Else
       End If
    End While

Here is how I append the data and eventually flush.

If Not String.IsNullOrEmpty(_html) Then
            sbuilder.Append(_html)
End If
If sbuilder.ToString.EndsWith(vbCr) And sbuilder.Length > 6000000 Then
        writeToFile(sbuilder)
End If

should I increase the buffer size and if so what is the max? that would mean I can read more data at a time and keep up with incoming push. Desparate here.

ACtually also I noticed that it gets stuck on this line

numbytesread = responseStream.Read(bufferread, 0, BUFFER_SIZE)

and then eventually I get disconnected.

[Oxiegen]

I'm thinking that the bottleneck in your code is that you're connecting to the URL on each cycle.
Wouldn't it work if you connect just once and perform the reading of the stream inside the While loop.

What I mean is, move the connection and configuration code to outside the first While but keep the reading of the responsestream inside it.
I'm also thinking that it could save you from having nested While loops.

Perhaps this could be of use: Reading the Response Stream Asynchronously