I am connecting to a data stream that pushes data continuously and when I tried to set up a network stream I just could not connect and kept getting the 'not found' error. I used the httpwebrequest with webrequest.keepalive = true and started to consume the data by reading into a buffer and then appending to a stringbuilder. Once the stringbuilder reaches max size I flush it to a text file. The problem is that I get disconnected from the data stream because I am not reading the data fast enough. I need to keep the connection open and read the data while ensuring data gets stored in text files. Here is what I have and would appreciate some advice on this. If I use method "CONNECT" it does not connect to the stream.
While (True) webrequest = DirectCast(Net.WebRequest.Create(url), HttpWebRequest) webrequest.Credentials = New System.Net.NetworkCredential(username, password) webrequest.Method = "GET" webrequest.KeepAlive = True webrequest.Timeout = 300000 webrequest.ContentType = "application/json" webrequest.Headers.Add("Accept-Encoding: Deflate, gzip") responseStream = webrequest.GetResponse().GetResponseStream responseStream = New GZipStream(responseStream, CompressionMode.Decompress) Dim bufferread(8100) As Byte While (True) Try numbytesread = responseStream.Read(bufferread, 0, BUFFER_SIZE) Dim dataStream As StreamReader = New StreamReader(responseStream) If numbytesread > 0 Then responseData = Encoding.UTF8.GetString(bufferread, 0, numbytesread) numbytesread = responseData.Length parsingTools.appendXMLtoFile(responseData, apiKey) timestamp = DateTime.Now ElseIf numbytesread = 0 Then Tools.forceFileclosing() Continue While End If If DateTime.Now.Subtract(timestamp).TotalSeconds > 30 Then Tools.forceFileclosing() webrequest.Abort() exit while End If Catch e As WebException If e.Status = WebExceptionStatus.ProtocolError Then response = DirectCast(e.Response, HttpWebResponse) response = e.Response Dim sreader As New StreamReader(response.GetResponseStream) Using sr As StreamReader = New StreamReader(response.GetResponseStream) responseData = WebUtility.HtmlDecode(sr.ReadToEnd) If responseData.Contains("Access Denied") Then Exit Sub End If End Using WebsRequest() Else End If End While
Here is how I append the data and eventually flush.
If Not String.IsNullOrEmpty(_html) Then sbuilder.Append(_html) End If If sbuilder.ToString.EndsWith(vbCr) And sbuilder.Length > 6000000 Then writeToFile(sbuilder) End If
should I increase the buffer size and if so what is the max? that would mean I can read more data at a time and keep up with incoming push. Desparate here.
ACtually also I noticed that it gets stuck on this line
numbytesread = responseStream.Read(bufferread, 0, BUFFER_SIZE)
and then eventually I get disconnected.