Continuous data stream

[benkyma]

I'm trying to connect to the twitter streaming api. At the moment I'm just trying to grab the incoming data stream, keep the connection open and write the data to the console. I can do this easily from a terminal using curl. The following code however runs, throws no errors, but outputs no text to the console...

StringBuilder sb = new StringBuilder();

            // used on each read operation
            byte[] buf = new byte[8192];

            String domain = "http://stream.twitter.com/1/statuses/sample.json";

            HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(domain);

            request.Credentials = new NetworkCredential("user_name", "password");


            HttpWebResponse response = (HttpWebResponse)
            request.GetResponse();

            Stream resStream = response.GetResponseStream();

            string tempString = null;
            int count = 0;

            do
            {
                // fill the buffer with data
                count = resStream.Read(buf, 0, buf.Length);

                // make sure we read some data
                if (count != 0)
                {
                    // translate from bytes to ASCII text
                    tempString = Encoding.ASCII.GetString(buf, 0, count);

                    // continue building the string
                    sb.Append(tempString);
                }
            }
            while (count > 0); // any more data to read?

            // print out page source
            Console.WriteLine(sb.ToString());

[Geekitygeek]

I'm not 100% sure, but surely if resStream is a continuous stream of data then count will never reach 0 and your loop wont exit to print to the console?

Try putting the Console.WriteLine inside the Do..While loop :)

[benkyma]

:) Aah yes, many thanks. More coffee required I think....

As an aside, there is a separate channel that allows filtering of the data. Do you know how I can create the equivalent of:

curl -d @tracking http://stream.twitter.com/1/statuses/filter.json

Where 'tracking' is the name of a file containing filter terms. Can I pass these terms into the HttpWebRequest somehow? I've tried adding them to the url, but without success so far.

[sknake]

I coded up a little asynchronous twitter API. It looked fun:

using System;
using System.IO;
using System.Net;
using System.Runtime.Remoting.Messaging;

namespace daniweb
{
  public class TwitterReader : IDisposable
  {
    //
    private string username;
    private string password;
    private bool running;
    //
    private const string url = @"http://stream.twitter.com/1/statuses/sample.json";
    //
    public event OnTwitterLineReceived OnLineReceived;
    public event OnTwitterError OnError;
    //
    private TwitterReader()
    {
      running = false;
    }
    public TwitterReader(string username, string password)
      : this()
    {
      this.username = username;
      this.password = password;
    }
    public void Start()
    {
      if (running)
        throw new InvalidOperationException("Twitter streamer is already running");

      running = true;
      HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
      request.Method = "GET";
      request.Credentials = new NetworkCredential(this.username, this.password);
      HttpWebResponse response = (HttpWebResponse)request.GetResponse();
      new Action<Stream>(BeginRead).BeginInvoke(
        response.GetResponseStream(),
        new AsyncCallback(EndRead),
        null);
    }
    private void BeginRead(Stream stream)
    {
      using (stream)
      {
        using (StreamReader sr = new StreamReader(stream))
        {
          while (!sr.EndOfStream && running)
          {
            string s = sr.ReadLine();
            var del = this.OnLineReceived;
            if (del != null)
            {
              del(this, new TwitterLineReceivedArgs(s));
            }
          }
        }
      }
    }
    private void EndRead(IAsyncResult ar)
    {
      AsyncResult result = (AsyncResult)ar;
      Action<Stream> action = (Action<Stream>)result.AsyncDelegate;
      try
      {
        action.EndInvoke(ar);
      }
      catch (Exception Ex)
      {
        var del = this.OnError;
        if (del != null)
        {
          del(this, new TwitterErrorArgs(Ex));
        }
      }
      finally
      {
        running = false;
      }
    }
    public void Stop()
    {
      running = false;
    }
    #region IDisposable Members
    public void Dispose()
    {
      Dispose(true);
      GC.SuppressFinalize(this);
    }
    protected virtual void Dispose(bool disposing)
    {
      if (disposing)
      {
        if (running)
        {
          this.Stop();
        }
      }
    }
    #endregion
  }
  public delegate void OnTwitterLineReceived(object sender, TwitterLineReceivedArgs e);
  public class TwitterLineReceivedArgs : EventArgs
  {
    private string line;
    public string Line { get { return line; } }
    private TwitterLineReceivedArgs()
    {
    }
    public TwitterLineReceivedArgs(string Line)
      : this()
    {
      this.line = Line;
    }
  }
  public delegate void OnTwitterError(object sender, TwitterErrorArgs e);
  public class TwitterErrorArgs : EventArgs
  {
    private Exception exception;
    public Exception Exception { get { return exception; } }
    private TwitterErrorArgs()
    {
    }
    public TwitterErrorArgs(Exception Exception)
      : this()
    {
      this.exception = Exception;
    }
  }
}

Calling it:

private TwitterReader reader;
    private int linesRead;
    private void button2_Click(object sender, EventArgs e)
    {
      linesRead = 0;
      reader = new TwitterReader(username, password);
      reader.OnLineReceived += new OnTwitterLineReceived(reader_OnLineReceived);
      reader.OnError += new OnTwitterError(reader_OnError);
      reader.Start();
    }

    private void button3_Click(object sender, EventArgs e)
    {
      reader.OnLineReceived -= new OnTwitterLineReceived(reader_OnLineReceived);
      reader.OnError -= new OnTwitterError(reader_OnError);
      reader.Stop();
      reader.Dispose();
    }


    void reader_OnError(object sender, TwitterErrorArgs e)
    {
      System.Diagnostics.Debugger.Break();
    }

    void reader_OnLineReceived(object sender, TwitterLineReceivedArgs e)
    {
      this.Invoke(new MethodInvoker(
        delegate()
        {
          System.Threading.Interlocked.Increment(ref linesRead);
          textBox1.Text += e.Line;
          label1.Text = string.Format("Lines Read: {0:F0}", linesRead);
        }
      ));
    }

[Geekitygeek]

Not worked with the Twitter API, but you need to send your tracking data as a POST request.

Try this:

HttpWebResponse response = (HttpWebResponse)request.GetResponse();
            response.Method = "POST";
            // Create POST data and convert it to a byte array.
            string postData = "track=basketball,football,baseball,footy,soccer";
            byte[] byteArray = Encoding.UTF8.GetBytes(postData);
            // Set the ContentLength property of the WebRequest.
            request.ContentLength = byteArray.Length;
            // Get the request stream.
            Stream dataStream = request.GetRequestStream();
            // Write the data to the request stream.
            dataStream.Write(byteArray, 0, byteArray.Length);
            // Close the Stream object.
            dataStream.Close();

            Stream resStream = response.GetResponseStream();

havent tested it as i dont have twitter (shock horrer!)...
Let me know how you get on :)

[benkyma]

Wow, that's pretty cool. Thanks a lot. :)

[sknake]

Don't forget to mark this thread as solved if you have found an answer to your question and good luck!

[benkyma]

Thanks guys. Wicked response.

[benkyma]

Arg, still struggling to get a response from this Streaming API. My connection code is now:

String domain = "http://stream.twitter.com/1/statuses/filter.json";
            
            HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(domain);
            request.Method = "POST";
            // Create POST data and convert it to a byte array.
            string postData = "track=basketball,football,baseball,footy,soccer";

            byte[] byteArray = Encoding.UTF8.GetBytes(postData);

            // Set the ContentLength property of the WebRequest.
            request.ContentLength = byteArray.Length;
            // Get the request stream.
            Stream dataStream = request.GetRequestStream();
            // Write the data to the request stream.
            dataStream.Write(byteArray, 0, byteArray.Length);
            // Close the Stream object.
            dataStream.Close();

It is always returning a 406 error which means there's something wrong with the POST data. Could it5 be in the way it's being encoded?

[Geekitygeek]

Quite possibly...i couldnt find anywhere in the API documentation that says what format to use...it only gives instructions for using curl.

try this:

HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(domain);
            request.Method = "POST";
            // Create POST data and convert it to a byte array.
            string postData = "track=basketball,football,baseball,footy,soccer";

            byte[] byteArray = Encoding.UTF8.GetBytes(postData);

            // Set the ContentLength property of the WebRequest.
            request.ContentLength = byteArray.Length;

// Set the content type of the data being posted.
myHttpWebRequest.ContentType = "application/x-www-form-urlencoded";

            // Get the request stream.
            Stream dataStream = request.GetRequestStream();
            // Write the data to the request stream.
            dataStream.Write(byteArray, 0, byteArray.Length);
            // Close the Stream object.
            dataStream.Close();

Alternatively, you can try ContentType = "text/html";
Other option would be to try the Twitter API forums to see if anyone can tell you the required ContentType :)

[benkyma]

Thanks a lot Ryshad. That's working nicely. Apologies for needing so much hand-holding with this. It's a bit of a new direction for me.

All the best

Ben

[Geekitygeek]

No problem at all :) Work is slow so im gettin paid to sit here helping you :D

The twitter API and POST thing is all new to me too, i'm just a whizz with google and have a reasonable understanding of c#....the rest is just fitting the pieces together. Enjoy :)

[sknake]

Have you found a solution to your problem yet? If not post the _full code_ you are using so far and we'll see. You can use tcpdump to capture the HTML headers from CURL and see how it is formatting the request.

[VinayRok]

Not worked with the Twitter API, but you need to send your tracking data as a POST request.

Try this:

HttpWebResponse response = (HttpWebResponse)request.GetResponse();
            response.Method = "POST";
            // Create POST data and convert it to a byte array.
            string postData = "track=basketball,football,baseball,footy,soccer";
            byte[] byteArray = Encoding.UTF8.GetBytes(postData);
            // Set the ContentLength property of the WebRequest.
            request.ContentLength = byteArray.Length;
            // Get the request stream.
            Stream dataStream = request.GetRequestStream();
            // Write the data to the request stream.
            dataStream.Write(byteArray, 0, byteArray.Length);
            // Close the Stream object.
            dataStream.Close();

            Stream resStream = response.GetResponseStream();

havent tested it as i dont have twitter (shock horrer!)...
Let me know how you get on :)

Check in Code project