Open Source .NET (C#) Twitter Streaming API Client

April 26, 201054 Comments

UPDATE: Twitter removed basic authentication and converted to oAuth. This code will no longer work as written. 04/05/2013

It can take some time to write a client to consume data from the Twitter Streaming API.  Although this has been done a few times already, I wanted to build an application using only .NET code.  This means that the connection to Twitter is made using the HttpWebRequest object and the resulting JSON is parsed with a Data Contract.  There were quite a number of pitfalls along the way, so I hope this code will help you avoid some of the issues that I uncovered.  For example, HttpWebRequest hung if I didn’t abort the request prior to a close, and parsing JSON using .NET objects can be tricky unless you’ve done it before.

The code can be found on GitHub:  https://github.com/swhitley/TwitterStreamClient

Run the application with the “/encrypt {password}” parameters to generate the encrypted password for the app.config file.

    <appSettings>
        <add key="loglevel" value="ALL"/>
        <add key="use_queue" value="true"/>
        <add key="multithread" value="false"/>
        <add key="twitter_username" value="{Twitter Username"/>
        <add key="twitter_password_encrypted" value="{Encrypted Twitter Password"/>
        <add key="stream_url" value="http://stream.twitter.com/1/statuses/sample.json"/>
    </appSettings>

This is a console application that accepts command line arguments.  Use “/?” to see the options.  You can execute “twitterstreamclient” without any arguments and the application will connect to Twitter using the supplied username and password.  If you have “use_queue” set to false, the output will only go to the screen and you’ll see the results of the stream fly by.

If you set “use_queue” to “true,” the messages will be written to MSMQ.  You can then run a second instance of the application with the “/p” command line argument.  That argument instructs the application to process the message queue.  Write additional code to store the messages in a database.  The current code will parse the JSON and write the string to the screen.

I’ve included the text of the main class below.  As you can see, I’ve tried to implement the recommended backoff procedures from Twitter’s documentation.  I’m looking forward to your feedback on this and want to see recommendations for improvement.  I know that these features are already built into some of the libraries out there, but if you’re like me, you don’t always want to be tied to a library and you’d rather have something that’s easy to tweak for your own needs.

using System;
using System.Text;
using System.IO;
using System.Net;
using System.Configuration;
using System.Threading;
using System.Diagnostics;
using System.Web;
using System.Reflection;
using System.Runtime.Serialization.Json;
using System.Messaging;
using System.Security.Cryptography;

namespace TwitterStreamClient
{
    public class TwitterStream
    {
        public void Stream2Queue()
        {
            string username = ConfigurationManager.AppSettings["twitter_username"];
            string password = Common.Decrypt( ConfigurationManager.AppSettings["twitter_password_encrypted"]);
            //Twitter Streaming API
            string stream_url = ConfigurationManager.AppSettings["stream_url"];

            HttpWebRequest webRequest = null;
            HttpWebResponse webResponse = null;
            StreamReader responseStream = null;
            MessageQueue q = null;
            string useQueue = ConfigurationManager.AppSettings["use_queue"];

            int wait = 250;
            string jsonText = "";

            Logger logger = new Logger();

            try
            {
                //Message Queue
                if (useQueue == "true")
                {
                    if (MessageQueue.Exists(@".\private$\Twitter"))
                    {
                        q = new MessageQueue(@".\private$\Twitter");
                    }
                    else
                    {
                        q = MessageQueue.Create(@".\private$\Twitter");
                    }
                }

                while (true)
                {

                    try
                    {
                        //Connect
                        webRequest = (HttpWebRequest)WebRequest.Create(stream_url);
                        webRequest.Credentials = new NetworkCredential(username, password);
                        webRequest.Timeout = -1;
                        webResponse = (HttpWebResponse)webRequest.GetResponse();
                        Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
                        responseStream = new StreamReader(webResponse.GetResponseStream(), encode);

                        //Read the stream.
                        while (true)
                        {
                            jsonText = responseStream.ReadLine();
                            //Post each message to the queue.
                            if (useQueue == "true")
                            {
                                Message message = new Message(jsonText);
                                q.Send(message);
                            }

                            //Success
                            wait = 250;

                            //Write Status
                            Console.Write(jsonText);

                        }
                        //Abort is needed or responseStream.Close() will hang.
                        webRequest.Abort();
                        responseStream.Close();
                        responseStream = null;
                        webResponse.Close();
                        webResponse = null;

                    }
                    catch (WebException ex)
                    {
                        Console.WriteLine(ex.Message);
                        logger.append(ex.Message, Logger.LogLevel.ERROR);
                        if (ex.Status == WebExceptionStatus.ProtocolError)
                        {
                            //-- From Twitter Docs --
                            //When a HTTP error (> 200) is returned, back off exponentially.
                            //Perhaps start with a 10 second wait, double on each subsequent failure,
                            //and finally cap the wait at 240 seconds.
                            //Exponential Backoff
                            if (wait < 10000)
                            {
                                wait = 10000;
                            }
                            else
                            {
                                if (wait < 240000)
                                {
                                    wait = wait * 2;
                                }
                            }
                        }
                        else
                        {
                            //-- From Twitter Docs --
                            //When a network error (TCP/IP level) is encountered, back off linearly.
                            //Perhaps start at 250 milliseconds and cap at 16 seconds.
                            //Linear Backoff
                            if (wait < 16000)
                            {
                                wait += 250;
                            }

                        }
                    }
                    catch (Exception ex)
                    {
                        Console.WriteLine(ex.Message);
                        logger.append(ex.Message, Logger.LogLevel.ERROR);
                    }
                    finally
                    {
                        if (webRequest != null)
                        {
                            webRequest.Abort();
                        }
                        if (responseStream != null)
                        {
                            responseStream.Close();
                            responseStream = null;
                        }

                        if (webResponse != null)
                        {
                            webResponse.Close();
                            webResponse = null;
                        }
                        Console.WriteLine("Waiting: " + wait);
                        Thread.Sleep(wait);
                    }
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
                logger.append(ex.Message, Logger.LogLevel.ERROR);
                Console.WriteLine("Waiting: " + wait);
                Thread.Sleep(wait);
            }
        }

        public void QueueRead()
        {
            MessageQueue q;
            string multiThread = ConfigurationManager.AppSettings["multithread"];
            Logger logger = new Logger();

            try
            {
                if (MessageQueue.Exists(@".\private$\Twitter"))
                {
                    q = new MessageQueue(@".\private$\Twitter");
                }
                else
                {
                    Console.WriteLine("Queue does not exist.");
                    return;
                }

                while (true)
                {
                    Message message;
                    try
                    {
                        message = q.Receive();
                        message.Formatter =
                            new XmlMessageFormatter(new String[] { "System.String" });
                        if (multiThread == "true")
                        {
                            ThreadPool.QueueUserWorkItem(MessageProcess, message);
                        }
                        else
                        {
                            MessageProcess(message);
                        }
                    }
                    catch { continue; }
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
                logger.append(ex.Message, Logger.LogLevel.ERROR);
            }
        }

        public void MessageProcess(object objMessage)
        {
            status status = new status();
            Logger logger = new Logger();
            DataContractJsonSerializer json = new DataContractJsonSerializer(status.GetType());

            try
            {
                Message message = objMessage as Message;

                byte[] byteArray = Encoding.UTF8.GetBytes(message.Body.ToString());
                MemoryStream stream = new MemoryStream(byteArray);

                //TODO:  Check for multiple objects.
                status = json.ReadObject(stream) as status;

                Console.WriteLine(message.Body.ToString());

                //TODO: Store the status object
                DataStore.Add(status);

            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
                logger.append(ex.Message, Logger.LogLevel.ERROR);
            }
        }
    }
}

Technorati Tags: ,,,,
Share

52 Responses to “Open Source .NET (C#) Twitter Streaming API Client”

  1. Twitter Comment


    RT @swhitley: “Open Source .NET (C#) Twitter Streaming API Client” [new post] – [link to post]

    Posted using Chat Catcher

  2. Twitter Comment


    “Open Source .NET (C#) Twitter Streaming API Client” [new post] – [link to post]

    Posted using Chat Catcher

  3. tester says:

    hi shannon,

    below bit create exception, and spit “Length of the data to decrypt is invalid” in the log

    CryptoStream(ms, cryptoProvider.CreateDecryptor(KEY_64, IV_64), CryptoStreamMode.Read);

    any advise?

  4. tester says:

    nevermind, it’s my bad. :)
    your streaming client rocks. and thanks for sharing

  5. @tester – You’re welcome. I’m glad to hear it’s working for you.

  6. ashigakari says:

    uhm, sorry for my dumb question.
    how can I use this with oAuth?

  7. Ian Black says:

    Thanks for the code Shannon it was a great help in connecting to the streaming API in .NET which I’ve struggled to find any other examples of.

    I’m adding each tweet to a database but every now and again I get an error in my log file that says it’s failed to insert. I printed out the actual data from the tweet and it is all there and correct so I can’t see a reason why it would error.

    I’m not using a queueing system like you have in place. Do you think it could be that in periods of high “tweetage” that there are just too many coming in for my program to process? Is that why you use the queuing system?

    Many thanks,
    Ian.

  8. @Ian Black – Hi Ian. I’m not sure. I’d have to see the error. Using a queue isn’t mandatory, but I can see where it would help during periods of high database utilization. I’m not using this code in production right now so I haven’t run into any similar issues.

  9. Khash says:

    One thing about the multithreaded read from the queue: It seems to me that picking up messages from the queue and then queueing them again on the threadpool is not so good: Firstly it picks up messages that might not be processed immediately by the client and therefore deprives free clients from processing them. Secondly you lost all the nice transactionality and persistence that comes with MSMQ for no apparent gain.

    May I’m failing to see the point.

  10. MeteoDenhelder says:

    Great code, but getting this in my log files:

    2010-09-09T23:55:31 Ongeldige gegevens.

    it’s in dutch, translated: Invalid Data.

  11. Robin says:

    Sorry for the idiot question, I’m just getting started. I tried to open the project with Visual Studio and run it, but it says:

    Source file:TwitterStreamClientDistrib\Properties\AssemblyInfo.cs’ could not be opened (‘Unspecified error ‘)

    I’m falling at the first hurdle!

  12. EdGruberman says:

    Shannon, thank you for inspiring me to create my own PowerShell solution. Having it all in .NET already gave me a huge leap forward in my efforts.

    http://code.google.com/p/pstwitterstream/

    I found one thing with my testing that might be good to check for. When the stream was closed (I’m assuming Twitter automatically closed the previous stream when I had more than one stream opened on the same user and IP) my code would loop infinitely instead of error out. I found checking the EndOfStream property on the response stream helped me break out appropriately.

  13. Abhi says:

    Hi Shannon,

    I’ve been dabbling with Twitter application development lately, and must say your post has been a tremendous help ! Thanks :)
    I got around your code to running it, however, I keep running into a tiny error:
    “Invalid character in a Base-64 string.”

    I ran the console application as:
    “TwitterStreamClientDistrib.exe /encrypt twitter_password”

    The app.config has twitter username and “” for twitter_password_encrypted key value. What am I missing here ?

  14. Jasnira says:

    Is it possible to unit test methods like Stream2Queue?
    If there is a way, I’d really like to know.
    I have a stream client derived from this code, with some extra stuff, and been having a hard time guaranteeing correctness.

  15. Jonathan says:

    Thanks for the great sample. This has been an awesome starting point for my project!

    One bug I just found: in status.cs, you have “contributors” as a string type when the Twitter API actually returns an array of strings. It doesn’t usually matter since this value is almost always null, but it creates an interesting bug for the few tweets that use it.

    Thanks again!
    -Jonathan

  16. @Jasnira Sorry, I don’t have anything written that would help with that. I usually just step through using debug mode or dump values out to a file that I can review.

    @Jonathan – Thanks for the feedback. I’m not surprised to see a bug like that. Some of the less-often-used items were harder to locate in the examples. I’ll update the code when I get a chance.

  17. Helen Neely says:

    Thanks for this example, I just started working with Twitter API in Python. So, I will adapt your example here to see if I can port it.

    Will post here this weekend if I manage to get it working.

  18. Pankaj says:

    hi i am passing correct username/password but getting 401 unauthorised please help do i have to change some setting on twitter or send me working code with test twitter id which is able to login

  19. Pankaj says:

    Hi
    is there any way that i get the notification automatically whenever any1 tweet instead of calling the service in a while loop as in facebook i can subscribe and it will send a notification if any update happen on my profile?

  20. TS says:

    I am getting 401 error as well. Username and password are correct for sure. Any idea how to fix that?
    Thanks.

  21. I haven’t had a chance to look at the code, but I wonder if the 401 errors are due to Twitter’s change to exclusively using https for the streaming services.

  22. Tony says:

    hi, i got serach url like: http://stream.twitter.com/1/statuses/filter.json?track=avatar%20movie

    Problem is that server hang for a long time between each tweet. However, its quite fast by searching one word(e.g.avatar) only. Do you have any idea?

  23. Thanks for a great library, Shannon. I was wondering, if there’s a way to parse the multiple objects in JSON string?

  24. Joe says:

    Sorry for newbie question but when you are reading the stream one line at a time and posting to a queue; dont you risk losing content. Do you have any ideas on how one can consume the stream data and still store the results to a file or DB at the same time?

  25. Hi @Joe, Because you have an open connection, it’s not like you miss messages. The messages queue on Twitter’s side. As long as the connection is open, you can read through the Twitter stream and catch up pretty quickly to the end of the stream. You could write to a file or DB in place of writing to the queue. I only wrote it that way because it’s a very quick way to pull the messages out of the stream and then continue reading messages. Although the messages won’t pass by, you also don’t want to get too far behind in reading the stream.

  26. Joe says:

    Thank you so much for clarifying that.

  27. Joe says:

    Will your code work with the new Streaming API (firehose, power track)?

  28. @Joe You’re welcome. I haven’t been following recent changes to the api (hadn’t heard of ‘power track’). I’ll take a look, but you might get to it before I do.

  29. memet says:

    hi shannon,

    thajs for your effort. but i amfacing with a trouble. I run your project but it prints an error message to console and before i see the message console closes and program finishes its execution. do you have any idea what is my problem.

    I want to stream a spesific hashtag. how can i do this with your project

  30. Hari says:

    Hi, shannon

    whether ur code wil work now?..A couple of months later, Twitter stopped supporting basic authentication (username/password). This meant that the old (simple) way of sending notifications stopped working. ..

  31. Joe says:

    Hi
    Just wanted to follow up with another question. When reading the stream as responsestream.readline; I assumed that from the twitter doc it separated tweet with the end of line/carriage return character. So as you read per line you are getting one tweet which helps in the parsing done later; but when looking at the content it seems that is not the case. The tweet is split up as you either write to a file or add to the queue which makes it very hard to compile the tweet at a later point. Is there a way around this?

  32. nee says:

    Hello Mr.Shannon,
    Thank you very much for your code.
    I would like to use your code for my work. I open your code without editing any code then run it. I found “The remote server returned an error: Unauthorized”.
    I am a newbie. Could you please suggest me?

  33. weikang says:

    @Shannon – Hi I have setup an application on Twitter and which part of the file do I nee to to modify? (app.config file only ask for username and password)

  34. @weikang -I’m sorry. I was thinking of another project. Yes, you should be able to enter your Twitter id and password into the app.config and get the project to work. Make sure you’re not behind a proxy server as I haven’t tested this code with a proxy server.

  35. weikang says:

    @Shanon, I have check that there is no problem with the proxy, and the error message I am getting is Unauthorised. Is it because, the Twitter is now using OAuth?

  36. Joe says:

    Shannon, sorry to be a nuisance but just want to clarify. Reading by line as you have it should be a complete post for each read, correct?

  37. Chris says:

    Hi, i see that you need to be checking the json state every 250 ms, there’s not a way to add a kind of listener to de json object? i mean omit the awkward timer?, great job btw.

  38. Thanks, Chris. I haven’t really focused on that part of the code in a couple of years. If you find something that works better, I’d be more than happy to look at a pull request.

  39. Robert says:

    Great code, great sample !
    One question: the sample outputs all sorts of fields to the console.
    Wehere would I limit it to show for example only the id of the tweeter and the message itself ?

  40. Gowri says:

    HI Shannon,

    I just follow your code..
    It’s giving me a error

    The remote server returned an error: (401) Unauthorized.

    I double check my credentials all are correct. How can I solve this problem..?

  41. marta says:

    Hi Shannon,
    I get the same error: (401) Unauthorized. Is there a way to resolve it?

  42. Shannon says:

    Hi @marta

    Twitter removed basic authentication and converted to oAuth. This code will no longer work as written. 04/05/2013

    I have updated the post to reflect the change.

  43. Patricia says:

    Oh my goodness! Incredible article dude! Thanks, However I am
    encountering troubles with your RSS. I don’t know why I cannot subscribe to
    it. Is there anybody getting identical RSS issues?

    Anyone who knows the answer can you kindly respond? Thanx!!

    Feel free to visit my web-site … pulidor de suelos (Patricia)

  44. I all the time used to read piece of writing in news papers but now as I am a user of net therefore from now I am using net for content, thanks to web.

  45. We absolutely love your blog and find many of your post’s to be just what I’m looking for.
    Do you offer guest writers to write content in your case?
    I wouldn’t mind creating a post or elaborating on a few
    of the subjects you write about here. Again, awesome weblog!

  46. Good day! I know this is kinda off topic however , I’d figured I’d ask.
    Would you be interested in exchanging links or maybe
    guest authoring a blog article or vice-versa? My site discusses a lot of the
    same subjects as yours and I feel we could greatly benefit from
    each other. If you’re interested feel free to shoot me an e-mail.

    I look forward to hearing from you! Great blog
    by the way!

  47. Camilla says:

    With this device, it will not matter which room of the house you are
    in, you will definitely hear the door bell the first time your visitor presses on it.
    Door – Bot is an interesting concept from BOT Home Automation. There are also so many models you can choose between, there will unquestionably
    be a doorbell out there suitable for you.

  48. You really make it appear so easy together with your presentation but I to find
    this matter to be really something that I believe I might by no means understand.
    It seems too complex and extremely broad for me. I am looking ahead on your
    subsequent post, I will attempt to get the hold of it!

Leave a Reply

Twitter Tweet This

2 Trackbacks

  1. eclipsed4utoo (Ryan Alford)

    Twitter Comment


    RT @swhitley: “Open Source .NET (C#) Twitter Streaming API Client” [new post] – [link to post]

    Posted using Chat Catcher

  2. swhitley (Shannon Whitley)

    Twitter Comment


    “Open Source .NET (C#) Twitter Streaming API Client” [new post] – [link to post]

    Posted using Chat Catcher