Open Source .NET (C#) Twitter Streaming API Client
UPDATE: Twitter removed basic authentication and converted to oAuth. This code will no longer work as written. 04/05/2013
It can take some time to write a client to consume data from the Twitter Streaming API. Although this has been done a few times already, I wanted to build an application using only .NET code. This means that the connection to Twitter is made using the HttpWebRequest object and the resulting JSON is parsed with a Data Contract. There were quite a number of pitfalls along the way, so I hope this code will help you avoid some of the issues that I uncovered. For example, HttpWebRequest hung if I didn’t abort the request prior to a close, and parsing JSON using .NET objects can be tricky unless you’ve done it before.
The code can be found on GitHub: https://github.com/swhitley/TwitterStreamClient
Run the application with the “/encrypt {password}” parameters to generate the encrypted password for the app.config file.
<appSettings>
<add key="loglevel" value="ALL"/>
<add key="use_queue" value="true"/>
<add key="multithread" value="false"/>
<add key="twitter_username" value="{Twitter Username"/>
<add key="twitter_password_encrypted" value="{Encrypted Twitter Password"/>
<add key="stream_url" value="http://stream.twitter.com/1/statuses/sample.json"/>
</appSettings>
This is a console application that accepts command line arguments. Use “/?” to see the options. You can execute “twitterstreamclient” without any arguments and the application will connect to Twitter using the supplied username and password. If you have “use_queue” set to false, the output will only go to the screen and you’ll see the results of the stream fly by.
If you set “use_queue” to “true,” the messages will be written to MSMQ. You can then run a second instance of the application with the “/p” command line argument. That argument instructs the application to process the message queue. Write additional code to store the messages in a database. The current code will parse the JSON and write the string to the screen.
I’ve included the text of the main class below. As you can see, I’ve tried to implement the recommended backoff procedures from Twitter’s documentation. I’m looking forward to your feedback on this and want to see recommendations for improvement. I know that these features are already built into some of the libraries out there, but if you’re like me, you don’t always want to be tied to a library and you’d rather have something that’s easy to tweak for your own needs.
using System;
using System.Text;
using System.IO;
using System.Net;
using System.Configuration;
using System.Threading;
using System.Diagnostics;
using System.Web;
using System.Reflection;
using System.Runtime.Serialization.Json;
using System.Messaging;
using System.Security.Cryptography;
namespace TwitterStreamClient
{
public class TwitterStream
{
public void Stream2Queue()
{
string username = ConfigurationManager.AppSettings["twitter_username"];
string password = Common.Decrypt( ConfigurationManager.AppSettings["twitter_password_encrypted"]);
//Twitter Streaming API
string stream_url = ConfigurationManager.AppSettings["stream_url"];
HttpWebRequest webRequest = null;
HttpWebResponse webResponse = null;
StreamReader responseStream = null;
MessageQueue q = null;
string useQueue = ConfigurationManager.AppSettings["use_queue"];
int wait = 250;
string jsonText = "";
Logger logger = new Logger();
try
{
//Message Queue
if (useQueue == "true")
{
if (MessageQueue.Exists(@".\private$\Twitter"))
{
q = new MessageQueue(@".\private$\Twitter");
}
else
{
q = MessageQueue.Create(@".\private$\Twitter");
}
}
while (true)
{
try
{
//Connect
webRequest = (HttpWebRequest)WebRequest.Create(stream_url);
webRequest.Credentials = new NetworkCredential(username, password);
webRequest.Timeout = -1;
webResponse = (HttpWebResponse)webRequest.GetResponse();
Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
responseStream = new StreamReader(webResponse.GetResponseStream(), encode);
//Read the stream.
while (true)
{
jsonText = responseStream.ReadLine();
//Post each message to the queue.
if (useQueue == "true")
{
Message message = new Message(jsonText);
q.Send(message);
}
//Success
wait = 250;
//Write Status
Console.Write(jsonText);
}
//Abort is needed or responseStream.Close() will hang.
webRequest.Abort();
responseStream.Close();
responseStream = null;
webResponse.Close();
webResponse = null;
}
catch (WebException ex)
{
Console.WriteLine(ex.Message);
logger.append(ex.Message, Logger.LogLevel.ERROR);
if (ex.Status == WebExceptionStatus.ProtocolError)
{
//-- From Twitter Docs --
//When a HTTP error (> 200) is returned, back off exponentially.
//Perhaps start with a 10 second wait, double on each subsequent failure,
//and finally cap the wait at 240 seconds.
//Exponential Backoff
if (wait < 10000)
{
wait = 10000;
}
else
{
if (wait < 240000)
{
wait = wait * 2;
}
}
}
else
{
//-- From Twitter Docs --
//When a network error (TCP/IP level) is encountered, back off linearly.
//Perhaps start at 250 milliseconds and cap at 16 seconds.
//Linear Backoff
if (wait < 16000)
{
wait += 250;
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
logger.append(ex.Message, Logger.LogLevel.ERROR);
}
finally
{
if (webRequest != null)
{
webRequest.Abort();
}
if (responseStream != null)
{
responseStream.Close();
responseStream = null;
}
if (webResponse != null)
{
webResponse.Close();
webResponse = null;
}
Console.WriteLine("Waiting: " + wait);
Thread.Sleep(wait);
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
logger.append(ex.Message, Logger.LogLevel.ERROR);
Console.WriteLine("Waiting: " + wait);
Thread.Sleep(wait);
}
}
public void QueueRead()
{
MessageQueue q;
string multiThread = ConfigurationManager.AppSettings["multithread"];
Logger logger = new Logger();
try
{
if (MessageQueue.Exists(@".\private$\Twitter"))
{
q = new MessageQueue(@".\private$\Twitter");
}
else
{
Console.WriteLine("Queue does not exist.");
return;
}
while (true)
{
Message message;
try
{
message = q.Receive();
message.Formatter =
new XmlMessageFormatter(new String[] { "System.String" });
if (multiThread == "true")
{
ThreadPool.QueueUserWorkItem(MessageProcess, message);
}
else
{
MessageProcess(message);
}
}
catch { continue; }
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
logger.append(ex.Message, Logger.LogLevel.ERROR);
}
}
public void MessageProcess(object objMessage)
{
status status = new status();
Logger logger = new Logger();
DataContractJsonSerializer json = new DataContractJsonSerializer(status.GetType());
try
{
Message message = objMessage as Message;
byte[] byteArray = Encoding.UTF8.GetBytes(message.Body.ToString());
MemoryStream stream = new MemoryStream(byteArray);
//TODO: Check for multiple objects.
status = json.ReadObject(stream) as status;
Console.WriteLine(message.Body.ToString());
//TODO: Store the status object
DataStore.Add(status);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
logger.append(ex.Message, Logger.LogLevel.ERROR);
}
}
}
}
data-text=”Open Source .NET (C#) Twitter Streaming API Client (Shannon Whitley)”
data-count=”vertical”
>Tweet
46 Responses to “Open Source .NET (C#) Twitter Streaming API Client”
Leave a Reply
2 Trackbacks
- eclipsed4utoo (Ryan Alford)
RT @swhitley: “Open Source .NET (C#) Twitter Streaming API Client” [new post] – [link to post] - swhitley (Shannon Whitley)
“Open Source .NET (C#) Twitter Streaming API Client” [new post] – [link to post]
-
App.net Status
swhitley
Available
Twitter Comment
RT @swhitley: “Open Source .NET (C#) Twitter Streaming API Client” [new post] – [link to post]
– Posted using Chat Catcher
Twitter Comment
“Open Source .NET (C#) Twitter Streaming API Client” [new post] – [link to post]
– Posted using Chat Catcher
hi shannon,
below bit create exception, and spit “Length of the data to decrypt is invalid” in the log
CryptoStream(ms, cryptoProvider.CreateDecryptor(KEY_64, IV_64), CryptoStreamMode.Read);
any advise?
nevermind, it’s my bad.
your streaming client rocks. and thanks for sharing
@tester – You’re welcome. I’m glad to hear it’s working for you.
uhm, sorry for my dumb question.
how can I use this with oAuth?
Hi ashigakari — oAuth doesn’t apply to the streaming client yet. They haven’t announced any plans to change that.
Thanks for the code Shannon it was a great help in connecting to the streaming API in .NET which I’ve struggled to find any other examples of.
I’m adding each tweet to a database but every now and again I get an error in my log file that says it’s failed to insert. I printed out the actual data from the tweet and it is all there and correct so I can’t see a reason why it would error.
I’m not using a queueing system like you have in place. Do you think it could be that in periods of high “tweetage” that there are just too many coming in for my program to process? Is that why you use the queuing system?
Many thanks,
Ian.
@Ian Black – Hi Ian. I’m not sure. I’d have to see the error. Using a queue isn’t mandatory, but I can see where it would help during periods of high database utilization. I’m not using this code in production right now so I haven’t run into any similar issues.
One thing about the multithreaded read from the queue: It seems to me that picking up messages from the queue and then queueing them again on the threadpool is not so good: Firstly it picks up messages that might not be processed immediately by the client and therefore deprives free clients from processing them. Secondly you lost all the nice transactionality and persistence that comes with MSMQ for no apparent gain.
May I’m failing to see the point.
Great code, but getting this in my log files:
2010-09-09T23:55:31 Ongeldige gegevens.
it’s in dutch, translated: Invalid Data.
Sorry for the idiot question, I’m just getting started. I tried to open the project with Visual Studio and run it, but it says:
Source file:TwitterStreamClientDistrib\Properties\AssemblyInfo.cs’ could not be opened (‘Unspecified error ‘)
I’m falling at the first hurdle!
@Robin, Sorry about that. You can just remove that reference from the project or import all of the other files into a new project.
Shannon, thank you for inspiring me to create my own PowerShell solution. Having it all in .NET already gave me a huge leap forward in my efforts.
http://code.google.com/p/pstwitterstream/
I found one thing with my testing that might be good to check for. When the stream was closed (I’m assuming Twitter automatically closed the previous stream when I had more than one stream opened on the same user and IP) my code would loop infinitely instead of error out. I found checking the EndOfStream property on the response stream helped me break out appropriately.
Hi Shannon,
I’ve been dabbling with Twitter application development lately, and must say your post has been a tremendous help ! Thanks
I got around your code to running it, however, I keep running into a tiny error:
“Invalid character in a Base-64 string.”
I ran the console application as:
“TwitterStreamClientDistrib.exe /encrypt twitter_password”
The app.config has twitter username and “” for twitter_password_encrypted key value. What am I missing here ?
Is it possible to unit test methods like Stream2Queue?
If there is a way, I’d really like to know.
I have a stream client derived from this code, with some extra stuff, and been having a hard time guaranteeing correctness.
Thanks for the great sample. This has been an awesome starting point for my project!
One bug I just found: in status.cs, you have “contributors” as a string type when the Twitter API actually returns an array of strings. It doesn’t usually matter since this value is almost always null, but it creates an interesting bug for the few tweets that use it.
Thanks again!
-Jonathan
@Jasnira Sorry, I don’t have anything written that would help with that. I usually just step through using debug mode or dump values out to a file that I can review.
@Jonathan – Thanks for the feedback. I’m not surprised to see a bug like that. Some of the less-often-used items were harder to locate in the examples. I’ll update the code when I get a chance.
Thanks for this example, I just started working with Twitter API in Python. So, I will adapt your example here to see if I can port it.
Will post here this weekend if I manage to get it working.
hi i am passing correct username/password but getting 401 unauthorised please help do i have to change some setting on twitter or send me working code with test twitter id which is able to login
Hi
is there any way that i get the notification automatically whenever any1 tweet instead of calling the service in a while loop as in facebook i can subscribe and it will send a notification if any update happen on my profile?
I am getting 401 error as well. Username and password are correct for sure. Any idea how to fix that?
Thanks.
I haven’t had a chance to look at the code, but I wonder if the 401 errors are due to Twitter’s change to exclusively using https for the streaming services.
hi, i got serach url like: http://stream.twitter.com/1/statuses/filter.json?track=avatar%20movie
Problem is that server hang for a long time between each tweet. However, its quite fast by searching one word(e.g.avatar) only. Do you have any idea?
Thanks for a great library, Shannon. I was wondering, if there’s a way to parse the multiple objects in JSON string?
Sorry for newbie question but when you are reading the stream one line at a time and posting to a queue; dont you risk losing content. Do you have any ideas on how one can consume the stream data and still store the results to a file or DB at the same time?
Hi @Joe, Because you have an open connection, it’s not like you miss messages. The messages queue on Twitter’s side. As long as the connection is open, you can read through the Twitter stream and catch up pretty quickly to the end of the stream. You could write to a file or DB in place of writing to the queue. I only wrote it that way because it’s a very quick way to pull the messages out of the stream and then continue reading messages. Although the messages won’t pass by, you also don’t want to get too far behind in reading the stream.
Thank you so much for clarifying that.
Will your code work with the new Streaming API (firehose, power track)?
@Joe You’re welcome. I haven’t been following recent changes to the api (hadn’t heard of ‘power track’). I’ll take a look, but you might get to it before I do.
hi shannon,
thajs for your effort. but i amfacing with a trouble. I run your project but it prints an error message to console and before i see the message console closes and program finishes its execution. do you have any idea what is my problem.
I want to stream a spesific hashtag. how can i do this with your project
Hi @memet – Try changing the url for twitter to “https” — that might be your issue. The code hasn’t been updated in awhile.
Hi, shannon
whether ur code wil work now?..A couple of months later, Twitter stopped supporting basic authentication (username/password). This meant that the old (simple) way of sending notifications stopped working. ..
Hi @Hari,
I just went through an update of the code and moved everything to GitHub. It’s all working with basic auth.
https://github.com/swhitley/TwitterStreamClient
Hi
Just wanted to follow up with another question. When reading the stream as responsestream.readline; I assumed that from the twitter doc it separated tweet with the end of line/carriage return character. So as you read per line you are getting one tweet which helps in the parsing done later; but when looking at the content it seems that is not the case. The tweet is split up as you either write to a file or add to the queue which makes it very hard to compile the tweet at a later point. Is there a way around this?
Hi Joe,
I don’t have the code up for testing, but I never had a problem with tweets being split in the stream.
Hello Mr.Shannon,
Thank you very much for your code.
I would like to use your code for my work. I open your code without editing any code then run it. I found “The remote server returned an error: Unauthorized”.
I am a newbie. Could you please suggest me?
@nee – Make sure you setup an application on Twitter and include your api credentials in the app.config file.
@Shannon – Hi I have setup an application on Twitter and which part of the file do I nee to to modify? (app.config file only ask for username and password)
@weikang -I’m sorry. I was thinking of another project. Yes, you should be able to enter your Twitter id and password into the app.config and get the project to work. Make sure you’re not behind a proxy server as I haven’t tested this code with a proxy server.
@Shanon, I have check that there is no problem with the proxy, and the error message I am getting is Unauthorised. Is it because, the Twitter is now using OAuth?
Shannon, sorry to be a nuisance but just want to clarify. Reading by line as you have it should be a complete post for each read, correct?
Hi, i see that you need to be checking the json state every 250 ms, there’s not a way to add a kind of listener to de json object? i mean omit the awkward timer?, great job btw.
Thanks, Chris. I haven’t really focused on that part of the code in a couple of years. If you find something that works better, I’d be more than happy to look at a pull request.
Great code, great sample !
One question: the sample outputs all sorts of fields to the console.
Wehere would I limit it to show for example only the id of the tweeter and the message itself ?
HI Shannon,
I just follow your code..
It’s giving me a error
The remote server returned an error: (401) Unauthorized.
I double check my credentials all are correct. How can I solve this problem..?
Hi Shannon,
I get the same error: (401) Unauthorized. Is there a way to resolve it?
Hi @marta
Twitter removed basic authentication and converted to oAuth. This code will no longer work as written. 04/05/2013
I have updated the post to reflect the change.