ASP.NET Twitter Link Tracker for Friend Posts via Twitter and Ottter APIs

While experimenting again with the Topsy Otter Twitter API, I wondered how easy it might be to specify any twitter username, get a list of their friends ("followees?"), and then put together a nice grid of all those who had recently tweeted a link to something. This has real value since you can look at all your friends (people you "follow") or even anybody else's friends, and instantly see what interesting resources they are tweeting about in a simple ASP.NET Web app.

The first thing we need is a list of the twitter usernames who our entered user follows. It turns out that this is so easy, I didn't even use one of the Twitter Client APIs such as Twitterizer, TweetSharp or Linq To Twitter. There is no oAuth required. Here is the Format:

string url ="http://api.twitter.com/1/statuses/friends.xml?screen_name=<twitterUserName>&cursor=-1"

For JSON, just replace the .xml? with .json?

The cursor=-1 parameter returns the first set of friends, newest first. By capturing the next_cursor element in the returned XML (or JSON) you can append this to the cursor parameter and get the next "set" of results for older friends.

Parsing the XML that comes back is simple:

            WebClient wc = new WebClient();
            string stuff = wc.DownloadString(url);
            wc.Dispose();
            XmlDocument doc = new XmlDocument();
             doc.LoadXml(stuff);
             // you need the next line in order to get more, set the &cursor= to this new value.
            next_cursor = doc.DocumentElement.SelectSingleNode("//next_cursor").InnerText;
            XmlNodeList nodes = doc.DocumentElement.SelectNodes("//screen_name");
            var users = new List<string>();
            foreach (XmlNode nod in nodes)
                 users.Add(nod.InnerText);
            numusers = users.Count;

Now we've got a list of our user's friends, and we can use the Topsy Otter API to  get a list of "LinkPost" objects for each of the users. I'm using the server-side Otter API wrapper that I originally created  for this older article. I won't go into the code for that here, if you are interested, have a look at the OtterApi project in the downloadable solution below.

So the next step is that I must create a List of type "list" (the object that my Otter API C# wrapper returns for a GetLinkPosts query), and keep filling this up with posts that have links in them for each of the list of users.  The fastest and most efficient way to do this is to use the .NET Threadpool:
  
  private ManualResetEvent mre = new ManualResetEvent(false);
foreach (string s in users)
             {
                 ThreadPool.QueueUserWorkItem(new WaitCallback(ProcessItem), s);
            }

            mre.WaitOne();
           List<list> finalPosts = new List<list>();
            linkposts = linkposts.OrderByDescending(x => x.date).ToList();
            finalPosts = linkposts.Distinct(new MyComparer()).ToList();
           finalPosts= FixLinks(finalPosts);
             this.GridView1.DataSource = finalPosts;
             Session["data"] = finalPosts;
            GridView1.DataBind();
         } // end of Button click handler
             private int ctr = 0;
        void ProcessItem(object state)
        {
            var linkpost = Otter.GetLinkPosts((string)state, 0, "http://");
            if (linkpost == null || linkpost.list == null)
            {
                 ctr++;
                 return;
            }
            foreach (var l in linkpost.list)
                 linkposts.Add(l);
            ctr++;
            // you can set this to a higher number, but it may blow up
            if (ctr == 20)
                mre.Set();
        }


With the above code, you can set an arbitrary number (here i use 20 users' lists of posts) and call the Set method on the ManualResetEvent which releases the WaitOne call in the first block of code and proceeds to fix up the total results and bind it to a GridView. For 20 users, I found this takes anywhere from 8 to 15 seconds which is acceptable. If you use a higher number than 20, it can either take too long or it may blow up and there isn't really anything you can do about that - The Otter API does have some rate limits, although no user key is required.

The rest of the code just performs some REGEX to create an <a href around each link, and ensure there are no duplicates. The de-dupe code uses a custom Comparer:

public class MyComparer : IEqualityComparer<OtterApi.list>
    {
        public bool Equals(OtterApi.list x, OtterApi.list y)
         {
             return x.content == y.content;
        }

        public int GetHashCode(OtterApi.list obj)
         {
             return obj.content.GetHashCode();
        }
    }

The FixLinks code is as follows:

protected List<list> FixLinks(List<list> items)
        {
           List<list> newPosts = new List<list>();
            Regex urlregex = new Regex(@"(http:\/\/([\w.]+\/?)\S*)",
                     RegexOptions.IgnoreCase | RegexOptions.Compiled);
            foreach (var l in items)
             {
                 if (l.content.Contains("http://"))
                {
                    l.content = urlregex.Replace(l.content,
                                                 "<a href=\"$1\" target=\"_blank\">$1</a>");
                     newPosts.Add(l);
                 }
            }
            return newPosts;
        }

    Finally, I order the items by date descending (most recent first), store the datasource in Session and set up Paging for the GridView so you can look at 10 rows at a time.

    The result should look something like this:

By the way, "Rickasaurus" is Rick Minerich, a friend, fellow MVP, and real F# genius with a new book out that I highly recommend: Professional F# 2.0

You can download the full Visual Studio 2010 Solution, which includes my OtterAPi wrapper project.

By Peter Bromberg   Popularity  (2967 Views)