Thoughts on technology and social web

March 2, 2010

Pubsubhubbub with .NET and WCF – Part 1

Filed under: Uncategorized — Ravikant Cherukuri @ 12:42 am

Pubsubhubbub (PuSH) has been gaining momentum in the real-time space as a protocol for server to server notifications on changes to feeds. Google Buzz API is based on PuSH. Several google services have adopted this and its now catching momentum among other services as well. Superfeedr is my favorite startup in this space. They host hubs for your feeds, handling the complexity of hub implementation for you. The real-time web is becoming truly real-time now and companies like superfeedr could provide the backbone for this new model.

The PuSH protocol itself simple to implement and defined in an open spec. Most of the code samples available for this are in ruby/python/php etc. I started working on a PuSH subscriber in C# with a WCF REST service. WCF provides an easy to use framework to build web services in .NET. To implement PuSH, there are some WCF quirks to getting this done. In this post I explain how to build a WCF subscriber service.

These are couple of good posts about how to use and implement PuSH.

http://blog.superfeedr.com/API/pubsubhubbub/getting-started-with-pubsubhubbub/
http://josephsmarr.com/2010/03/01/implementing-pubsubhubbub-subscriber-support-a-step-by-step-guide/

We need a service that supports the following REST APIs to behave as a PuSH subscriber.

  • Verify
    This will receive a GET request to verify the subscription once we POST a subscribe to the hub.
  • Callback
    This will receive new content posted to the feed.

Note that PuSH is a server to server protocol and so the subscribing entity is also a server. There are two pieces of code we need to write for the subscriber. One is the client code that posts a subscribe onto the hub and the second is the service that responds to the verification and data callbacks. The service is implemented using WCF and the subscriber client library using HttpWebRequest. I used WcfContrib library to encode/decode requests and responses. The WCF service interface would look like this.

[ServiceContract]
public interface IPubSubHubbub
{
    [OperationContract]
    [WebInvoke(Method = Verbs.Get, UriTemplate = "/callback3?hub.mode={mode}&hub.topic={topic}&hub.challenge={challenge}&hub.lease_seconds={leaseSeconds}&hub.verify_token={verifyToken}")]
    Stream Verify(SubscriptionMode mode, Uri topic, string challenge, int leaseSeconds, string verifyToken);

    [OperationContract]
    [WebInvoke(Method=Verbs.Post, UriTemplate = "/callback3")]
    [ServiceKnownType(typeof(Atom10FeedFormatter))]
    void SubscriberCallback(SyndicationFeedFormatter formatter);
}

When a subscriber subscribes to a hub, the subscriber provides a callback URL to the hub. The hub calls this URL to verify the subscription. Later, when the topic has new content published, the hub will call the same URL with the content (PuSH does a fat ping). This is why both methods above gave the same base URI. Verification is a GET request while the data callback is a POST. Subscription mode is defined using DataContract as below. Notice that the Verify method uses a Stream return value. This is because the Verify method needs to return the challenge that the hub sends in “text/plan” format. The new content notification happens as a POST in ATOM format.

[DataContract]
public enum SubscriptionMode
{
    [EnumMember(Value = "subscribe")]
    Subscribe,
    [EnumMember(Value = "unsubscribe")]
    Unsubscribe,
}

Below is the WCF server implementation.

[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
[WebDispatchFormatterConfiguration("application/x-www-form-urlencoded")]
[WebDispatchFormatterMimeType(typeof(WcfRestContrib.ServiceModel.Dispatcher.Formatters.FormUrlEncoded), "application/x-www-form-urlencoded")]
[WebDispatchFormatterMimeType(typeof(WcfRestContrib.ServiceModel.Dispatcher.Formatters.PoxDataContract), "application/atom+xml")]
public class Service1 : IPubSubHubbub
{
    public Stream Verify(SubscriptionMode mode, Uri topic, string challenge, int leaseSeconds, string verifyToken)
    {
        WebOperationContext.Current.OutgoingResponse.ContentType = "text/plain";
        return new MemoryStream(Encoding.UTF8.GetBytes(challenge));
    }

    public
void Callback(SyndicationFeedFormatter formatter)
    {
        ServiceReference1.Service1Client service = new ServiceReference1.Service1Client();

        foreach (SyndicationItem item in formatter.Feed.Items)
        {
       
        }
    }
}

Things of note here.

  • The "application/x-www-form-urlencoded" encoding is handled by the WcfContrib library’s FormUrlEncoded formatter class.

  • The verify method returns the challenge back as a memory stream and sets the encoding as "text/plain" on the WebOperationContext. Without this, the default encoding will be "text/xml" and hub would fail the subscribe.

  • The Callback method would received a parsed feed. The SyndicationFeedFormatter should be able handle both ATOM and RSS feeds, though I tested it only with ATOM.

To establish the subscription, the subscriber needs to make a REST call to the hub. The initial input that the subscriber would have about the topic that needs to be subscribed to is the URL to the feed. The feed header would have a link of type “hub” that would point to the hub for this feed.

String GetHubFromFeed(String feed)
{
    WebRequest r = HttpWebRequest.Create(feedUrl);
    WebResponse re = r.GetResponse();

    using (StreamReader sr = new StreamReader(re.GetResponseStream()))
    {
        XmlDocument doc = new XmlDocument();

        doc.LoadXml(sr.ReadToEnd());
        XmlNamespaceManager nsMgr = new XmlNamespaceManager(doc.NameTable);
        nsMgr.AddNamespace("atom", http://www.w3.org/2005/Atom);
        XmlElement hubElement = (XmlElement)doc.SelectSingleNode(
                                                                  "//atom:feed/atom:link[@rel='hub']", nsMgr);
        return hubElement.SelectSingleNode("@href", nsMgr).Value;
    }
}

Once we have the hub URL, we need to setup a subscription to the hub for our topic URL. The topic URL is the URL in the atom:link with rel=”self” (There has been discussion in the PuSH group about being able to subscribe with any URL that points to the feed and in the latest version of the spec, its no longer a requirement to subscribe to the “self” URL).

public String Subscribe(string feedUrl, string hubUrl, string callbackUrl)
{
    StringBuilder contentBuilder = new StringBuilder();
    NameValueCollection collection = new NameValueCollection();
    byte[] responseContent = null;

    collection.Add(hubCallbackParamName, callbackUrl);
    collection.Add(hubMode, "subscribe");
    collection.Add(hubTopic, feedUrl);
    collection.Add(hubVerify, "sync");
    collection.Add(hubVerifyToken, "test_token");

    for (int i = 0; i < collection.Keys.Count; i++)
    {
        contentBuilder.AppendFormat("{0}={1}&",
        HttpUtility.UrlEncode(collection.GetKey(i)),
        HttpUtility.UrlEncode(collection.Get(i)));
    }

    int stringLength = contentBuilder.Length;
    byte[] requestContent = Encoding.UTF8.GetBytes(
    contentBuilder.ToString(0, stringLength > 0 ? stringLength – 1 : 0));

    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(hubUrl);
    request.Method = "POST";
    request.ContentType = "application/x-www-form-urlencoded";
    request.ContentLength = requestContent.LongLength;

    using (Stream requestStream = request.GetRequestStream())
    {
        requestStream.Write(requestContent, 0, requestContent.Length);
        requestStream.Flush();
    }

    using
(HttpWebResponse response = (HttpWebResponse)request.GetResponse())
    {
        if (request.HaveResponse)
        {
             responseContent = new byte[response.ContentLength];
             using (Stream responseStream = response.GetResponseStream())
             {
                 responseStream.Read(responseContent, 0, responseContent.Length);
             }
        }
    }

    return Encoding.UTF8.GetString(responseContent);
}

This completes a PuSH client implementation. In a later post I will go over the implementation of a hub in WCF.

About these ads

9 Comments »

  1. Great post! Thanks a lot for writing it.

    Comment by Brett Slatkin — March 2, 2010 @ 9:39 pm

  2. Indeed :) and thanks for the thumbs up!

    Comment by julien — March 24, 2010 @ 4:00 am

  3. Thanks, would it be too much to ask that you make a ASP.NET MVC version of this. We summarily dumped all our WCF implementations and converted them all to ASP.NET MVC REST.
    WCF is dead in our opinion, long live MVC.

    Comment by George — April 13, 2010 @ 1:34 pm

    • I would definitely try to do that. I am new to MVC so some learning needed there. I was thinking of having an azure version of a hub. Maybe MVC would be a good idea for that.

      Comment by Ravikant Cherukuri — April 13, 2010 @ 2:49 pm

  4. Pardon me but I am a junior programmer and this is the first time I am getting into API applications. Can Push be implemented as a desktop application sending requests to google buzz API? Or is there some other way to implement this?

    Comment by Joe — May 24, 2010 @ 6:57 am

    • Pubsubhubbub is a server to server protocol. That is, you could only receive buzz updates from a web server as you have to expose a callback URL that can be called by the hub.

      You could publish to a hub from a desktop aplication though. Publish is just an http post. If you want to receive buzz updates from a desktop application, I think you can use the google feed API for that.

      Comment by Ravikant Cherukuri — May 24, 2010 @ 8:18 am

      • Thanks, I will look into the rss reader.

        Comment by Joe — May 24, 2010 @ 10:59 am

  5. Is it possible for you to post or email me the source files for this project. By the way this was an EXCELLENT article. Thank you so much

    Jeffrey

    Comment by Jeff — July 28, 2010 @ 8:20 am

    • If it possible please share this source file or send email to me also… Big Thanks.

      Comment by Worawut — October 29, 2010 @ 11:01 pm


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The WordPress Classic Theme Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: