Message Queue over HTTP: Usage Scenarios
By Peter A. Bromberg, Ph.D.

Peter Bromberg

I have a pet peeve about Managers (Project Managers, IT Department Managers, and the like). It seems that some otherwise perfectly intelligent people, when they become managers, suddenly go through this Kafkaesque transformation and become - well, ogres (I can say this without hesitation, because I was once a Manager, and it didn't last long, because I could feel it starting to happen to me). Kind of Like Jeff Goldblum staring at his crawling skin as he began to turn into -- The Fly! Their personalities change; they seem to get these "hardened" views about certain technologies and become ultra-dogmatic about whether or not their development people are going to be allowed to use them to solve business problems. (Thankfully, the boogers let me go back to being a developer and I remained happy. My personality didn't change - it's still just perfect.)



Let me give you a couple of real - life examples. These both happen to relate to MSMQ, and -- not coincidentally -- that's what this short article is about, too. I was a Senior Developer at one outfit which had a humongous e-commerce application with over 1200 SQL Server tables spread out across at least five or six linked servers. One of the needs was a reliable disconnected messaging scenario involving the clients dropping notification messages about having uploaded a new product catalog, which our humongous application was supposed to process and then update these 1200 or so tables - but first by sticking everything in a half dozen "staging tables" for preprocessing. There really wasn't a reliable way to deposit notification and status messages that could be picked up later. Naturally, this is an MSMQ scenario, and when I proposed that we look into it to the "Manager", you can guess what the response was. "Oh, no, we've had REAL BAD EXPERIENCES with Message Queue. Absolutely not!". Doh!

The other one was with another company that had a pretty big operation that was almost totally centered around SQL Server. SQL Server did everything in this enterprise - it made the notifications, it polled stuff with timed jobs running out of SQL Agent, it sent the emails, everything with SQL Server except maybe cleaning the men's room - and I bet if the manager could have figured a way to make SQL Server do that, he would have added it to the list. There was so much contention on their network what with different jobs running at the same time as all the heavy data access, that stuff would fail or timeout. I suggested several alternatives here, including Windows Services, Message Queue, and even Multicast IP for subscribing to notifications-- all of which were tossed out with comments like "No. I don't like WIndows Services. Absolutely not. " Jeesh! Don't like Windows Services? Better not turn on your computer in the morning then, cause it runs about twenty of them. . .

The bottom line is this: If the only tool you have to deal with programming situations is a hammer, every problem starts to look like a nail. That's a pretty heavy-handed, tunnel - vision approach to programming, and to life, in general. When somebody says something like "We've had bad experiences with ..." or "I don't like ...", what they are really saying is, "I'm the boss, and since I'm too dumb (or unwilling) to figure out how to use this technology reliably, we simply won't use it so that I don't have to put my ego in my back pocket in the name of quality software development". These guys could all use a good course in lateral thinking.

One of the coolest features of MSMQ 3.0 is the ability to send messages over HTTP - right through firewalls. This works great in workgroup mode, and the Queue is simply exposed as an IIS Vroot. But before I get into that, lets take a quick look at the feature matrix table (from MSDN online) for Message Queue. This is broken down into New Features, Improved Features, and Existing Features:

New features
Feature name Description
Delivering messages over HTTP transport SOAP Reliable Messaging Protocol (SRMP), an XML-based messaging protocol, has been developed for delivering high Quality of Service (QoS) messages. The direct, public, and private format names of administration and response queues can be included in messages sent over HTTP transport. Similarly, the names of administration and response queues in HTTP format can be included in messages sent over ordinary (non-HTTP) transport.
Triggers Now a component in Message Queuing 3.0, rather than an add-on module that existed in MSMQ 1.0 and MSMQ 2.0. Triggers provide a mechanism that associates the arrival of each incoming message at a queue with a response that depends on the contents of the message and may invoke either a COM component or a stand-alone executable program. Business rules can be defined and invoked in response to such messages without any additional programming.
Sending messages to multiple destinations Clients are able to send the same message to multiple recipient queues. Lists of destination queues can be specified explicitly by means of distribution group objects (distribution lists) in Active Directory® as well as in the form of multiple-element format names. Support for ensuring that messages sent to distribution lists and multiple-element format names will reach queues on downlevel computers is provided. In addition, message delivery to IP multicast destinations using the PGM protocol is supported.
Message lookup A way to peek at or retrieve a specific message without using cursors to navigate through the queue until the message sought is located. This functionality is based on a 64-bit lookup identifier that is assigned to each message when it is placed in a destination queue.
Improved Features
Feature name Description
Improved Active Directory Integration Although Active Directory integration exists in MSMQ 2.0, Message Queuing 3.0 improves integration with, and extension of, the Active Directory service to store all configuration, security, and status information. Message Queuing clients now use the Lightweight Directory Access Protocol (LDAP) to access domain controllers and global catalog servers for Message Queuing-specific information in Active Directory directly without assistance from a Message Queuing server on a domain controller.
Improved Microsoft Management Console (MMC) support Message Queuing is now administered more completely using snap-ins hosted in MMC . Administrative tasks formerly performed in Control Panel are now performed in snap-ins.
Workgroup support Message Queuing can be installed in workgroup mode on computers belonging to a Windows Server 2003 workgroup, rather than to a domain. In addition, a computer on which Message Queuing is installed in a workgroup can later join a domain, and then separate from the domain.
Improved Active/active cluster support In addition to MSMQ 3.0 full support of the active/active paradigm in a server cluster, which means that Message Queuing can run on all nodes in a server cluster simultaneously, triggers in MSMQ 3.0 are also integrated with active/active cluster support.
Windows CE support Although MSMQ 2.0 was also available on handheld and palm-sized computers running the Windows CE 3.0 or later operating system, Windows CE also now supports the Message Queuing SRMP protocol for HTTP messaging, available in MSMQ 3.0.
Enhanced Windows security integration MSMQ 3.0, like MSMQ 2.0, leverages security features of the Windows operating system through the use of access control, auditing, encryption, and authentication, and using security features such as the Kerberos security protocol. In addition, MSMQ 3.0 provides for the use of more secured remote reading, and the use of SSL authentication for HTTP/HTTPS messaging.
Existing Features
Feature name Description
Workgroup support MSMQ 3.0 can be installed and run in a Windows Server 2003 workgroup, rather than in a domain. In addition, Message Queuing that is installed in a workgroup can later join a domain, and then separate from the domain again.
Transactional messaging Using transactional capabilities, you can couple several related actions in a single transaction, ensure messages are delivered in order, ensure messages are delivered only once, and confirm that messages were successfully retrieved from the destination queue.
Message backup and restore Message storage files, log files, transaction log files, and registry settings can be backed up and restored in case of computer failure.
Message prioritization Message prioritization allows urgent or important messages to to be sent before less important messages, so you can guarantee adequate response time for critical applications at the expense of less important applications.
Guaranteed message delivery Messages can be stored on a disk-based queue and then later forwarded to provide guaranteed delivery.
Connectionless messaging With store-and-forward message queuing, computers are not affected by network disturbances and do not have to establish sessions. Because a sessionless model is used at the application level, the source computer and the destination computer do not need to support the same network protocol. Unlike MSMQ 2.0, MSMQ 3.0 does not support the Internet Packet Exchange (IPX) protocol.
Message routing Message Queuing provides message routing based on the physical topology of the network, session concentration needs, and transport connectivity. Session concentration facilitates the efficient usage of slow communication links.
Dynamic queue creation You can create queues or change queue properties as you go without affecting messaging applications.
Disparate system integration Message Queuing can be used across a wide variety of hardware platforms using connectivity products provided by other vendors.
Mixed-mode operation MSMQ 3.0, unlike MSMQ 2.0 does not support a mixed-mode domain environments, which means that MSMQ 1.0 controller servers running on Windows NT 4.0 cannot exist with in a Windows Server 2003 domain with Windows Server 2003 domain controllers running MSMQ 3.0.

To me, the two most interesting features are the very first two: delivering messages over HTTP transport, and triggers. HTTP, for all the obvious reasons, not to forget SOAP formatting. Triggers, because we now have an easy way to create rules and associate the arrival of each incoming message at a queue with a response that depends on the contents of the message. We can either invoke a COM component or a stand-alone .NET executable program. Our .NET executable for example, can read the message in the Queue and do all kinds of neat stuff "behind the scenes". A third and also very interesting new feature is message delivery to IP multicast destinations using the PGM protocol. That can get somewhat complex, so I'll leave it for a subsequent visit. So, let's discover how to use the first two features, HTTP and Triggers.

HTTP Messages with MSMQ 3.0:

MSMQ 3.0 provides support for referencing URL-named queues, creating SOAP-compliant (SRMP) messages that can be sent over HTTP, and authenticating HTTPS messages.

To send messages to a destination queue over HTTP, the target computer must have the Message Queuing HTTP Support subcomponent and IIS installed. Access to the queue can be restricted to specific users by changing the permissions of the virtual directory in the IIS snap-in. If you have it installed, you will see an IIS application with the name (Surprise!) "MSMQ", and it will have nothing in it.

Destination queues for HTTP messages are opened using direct format names that include the URL address of the target computer, the virtual directory name, and the queue name separated by slashes. The default virtual directory name is msmq, but Message Queuing can be configured by IIS to use a different virtual directory. A direct format name containing HTTPS as the protocol name invokes a secure HTTP transport through a Secure Sockets Layer (SSL) connection.

The following are some examples of direct format names for sending HTTP messages.

DIRECT=HTTP://Mike01/msmq/MyQueue (Within the enterprise)
DIRECT=HTTP://www.northwindtraders.com/msmq/MyQueue (Over the Internet)
DIRECT=HTTPS://www.northwindtraders.com/msmq/MyQueue (Over the Internet)

HTTP is automatically chosen as the transport for any Message Queuing message sent using an HTTP format name. In this case, the message will be formatted using the SOAP extensions for reliable messaging in the SOAP Reliable Messaging Protocol (SRMP).

Note You cannot read messages in a queue on a remote computer if the queue is opened using an HTTP direct format name; you can only use HTTP to send messages.

A complete HTTP message includes a SOAP envelope and the SOAP attachments associated with it. Here is a sample actual SOAP Message.

When sending an HTTP message, Message Queuing automatically transforms the individual binary message properties to specific WS-Routing elements, SRMP elements, and elements of the MSMQ namespace, and includes them in the header portion of the SOAP envelope. On the receiver side, Message Queuing maps these XML elements to the respective binary message properties before storing the message. An HTTP message maintains the XML elements specifying the message properties in the read-only PROPID_M_SOAP_ENVELOPE or MSMQMessage.SOAPEnvelope property.

When authentication is requested, Message Queuing automatically signs HTTP messages using an XML digital signature, and automatically includes the correct signature - related elements in the SOAP message. All of this is done automatically, without you having to write a single line of code!

Setting up a Queue for HTTP access with a Trigger

In our sample project, we will make some very simple assumptions that will enable you to get MSMQ via HTTP access with a trigger up and running as a "learning proof of concept". We will assume that subscribers wish to have their name and email addresses entered into your Notifications table for further processing, and that during the course of the normal operation of some sort of client program located outside your network, the program will make an HTTPRequest, sending an MSMQ subscribe message that will execute your MSMQ Trigger upon receipt. On the server, your .NET executable will read the Message out of the Queue, cast it to an instance of the "test" class, get the Name and Email properties, and make a SQL insert into your Notifications table for further processing.

Follow these steps to create the infrastructure for our sample project:

  1. Create a new Private Queue called "notifications"
  2. This should be a Transactional Queue.
  3. Create a new Trigger for Peeking, enabled, serialized, called "Notify"
  4. Create a new Rule called "Notify" with a Rule Condition of "Message Label contains Notification"
  5. For the Rule Action, specify the path to the TriggerHappy project executable (on my machine, this is "C:\CSHARPBIN\HTTPQueueSender\TriggerHappy\bin\Debug\HTTPSender.exe"
  6. In Parameters, pass the MessageId as Variant, and the MessageBody as String.
  7. In your Trigger's attached rules, ensure that the Rule Notify is attached.
  8. Ensure that both the MessageQueue and the MessageQueue Trigger services are running.
  9. Create a new (local) Sql Server (or MSDE) database named "TEST".
  10. Execute the Sql Script "NotificationsTable.sql" in Query Analyzer to add the Notifications table to your database

Ok. We are now ready to explore the code, and how it works. First, let's take a look at the HTTPSender Client UI so that you can get an idea of what this is about:

We can specify the Queue name, and using the FormatName directive, the full HTTP endpoint is visible. For the Info to send, we will specify the user's name and email address as explained above. The SEND button will intitiate the sending of the Message to the queue. Under the hood, here is the code that actually performs this action:

private void button1_Click(object sender, System.EventArgs e)
  {
   try
   {
    System.Messaging.MessageQueue mq = 
       new System.Messaging.MessageQueue (label5.Text);
        mq.DefaultPropertiesToSend.Recoverable=true;
    mq.DefaultPropertiesToSend.Label="Notification";
    Test myTest = 
       new  Test(textBox2.Text,textBox3.Text);
    System.Messaging.Message msg=new System.Messaging.Message();     
    msg.Formatter =new System.Messaging.BinaryMessageFormatter();     
    msg.Body=myTest;    
    mq.Send(msg,"Notification",System.Messaging.MessageQueueTransactionType.Single );
   }  
   catch (System.Messaging.MessageQueueException mqx)
   {
    MessageBox.Show (mqx.Message) ;
   }
  }


"Test", above, is simply an instance of a simple serializable class that has two string properties, Name and Email. This class is included in both the Sender and the Receiver (our .NET Executable that is run by the Trigger) so that correct serialization and deserialization can be performed on the Message Body.

At the receiving end, recall that our Trigger is going to execute an instance of an "HTTPSender.exe" console app that will read the Message from the Queue, and do the SQL insert. This is, of course, just a simple surrogate for some business process that in production would obviously be much more complex:

static void Main(string[] args)
  { 
   // you can use the next line for debugging purposes
   // System.Diagnostics.Debugger.Launch(); 
    object MessageId="";
    object MessageBody="";
    if(args.Length>0)
    {
     MessageId=args[0];
     // note we are passing the message body too, but we don't use it here
     MessageBody=args[1];
    }
    MessageQueue q  = new MessageQueue(@".\Private$\notifications");
     if(MessageId!=null)
    {
     string msgId=((string)MessageId).Substring(((string)MessageId).IndexOf(@"\"));
    }
    System.Messaging.Message msg=null;
    Test tst=null;
    if(MessageId!=null)
    {
     // need to strip off the curly brackets to use ReceiveById!
     string msgid=((string)MessageId).Replace("}","");
     msgid=msgid.Replace("{","");   
     msg= q.ReceiveById(msgid);
     Type[] expectedTypes= new Type[]{typeof(string)};
     msg.Formatter =new System.Messaging.BinaryMessageFormatter();
     System.Diagnostics.Debug.WriteLine(msg.Id.ToString());
     tst=(Test)msg.Body;
    }     
   string sSql="Insert into Notifications (NotifyDateTime,Name,Email)";
   sSql+="Values('"+System.DateTime.Now+"','"+tst.Name +"','"+tst.Email +"')";
   SqlConnection cn =new SqlConnection("server=(local);database=Test;user id=sa;password=;");
    cn.Open();
    SqlCommand cmd=new SqlCommand(sSql,cn);
    cmd.CommandType=CommandType.Text;
    cmd.ExecuteNonQuery();
    cmd.Dispose();
    cn.Close();    
   }   

With that, we have completed the entire implementation of sending an MSMQ message over the Internet via HTTP to a remote queue, and having a Trigger on the remote queue spawn an instance of an Executable that handles some required business logic in the enterprise.

There is much more to MSMQ 3.0 than described here. You have Acknowledgement Queues, where you can set the Administration Queue to the queue that receives acknowledgements of messages, and the AcknowledgementType defines the acknowledgements that we are interested in . There is also a Mapping XML file that you can set for HTTP to a secondary URL where acknowledgements can be picked up. Routing of messages to other queues, even other machines, can also be set up here. You determine which acknowledgement corresponds to a sent messages with the CorrelationId and MessageQueue.ReceiveByCorrelationId method. Messages can be involved in Transactions, and they can be Encrypted. We can also turn on Tracing to get reports on what is happening to messages in an enterprise environment.

Finally, we can use Queued Components derived from System.EnterpriseServices in COM+ to make it easier to call methods across a Message Queue. However, by using this simplified model, we also lose the ability to programmatically do fine-grained Message control as in the above example.

Conclusion: MSMQ 3.0 is an Enterprise - Level tool to handle a wide range of both disconnected and connected Messaging scenarios. Developers should carefully study the documentation. Contrary to what some "Managers" may have to say about it, when properly configured and tested, MSMQ is both reliable and fast, and can save many hours of custom programming time.

Download the Solution that accompanies this article


Peter Bromberg is a C# MVP, MCP, and .NET consultant who has worked in the banking and financial industry for 20 years. He has architected and developed web - based corporate distributed application solutions since 1995, and focuses exclusively on the .NET Platform.
Article Discussion: