WebClient Class: Gotchas and Basics

Some basics of the useful WebClient class along with some things to watch out for.

For Oscar Peterson, who died today. We will miss you.

Trailing Slashes

When making a request for an HTTP resource, the trailing slash is significant in a URI and makes a difference in how the webserver processes the request.

For example, if you have the URI http://www.mysite.com/userstuff/, you can expect the server to look in the userstuff subdirectory and return the default document (provided directory browsing is turned off). But without the trailing slash, the server will look for a file called "userstuff" in the site's root folder - not what you want. If no such file is present, most webservers will return a 301 Permanent Redirect response, suggesting that the client try again with the trailing slash. A .NET WebClient or WebRequest will respond just like a browser would - by re-requesting the resource with the trailing slash appended.

What this means is that when you leave out a trailing slash when it should have been included, your request will still work, but you'll have created an extra round trip to the server.

WebProxy

If you do not have a proxy in use, you should set the Proxy property on all your WebClient and WebRequest objects to null. If you do not do this, you run the risk of having the Framework attempt to "autodetect" proxy settings, which can add up to an additional 30 seconds to your requests. As an alternative to having to set the Proxy property on every request, you can set global defaults like so:

WebRequest.DefaultWebProxy =null;

The above will apply for the lifetime of the application domain.

WebClient Class

WebClient is simply a wrapper over the WebRequest and WebResponse classes which can save a lot of extra lines of code. WebClient allows you to deal with your requests and responses with strings, byte arrays, files, or streams. Unfortunately, you can't use WebClient all the time; some features such as cookies are only available with WebRequest and WebResponse. You can't use the same instance of WebClient to perform repeated requests in sequence. For example if you are using a ThreadPool to handle multiple requests, you need to create a separate WebClient object in each thread.

The following are the steps to use WebClient:

1. Instantiate a WebClient instance

2. Assign the Proxy property

3. Assign the Credentials property if authentication is required.

4. Call one of the DownloadXXX or UploadXXX methods with the correct URI.

Each method of WebClient is overloaded to accept a Uri instance instead of a string address. The Upload methods are similar and each returns the response from the server:

UploadValues is used with the POST verb to upload a NameValueCollection just like a Form Post. The Content-Type header must not be null , you must set it to "application/x-www-form-urlencoded".

The Download methods are similar:

Since the asynchronous methods are new in .NET Framework 2.0, here is a short example of a Windows Form with a textbox for the url, a button and a multiline textbox for display, using DownloadStringAsync:

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Net;
using System.Text;
using System.Windows.Forms;

namespace WebClientTest
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();
        }

        private void button1_Click(object sender, EventArgs e)
        {
            WebClient wc = new WebClient();
            wc.DownloadProgressChanged += new DownloadProgressChangedEventHandler(wc_DownloadProgressChanged);
            wc.DownloadStringCompleted += new DownloadStringCompletedEventHandler(wc_DownloadStringCompleted);
            wc.DownloadStringAsync(new Uri(textBox2.Text),textBox2.Text);
            while(wc.IsBusy)
                System.Threading.Thread.Sleep(0);
            wc.Dispose();
        }

        void wc_DownloadStringCompleted(object sender, DownloadStringCompletedEventArgs e)
        {
            textBox1.Text += "Result: " + e.Result + " State:" + e.UserState.ToString() + "\r\n";
        }

        void wc_DownloadProgressChanged(object sender, DownloadProgressChangedEventArgs e)
        {
            textBox1.Text += e.BytesReceived.ToString() + " bytes  of " + e.TotalBytesToReceive.ToString() + " total\r\n";
        }
    }
}

Authentication

You can provide a username / password combination to an HTTP or FTP site by creating the NetworkCredential object and assigning it to the Credentials property of your WebClient instance:

using (WebClient wc = new WebClient() )
{
wc.Proxy=null;
wc.BaseAddress = "ftp://yoursite.com/files/";

string userName="me@mysite.com";
string password = "dorkusMcDweeb";
wc.Credentials = new NetworkCredential(userName, password);
wc.DownloadFile("mydocument.doc", "mydocument.doc");
wc.UploadFile("userlist.txt", "userlist.txt");
}

The above works with Basic and Digest authentication, and you can extend it via the AuthenticationManager class. You can also use NTLM and Kerberos if you supply the domain name before the userName. However this does not work with FormsAuthentication as it requires the Forms cookie ticket which WebClient does not support. For that, you must use WebRequest / WebResponse.

Header Support

WebClient (as does WebRequest) allows you to add custom HTTP headers as well as to enumerate the headers in a response. Headers are just key-value pairs. You add a header like this:

wc.Headers.Add ("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");

If your request contains a query, you'll need to add a user-agent header.

WebClient has a Dispose method which should be called as soon as your code has obtained the return data or any headers that are to be examined. Or, you can employ the using ( ... ) {   } construct as shown above.

 Find out more about the WebClient class, its methods, properties and events.