Logging HTTP Referals with asp.net

Source Code

Before using any sample code published here please read the disclaimer.

View Object Source	View SQL Source
Download Object Source	Download SQL Source

Referer [sic]

The specifications for HTTP, RFC 2616, state in 14.36

The Referer[sic] request-header field allows the client to specify, for the server's benefit, the address (URI) of the resource from which the Request-URI was obtained (the "referrer", although the header field is misspelled.) The Referer request-header allows a server to generate lists of back-links to resources for interest, logging, optimized caching, etc.

As you click a link, if your browser supports it (and 99.99% of them all do) and you don't have a privacy proxy which strips it, your browser tells the web server where it came from.

So how is it logged?

IIS logs the referring URLs as part of its log files. Unless you are logging into an ODBC database (which Microsoft admits can slow IIS down ) you have to rely on something that parses the log files to get your referring URLs into readable form.

So what if you want instant gratification? Log it yourself. There is one caveat to this process; it only logs referrals to pages processed by the ASP.Net engine (.aspx, .ashx and anything else configured in your machine.config's <httphandlers> settings).

Creating a database for referrals

First we need to create a suitable database structure and stored procedures to access the tables. referralLogger.sql contains

• a suitable table for logging the URL, [referral] (and the URL's component break downs, should you wish to query on those elements)
• a table for sites excluded from any stored procedures you use to display information [referralExclusions] (remember that every referrer is logged, this includes clicks generated from pages on your own site so you will want, at a minimum, to add records of your own site name into this table)
• a stored procedure [insertReferral] for inserting the referral information
• two sample stored procedures [listReferralsByClickCount] and [listReferralsbyReferrer] which can be used in a DataAdapter to provide a suitable DataSource for binding to data bound controls.

Rather than log each referral "hit" as a separate row the [referral] table has a count column and [insertReferral] increments this count as a new hit is logged.

The referral class

So we have a table, how do we log referring urls for each page hit? referralLogger.cs contains a simple class for logging the referral "hits".

You can see from the code that all the work is done in the constructor, you simple pass in the page url being displayed and the referring URL. The referrer is then broken down, using the properties provided by the Uri class.

The class excludes any referrals from localhost, if you wish to stop this behaviour you can remove the !uriReferrer.IsLoopback check (line 22).

The database connection string is read from the database.referrals application setting. (line 27).

Logging the referral "hits"

So once you have the sql database installed, and the class compiled and you have added a suitable database connection key into the appSettings section of your web.config how do you actually log?

Global.asxa provides a suitable (undocumented) event,

Application_OnPostRequestHandlerExecute(Object sender,
  EventArgs e)



This event fires after any request serviced by the ASP.Net engine has been handled and so is a suitable place to put code you want to run with every request, such as logging referrals.

So all we need to do is add a suitable object constructor into this event.

public void Application_OnPostRequestHandlerExecute(Object sender, EventArgs e)
{
  ReferralLogger rlLogger;
  if (String.Empty != Request.QueryString.ToString()) 
    // We have query parameters so we need to log those as well
    // as the current page name
    rlLogger = new ReferralLogger(Request.Path.ToString()+"?"
                                  Request.QueryString.ToString(),
                                  Request.UrlReferrer);
  else 
    // No query string, so simply log the request and the referring URL
    rlLogger = new ReferralLogger(Request.Path.ToString(),
                                  Request.UrlReferrer);
}

And now, every time the ASP.Net engine displays a file its referral information will be stored in your database.

Note: Displaying every referring URL may not be a good idea. In fact, it may well help spammers.