Making URIs safe

One of the things I try to hammer home in my presentation is you should never emit user input without making it safe. The HttpUtility class provides developers with two main methods for this, HtmlEncode and UrlEncode.

HtmlEncode will take a string and escape it so that it is safely displayable on screen, removing the risk of Cross Site Scripting attacks. UrlEncode takes a string and escapes it to a format suitable for use in a URL and is usually used to encode query values, escaping such characters as = and & into their encoded values; but what happens if you have a complete URI and push it through UrlEncode?

Well, it gets mangled. If I take the following URI from my referral logs

http://google.com/search?hl=en&lr=&safe=off&q="key+
renewal+period"+wpa

and push it through UrlEncode I get

http%3a%2f%2fgoogle.com%2fsearch%3fhl%3den%26lr%3d
%26safe%3doff%26q%3d%22key%2brenewal%2bperiod
%22%2bwpa

The UrlEncode method has escaped everything, not, as I probably wanted, just the query values. That's really not much good to me. Obviously I don't want to echo a potentially unsafe URL into an <a href> tag as judicious use of a single quote, such as that in the example above would open me up to XSS.

With version 2 of the .NET framework Microsoft delivered a function to escape a complete URL correctly, a method on the URI class. The URI class has been long used to get at various parts of a URI without having to do any of the heaving lifting yourself. Unfortunately in earlier versions of the framework you'd still have to do a lot of work yourself as the ToString() representation of a URI can still be unsafe. For example creating a new URI object and loading the example URI above then calling ToString() gives me

http://google.com/search?hl=en&lr=&safe=off&q="key+
renewal+period"+wpa

which, whilst correct, isn't safe, in fact it's just the original string I loaded in. Help is at hand however, in the shape of the EscapeUriString method. A call to that gives us

http://google.com/search?hl=en&lr=&safe=off&
q=%22key+renewal+period%22+wpa

Safety at last. EscapeUriString is also static, so you don't need to construct a URI object to use it, simply call Uri.EscapeUriString(badUrl);

What can you do if you're using .NET 1.0 or 1.1? Use the EscapeString method. Both these methods do pretty much the same thing, calling a private unsafe method to do the work for you.

Sunday, January 07, 2007 4:26 PM

Your Comments.

# re: Making URIs safe

"..ToString() representation of a URI can still be unsafe.."

In fact you should rarely use Uri.ToString() because it gives you a "more human-readable" but essentially incorrect representation of the URI. You should use the Uri instance's AbsoluteUri property e.g.:

string q = "X&Y";
Uri googleUrl = new Uri("http://google.co.uk/search?q=" + HttpUtility.UrlEncode(q));

Console.WriteLine(googleUrl.ToString());
// displays "q=X&Y" - WRONG

Console.WriteLine(googleUrl.AbsoluteUri);
// displays "q=X%26Y" - Correct

So even if you've gone to the trouble of encoding your querystring params, if you ever load your URL in a Uri and ToString it they get unescaped!

Also I think you should use the most appropriate encoding/escaping method depending on the context – rather than rely on some overall "make it safe" routine. For example if you’re embedding a URL in an <a href="..."> then you’re writing an HTML attribute - therefore use HttpUtilty.HtmlAttributeEncode. If you’re injecting a value into a querystring, use UrlEncode, if it’s going to the text of a page use HtmlEncode, etc…

Left by Duncan at 1/7/2007 8:05 PM

# re: Making URIs safe

Ironically, on the page http://idunno.org/archive/2007/01/07/Making-URIs-safe.aspx the "previous page" link at the top reads "nxtgenug Oxford January Meeting : "Object Orientation"", double encoding which may be indicative of encoding too early, and of encoding again later on. I think that you shouldn't try to make user data "safe" too early on, and instead do encoding as late as possible.

Left by Duncan at 1/7/2007 8:13 PM

# re: Making URIs safe

Ah well the URL of the previous page is down to a little subroutine that attempts to make more search engine friendly URLs :) It's actually done after the creation of a blog post, which is as late as it can do it during that process.

Left by BarryD at 1/7/2007 8:26 PM

# re: Making URIs safe

I actually just checked in a fix for the problem that Duncan pointed out in his second comment.

The problem here was the Title string of the Post object was already encoded when it was added to the database. However Subtext can't assume that all Titles are encoded when they are inserted in the db.

So, the fix was to call Decode when pulling the string out off of the Post and then Encoding it just before we push the string out to the page.

Does that make sense?

Left by Steve Harman at 1/7/2007 9:57 PM

# re: Making URIs safe

Always store input exactly as provided by the user.

Encode for safety only on display.

Left by Spanky at 10/22/2007 9:18 PM

# re: Making URIs safe

Unfortunately neither the ASP.NET Server.UrlEncode nor the System.Uri.EscapeUriString methods seem to encode the single quote for safe use in URL query strings and JavaScript.

Server.UrlEncode("Women's Rights") =
"Women's+Rights"
and
System.Uri.EscapeUriString("woman's rights") =
"woman's%20rights"

So we're not totally safe with these encoding utilities yet. Guess we have to write our own additional encoding methods to cover the rest.
(good list at http://www.w3schools.com/tags/ref_urlencode.asp)

Does anyone know a full list of the characters that are ignored by these encoding methods?

Thanks everyone for the tips!

Left by John at 2/12/2008 3:50 PM

# re: Making URIs safe

Both Uri.EscapeUriString() and HttpUtility.UrlEncode() can do part of the job.

For example: Uri.EscapeUriString() escapes double equotation, HttpUtility.UrlEncode() encode &. None of both can do it all.

Left by Steve at 12/9/2010 2:58 AM

Your Reply.

Comment Form.

Fields denoted with a "*" are required.

*Your name:

*Subject:

You may also like to leave your email or website.

*Your message:

Please add 3 and 3 and type the answer here:

Enter the code shown above:

Preview Your Comment.

# re: Making URIs safe
"..ToString() representation of a URI can still be unsafe.."

In fact you should rarely use Uri.ToString() because it gives you a "more human-readable" but essentially incorrect representation of the URI. You should use the Uri instance's AbsoluteUri property e.g.:

string q = "X&Y";
Uri googleUrl = new Uri("http://google.co.uk/search?q=" + HttpUtility.UrlEncode(q));

Console.WriteLine(googleUrl.ToString());
// displays "q=X&Y" - WRONG

Console.WriteLine(googleUrl.AbsoluteUri);
// displays "q=X%26Y" - Correct

So even if you've gone to the trouble of encoding your querystring params, if you ever load your URL in a Uri and ToString it they get unescaped!

Also I think you should use the most appropriate encoding/escaping method depending on the context – rather than rely on some overall "make it safe" routine. For example if you’re embedding a URL in an <a href="..."> then you’re writing an HTML attribute - therefore use HttpUtilty.HtmlAttributeEncode. If you’re injecting a value into a querystring, use UrlEncode, if it’s going to the text of a page use HtmlEncode, etc…

Left by Duncan at 1/7/2007 8:05 PM
# re: Making URIs safe
Ironically, on the page http://idunno.org/archive/2007/01/07/Making-URIs-safe.aspx the "previous page" link at the top reads "nxtgenug Oxford January Meeting : "Object Orientation"", double encoding which may be indicative of encoding too early, and of encoding again later on. I think that you shouldn't try to make user data "safe" too early on, and instead do encoding as late as possible.

Left by Duncan at 1/7/2007 8:13 PM
# re: Making URIs safe
Ah well the URL of the previous page is down to a little subroutine that attempts to make more search engine friendly URLs :) It's actually done after the creation of a blog post, which is as late as it can do it during that process.

Left by BarryD at 1/7/2007 8:26 PM
# re: Making URIs safe
I actually just checked in a fix for the problem that Duncan pointed out in his second comment.

The problem here was the Title string of the Post object was already encoded when it was added to the database. However Subtext can't assume that all Titles are encoded when they are inserted in the db.

So, the fix was to call Decode when pulling the string out off of the Post and then Encoding it just before we push the string out to the page.

Does that make sense?

Left by Steve Harman at 1/7/2007 9:57 PM
# re: Making URIs safe
Always store input exactly as provided by the user.

Encode for safety only on display.

Left by Spanky at 10/22/2007 9:18 PM
# re: Making URIs safe
Unfortunately neither the ASP.NET Server.UrlEncode nor the System.Uri.EscapeUriString methods seem to encode the single quote for safe use in URL query strings and JavaScript.

Server.UrlEncode("Women's Rights") =
"Women's+Rights"
and
System.Uri.EscapeUriString("woman's rights") =
"woman's%20rights"

So we're not totally safe with these encoding utilities yet. Guess we have to write our own additional encoding methods to cover the rest.
(good list at http://www.w3schools.com/tags/ref_urlencode.asp)

Does anyone know a full list of the characters that are ignored by these encoding methods?

Thanks everyone for the tips!

Left by John at 2/12/2008 3:50 PM
# re: Making URIs safe
Both Uri.EscapeUriString() and HttpUtility.UrlEncode() can do part of the job.

For example: Uri.EscapeUriString() escapes double equotation, HttpUtility.UrlEncode() encode &. None of both can do it all.

Left by Steve at 12/9/2010 2:58 AM

Jump Menu

So what are all these buttons for?

Personalize

Text Size

Layout

Navigation Position

idunno.org

now with extra subtext goodness

Making URIs safe

Your Comments.

Your Reply.

Preview Your Comment.

Buy My Book

News

Tag Cloud

Article Categories

Archives

Post Categories

Syndicate

Shiny Badge Things