Web Application Functional Regression Testing Using Selenium

Add to FacebookAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to TwitterAdd to TechnoratiAdd to Yahoo BuzzAdd to Newsvine

At Foliotek, we use a rapid development methodology.  Typically, a new item will go from definition through coding to release in a month’s time (bucketed along with other new items for the month).  A bugfix will nearly always be released within a week of the time it was reported.  In fact, we are currently experimenting with a methodology that will allow us to test and deploy new items individually as well – which means that a new (small) item can go from definition to release in as little as a week, too.

Overall, this kind of workflow is great for us, and great for our customers.  We don’t need to wait a year to change something to make our product more compelling, and customers don’t have to wait a year to get something they want implemented.  We also avoid the shock of suddenly introducing a year’s worth of development to all our customers all at once – a handful of minor changes every month (or week) is much easier to cope with.

However, it also means that Foliotek is never exactly the same as it was the week before.  Every time something changes, there is some risk that something breaks.   We handle this risk in two ways:

  1. We test extremely thoroughly
  2. We fix any problems that arise within about a week (severe problems usually the same day)

At first, we did all testing manually.  This is the best way to test, assuming you have enough good testers with enough time to do it well.  Good testers can’t be just anyone – they have to have a thorough knowledge of how the system should work, they have to care that it does work perfectly, and they have to have a feel for how they might try to break things.  Having enough people like this with enough time to do testing is expensive.

Over time two related things happened.  One was that we added more developers to the project, and started building more faster.  Two was that the system was growing bigger and more complex.

As more people developed on it and the system grew more complex, our testing needs grew exponentially.  The rise in complexity and people developing led to much, much more potential for side-effects – problems where one change affects a different (but subtly related) subsystem.  Side-effects by their nature are impossible to predict.  The only way to catch them was to test EVERYTHING any time ANYTHING changed.

We didn’t have enough experienced testers to do that every month (new development release) let alone every week (bugfix release).

To deal with that, we started by writing a manual regression test script to run through each week.  While this didn’t free up any time overall – it did mean that once the test was written well, anyone could execute it.  This was doable, because we had interns who had to be around to help handle support calls anyways – and they were only intermittently busy.  In their free time they could execute the tests.

Another route we could have gone would have been to write automated unit tests (http://en.wikipedia.org/wiki/Unit_testing).  Basically, these are tiny contracts the developers would write that say something like “calling the Add function on the User class with name Luke will result in the User database table having a new row with name Luke”.  Each time the project is built, the contracts are verified.  This is great for projects like code libraries and APIs where the product of the project IS the result of each function.  For a web application, though, the product is the complex interaction of functions and how they produce an on screen behavior.  There are lots of ways that the individual functions could all be correct and the behavior still fails.  It is also very difficult to impossible to test client-side parts of a web application – javascript, AJAX, CSS, etc.  Unit testing would cost a non trivial amount (building and maintaining the tests) for a trivial gain.

Eventually, we discovered the Selenium project (http://seleniumhq.org/download/).  The idea of Selenium is basically to take our manual regression test scripts, and create them such that a computer can automatically run the tests in a browser (pretty much) just like a human tester would.  This allows us to greatly expand our regression test coverage, and run it for every single change we make and release.

Here are the Selenium tools we use and what we use them for:

  • Selenium IDE (http://release.seleniumhq.org/selenium-ide/) : A Firefox plugin that lets you quickly create tests using a ‘record’ function that builds it out of your clicks, lets you manually edit to make your tests more complex, and runs them in Firefox.
  • Selenium RC (http://selenium.googlecode.com/files/selenium-remote-control-1.0.3.zip):  A java application that will take the tests you create with Selenium IDE, and run them in multiple browsers (firefox, ie, chrome, etc).  It runs from the command line, so its fairly easy to automate test runs into build actions/etc as well.
  • Sauce RC (http://saucelabs.com/downloads): A fork of RC that adds a web ui on top of the command line interface.  It’s useful for quickly debugging tests that don’t execute properly in non-firefox browsers.  It also integrates with SauceLabs – a service that lets you run your tests in the cloud on multiple operating systems and browsers (for a fee).
  • BrowserMob (http://browsermob.com/performance-testing): An online service that will take your selenium scripts and use them to generate real user traffic on your site.  Essentially, it spawns off as many real machines and instances of FireFox at once to run your test – each just as you would do locally – for a fee.  It costs less than $10 to test up to 25 “real browser users” – which actually can map to many more users than that since the automated test doesn’t have to think between clicks.  It gets expensive quickly to test more users than that.

Selenium is a huge boon for us.  We took the manual tests that would occupy a tester for as much as a day, and made it possible to run those same tests with minimal interaction in a half hour or less.  We’ll be able to cover more test cases, and run it more – even running them as development occurs to catch issues earlier.

In my next post, I’ll talk about the details of how you build tests, run them, maintain them, etc. with the tools mentioned above.

Accessible Custom AJAX and .NET

Add to FacebookAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to TwitterAdd to TechnoratiAdd to Yahoo BuzzAdd to Newsvine

One general rule in making an accessible web application is that you shouldn’t change content of the page with javascript. This is because screen readers have a tough time monitoring the page and notifying the user of dynamic changes. Usually, the reason you would use Ajax is exactly that – do something on the server, and then update some small part of the page without causing the whole page to refresh. That means if you have an “ajaxy” page and you care about accessibility, you have to provide a non-ajax interface as a fallback for screen readers.

Ajax.NET, Microsoft’s library for Ajax, makes this fallback easy to implement. Microsoft has you define your AJAX properties in an abstraction layer – the page XAML – which means that the framework can decide to not render the AJAX code to certain browsers (and screen readers) that will not support it, and instead use the standard postback method.

The problem with Ajax.NET is that the communication can be bloated (mostly because of the abstraction layer, it sends more post values than you might need – like the encrypted viewstate value), which negates many of the benefits of an Ajax solution. I really wanted to roll my own ajax communications to make them as lightweight as possible.

My solution was to write the page in a standard .NET postback manner, and then use a user-defined property that would allow javascript to replace the postback in javascript/jQuery with an Ajax call.

Here’s my code:

$(function() {
            if (serverVars.uiVersion != "accessible") { // use js/ajax for checking/unchecking where possible
                var $todos = $("#chkToDo");
               $todos.removeAttr("onclick"); // remove postback
                    function() {
                        //some stuff altering the document, and an ajax call to report the change to the db

This works great, although you need to be careful about your server-side events. In my case, I had an OnCheckChanged event to handle the postback/accessible mode. Even though checking/unchecking the box no longer fired an autopostback – ASP.NET will still fire the checkchanged event if you postback later for some other reason (e.g. – a linkbutton elsewhere on the page) after the checked status had changed. So, if a user had changed the state of a checkbox, then clicked another link button on the page – instead of sending the user to the intended page,my app just refreshed the whole page(because my CheckChanged event redirected to reload the page – which caused it to skip the ‘click’ event of the linkbutton). Once I realized this was happening, it was easy enough to fix – I just needed to only run the event logic if the user was in accessibility mode. I spent a little time running in circles on that one though, at first I thought my client side document changes were causing a ViewState validation error on the server.

Some Good Uses for the ASP.NET .ASHX/”Generic Handler”

Add to FacebookAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to TwitterAdd to TechnoratiAdd to Yahoo BuzzAdd to Newsvine

I’ve been developing ASP.NET applications since 2002.  Up until recently, I’d overlooked a pretty useful part of the ASP.NET Framework : Generic Handlers.

Generic Handlers are basically HTTP Handlers that can process requests to exactly one url (Http Handlers are classes that need to be registered in the web.config for what urls they run on, and often manipulate requests/responses that go to/come from .aspx pages). They are “closer to the wire” than .ASPX WebForms or .ASMX WebServices.  They are built with only two requirements, implement:

public void ProcessRequest(HttpContext context);

(this contains the logic for when this page is requested) and, implement:

public bool IsReusable;

(this tells ASP.NET whether it can use one instance to serve multiple requests).

Generic Handlers are similar to web forms – they are reachable by a standard get/post to the url they are located in.  ASP.NET starts you off with context object (passed into the ProcessRequest() function you implement), which you then use to communicate with the request (form and query vals, etc) and the response object (to write info to the browser).  You can decide in what capacity you want that context to have access to the session:read/write, readonly, or none. Unlike web forms, generic handlers skip some of the niceties: the designer/xaml/servercontrol model, themes, viewstate, etc. Thus, generic handlers can be much more efficient for many tasks that make no use of those features.

Here’s a few common tasks where a generic handler would be more suitable than a web form:

  • Writing out the contents of files stored in the db/elsewhere for download
  • Writing out manipulated images, generated graphics, etc
  • Writing out generated PDFs/CSVs etc
  • Writing out data without the webservice overhead
    • xml for other systems or sites
    • html for simple AJAX get requests to replace page content
    • JSON for more advanced ajax data communication
  • Callback urls for data sent from other sites through a http post
  • Building a “REST API”
  • A url that outputs status info about the website, for use by a content switch or other monitoring system
  • Writing out dynamic JS/CSS based on server variables, browser capabilities, etc

Basically, anywhere I found myself before creating an ASPX page, but clearing everything on the .ASPX and using Response.Write()/Response.BinaryWrite() to talk to the browser or another system from the code behind, I should have been using a .ASHX handler instead and saving my webserver some work.

Here is a sample GenericHandler to get you started. It writes out a simple js file that gives you some useful globals you can use in other js files. Its meant to just be an example of the types of things you can do with generic handlers, and doesn’t necessarily produce what you might want in your own dynamic JS file.

    public class SiteJSGlobals : System.Web.IHttpHandler, System.Web.SessionState.IReadOnlySessionState
        public void ProcessRequest(HttpContext context)
            context.Response.ContentType = "text/javascript";

            context.Response.Write("var _g_username='" + context.User.Identity.Name + "';\n");
            context.Response.Write("var _g_usertype='" + context.Session["UserType"] + "';\n");
            context.Response.Write("var _g_browserheight=" + context.Request.Browser.ScreenPixelsHeight + ";\n");
            context.Response.Write("var _g_browserwidth=" + context.Request.Browser.ScreenPixelsWidth + ";\n");
            context.Response.Write("var _g_queryparam='" + context.Request["queryparam"] + "';\n");
            context.Response.Write("var _g_developmentmode=" + context.Request.Url.Host.ToLower().StartsWith("localhost") + ";\n");
            context.Response.Write("var _g_approoturl='" + context.Request.ApplicationPath + "';\n");
        public bool IsReusable
                return false;