For those that subscribe, please remember to update your bookmarks and feeds.

STANDARD DeViATiON’s new home is at http://www.stdeviation.com/blog

And you can find the new STANDARD DeViATiON RSS feed here http://www.stdeviation.com/rss

Stay tuned for some new content and enjoy the new look!

Documentation is lacking (apart from my blog posts) so I’m happy to provide assistance via email to anyone who has trouble configuring the beacon / viewer. Requirements wise, I’d recommend you run IIS 7, MSSQL 2008 and .Net 3.5. If you prefer to run IIS 6, you can still host the beacon in ASPX page mode (unlike IIS 7 where you can host it as a HTTP listener).

Finally, I’d like to acknowledge the support of my employer seek.com.au in the development and release of this project.
You can find the source code for the beacon and viewer components at my github repo here:

http://github.com/stoja

If you’re interested in following up on the previous posts in the series, take a look at:
Host Based Analytics Testing – Part 1
Host Based Analytics Testing – Part 2
Host Based Analytics Testing – Part 3

The final product: Our custom web analytics viewer - supported by web analytics data collected via our internal collection beacon.

One item that I’ve failed to discuss in any significant detail is the role that functional test automation plays in driving this approach. Host based analytics testing will function with or without automation – but it’s potential is greatly increased in environments where test automation already exists. By relying on test automation to drive the application under test, organisations can maintain a high frequency of test resulting in more current data and more relevant results. Without functional automation, testers are required to manually drive the application under test on a regular basis in order to maintain a current representation of the web application and its associated web analytics implementation. The ideal approach of course, is to use both – test automation in conjunction with manual testing. By capturing analytics traffic from all testers, regardless of what they’re testing (they don’t need to be testing web analytics specifically) and combining it with data captured during automated testing, you can ensure the ongoing currency of your dataset.
Finally, considering that I’ve specifically mentioned Selenium in the title of my posts, I should clarify that you don’t *need* to be using Selenium. You can use whichever functional automation tool you prefer, provided it has the ability to drive a web application via a web browser. I mentioned Selenium because I personally use it on a daily basis and I consider it one of the most capable tools on the market but ultimately, the choice is yours.

The baseline value edit dialog.

By implementing the techniques descibed in this series, organisations can reduce the typical costs associated with maintaining their web analytics implementations while also reducing their dependency on manual test resources. For the record, I do intend to release all the source code associated with this project eventually. I’ll post a few updates as our internal trial wraps up.

For those interested in the details, here’s a quick overview of our trip (courtesy of tripit.com).
Days 41
Distance 30,754 miles / 49,494 km
Cities 28
Countries 13

Places visited (in no particular order):
Singapore, Singapore; Athens, Greece; Barcelona, Spain; Brugge, Belgium; Cannes, France; Nice, France; Corfu, Greece; Dublin, Ireland; Dubrovnik, Croatia; Florence, Italy; Genova, Italy; Milan, Italy; Edinburgh, Scotland; Kusadasi, Turkey; Lisbon, Portugal; London, United Kingdom; Mykonos, Greece; New York, NY; Paris, France; Rhodes, Greece; Rome, Italy; Southampton, United Kingdom; Sydney, Australia; Santorini, Greece; Venice, Italy

http://stdev.wordpress.com/2010/01/15/selenium-automation-webanalytics/

Since establishing the beacon, it has collected over 50,000 unique variables across countless pages, multiple test environments and from a combination of automated and manual test clients. While performance was a consideration in the original solution design, exposing the beacon to multiple test clients and introducing the possibility of parallel result processing quickly revealed a scalability problem. Given this (and for maintainability reasons), I decided to port the beacon from PHP, MySQL and Apache to .Net, MSSQL and IIS6. If you’re interested in the details, let me know and I can email you some info.

Below is a summary of some of my findings from the project with some important considerations for anyone wishing to develop their own custom analytics beacon.

Be Selective

In the beginning, it’s likely that your beacon will be logging ALL analytics data being sent to it. This isn’t a problem, but it can turn into one if left too long. Logging duplicate or irrelevant data will unnecessarily increase the size of your database and potentially increase processing times.
After you’ve gathered a good amount of data, perform some analysis to determine whether there’s anything being logged that you don’t care about. For example, it’s very likely that you’re beacon is storing analytics variables which detail page access times. In terms of testing your analytics code, these dynamic variables probably don’t reveal much and can be ignored.
In addition, ensure that any url’s you’re using to identify pages are stripped of any variable query string data before logging. You’ll likely be using these urls to identify unique pages so this variable data can mislead your beacon into logging another instance of what is actually an existing page.

Manage Undefined Variables

Undefined variables might show up in your analytics query string data. You wont know where and you won’t know when so it’s important to handle this data appropriately within your beacon code. Before deciding whether or not to store undefined variables (in some fashion), think of the situations where they’ll be used. For example, if the undefined variable is a host, url or page name field, it’s very likely that these core attributes will be used to query the database at a later stage. Ultimately, analytics data doesn’t mean much if you don’t know where it came from or what page it relates too but use your own judgement as to whether to keep or discard these records.

Single Page, Multiple Instances

In the web analytics world, a single page can have several different instances based on the path taken through the site to reach it. In order to log data for each of these instances, ensure that you log based on unique url and referrer, otherwise your beacon will overwrite the same instance of the page with each set of new variables that it receives.

Database Indexing

Naturally, exposing your beacon to multiple test clients increases the amount of data being processed and written to the database. Combine this with an all-inclusive logging approach (at least in the early stages of data collection) and chances are, you’ll be storing thousands of records. To prove the importance of indexing, my performance tests revealed that once the database grew to 100,000+ records, beacon processing and response times jumped from < 1 second to approximately 10 seconds. In my specific example, this ultimately resulted on our test automation timing out waiting for the analytics request to complete. In this particular case, indexing was a simple solution.

For those considering automating their manual analytics testing effort, the above steps should help you develop an effective and efficient analytics logging solution. In the last post of this series, I’ll detail the final steps required to fully automate your analytics testing, including baselining your resultset and displaying your test results.

You can visit the official Melbourne Selenium Meetup group page by clicking on the link below.

Melbourne Selenium Meetup Group

In todays world, web analytics play an important role in developing & maintaining any successful web presence.
As more and more organisations begin to rely on their web analytics data to drive strategic decision making, the importance of verifying your website’s web analytics code becomes paramount.

From a testing perspective, web analytics testing is typically done manually. In fact, I’ve never heard of if being done any other way. The process involves scouring your website with a browser based analytics profiler like WASP and verifying that data being sent to your analytics provider is accurate. While this type of testing is relatively simple (and tedious), it’s repetitive nature makes it the perfect candidate for automation.

Fresh from integrating yslow performance analysis into our Selenium tests, the next step was to analyse what part Selenium could play in validating our onsite analytics code.

The Concept

The concept for automated web analytics testing is relatively simple. As Selenium drives GUI based tests through your web application, we intercept any outgoing analytics data, capture it and store it in a database for further processing. The process isn’t actually all that different to what’s already happening under the hood with the exception of the “intercept” component – typically, the data hits your analytics provider where its stored and processed in a similar manner. Once you’ve captured this data, it’s simply a matter of comparing it against a production baseline and identifying any differences every time you execute a test run.

Environment Setup

In order to implement the above, here’s a list of what I needed:

1) A website… with web analytics code embedded.
In my specific context, the target site includes a significant implementation of Omniture web analytics code. The concept will work with other analytics providers too so this solution isn’t confined to a single flavour of web analytics. Ultimately, the bigger your target site and the more analytics code it includes, the more value that you’ll derive from this exercise.

2) A machine that runs your Selenium based automation against your target site.
My guess is you already have something in place here so I wont go into any further detail.
For the sake of transparency, I run in a Selenium Grid environment where I’ve configured multiple clients with this setup.

3) A custom analytics host – essentially, a machine that will collect your analytics data.
This machine will impersonate your actual analytics provider. It needs to be running apache web server and must run a database (mysql or similar). PHP is recommended but optional (you’ll need it for (5) listed below but you can replace it with whatever technology you prefer).
NOTE: I’d strongly recommend implementing this on the same machine running your showslow server as the prerequisites are the same. In fact, if you’ve already got showslow setup, this should be a pretty simple extension for you.

4) A hosts file redirect which hijacks your analytics traffic and directs it to your custom analytics host in step 3. This needs to be completed on the machine performing your automated tests. Analysing your target site’s html should reveal your analytics host, for example analytics.company.com. Once you have this information, redirect all traffic accessing this host to your custom analytics host.

5) An analytics beacon to capture, parse and store your analytics data on your custom analytics host.
You need a little bit of development knowledge for this bit. Similar to how yslow / showslow work, your analytics data is usually sent to your analytics provider in a query string containing a series of name / value pairs. These pairs hold the analytics variable name and associated value. In order to strip these values, you’ll need to setup a page that you can direct traffic to (this page will be hosted on your custom analytics host) which strips parameters from the query string and logs them to the database. I’d strongly recommend having a look at how showslow’s yslow beacon has been setup as the concept is practically the same. Also, I have some code in this area that I could possibly share (after I clean it up a little) so feel free to get in touch if you need some assistance. Once you’ve setup the beacon, ensure you configure apache to load the beacon page automatically when you hit the location of the page in your browser.

Finally, you need to actually verify the format being used to send analytics data to your analytics host. If the URL actually includes subdirectories for example, “http://analytics.company.com/site/data?test=test&test1=test1&”, you’ll need to set a rule in apaches httpd.conf alias_module so it can redirect any traffic including the “/site/data” component of the URL. This will ensure your analytics data is directed to the right place.

Once you’ve configured the above, you should have most of the pieces in place to capture and log your analytics traffic. Part 2 of this post will include details on the pro’s and con’s of this implementation as well as some important factors which need to be considered before entirely eliminating your manual web analytics test effort.

The Selenium / YSlow integration concept itself isn’t new. There’s a great post on Adam Goucher’s blog discussing the idea and while I wasn’t keen on tying it to our CI system, the idea of getting page performance data at the completion of our automated tests sounded great.
In terms of setup, its relatively painless. The solution is based around the free Yslow firefox addon and the (opensource) ShowSlow reporting server. Given that there isn’t a lot of information available on the setup process, here’s a brief overview:

ShowSlow Server Setup

1. Download and install Apache, PHP and MySQL (essentials will do).
2. Once all of the above are setup, checkout the ShowSlow project source from here and host it via Apache.
3. You’ll also need to create a new database in MySQL and run in tables.sql (from the ShowSlow directory) to create all the relevant tables for ShowSlow.
4. Edit config.php and supply your database credentials.
5. Load the ShowSlow site in your browser on your webserver and click on the “configuring your yslow” link – you’ll need the values from this page later.

 YSlow Client Setup

1. Create a custom Firefox profile on your Selenium RC machines (using firefox.exe -ProfileManager). The profile needs to be setup with the following attributes (at minimum):

• YSlow add-on installed and setup to run perf analysis on page load (set in Yslow options).
• In about:config, set “extensions.yslow.optinBeacon” to true and “extensions.yslow.beaconUrl” to the value from your “configuring your yslow” page.

2. On each Selenium RC box (the machine you’ll use to run automated tests), you’ll need to setup Selenium to use your new custom profile (using DseleniumArgs=”-firefoxProfileTemplate ” argument). In order to achieve this, modify the Selenium RC launch command to include the path to the new Firefox profile you just created. For example:

java -jar selenium-server.jar -firefoxProfileTemplate “C:/SELENI~2/SELENI~1″

NOTE: You need to provide the profile path using the DOS short name format. For more info on finding your firefox profile directory, see: http://support.mozilla.com/en-US/kb/profiles

Once you’ve completed the setup, running your automated tests should result in some YSlow performance statistics being sent to your ShowSlow server.

It’s worth noting that there are some downsides to this setup. Firstly, its Firefox only (currently) so any tests run against other browsers wont record any results with ShowSlow. And finally, it records page performance data for every page that your browser loads. This means that you’ll get performance data for URL’s that you don’t necessarily care about like those used by Selenium and those with variable query strings. While you can’t explicitly tell YSlow to not record performance data for these URL’s, you can modify the code within ShowSlow to prevent these URL’s from contaminating your results (eg./ I’m currently stripping some variable data out of certain URL’s as well as filtering the ShowSlow ouput).

If you’re already using Selenium as the basis for your automated testing, I’d definitely recommend tying it in with YSlow. With a little extra effort up front, it’ll add ongoing value to your automated tests.

UPDATE: It seem that 2.0b6 is the last that works correctly with the current version of ShowSlow. Version 2.0.1 of Yslow seems to rely on POST’s to send the full set of performance data to the beacon server. Version 2.0.1 will still work, you’ll just notice that the details page has a “0″ value for all of the Yslow Breakdown results.

Big vendors like HP & IBM have capitalised on a long period where test automation as a profession was immature. At the time, test automation was a niche dominated by the big players. Choice was limited and if you managed to settle on a tool, it’s very likely that it was a costly decision. In today’s world, test automation as a profession is no longer immature… and its no longer a niche market.

Open source tools like Selenium are now competing head to head with the major players in the test automation space. Granted, most open source tools don’t come shrink wrapped and they don’t come with a bundle of brochures but does it really matter?
Functionally, open source tools allow unparalleled flexibility. Don’t like the reporting component that comes with your tool? Then just switch it to something more suitable… or write your own. Also, you can forget the support contracts and maintenance agreements. Upgrades are free and support is an email / forum post / tweet away.

The test automation game is changing… in fact, its been changing for a while.
Open source test automation tools make automation available to the masses. Not only that, but in most cases, they do it better.
So, if you’re in a position to be considering the software that will form the core of your test automation framework, think twice before reaching for your wallet (or more appropriately, asking your boss to reach for theirs). Times have changed and you have a choice. Personally, I see no compelling reason to invest thousands of dollars into something that I can get for free.

Firstly… the format of the conference.
The “open spaces” format is quite effective. It seemed to inspire a lot more discussion than a more typical presentation format.
Given that topics are defined based on input from the attendees, I found that there was rarely a case where we weren’t discussing something that was of interest or value to me. In contrast, there were a few occasions where discussions diverted from the “agreed” topic to other matters. The general rule in these cases is that everyone has the right to walk out and sit in on another session… and I did on a couple of occasions.

In terms of content, we discussed everything from BDD to cloud based testing solutions. Believe me, even if you’re fairly confident about your position in the testing world, you’ll learn something new at CITCON. It’s worth mentioning that the conference is opensource-centric, so don’t expect to hear much about .Net. That’s not to say you cant talk about it, just that its probably not at the top of everyone’s list.
Regardless, most of the examples shown at CITCON aren’t technology specific and the same patterns can be applied in your organisation’ technology of choice.

Overall, CITCON was well worthwhile. The opportunity to share experiences with other testers was invaluable. CITCON is the perfect forum to raise and discuss the challenges we regularly encounter in our roles. Even if you don’t think you have much to contribute, you’ll be surprised at how much you actually have in common with other testers.

In closing, I should mention that I was really impressed by the number of organisations pushing the envelope in terms of test automation. The fact that most of these organisations were relatively small was inspirational… proof that you don’t need a big budget to implement a successful test automation strategy (I’ll talk more about this later).

If you have the opportunity, CITCON is well worth your time.
For info on the 2010 event in New Zealand, go to: http://citconf.com