STANDARD DeViATiON

It’s been well over six months since we started developing the concepts behind automated web analytics testing. It’s fair to say that much has changed in the six months since we began this journey. Our test dataset has grown tremendously and through detailed analysis, we’ve gained a thorough understanding of the data which supports our automated analysis system. Through increasing our understanding, it’s also enabled us to refine the very rules that govern what data we collect and which data we ignore, further improving the relevance and accuracy of our dataset. We’ve discovered and treated several scalability and performance issues, and today have a much more robust and scalable solution than when we first started. And finally, we’ve moved from an new and unproven concept to a practical, functional implementation which were currently trialling internally -  and as a consequence, we’ve formally named the approach “Host Based Analytics Testing”.

The final product: Our custom web analytics viewer - supported by web analytics data collected via our internal collection beacon.

One item that I’ve failed to discuss in any significant detail is the role that functional test automation plays in driving this approach. Host based analytics testing will function with or without automation – but it’s potential is greatly increased in environments where test automation already exists. By relying on test automation to drive the application under test, organisations can maintain a high frequency of test resulting in more current data and more relevant results. Without functional automation, testers are required to manually drive the application under test on a regular basis in order to maintain a current representation of the web application and its associated web analytics implementation. The ideal approach of course, is to use both – test automation in conjunction with manual testing. By capturing analytics traffic from all testers, regardless of what they’re testing (they don’t need to be testing web analytics specifically) and combining it with data captured during automated testing, you can ensure the ongoing currency of your dataset.
Finally, considering that I’ve specifically mentioned Selenium in the title of my posts, I should clarify that you don’t *need* to be using Selenium. You can use whichever functional automation tool you prefer, provided it has the ability to drive a web application via a web browser. I mentioned Selenium because I personally use it on a daily basis and I consider it one of the most capable tools on the market but ultimately, the choice is yours.

The baseline value edit dialog.

By implementing the techniques descibed in this series, organisations can reduce the typical costs associated with maintaining their web analytics implementations while also reducing their dependency on manual test resources. For the record, I do intend to release all the source code associated with this project eventually. I’ll post a few updates as our internal trial wraps up.

Tags: automation, google analytics, host based analytics, omniture, selenium, web analytics