“What gets measured, gets done.”
I remember this mantra from my first job in tech where I worked on construction project management software. On the surface it made sense. Someone older at the table quipped “Do you measure the measuring?” The client joked back “Oh we don’t talk about that.” There was a dance between taking this mantra seriously and not-seriously that my younger self didn’t quite get at the time.
Over the years the growing complexity of software has always included measurements, usually called metrics or analytics. Today, nearly every website and app generates extensive logs about usage and metadata scraps of context about the people behind the clicks.
Apps and websites have an inherent remoteness between their makers and the people using them. It’s that void that frustrates our desire to iterate and improve, and it’s the origin of what, to us, becomes an over-reliance on just the numbers.
That over-reliance is where mis-use creeps in, and the two ways we see metrics mislead are through universal value assessments of changes in data, and making simple numbers the arbiter of design decisions.
Up is Good?
When it comes to web analytics, there are two numbers that can be mislead: Time on Site and Pages per Visit.
Sometimes after a website replacement we’ll get questions about drops in these two numbers. Two quick case studies:
In one example, before the redesign we had noticed that a lot of people came to the site, navigated to the contact page, then to a third page with the organization address and a map, then left the site. We inferred that people were after that address and incorporated it into the footer. Visits to the contact page dropped by quite a lot, and the Pages per Visit took a hit.
In another project we made a bigger change by moving a central piece – the shows program – to the homepage and surfaced short descriptions, performance dates, and direct links to booking that bypassed the full show page. This time, total time on site and pages per visit took a hit, but more shows were viewed at the summary level than in the previous design, where checking out a show required visiting a whole other page.
These changes can look bad if we only look at the numbers and assume that up is better. But if we think about what the website is for, what its visitors are looking to do, the picture is different. It’s now evidence that people are getting what they need faster, and moving on with their day. It’s the kind of thing that we actually want especially when navigation has undergone major changes, but for website managers not ready for it the drop in these numbers can be a shock.
The only kind of website that should fret about time on site and pages per visit is one primarily monetized through display advertising. In almost all other cases, we should be striving to let people do what they want and move along.
A/B Testing
If you’ve ever been in a design discussion where differing options reach an impasse, you’ve likely heard someone chime in with “Let’s A/B test it,” assuming that the data will show the way to the correct design.
A/B testing is informative and it can feel like a fair way to settle an argument, but it comes with a hidden cost of understanding why. In other words the test tells you which choice people interacted with more, but it will never tell you why, and without the why there’s no chance at building a coherent system of interaction through a website or app.
The Map is not the Territory
This is another mantra that we’ve picked up over the years, and it’s something we keep in mind when spending time with analytics.
The numbers are clues, they’re indicators of something going on, but to illuminate any kind of truth they need to be connected with purpose, context, and some kind of explanatory framework.
Otherwise they’re mirages that leave us reactive and adrift, waiting for the next report to tell us what’s happening without a hint of how we can make sense of it all.