Wednesday, June 23, 2010
Sunday, May 16, 2010
A/B Testing
A/B testing, consists in the development of two versions of a page, and show different versions to users randomly.
These users will be identified in either the A or B track group, and the analytics evaluate how do they perform to go with the winning page version.
This is useful to explain why the conversion rate of visitors into customers of a website is not as good as expected and helps us to take the necessary actions to improve it just by applying some small changes like for example:
- Removing form fields.
- Adding relevant form fields.
- Marketing landing pages.
- Different explanations.
- Having interstitial pages.
- Email contents / subject.
This kind of testing doesn't substitute: talking to users, usability tests, acceptance tests, unit tests and of course... thinking.
Behind the scenes
The first time a visitor lands to the web site a randon version of the page is created, saved and displayed on the browser. The following times that the visitor comes to the site, the previous version is found and returned (usually by a cookie value stored on the client device) so the random pick is permanent.
This behavior leads to some development considerations. The most important of them is that the tests are temporary and when one A/B test finishes, one version of the page wins and the rest of them are deleted. As developers normally want maintainable code and they will need to write a lot of A/B tests the process should be as easy as possible and eliminate annoying steps by building a framework for this particular purpose.
Conversion funnels improved
To create useful A/B tests we should know the particular steps in our site to convert visitors into customers, that is, the conversion funnels.
Most A/B tests concetrate on only one step in the funel and each step can be tracked with some metric like for example:
- Sessions, sessions with registration
- People who searched, who viewed detail page, contacted, leased
- People who saved favorites, started a cart, completed purchase
- People who saw at least 3 pages, clicked on an ad
When we are analyzing the results of an A/B test we should take into account to compare apples to apples. That means that for example traffic behaves differently at different times and we might not compare visits received on a friday night with the ones from a monday morning.
In case of our portal interface is changed and tested using A/B we should only include in our analysis the new users of each page version, as we know that people don't like UI changes. A priori, it can cause the new version to lose on the A/B test because our old visitors are used to the older interface.
Running multiple tests
A/B tests can take a while to finish and we have many A/B tests that we'd like to run, so it would be nice not having to wait. It can be achieved by running multiple tests at once. To do this we need to assign people to tests randomly, so different tests will be run at the same time.
Google's Website Optimizer
If you want to get going on A/B testing and don't want to worry about doing any of the reporting then use it.
However be aware that it has a number of major limitations.
- It works by using JavaScript to inject static content into your page which can be a problem when doing dynamic A/B testing.
- It doesn't support multiple metrics.
- It can't be used to A/B test email content.
These users will be identified in either the A or B track group, and the analytics evaluate how do they perform to go with the winning page version.
This is useful to explain why the conversion rate of visitors into customers of a website is not as good as expected and helps us to take the necessary actions to improve it just by applying some small changes like for example:
- Removing form fields.
- Adding relevant form fields.
- Marketing landing pages.
- Different explanations.
- Having interstitial pages.
- Email contents / subject.
This kind of testing doesn't substitute: talking to users, usability tests, acceptance tests, unit tests and of course... thinking.
Behind the scenes
The first time a visitor lands to the web site a randon version of the page is created, saved and displayed on the browser. The following times that the visitor comes to the site, the previous version is found and returned (usually by a cookie value stored on the client device) so the random pick is permanent.
This behavior leads to some development considerations. The most important of them is that the tests are temporary and when one A/B test finishes, one version of the page wins and the rest of them are deleted. As developers normally want maintainable code and they will need to write a lot of A/B tests the process should be as easy as possible and eliminate annoying steps by building a framework for this particular purpose.
Conversion funnels improved
To create useful A/B tests we should know the particular steps in our site to convert visitors into customers, that is, the conversion funnels.
Most A/B tests concetrate on only one step in the funel and each step can be tracked with some metric like for example:
- Sessions, sessions with registration
- People who searched, who viewed detail page, contacted, leased
- People who saved favorites, started a cart, completed purchase
- People who saw at least 3 pages, clicked on an ad
When we are analyzing the results of an A/B test we should take into account to compare apples to apples. That means that for example traffic behaves differently at different times and we might not compare visits received on a friday night with the ones from a monday morning.
In case of our portal interface is changed and tested using A/B we should only include in our analysis the new users of each page version, as we know that people don't like UI changes. A priori, it can cause the new version to lose on the A/B test because our old visitors are used to the older interface.
Running multiple tests
A/B tests can take a while to finish and we have many A/B tests that we'd like to run, so it would be nice not having to wait. It can be achieved by running multiple tests at once. To do this we need to assign people to tests randomly, so different tests will be run at the same time.
Google's Website Optimizer
If you want to get going on A/B testing and don't want to worry about doing any of the reporting then use it.
However be aware that it has a number of major limitations.
- It works by using JavaScript to inject static content into your page which can be a problem when doing dynamic A/B testing.
- It doesn't support multiple metrics.
- It can't be used to A/B test email content.
Sunday, April 4, 2010
Who will give you the best return on your investment?
Market segmentation or dividing the market into groups with similar wants or needs under segmentation bases that include:
- Geographic factors (location, climate)
- Demographic variables (gender, age, household income, occupation, education, religion, nationality, family size, marital status, presence of children)
- Psychographic variables (lifestyle, need for status, role of money, ethics and morals, risk taker vs conservative, spendthrift vs thrifty, AIO’s – attitudes, interests, opinions)
- Behaviouristic or user status (nonuser, ex-user, potential user, first-time user, regular user)
Take a look at the Nielsen Claritas PRIZM customer segmentation system and try to know in which cluster will you be clasified from Low-Rise Living to Upper Crust.
Market segmentations will help us to discover the appropriate target market(s) for our product or service. Targeting consists of knowing the target audience (audience targeting)
By understanding the audience: What are they reading? What sites and pages are they looking at? What are they shopping for? Which search terms are they using? What behaviors are they exhibiting that represent intent? we can position how our product or company is perceived in the minds of consumers.
Predictive analytics encompasses a variety of techniques from statistics, data mining and game theory that analyze current and historical facts to make predictions about future events, identify risks and opportunities.
It uses the theory that “birds of a feather flock together”, the approach is based on the assumption that active customers will have similar retention outcomes as those of their comparable predecessors.
It can be used to make predictions about active customers regarding:
* Whether they are at high risk of canceling their service
* Whether they are profitable to retain
* What retention tactics are likely to be most effective
One of the better known applications of predictive analysis is credit scoring, used in financial services, in order to rank-order individuals by their likelihood of making future credit payments on time.
- Geographic factors (location, climate)
- Demographic variables (gender, age, household income, occupation, education, religion, nationality, family size, marital status, presence of children)
- Psychographic variables (lifestyle, need for status, role of money, ethics and morals, risk taker vs conservative, spendthrift vs thrifty, AIO’s – attitudes, interests, opinions)
- Behaviouristic or user status (nonuser, ex-user, potential user, first-time user, regular user)
Take a look at the Nielsen Claritas PRIZM customer segmentation system and try to know in which cluster will you be clasified from Low-Rise Living to Upper Crust.
Market segmentations will help us to discover the appropriate target market(s) for our product or service. Targeting consists of knowing the target audience (audience targeting)
By understanding the audience: What are they reading? What sites and pages are they looking at? What are they shopping for? Which search terms are they using? What behaviors are they exhibiting that represent intent? we can position how our product or company is perceived in the minds of consumers.
Predictive analytics encompasses a variety of techniques from statistics, data mining and game theory that analyze current and historical facts to make predictions about future events, identify risks and opportunities.
It uses the theory that “birds of a feather flock together”, the approach is based on the assumption that active customers will have similar retention outcomes as those of their comparable predecessors.
It can be used to make predictions about active customers regarding:
* Whether they are at high risk of canceling their service
* Whether they are profitable to retain
* What retention tactics are likely to be most effective
One of the better known applications of predictive analysis is credit scoring, used in financial services, in order to rank-order individuals by their likelihood of making future credit payments on time.
Wednesday, March 10, 2010
Raw data
Nowadays, in 2010, we can use different ways to get the raw data needed to feed a web analytics solution such as: server log files, page tagging, packet sniffing or integrating web analytics into the web server software itself.
The massive use of Google Analytics has made very popular the page tagging, but the other methods are still valid and it is worthy to know the pros and the cons of each of them. This way we might find an hybrid solution that adapts better to our needs.
Let's talk in detail about the methods to get this information.
SERVER LOG FILES
The web server reliably records every transaction it makes.
If a person revisits a page, the second request will often be retrieved from the browser's or proxy cache, and so no request will be received by the web server. This means that the person's path through the site is lost. Caching can be defeated by configuring the web server, but this can result in degraded performance for the visitor to the website.
The data is on the company's own servers, and is in a standard, rather than a proprietary, format. This makes it easy for a company to switch programs later and analyze historical data with a new program.
Logfiles contain information on visits from search engine spiders. Although these should not be reported as part of the human activity, it is useful information for search engine optimization.
Logfiles require no additional DNS Lookups. Thus there are no external server calls which can slow page load speeds, or result in uncounted page views.
PAGE TAG
Involves including a small invisible image and, by using a component ("tag"), to pass along with the image request certain information about the page and the visitor. It is usually written in JavaScript, though Java can be used, and increasingly Flash is used.This information can be processed remotely by a web analytics solution.
With the increasing popularity of Ajax-based solutions, an alternative to the use of an invisible image, is to implement a call back to the server from the rendered page. In this case, when the page is rendered on the web browser, a piece of Ajax code would call back to the server and pass information about the client that can then be aggregated by a web analytics company.
It requires changes to the web site to be analyzed which can be difficult to maintain, but many applications blogs, CRM tools, E-commerce solutions, site builders already provide complete tagging solutions.
The tag is specific for the web analytics vendor, so it won't be easy to change it.
Page tagging may not be able to record all transactions:
- Page tagging relies on the visitors' browsers co-operating, which a certain proportion may not do (for example, if JavaScript is disabled like in most mobile devices, or a hosts file prohibits requests to certain servers).
- Tags may be omitted from pages either by oversight or between bouts of additional page tagging.
- It may not be possible to include tags in all pages. Examples include static content such as PDFs or application-generated dynamic pages where re-engineering the application to include tags is not an option.
It is easier to add additional information to the tag, like the visitors' screen resolution, or the price of the goods they purchased.
Page tagging can report on events which do not involve a request to the web server, such as interactions within Flash movies, partial form completion, mouse events such as onClick, onMouseOver, onFocus, onBlur etc.
Page tagging is available to companies who do not have access to their own web servers.
Tagging will slightly slow down your pages.
PACKET SNIFFING
Packet sniffing collects data by sniffing the network traffic passing between the web server and the outside world. Once the packet sniffer has recreated the HTTP and HTTPS traffic it can then create a log file, similar to one created by a web server. From this you can use your favourite web analytics solution to process the log files.
Packet sniffing involves no changes to the web pages or web servers.
It can be implemented:
a) Installing a separate server at your data center which you connect either to a SPAN port on your switch (a software change) or to a network tap (a physical device that basically duplicates all the packets that cross your cables. Unlike proxy servers or server plugins (or even page tags), both of these methods are completely passive and are physically incapable of impacting your traffic in anyway.
b) Installing the software on your web servers. This approach is still much less intrusive than server plugins, etc. but it does add additional CPU and memory usage to your web servers. Unless your servers are idle (and you have only one), this isn't recommended. But it is an easy way to get up and running quickly.
There are certain things you cannot capture using packet sniffing alone. In particular, interactions that occur client-side (within a visitor's browser only) and do not generate any corresponding traffic to the server. One example is if you download a movie and pause/rewind/fast-forward, and actually want to know that user performed those interactions.
Atomic Labs' Pion takes a completely approach by capturing all your web traffic passively using packet sniffing, and provides a visual, web-based interface that integrates with some major vendors like Omnitures Genesis, Google Analytics, WebTrends and Unica.
The massive use of Google Analytics has made very popular the page tagging, but the other methods are still valid and it is worthy to know the pros and the cons of each of them. This way we might find an hybrid solution that adapts better to our needs.
Let's talk in detail about the methods to get this information.
SERVER LOG FILES
The web server reliably records every transaction it makes.
If a person revisits a page, the second request will often be retrieved from the browser's or proxy cache, and so no request will be received by the web server. This means that the person's path through the site is lost. Caching can be defeated by configuring the web server, but this can result in degraded performance for the visitor to the website.
The data is on the company's own servers, and is in a standard, rather than a proprietary, format. This makes it easy for a company to switch programs later and analyze historical data with a new program.
Logfiles contain information on visits from search engine spiders. Although these should not be reported as part of the human activity, it is useful information for search engine optimization.
Logfiles require no additional DNS Lookups. Thus there are no external server calls which can slow page load speeds, or result in uncounted page views.
PAGE TAG
Involves including a small invisible image and, by using a component ("tag"), to pass along with the image request certain information about the page and the visitor. It is usually written in JavaScript, though Java can be used, and increasingly Flash is used.This information can be processed remotely by a web analytics solution.
With the increasing popularity of Ajax-based solutions, an alternative to the use of an invisible image, is to implement a call back to the server from the rendered page. In this case, when the page is rendered on the web browser, a piece of Ajax code would call back to the server and pass information about the client that can then be aggregated by a web analytics company.
It requires changes to the web site to be analyzed which can be difficult to maintain, but many applications blogs, CRM tools, E-commerce solutions, site builders already provide complete tagging solutions.
The tag is specific for the web analytics vendor, so it won't be easy to change it.
Page tagging may not be able to record all transactions:
- Page tagging relies on the visitors' browsers co-operating, which a certain proportion may not do (for example, if JavaScript is disabled like in most mobile devices, or a hosts file prohibits requests to certain servers).
- Tags may be omitted from pages either by oversight or between bouts of additional page tagging.
- It may not be possible to include tags in all pages. Examples include static content such as PDFs or application-generated dynamic pages where re-engineering the application to include tags is not an option.
It is easier to add additional information to the tag, like the visitors' screen resolution, or the price of the goods they purchased.
Page tagging can report on events which do not involve a request to the web server, such as interactions within Flash movies, partial form completion, mouse events such as onClick, onMouseOver, onFocus, onBlur etc.
Page tagging is available to companies who do not have access to their own web servers.
Tagging will slightly slow down your pages.
PACKET SNIFFING
Packet sniffing collects data by sniffing the network traffic passing between the web server and the outside world. Once the packet sniffer has recreated the HTTP and HTTPS traffic it can then create a log file, similar to one created by a web server. From this you can use your favourite web analytics solution to process the log files.
Packet sniffing involves no changes to the web pages or web servers.
It can be implemented:
a) Installing a separate server at your data center which you connect either to a SPAN port on your switch (a software change) or to a network tap (a physical device that basically duplicates all the packets that cross your cables. Unlike proxy servers or server plugins (or even page tags), both of these methods are completely passive and are physically incapable of impacting your traffic in anyway.
b) Installing the software on your web servers. This approach is still much less intrusive than server plugins, etc. but it does add additional CPU and memory usage to your web servers. Unless your servers are idle (and you have only one), this isn't recommended. But it is an easy way to get up and running quickly.
There are certain things you cannot capture using packet sniffing alone. In particular, interactions that occur client-side (within a visitor's browser only) and do not generate any corresponding traffic to the server. One example is if you download a movie and pause/rewind/fast-forward, and actually want to know that user performed those interactions.
Atomic Labs' Pion takes a completely approach by capturing all your web traffic passively using packet sniffing, and provides a visual, web-based interface that integrates with some major vendors like Omnitures Genesis, Google Analytics, WebTrends and Unica.
Tuesday, February 23, 2010
A perspective on web analytics solutions
There are multiple solutions when trying to analyze a web site and it is not always easy to find the one which will fit your needs perfectly. In order to help find the way through it all, I’ll try to describe the possible options the best I can.
Please let me know if I’m missing a solution or provider you think should be listed, so I can check it out and add it.
The VIPs
If performance, innovation and support are the main priorities when selecting our solution for web analytics we should refer to the Forrester Research report. The last update dates from the 3rd quarter of 2009 an it compares the most relevant vendors at the moment. The next graphic shows the final clasification where Omniture, Unica and Coremetrics achive the best ranking positions.
Omniture: an Adobe company, Site Catalyst stands out for its core data handling, reporting, and analysis capabilities. Although it is comparatively high pricing, it has a big market presence, with clients like eBay or Microsoft.
Coremetrics: Small in total client count, but it offers oversized capabilities like its ability to handle data efficiently, customize reporting and analysis, and integratation with multiple marketing functions (cm tags).
Unica: The NetInsight offering is one of the most flexible and customizable offerings that the Forrester report evaluated due to an open relational database.
Although the Forrester report is based on the version 7.6, the last one at the moment is 8.0 which has improved features at reporting and ad hoc analysis.
Webtrends: After restructuring its management team and refocused its efforts in web analytics the company scored high marks because of its ability to create unlimited custom and calculated metrics, as well as the capacity to perform unlimited correlations.
Nedstat: With a strong presence on the European market. The variables/metrics customization to meet the client needs is one of the company’s strengths.
Other good choices
If you want to minimize expenses on technology in favor of dedicating resources to in-house expertise take a look at these solutions:
Google Analytics: Considered the most powerful free application. It provides a wealth of statistics and it's tie in with Adwords. Google Analytics uses page tagging to report the data so it's not necesary to deal with hosting your own log files, but the downside is that the data is 12 hours behind, which is not suitable for everyone.
AT Internet: This company, primarily focused on Europe, provides a free solution called XiTi, yet the vendor augments that solution with a fee-based offering. It offers real-time data processing, strong segmentation, and some customization capabilities for its clients.
Yahoo! Web Analytics: Since the acquisition of the IndexTools Web analytics solution in 2008, it is free for use by Yahoo! small business merchants and all clients that are supported by a Yahoo! account team. This limited distribution strategy makes the tool somewhat exclusive but provides the opportunity for Yahoo! Web Analytics to focus on the core needs of its immediate client base.
New analytics for the new ways of communication.
Web 2.0 applications are discovering new ways to analyze a web site efectively besides the use of log files and page tagging.
Final users can also guide us to analyze some situations if for instance we create a test laboratory with clients, generate our own QA forms or by using online surveys. There are free ones such as 4Q or tusencuestas. This kind of customer driven optimization can tell us whether the users are finding their way on the website or not.
There is another measurement system, very used in communication companies (it was inherited from the TV) which is to install in a group of user's systems a data recovery software that will provide with statistics to analyze the site audience. The better known panels are Nielsen (NetView) and Comscore (Media Metrix)
There are also tools that let us compare our website with other ones based on estimated audience data which can help us to get an idea of how are we doing. Examples of this as a free application Alexa whereas fee-based Compete.
It's possible to analyze a blog like this by using the Google Analytics page tag and use a social media search engine to search what is being said about us in blogs, comments, bookmarks, events, news and videos with socialmention.
The Google solution Feedburner provides custom RSS feeds, traffic analysis and an optional advertising system to bloggers, podcasters, and other web-based content publishers.
For a Twitter profile we can use some tool such as twittercounter,
Facebook lets us use Allfacebook to analyze pages with more followers, and statistics about applications used or Socialistics where you can, for example, see the political views or religious beliefs of the people in your network, get stats for friends of your friends (extended network) or see all of your friends' images in a big picture wall.
With Youtube we can use the own application to check if the videos are reaching their audience.
In case that our concern goes to the frequency of a Google search, Google Trends will help us to know better our users preferences, the why and how they reach our website and the reasons of them not choosing us.
Related to this, there are tools like Google Insights for Search that help us with Search Engine Optimization by comparing search volume patterns across specific regions, categories, time frames and properties.
And last but not least, there are applications that let us test different versions of the same page to understand what the site visitor wants and increase conversions. On the free side we have Google Website Optimizer but if we need a more sophisticated solution lets chose Optimost or the Omniture bet Test&Target.
Although these are the most important solutions, I was amazed to see that there are many more in the market right now. Have a look at this link to check them out.
Related Documents
The Forrester Wave™: Web Analytics, Q3 2009
Omniture, Coremetrics, Unica, And Webtrends Lead, With Nedstat Close
by John Lovett
Cuadernos de comunicación Evoca
Herramientas para la correcta utilización de la Analítica Web
by Gemma Muñoz
Subscribe to:
Posts (Atom)