Testing conversion rates

For a business running a Google AdWords campaign it is important that they endeavour to buy the most appropriate keywords, and that they do so at a suitable price. This can be helped by conversion rate tracking, which involves analysing the conversion rates of keywords. When assessing conversion rates a business needs to be aware that the statistical reliability of those rates will depend on the amount of data on which they are based.

For example, a conversion rate calculated from a very small number of clicks might be an inaccurate representation of its true value. As the keyword has so little data there is a chance it could have an unusually high or low number of conversions during the time the data was collected. Therefore it may not be appropriate to base a decision on the use or price of the keyword using this data.

Keywords can be split into two categories based on the data associated with them.

• Keywords which do have sufficient data to use their conversion rates with statistical confidence.
• Keywords which do not have sufficient data to use their conversion rates with statistical confidence.

The later of these is more common and therefore produces a problem when analysing the conversion rates. Statistical tests can be used to compare the conversion rate of a keyword with the population as a whole, to see if they are statistically significantly different. If they differ from each other statistically the conversion data may not be reliable and we should not use it.

Null hypothesis: The conversion rates are the same. Therefore it can be assumed the keyword has reliable conversion data.

Alternative hypothesis: The conversion rates are different, so we can reject the null hypothesis.

Confidence Interval for the Difference Between Proportions (for independent samples) is used to compare two proportions of large samples.
An interval estimate for the difference in proportions $p_1-p_2$ with confidence level $1-\alpha$ is given by

Where $SE_1$ and $SE_2$ are the standard errors of and , respectively and

For most keywords, this test fails because the sample size (number of clicks for the keyword) is too small for the test to be valid.

In this case you can use Fisher’s Exact Test.

For Fisher’s Exact Test we have variables X and Y, with observed states $m$ and $n$ respectively. Using these we form a matrix $n\times m$ where $a_{ij}$ is the number of observations when $x=i$ and $y=j$ (9). Row sum and column sum are $R_i$ and $C_j$ with

$N=\sum_i R_i =\mathbf{\sum_j C_j}$

the total sum of the matrix.
Next we calculate the conditional probability of getting the whole matrix, which is given by

For example

Where $R_1=6810, R_2=7$ and $C_1=25.8, C_2=6791.2$

p$\le0.0001$ so the conversion rate of the keyword ‘blue leather office chairs’ is statistically significantly different to the campaign conversion rate. Therefore we reject the null hypothesis, meaning we cannot say with statistical confidence if the keywords conversion rate is reliable.

In general, the smaller the P value is, the less likely the conversion data will be reliable. To do the test we can use a computer program because the computation can be huge.

Another problem can arise when testing the data because of the large range in sample sizes. In the table below if you compare the data of blue leather office chairs with the total data, the number of clicks is 21 and 6789 respectively which are very different in size. This could mean the C-difference test is inaccurate as some of the data is too small but Fishers Exact test has a large and very laborious computation.

4. Personal contact.

8. N. Drakos, Overview of Confidence Intervals

9. E.W. Weisstein, Fisher’s Exact Test, From MathWorld–A Wolfram Web Resource.