Testing conversion rates

For a business running a Google AdWords campaign it is important that they endeavour to buy the most appropriate keywords, and that they do so at a suitable price. This can be helped by conversion rate tracking, which involves analysing the conversion rates of keywords. When assessing conversion rates a business needs to be aware that the statistical reliability of those rates will depend on the amount of data on which they are based.

For example, a conversion rate calculated from a very small number of clicks might be an inaccurate representation of its true value. As the keyword has so little data there is a chance it could have an unusually high or low number of conversions during the time the data was collected. Therefore it may not be appropriate to base a decision on the use or price of the keyword using this data.

Keywords can be split into two categories based on the data associated with them.

  • Keywords which do have sufficient data to use their conversion rates with statistical confidence.
  • Keywords which do not have sufficient data to use their conversion rates with statistical confidence.

The later of these is more common and therefore produces a problem when analysing the conversion rates. Statistical tests can be used to compare the conversion rate of a keyword with the population as a whole, to see if they are statistically significantly different. If they differ from each other statistically the conversion data may not be reliable and we should not use it.

Null hypothesis: The conversion rates are the same. Therefore it can be assumed the keyword has reliable conversion data.

Alternative hypothesis: The conversion rates are different, so we can reject the null hypothesis.

Confidence Interval for the Difference Between Proportions (for independent samples) is used to compare two proportions of large samples.
An interval estimate for the difference in proportions p_1-p_2 with confidence level 1-\alpha is given by


Where SE_1 and SE_2 are the standard errors of phat1 and phat2, respectively and


For most keywords, this test fails because the sample size (number of clicks for the keyword) is too small for the test to be valid.

In this case you can use Fisher’s Exact Test.

For Fisher’s Exact Test we have variables X and Y, with observed states m and n respectively. Using these we form a matrix n\times m where a_{ij} is the number of observations when x=i and y=j (9). Row sum and column sum are R_i and C_j with

N=\sum_i R_i =\mathbf{\sum_j C_j}

the total sum of the matrix.
Next we calculate the conditional probability of getting the whole matrix, which is given by


For example

Where R_1=6810, R_2=7 and C_1=25.8, C_2=6791.2

p\le0.0001 so the conversion rate of the keyword ‘blue leather office chairs’ is statistically significantly different to the campaign conversion rate. Therefore we reject the null hypothesis, meaning we cannot say with statistical confidence if the keywords conversion rate is reliable.

In general, the smaller the P value is, the less likely the conversion data will be reliable. To do the test we can use a computer program because the computation can be huge.

Another problem can arise when testing the data because of the large range in sample sizes. In the table below if you compare the data of blue leather office chairs with the total data, the number of clicks is 21 and 6789 respectively which are very different in size. This could mean the C-difference test is inaccurate as some of the data is too small but Fishers Exact test has a large and very laborious computation.


4. Personal contact.

7. N. Drakos, Confidence Interval for the Difference Between Two Proportions

8. N. Drakos, Overview of Confidence Intervals

9. E.W. Weisstein, Fisher’s Exact Test, From MathWorld–A Wolfram Web Resource.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s