Analyze A/B Test Results
E-commerce website¶
Introduction¶
Should we choose the new page or the old page for our E-commerce website?¶
A/B tests are very commonly performed by data analysts and data scientists. For this project, we will be working to understand the results of an A/B test run by an e-commerce website. Our goal is to work through this notebook to help a company understand if they should:
- Implement the new webpage,¶
- Keep the old webpage, or¶
- Perhaps run the experiment longer to make their decision.¶
Dataset Overview¶
The dataset is composed of 294,478 entries containing different information like user id, the time user visited the website, the user group, landing page, and whether if they converted or not.¶
Dataset summary statistics and proportions¶
In [19]:
sns.countplot(data=df2,x='group');
print("Number of visitors in control group: ",df2.query('group=="control"').shape[0])
print("Number of visitors in Treatment group: ",df2.query('group=="treatment"').shape[0])
What is probability of conversion regardless of the group?¶
In [20]:
p_conv=df2['converted'].mean()
"{:.2%}".format(p_conv)
Out[20]:
Given that an individual was in the control group, what is the probability they converted?¶
In [21]:
control_cr=df2.query('group=="control"')['converted'].mean()
"{:.2%}".format(control_cr)
Out[21]:
Given that an individual was in the treatment group, what is the probability they converted?¶
In [22]:
treatment_cr=df2.query('group=="treatment"')['converted'].mean()
"{:.2%}".format(treatment_cr)
Out[22]:
What is the observed difference between the new_page conversions and old_page conversions in our sample dataset?¶
In [23]:
# Calculate the actual difference (obs_diff) between the conversion rates for the two groups.
obs_diff=treatment_cr - control_cr
"{:.2%}".format(obs_diff)
Out[23]:
Do the treatment group users lead to more conversions?¶
According to bayes theorem:
¶
If we want to get whether the new new_page leads to more conversion rate or not, then we need to find the proportion of the treatment group in the conversion pool, in other words we need to calculate $P (treatment | converted) $
And so the $P (treatment | converted) $ is:¶
In [26]:
P_treatmeant_conv=p_treatment * treatment_cr / p_conv
"{:.2%}".format(P_treatmeant_conv)
Out[26]:
And the $P (control | converted) $ is:¶
In [27]:
P_control_conv=p_control * control_cr / p_conv
"{:.2%}".format(P_control_conv)
Out[27]:
It looks like the old_page leads to slightly more conversion than the new_page, however they are almost the same, In the next section we will attempt to calculate how significant is this observation from our sample and try to infer that to the actual population (All the website users)¶
A/B Test¶
Null Hypothesis $H_0$ Testing¶
Since the difference between the two conversion rates is very small. Lets assume that our null hypothesis $H_0$ that the new_page conversion rate $p_{new}$ is the same or lower than conversion rate of old_page $p_{old}$ until the oppoiste proves to be true. And so, our assumption is:
Which translates into:¶
$$ H_0: p_{new} - p_{old} \leq 0 $$$$ H_1: p_{new}-p_{old} > 0 $$Simulation¶
In [60]:
null_dsprt=np.random.normal(0,np.std(p_diff),p_diff.size)
sns.displot(null_dsprt,color='#aaaaaa',height=4.5,aspect=2.4);
plt.title(r'Null distripution $ H_0: p_{new} - p_{old} \leq 0 $',fontsize=15);
In [61]:
null_dsprt=np.random.normal(0,np.std(p_diff),p_diff.size)
sns.displot(null_dsprt,color='#aaaaaa',height=4.5,aspect=2.4);
plt.axvline(np.percentile(null_dsprt,95),color='#3d4435',lw=3)
plt.title(r'Null distripution with significance level $\alpha$ of 5% (right tail test)',fontsize=15);
plt.legend(labels=[r'$\alpha$ of 5%']);
In [63]:
obs="{:.2%}".format(obs_diff)
sns.displot(data=null,x='differences',color='#aaaaaa',height=4.5,aspect=2.4);
plt.axvline(obs_diff,lw=3)
plt.axvline(np.percentile(null_dsprt,95),color='#3d4435',lw=3)
plt.title('Observed difference @ {} under the null hypothesis'.format(obs),fontsize=15)
plt.legend(labels=['observed difference',r'$\alpha$ of 5%'],loc='upper right');
In [64]:
null['p-value']=null['differences'].apply(lambda x: 1 if ((x > obs_diff)) else 0)
strin = 'The probability of observing a difference of {}\n or more extreme cases under the null hypothesis'.format(obs)
strin = strin + r" $ H_0: p_{new} - p_{old} \leq 0 $"
sns.displot(data=null,x='differences',hue='p-value',palette=['#aaaaaa','#0b5394'],legend=False,height=4.5,aspect=2.4);
plt.axvline(np.percentile(null_dsprt,95),color='#3d4435',lw=3)
plt.title(strin,fontsize=15);
plt.legend(labels=[r'$\alpha$ of 5%',"p-value = {:.1%}".format(null['p-value'].mean())],loc='upper right');
And so our calculated p-value is¶
In [65]:
"{:.2%}".format(null['p-value'].mean())
Out[65]:
A p-value of 90.02% is way too large to say that our observed difference of conversions came from the realm of chance, since our significance level $\alpha$ is 5%, and our p-value is large than $\alpha$, therefore our conclusion is:¶
- We fail to find enough evidence to reject the null hypothesis that we've assumed previously to be true stating that¶
- The conversion rate of the new_page is the same or lower than the conversion rate of the old_page $ H_0: p_{new} \leq p_{old} $¶
So now we know for a fact that we can't favour the alternate hypothesis $ H_1: p_{new} > p_{old} $ over our null hypothesis $H_0: p_{new} \leq p_{old} $, we still have two question in mind.¶
- Is the conversion rate for both pages is the same $ p_{new} = p_{old}$ ?¶
- or is the conversion rate for the new_page lower than the old_page $ p_{new} < p_{old}$?¶
I'll be using built-in method to conduct a two-sided test, where our null hypothesis $H_0: p_{new} = p_{old} $ and our alternate is $H_1: p_{new} \neq p_{old} $¶
But we previously know from the right-tailed test that:¶
$$p_{new} \leq p_{old} $$
And this makes our current alternate hypothesis for the two-sided test should we reject the null:¶
$$ H_1: p_{new} < p_{old} $$