Statistics : Correlation and Regression Tutorials

Covariance between X and Y Let $(x_i, y_i), i=1,2, \cdots , n$ be $n$ pairs of observations. Covariance measures the simultaneous variability between the two variables. It indicates how the two variables are related. A positive value of covariance indicate that the two variables moves in the same direction, whereas a negative value of covariance indicate that the two variables moves on opposite direction. Formula The sample covariance between $x$ and $y$ is denoted by $Cov(x,y)$ or $s_{xy}$ and is defined as

Karl Pearson’s Correlation Coefficient Let $(x_i, y_i), i=1,2, \cdots , n$ be $n$ pairs of observations then the Karl Pearson’s coefficient of correlation between two variables $X$ and $Y$ is denoted by $r_{xy}$ or $r$ and is given by $r = \dfrac{Cov(X,Y)}{\sqrt{Var(X) Var(Y)}}$ where, the sample covariance between $x$ and $y$ is $$ \begin{aligned} Cov(x,y) =s_{xy}&=\frac{1}{n-1}\sum_{i=1}^{n}(x_i -\overline{x})(y_i-\overline{y})\\ &= \frac{1}{n-1}\bigg(\sum_{i=1}^n x_iy_i - \frac{(\sum_{i=1}^n x_i)(\sum_{i=1}^n y_i)}{n}\bigg) \end{aligned} $$ the sample variance of $x$ is $$ \begin{aligned} V(x) =s_{x}^2 &=\frac{1}{n-1}\sum_{i=1}^{n}(x_i -\overline{x})^2\\ &= \frac{1}{n-1}\bigg(\sum_{i=1}^n x_i^2 - \frac{(\sum_{i=1}^n x_i)^2}{n}\bigg) \end{aligned} $$

Simple linear regression from raw data Let $(x_i, y_i), i=1,2, \cdots , n$ be $n$ pairs of observations. The simple linear regression model of $Y$ on $X$ is $$y_i=\beta_0 + \beta_1x_i +e_i$$ where, $y$ is a dependent variable, $x$ is an independent variable, $\beta_0$ is an intercept, $\beta_1$ is the slope, $e$ is the error term. Formula By the method of least square, the model parameters $\beta_0$ and $\beta_1$ can be estimated as

Simple linear regression from sum and sum of squares Let $(x_i, y_i), i=1,2, \cdots , n$ be $n$ pairs of observations. The simple linear regression model of $Y$ on $X$ is $$y_i=\beta_0 + \beta_1x_i +e_i$$ where, $y$ is a dependent variable, $x$ is an independent variable, $\beta_0$ is an intercept, $\beta_1$ is the slope, $e$ is the error term. Formula By the method of least square, the model parameters $\beta_0$ and $\beta_1$ can be estimated as

Simple linear regression from summarize data Let $(x_i, y_i), i=1,2, \cdots , n$ be $n$ pairs of observations. The simple linear regression model of $Y$ on $X$ is $$y_i=\beta_0 + \beta_1x_i +e_i$$ where, $y$ is a dependent variable, $x$ is an independent variable, $\beta_0$ is an intercept, $\beta_1$ is the slope, $e$ is the error term. Formula The simple linear regression model parameters $\beta_0$ and $\beta_1$ can be estimated using the method of least square.

Spearman’s Rank correlation Coefficient Let $(x_1, y_1), (x_2, y_2), \cdots , (x_n, y_n)$ be the ranks of $n$ individuals in two characteristics $A$ and $B$ respectively. Formula Then the Spearman’s rank correlation coefficient is denoted by $\varrho$ and is given by $\varrho = 1- \dfrac{6 \sum_{i=1}^{n}d_i^2}{n(n^2-1)}$ where, $d_i = x_i - y_i$ is the difference between the pairs of ranks of the $i^{th}$ individual in the two characteristics and $n$ is the number of pairs.

Testing Correlation Coefficient Let $(x_i, y_i), i=1,2,\cdots, n$ be a random sample of $n$ pairs of observations drawn from a bivariate normal population with correlation coeffficient $\rho$. Let $r_{xy}$ or $r$ be the observed correlation coefficient between $x$ and $y$. A test of significance for a linear relationship between the variables $x$ and $y$ can be performed using the sample correlation coefficient $r_{xy}$. We wish to test the hypothesis $H_0 : \rho =0$ (there is no significant linear relationship between $x$ and $y$) against $H_a : \rho \neq 0$ (there is a significant linear relationship between $x$ and $y$).

Testing Significance of Linear Relationship A test of significance for a linear relationship between the variables $x$ and $y$ can be performed using the sample correlation coefficient $r_{xy}$. Example 1 For a sample of eight bears, researchers measured the distances around the bears’ chests and weighed the bears. The Sample correlation coefficient between the chest size and weight of bears is $r=0.744$ for $n=8$ bears. Using $\alpha=0.05$, determine if there is a positive linear correlation between chest size and weight.

Testing Correlation Coefficient Let $(X_i, Y_i), i=1,2,\cdots, n$ be a random sample of $n$ pairs of observations drawn from a bivariate population with correlation coeffficient $\rho$. Let $r$ be the observed correlation coefficient between $X$ and $Y$. We wish to test the hypothesis $H_0 : \rho =\rho_0$ against $H_a : \rho \neq \rho_0$. Assumptions The population from which, the samples drawn, is a bivariate normal. The relationship between $X$ and $Y$ is linear.

Testing Correlation Coefficient In this tutorial we will discuss step by step solution of numerical problems on testing whether the population correlation coefficient is $\rho_0$ or not. Example 1 The median records shows that the correlation between the age of the mother and the birth weight of their first child is less than -0.34. A random sample of 8 mother’s age and the birth weight of their first child are as follows:

Testing Homogeneity of Two Correlation Coefficient We wish to test the null hypothesis that the correlation $\rho_1$ between $X$ and $Y$ in one population is the same as the correlation $\rho_2$ between $X$ and $Y$ in another population. Let $n_1$ be the sample pairs of observation from first population with sample correlation coefficient $r_1$ and $n_2$ be the sample pairs of observation from second population with sample correlation coefficient $r_2$.

Testing Homogeneity of Two Correlation Coefficient In this tutorial we will discuss step by step solution to some problems on testing homogeneity of two correlation coefficients. Example 1 The correlation between the mathematics scores and statistics scores of group of 50 male students is 0.358 and the correlation between the mathematics scores and statistics scores of groups of 40 female students is 0.569. Is the relationship between mathematics scores and statistics scores for male students is weaker than for female students?