Preliminary Reading

• An Introduction to Statistical Learning with Applications in R, Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani, Chapter 2

Learning Objectives

• What is Statistical Learning

Keywords:

• Inputs: independent variable, covariates, predictors, features, regressors.

• Output: dependent variable, variates, labels, regressand.

• Reducible/Irreducible Errors

• Prediction and Inference

• Bias-Variance

Predictors and Response (Dependent, Output) Variables

• Let Y be the dependent variable and \(X_1, X_2,\cdots,X_p\) be \(p\) different predictors.

• Assume that there is some relationship between \(Y\) and \(X = (X_1, X_2,...,X_p)\):

\[Y=f(X)+\varepsilon\]

• Where, \(f(.)\) is some fixed but unknown function of \(X_1,...,X_p\), \(\varepsilon\) is a random error term, which is independent of \(X\) and has mean zero.

\[Y=f(X)+\varepsilon=Pattern+Error\]

Function \(f()\)

• Why estimate \(f()\)

• How do we estimate \(f()\)

• Prediction Accuracy and the ease of model interpretation

• Supervised vs Unsupervised Learning

• Regression vs Classification Problems

An Example, Salary & Age

*: Data may be obtained from Hull website

First Six Observations of Salary and Age Data

Age	Salary
25	135000
55	260000
27	105000
35	220000
60	240000
65	265000

An Example, Salary & Age

• Let’s split the data such that:
  • First 10 Observations are training data
    
  • Remaining Observations are test data

An Example, Salary & Age

• Model Ideas (Data Analytics I & II and Data Modeling Courses):
  • \(f():Linear \; Regression\)
    
  • \(f():Quadratic \; Regression\)
    
  • \(f():5^{th} \: Degree \; Polynomial \; Regression\)

An Example, Salary & Age (MIS 305, MIS 314, MIS 315, MIS 403)

Regression Models are basic building blocks of many complex models. This part is covered in MIS 305 course

• Model Results

Regression Results: 1-Linear, 2-Quadratic 3-Fifth Degree

	Dependent variable:
	Salary
	(1)	(2)	(3)
Age	3,827.305**	25,377.170**	-2,312,166.000**
	(1,246.189)	(7,251.933)	(705,471.800)
I(Age2)		-243.289**	107,539.500**
		(81.263)	(33,840.510)
I(Age3)			-2,403.642**
			(790.506)
I(Age4)			25.993**
			(9.005)
I(Age5)			-0.109*
			(0.040)
Constant	51,160.420	-382,171.100**	19,198,421.000**
	(56,360.290)	(150,139.600)	(5,723,899.000)
Observations	10	10	10
R2	0.541	0.799	0.969
Adjusted R2	0.484	0.741	0.930
Residual Std. Error	52,747.830 (df = 8)	37,341.470 (df = 7)	19,353.310 (df = 4)
F Statistic	9.432** (df = 1; 8)	13.892*** (df = 2; 7)	25.099*** (df = 5; 4)
Note:	p<0.1; p<0.05; p<0.01

Why Estimate \(f()\), Prediction

• For Prediction

\(X=(X_1, X_2, \cdots,X_p)\) are available but \(Y\) can not be obtained.

\(\hat Y=\hat f(X)\) since \(E(\varepsilon)=0\)

\(\hat f()\) may be a black box model where exact form is not important but it predicts \(Y\) accurately

Reducible (\(\hat f()\) is not perfect estimate of \(f\)) and irreducible error (\(\hat f()\) almost perfect estimate of \(f\), but \(Y\) is a function of \(\varepsilon\))

The expected value of the squared difference between actual and predicted value of \(Y\)

\(E(Y-\hat Y)^2=E[f(X)+\varepsilon -\hat f(X)]^2\)

\(=\underbrace{E[f(X) -\hat f(X)]^2}_{reducible} +\underbrace{Var(\varepsilon)}_{irreducible}\)

\(\varepsilon\) may contain (i) unmeasured variables and (ii) unmeasurable variation

The focus is to minimize the reducible error with different techniques for estimating \(\hat f()\)

An Example, Salary & Age

• Prediction: Let’s see the predictions for test data

An Example, Salary & Age

• Prediction: Let’s see the predictions for test data

An Example, Salary & Age

• Let’s Obtain Model Performance Summaries

An Example, Salary & Age

• Let’s Obtain RMSE for Training and Test Data

Why Estimate \(f()\), Inference

• For Inference

The main aim is to understand the relationship between \(X\) and \(Y\). The main aim is not necessarily to make prediction.

\(\hat f()\) should be chosen so that it is interpretable.

Questions are:

• Which predictors are associated with the response?

• What is the relationship between the response and each predictor?

• Is the relationship simple or complicated?

• If one of the predictors changes, how the response change?

Inference: An Example, Advertising Data (ISLR Book)

– Which media contribute to sales?

– Which media generate the biggest boost in sales?

– How much increase in sales is associated with a given increase in TV advertising?

How Do We Estimate \(f\)

• In this course, some of the linear and non-linear approaches will be covered

• Parametric methods (Estimating \(f()\) via estimating set of parameters):

  • Make an assumption about functional form or shape of \(f()\), for example, a linear model: \(f(X)=\beta_0+\beta_1x_1+ \beta_2x_2+ \cdots + \beta_kx_k\)
    
  • Use the training data to fit (or train) the model (for example using ordinary least squares)

If the selected model performance is poor, one may choose to select more flexible models. This may result in overfitting the data (model follow the error too closely)

Parametric Example (Fig 2.4 ISLR)

As an example, here we use income, education and seniority relationship where the true underlying relationship shown as below:

Parametric Example

• Let’s use linear function

\[income=\beta_0 + \beta_1 \times education + \beta_2 \times seniority\]

Then the estimated function:

Non-Parametric Example

• Non-parametric methods: no explicit assumptions about the functional form of \(f()\), suitable for wider range possible shapes for \(f()\).
  • Try to estimate \(f()\) as close the data points as possible

Compare this with a non-parametric approach shown below:

Flexibility of Models

Prediction Accuracy and Model Interpretability

• Some models are more flexible where some others are less flexible

• For example, Linear Regression is an example of less flexible model

• Lasso, Ridge regression (MIS 403) are examples of less flexible models and Trees, Bagging, Boosting (MIS 403) are examples of more flexible models

• If inference is important then there are clear advantages to using the less flexible models. Because it is easy to understand them

• Time to time, Partially linear models may be used for inference where some of the variables exhibit non-linear relationship with the dependent model. If the effects of the variables that have linear relationship with dependent variable are important to analyze then partial linear model may be suitable

Supervised vs Unsupervised Learning

• Most of the learning problems fall into

  • Supervised Learning
    
  • Unsupervised Learning
    
  • (Semi-Supervised Learning)
    
  • (Reinforcement Learning)

• Many classical statistical learning methods are examples of Supervised Learning (since \(Y \: and \: X_i\) are available)

• Unsupervised Learning describes the more challenging situation (since \(X_i\) are available but response variable \(Y\) is not available)

• Clustering Analysis is an example of Unsupervised learning

Regression vs Classification

• Some variables are Categorical (qualitative) and some others are Numerical (quantitative)

• Qualitative variables take different categories as values (sometimes called as different classes), such as, default/no default; cancer/no cancer etc.

• Regression Problem is used for when the response variable is quantitative

• Classification Problem is used for when the response variable is qualitative

• Some methods can be used either for quantitative or qualitative responses (for example Trees, Boosting)

Assessing Model Accuracy

• Quality of fit measures (in general for test data)
  • Mean Absolute Error, MAE, \(\text{mean}(|\varepsilon_j|)\)
    
  • Mean Absolute Percentage Error, MAPE, \(\text{mean}(|p_{j}|), \: p_j=100\varepsilon_j/Y_j\)
    
  • Mean Squared Error, MSE, \(\text{mean}(\varepsilon_j^2)\)
    
  • Root Mean Squared Error, RMSE, \(\sqrt{\text{mean}(\varepsilon_j^2)}\)
    
  • K-Fold Cross Validation
    
  • …

Low Bias and Low Variance

• Variance refers to the amount by which \(\hat f()\) would change if we estimated it using a different training data set.
  • Ideally the estimate for f should not vary too much between training sets
    
  • High variance leads to large changes in \(\hat f()\) even in the case when small changes in training data
    
  • In general, more flexible statistical methods have higher variance

Low Bias and Low Variance

• Bias refers to the error that is introduced by approximating a real-life problem, which may be extremely complicated, by a much simpler model

  • For example Linear Regression may perform poorer if the relationship is highly non-linear
    
  • In these cases, increasing the number of observations do not lead to better predictions

• As a general rule, as we use more flexible methods, the variance will increase and the bias will decrease

• Good test set performance of a statistical learning method quires low variance as well as low squared bias

An Example, Salary & Age

• Overfitting: Model closely follows noises. (Hence Variance refers to errors caused by overfitting)

• Undefitting: Model does not capture Pattern (or part of the patterns) successfully. (Hence Bias refers to error caused by underfitting)

Model Assessment and Model Selection

• The process of evaluating a model’s performance is known as model assessment

• The process of selecting the proper level of flexibility for a model is known as assessment model selection

• Two common Resampling Methods will be discussed later in this course

  • Cross-Validation
    
  • Bootstrap