Least Squares and the Fitted Model

Least Squares and the Fitted Model

In statistical modeling, particularly in linear regression, the least squares method is a technique used to estimate the parameters of a linear regression model. The objective is to find the line (or hyperplane in higher dimensions) that minimizes the sum of the squared differences between the observed values and the values predicted by the model. This line is known as the fitted model or regression line.

Let's break down the concept of least squares and how it leads to the fitted model.

1. The Least Squares Method

The least squares method is used to find the best-fitting line by minimizing the sum of squared residuals. Residuals are the differences between the observed values ($Y_i$) and the predicted values ($\hat{Y}_i$) from the regression line. In simple linear regression, the goal is to estimate the parameters (slope $\beta_1$ and intercept $\beta_0$) of the linear model:

Where:

• $Y_i$ are the observed values,
• $X_i$ are the values of the independent variable,
• $\beta_0$ is the intercept,
• $\beta_1$ is the slope,
• $\epsilon_i$ is the error term (residual).

The residual for each data point $i$ is given by:

2. Objective of Least Squares

The objective of the least squares method is to minimize the sum of squared residuals, which is mathematically expressed as:

Where:

• $SSE$ stands for Sum of Squared Errors (or residual sum of squares).
• $n$ is the number of data points.

The least squares method minimizes $SSE$ by adjusting the parameters $\beta_0$ (intercept) and $\beta_1$ (slope) of the linear equation.

3. Estimating the Parameters $\beta_0$ and $\beta_1$

Using calculus, we can find the values of $\beta_0$ and $\beta_1$ that minimize the sum of squared residuals. These values are computed as follows:

a. Formula for $\beta_1$ (Slope)

The slope $\beta_1$ is given by the formula:

Where:

• $\hat{\beta_1}$ is the estimated slope,
• $X_i$ and $Y_i$ are the individual data points,
• $n$ is the number of data points.

b. Formula for $\beta_0$ (Intercept)

Once the slope $\hat{\beta_1}$ is found, the intercept $\beta_0$ can be estimated as:

Where:

• $\bar{X}$ is the mean of the independent variable $X$,
• $\bar{Y}$ is the mean of the dependent variable $Y$.

Thus, the fitted model (regression line) is given by:

This is the fitted regression line that minimizes the sum of squared errors.

4. The Fitted Model: The Regression Line

The fitted model refers to the regression equation obtained after applying the least squares method. The line of best fit is the line that minimizes the difference between the observed values $Y_i$ and the predicted values $\hat{Y}_i$. The fitted model is represented by the equation:

Where:

• $\hat{Y}$ is the predicted value of $Y$,
• $X$ is the independent variable,
• $\hat{\beta_0}$ is the estimated intercept,
• $\hat{\beta_1}$ is the estimated slope.

The fitted line is used to make predictions about the dependent variable $Y$ for any given value of $X$. For example, if $X$ represents years of experience, and $Y$ represents salary, the fitted model can predict the expected salary for any given number of years of experience.

Example of Fitted Model:

Let’s say we have a dataset that represents the relationship between the number of study hours ($X$) and the test scores ($Y$) of a group of students. After performing linear regression, we obtain the fitted model:

This means that for each additional hour of study ($X$), the test score ($Y$) increases by 5 points. The intercept of 50 suggests that a student who does not study at all ($X = 0$) is expected to have a baseline test score of 50.

5. Assessing the Fit of the Model

Once the least squares method is used to estimate the parameters, it’s important to assess how well the fitted model represents the data. This can be done using several metrics:

a. Residuals

Residuals are the differences between the observed values and the predicted values:

By examining the residuals, we can check the assumptions of the regression model, such as homoscedasticity (constant variance) and independence of errors.

b. $R^2$ (Coefficient of Determination)

$R^2$ is a key metric that tells us how well the fitted model explains the variability in the dependent variable. It is the proportion of the variance in the dependent variable that is explained by the independent variable.

Where:

• $Y_i$ are the observed values,
• $\hat{Y}_i$ are the predicted values,
• $\bar{Y}$ is the mean of $Y$.

An $R^2$ value close to 1 indicates that the model explains most of the variance in the data, while an $R^2$ value close to 0 suggests that the model does not explain much of the variance.

c. Hypothesis Testing for Parameters

In addition to the fitted model, hypothesis tests can be performed on the parameters $\beta_0$ and $\beta_1$ to assess whether they are statistically significantly different from zero. Typically, this is done using t-tests for the individual regression coefficients.

6. Limitations of the Fitted Model

While the least squares method provides a useful tool for fitting a linear regression model, there are some important limitations:

• Linearity Assumption: Linear regression assumes a linear relationship between the independent and dependent variables. If the relationship is non-linear, a linear regression model may not provide a good fit.
• Outliers: Outliers can significantly affect the fitted model, especially in small datasets.
• Assumption of Homoscedasticity: The least squares method assumes constant variance of the residuals. If the variance of residuals changes with the independent variable, this assumption is violated (heteroscedasticity).
• Multicollinearity: In multiple regression, high correlation between independent variables can lead to instability in the coefficient estimates.

Summary

• The least squares method is used to estimate the parameters (slope and intercept) of a linear regression model by minimizing the sum of squared residuals.
• The fitted model represents the best-fitting line, which can be used for prediction and analysis.
• Key metrics for assessing the fit of the model include residuals, $R^2$, and hypothesis tests for the regression coefficients.
• Understanding the assumptions and limitations of the model is essential for accurate interpretation and use of the regression results.