Weighted Least Squares and Heteroskedasticity
Table of Contents
Context
One assumption in the linear regression is the error terms
- Transform the response
, for example, the residual plots are funnel shape, then we can do - Use weighted least squares, and set smaller weights to observations that have large variances. The ordinary least square attaches equal importance to every observation, but the WLS concerns more about fitting well on spots where noise is small.

Weighted Least Square
Minimize the weighted mean squared error:
To write this in a matrix format, we need to make
Differentiate the above with respect to
Suppose the true relationship is
Since
Key Points
- WLS estimator is still an unbiased estimator.
- The objective is minimized if we choose
. This is ideal because the true variance is unknown.
Conditional Variance
How to estimate
Squared Residuals Method
- Model the mean: maybe this is a regression itself, but you get
- Construct the squared residuals:
- Model the conditional mean of residual squared. This should be a non-parametric method (like KDE) that describes the distribution of the residual squared. This is a function of
, say . - The predicted value of the function is an estimates of the conditional variance.
. - Calculate the weights and fit the WLS.
Reference
Content summarized from the following notes: