Chapter 5: Multiple Regression Analysis - OLS Asymptotics

Asymptotic theory provides the foundation for statistical inference when sample sizes are large, describing the limiting behavior of OLS estimators as $n \to \infty$. This chapter establishes consistency and asymptotic normality of OLS estimators, develops large-sample inference procedures, and addresses practical complications including heteroskedasticity-robust standard errors and asymptotically efficient estimation methods.

The presentation builds from fundamental concepts to practical applications. We begin by establishing consistency and asymptotic normality under the Gauss-Markov assumptions (Section 5.1), demonstrate large-sample hypothesis testing and confidence interval construction (Section 5.2), examine asymptotic efficiency through the Gauss-Markov theorem’s large-sample analog (Section 5.3), and conclude with robust inference methods when assumptions fail (Section 5.4). Throughout, we emphasize the practical relevance of asymptotic approximations for finite samples typically encountered in applied econometric research.

Asymptotic Properties of OLS¶

Under the Gauss-Markov assumptions MLR.1-MLR.4 (from Chapter 3: linearity, random sampling, no perfect collinearity, and zero conditional mean), OLS estimators have the following asymptotic properties:

1. Consistency: $\hat{\beta}_j \xrightarrow{p} \beta_j$ as $n \to \infty$ for all $j = 0, 1, \ldots, k$

The OLS estimators converge in probability to the true parameter values as the sample size grows large. Consistency is a weaker property than unbiasedness--it only requires that the estimator approaches the true value asymptotically. Notably, consistency does not require normality (MLR.6) or homoscedasticity (MLR.5).

2. Asymptotic Normality: $\sqrt{n}(\hat{\beta}_j - \beta_j) \xrightarrow{d} N(0, \sigma^2_{\beta_j})$ as $n \to \infty$

By the Central Limit Theorem (CLT), the sampling distribution of $\hat{\beta}_j$ approaches a normal distribution as $n \to \infty$, even when the errors are not normally distributed. This requires:

• MLR.1-MLR.4 (especially zero conditional mean)

• Finite fourth moments of the errors: $E(u^4) < \infty$ (mild regularity condition)

• No normality assumption required

Practical Implications:

• With large samples ($n \geq 30$ typically), t-tests and F-tests are approximately valid even without normal errors (MLR.6)

• Consistency means OLS remains reliable in large samples even under violations of MLR.5 (heteroskedasticity)

• Asymptotic inference requires only MLR.1-MLR.4 plus regularity conditions, making it more robust than finite-sample inference

We will use simulations to visualize these concepts and then apply the Lagrange Multiplier (LM) test to a real-world example.

Connection to Previous Chapters:

• Chapters 2-3 established finite-sample properties under Gauss-Markov assumptions

• Chapter 4 showed exact inference requires normality (MLR.6) for finite samples

• This chapter demonstrates that normality is not necessary for large-sample inference

• These results justify the robustness claims made in Chapter 4 about t-tests with $n \geq 30$

5.1 Consistency of OLS¶

Consistency is a fundamental asymptotic property that describes whether an estimator converges to the true parameter value as the sample size grows infinitely large. While unbiasedness (Chapter 3) is a finite-sample property, consistency is a large-sample property.

5.1.1 Definition and Importance¶

Definition: An estimator $\hat{\theta}_n$ of parameter $\theta$ is consistent if:

This means: For any $\epsilon > 0$, $P(|\hat{\theta}_n - \theta| > \epsilon) \to 0$ as $n \to \infty$.

In words: As the sample size increases, the probability that the estimator is far from the true value approaches zero.

Why Consistency Matters:

1. Asymptotic reliability: With large enough samples, consistent estimators give us values close to the truth

2. Weaker than unbiasedness: An estimator can be biased but still consistent (bias vanishes as $n \to \infty$)

3. Minimum requirement: Consistency is the least we should expect from an estimator

4. Practical relevance: Most real-world datasets are large enough for asymptotics to apply

Example: Sample Mean

Consider estimating the population mean $\mu$ using the sample mean $\bar{x} = \frac{1}{n}\sum_{i=1}^n x_i$.

By the Law of Large Numbers (LLN):

The sample mean is consistent for the population mean under weak conditions (just need $E(x_i) = \mu$ and finite variance).

5.1.2 Consistency of OLS Estimators¶

Theorem (Consistency of OLS): Under assumptions MLR.1-MLR.4:

• MLR.1: Linear in parameters

• MLR.2: Random sampling

• MLR.3: No perfect collinearity

• MLR.4: Zero conditional mean ($E(u|x_1, \ldots, x_k) = 0$)

The OLS estimators are consistent:

Key Observations:

1. No normality required (MLR.6): We don’t need normal errors for consistency

2. No homoscedasticity required (MLR.5): Heteroscedasticity doesn’t affect consistency

3. Zero conditional mean is crucial (MLR.4): Omitted variable bias prevents consistency

Practical Implication: Even if errors are heteroskedastic or non-normal, OLS estimates converge to true values with large samples.

5.1.3 When Consistency Fails: Omitted Variable Bias¶

The zero conditional mean assumption (MLR.4) is critical for consistency. If it fails due to omitted variables, OLS is inconsistent.

Example: Wage Equation with Omitted Ability

True model:

Estimated model (ability unobserved):

where $v = \beta_2 \text{ability} + u$.

If $\text{Cov}(\text{educ}, \text{ability}) \neq 0$, then:

This violates MLR.4, so $\hat{\beta}_1$ is inconsistent even as $n \to \infty$!

Interpretation:

• Consistent estimator (includes ability): Bias approaches zero as $n$ increases

• Inconsistent estimator (omits ability): Bias persists even with n=10,000

• This demonstrates that no amount of data can fix omitted variable bias

5.2 Asymptotic Normality and Large Sample Inference¶

While consistency tells us that OLS converges to the true value, asymptotic normality describes the distribution of the OLS estimator in large samples. This is crucial for hypothesis testing and confidence intervals.

5.2.1 The Central Limit Theorem (CLT)¶

The Central Limit Theorem is one of the most important results in statistics. In the context of OLS:

Theorem (Asymptotic Normality of OLS): Under MLR.1-MLR.5:

Or equivalently:

where $a_{jj}$ is the j-th diagonal element of $(X'X)^{-1}$ and $\sigma^2 = \text{Var}(u)$.

What This Means:

• As $n \to \infty$, the t-statistic follows a standard normal distribution

• We can use the normal distribution to construct confidence intervals and test hypotheses

• This holds even if errors are not normally distributed (huge advantage!)

5.2.2 Why Normality Assumption is “Too Strong”¶

In Chapter 4, we assumed MLR.6: Normality ($u \sim N(0, \sigma^2)$) for exact finite-sample inference. But:

Problems with Normality Assumption:

1. Rarely true in practice: Economic variables often have skewed or heavy-tailed distributions

2. Restrictive: Rules out many plausible error distributions

3. Unnecessary for large samples: CLT provides normality asymptotically

4. Not testable in a useful way: We observe $\hat{u}_i$, not $u_i$

Examples Where Normality Fails:

• Income/wage data: Typically right-skewed (log-normal or Pareto distributions)

• Count data: Number of arrests, patent applications (Poisson-like distributions)

• Binary outcomes: 0/1 variables cannot be normally distributed

• Financial returns: Fat tails, excess kurtosis (not normal)

5.2.3 Large Sample Inference Without Normality¶

Key Result: For $n \geq 30$ (rule of thumb), we can use:

t-tests:

Confidence Intervals:

Unexpected end of input in a macro argument, expected '}' at end of input: …text{ (95% CI)}

\hat{\beta}_j \pm 1.96 \cdot \text{se}(\hat{\beta}_j) \text{ (95% CI)}

F-tests:

Advantages of Asymptotic Inference:

1. Robustness: Works with non-normal errors

2. Simplicity: Use standard normal critical values instead of t-distribution

3. Flexibility: Applies to heteroskedastic errors (with robust SEs)

4. Wide applicability: Most econometric methods rely on asymptotics

5.3 Asymptotic Efficiency of OLS¶

Efficiency concerns whether an estimator has the smallest possible variance among all consistent estimators. OLS has strong efficiency properties under certain conditions.

5.3.1 The Gauss-Markov Theorem (Revisited)¶

Recall from Chapter 3: Under MLR.1-MLR.5, OLS is BLUE (Best Linear Unbiased Estimator) in finite samples.

Asymptotic Version: Under MLR.1-MLR.5, OLS is asymptotically efficient among linear estimators:

where Avar denotes asymptotic variance.

5.3.2 When is OLS Inefficient?¶

OLS can be inefficient (not best) when:

1. Heteroskedasticity (MLR.5 fails):

   • OLS is still consistent but no longer BLUE

   • Weighted Least Squares (WLS) is more efficient

   • Use robust standard errors for valid inference (Chapter 8)

2. Serial correlation (time series, Chapter 12):

   • OLS standard errors are incorrect

   • Need heteroskedasticity and autocorrelation consistent (HAC) standard errors

3. Endogeneity (MLR.4 fails):

   • OLS is inconsistent

   • Use Instrumental Variables (IV) or 2SLS (Chapter 15)

   • IV is consistent but less efficient than OLS when OLS is valid

Practical Implication: If MLR.1-MLR.5 hold, you can’t systematically beat OLS in terms of efficiency. But if MLR.5 fails, there are more efficient estimators (though OLS remains consistent).

5.3.3 Trade-off: Efficiency vs Robustness¶

Interpretation:

• Both OLS and WLS are consistent (converge to 0.5)

• WLS has lower standard deviation (more efficient)

• Under heteroskedasticity, WLS can be substantially more efficient

5.4 Testing Normality of the Error Term¶

Given the importance (or lack thereof) of the normality assumption, you might wonder: Should we test whether errors are normally distributed?

5.4.1 Why Testing Normality is “Not Actionable”¶

The Dilemma: Suppose we test $H_0: u \sim N(0, \sigma^2)$ and:

1. Reject normality: What do we do?

   • Can’t “fix” non-normal errors

   • OLS is still consistent (MLR.1-MLR.4)

   • Use asymptotic inference (which doesn’t require normality anyway)

2. Fail to reject normality: What do we conclude?

   • Lack of evidence against normality $\neq$ evidence for normality

   • With small samples, test has low power

   • With large samples, we don’t need normality anyway (CLT)

The Bottom Line: Testing normality doesn’t lead to an actionable response in most cases.

5.4.2 Common Normality Tests¶

Despite the above, normality tests are sometimes reported:

1. Jarque-Bera Test:

where $S$ = skewness, $K$ = kurtosis of residuals.

2. Shapiro-Wilk Test: Based on correlation between residuals and expected order statistics

3. Anderson-Darling Test: Weighted version of Kolmogorov-Smirnov test

Example: Jarque-Bera Test

5.4.3 Practical Recommendations¶

What to do instead of testing normality:

1. Check for influential observations:

   • Look for outliers that might distort results

   • Use robust regression methods if needed

2. Examine residual plots:

   • Plot residuals vs fitted values

   • Look for patterns suggesting misspecification

3. Use robust inference:

   • Always use heteroskedasticity-robust standard errors (Chapter 8)

   • They’re valid under both homoskedasticity and heteroskedasticity

4. Check functional form:

   • Test for nonlinearities (Chapter 6)

   • Use RESET test for misspecification

5. Consider transformation:

   • Log transformation often reduces skewness

   • Can improve linearity and homoskedasticity simultaneously

5.5 Simulation Exercises¶

In this section, we will conduct simulation exercises to illustrate the asymptotic properties of the OLS estimator, particularly focusing on its distribution as the sample size increases under different scenarios.

5.1.1 Normally Distributed Error Terms¶

This simulation demonstrates the behavior of the OLS estimator when the error terms are normally distributed. Under the classical linear model assumptions, including normally distributed errors, the OLS estimators are not only BLUE (Best Linear Unbiased Estimator) but also have desirable properties even in small samples. Asymptotically, the OLS estimator is consistent and normally distributed. We will visualize how the distribution of the estimated coefficient $\hat{\beta}_1$ approaches a normal distribution as the sample size $n$ increases.

Interpretation of 5.1.1:

The plots above show the simulated density of the OLS estimator $\hat{\beta}_1$ for different sample sizes ($n = 5, 10, 100, 1000$) when the error term is normally distributed. We compare this simulated density to the theoretical normal distribution that $\hat{\beta}_1$ should asymptotically follow.

• Small Sample Sizes (n=5, 10): For very small sample sizes, the simulated density of $\hat{\beta}_1$ is somewhat close to the normal distribution, but there are noticeable deviations. This is because while OLS is BLUE under these conditions, the asymptotic normality is approached as $n \rightarrow \infty$.

• Larger Sample Sizes (n=100, 1000): As the sample size increases to $n=100$ and $n=1000$, the simulated density of $\hat{\beta}_1$ gets increasingly closer to the theoretical normal distribution. For $n=1000$, the simulated density is almost indistinguishable from the normal distribution.

This simulation visually confirms that when the errors are normally distributed, the distribution of the OLS estimator $\hat{\beta}_1$ approaches a normal distribution as the sample size $n$ increases, consistent with asymptotic theory and even showing reasonable approximation for smaller sample sizes in this ideal scenario.

5.1.2 Non-Normal Error Terms¶

In this simulation, we investigate what happens when one of the classical linear model assumptions is violated - specifically, the assumption of normally distributed errors. We will use error terms that follow a standardized Chi-squared distribution with 1 degree of freedom. Even when the error term is not normally distributed, under the Gauss-Markov conditions and assuming homoskedasticity and no autocorrelation, OLS is still BLUE. More importantly, even with non-normal errors, the OLS estimator is still consistent and asymptotically normally distributed under weaker conditions (CLT for sample averages). This simulation will demonstrate the asymptotic normality even with non-normal errors.

First, let’s visualize the shape of the standardized Chi-squared distribution compared to the standard normal distribution.

The plot above shows that the standardized Chi-squared distribution is skewed to the right and has a different shape compared to the standard normal distribution. Now, let’s perform the simulation with these non-normal errors.

Interpretation of 5.1.2:

These plots illustrate the distribution of $\hat{\beta}_1$ when the error terms are from a standardized Chi-squared distribution, which is non-normal.

• Small Sample Sizes (n=5, 10): For small sample sizes, the simulated density of $\hat{\beta}_1$ is noticeably skewed and deviates from the normal distribution, reflecting the non-normality of the error term.

• Larger Sample Sizes (n=100, 1000): As the sample size increases, the simulated density of $\hat{\beta}_1$ becomes progressively more symmetric and approaches the theoretical normal distribution. By $n=1000$, the simulated distribution is very close to normal, even though the underlying errors are non-normal.

This simulation demonstrates the power of the Central Limit Theorem in action. Even when the errors are not normally distributed, the OLS estimator $\hat{\beta}_1$, which is a function of the sample average of the error terms (indirectly), becomes approximately normally distributed as the sample size grows large. This is a key result in asymptotic theory, justifying the use of normal distribution based inference (t-tests, confidence intervals) in OLS regression with large samples, even if we suspect the errors are not normally distributed.

5.1.3 (Not) Conditioning on the Regressors¶

In previous simulations (5.1.1 and 5.1.2), we fixed the regressors $x$ across replications for each sample size $n$. This is akin to conditioning on the regressors. In econometric theory, we often derive properties of OLS estimators conditional on the observed values of the regressors. However, in reality, regressors are also random variables. This simulation explores the implications of not conditioning on the regressors by drawing new samples of $x$ in each replication, along with new error terms. We will see if the asymptotic normality of $\hat{\beta}_1$ still holds when both $x$ and $u$ are randomly drawn in each simulation run.

Interpretation of 5.1.3:

In this simulation, both the regressors $x$ and the error terms $u$ are randomly drawn in each replication.

• Small Sample Sizes (n=5, 10): Similar to the previous simulations, with very small sample sizes, the simulated distribution of $\hat{\beta}_1$ shows some deviation from the normal distribution.

• Larger Sample Sizes (n=100, 1000): As the sample size increases, even when we are not conditioning on the regressors, the distribution of $\hat{\beta}_1$ still converges to a normal distribution. By $n=1000$, the convergence is quite evident.

This simulation reinforces the asymptotic normality of the OLS estimator even when we consider the randomness of the regressors. The key conditions for asymptotic normality are related to the properties of the population and the law of large numbers and central limit theorem applying to sample averages, which hold true whether we condition on regressors or not, as long as certain regularity conditions are met (like finite variance of $x$ and $u$, and exogeneity). This is fundamental because in most econometric applications, regressors are indeed random variables.

5.6 The Lagrange Multiplier (LM) Test¶

The Lagrange Multiplier (LM) test, also called the score test, is an asymptotic test used to test restrictions on parameters in regression models. In econometrics, it is commonly used to test for omitted variables or other forms of model misspecification.

Key Advantage: The LM test only requires estimation of the restricted model (under the null hypothesis), making it computationally attractive when the unrestricted model is complex.

Asymptotic Equivalence: Under standard regularity conditions, the LM test, Wald test, and Likelihood Ratio (LR) test are asymptotically equivalent - they have the same limiting distribution under $H_0$. In practice with large samples, they often give similar conclusions.

For testing restrictions in a linear regression model, the LM test statistic is often computationally simpler than the Wald or F tests, especially when the null hypothesis involves multiple restrictions on coefficients.

For testing $q$ restrictions of the form $H_0: R\beta = r$ in a linear regression, where $R$ is a $q \times (k+1)$ matrix and $r$ is a $q \times 1$ vector, the LM test statistic can be calculated as:

where:

• $n$ is the sample size.

• $R^2_{\tilde{u}}$ is the R-squared from a regression of the residuals from the restricted model ($\tilde{u}$) on all the independent variables in the unrestricted model.

• $q$ is the number of restrictions being tested (degrees of freedom).

• $\chi^2_q$ denotes a Chi-squared distribution with $q$ degrees of freedom.

The steps to perform an LM test are typically as follows:

1. Estimate the Restricted Model: Estimate the regression model under the null hypothesis. Obtain the residuals from this restricted model ($\tilde{u}$).

2. Auxiliary Regression: Regress the residuals $\tilde{u}$ from the restricted model on all the independent variables from the unrestricted model. Calculate the $R^2$ from this auxiliary regression ($R^2_{\tilde{u}}$).

3. Calculate the LM Statistic: Compute the LM test statistic as $LM = n \cdot R^2_{\tilde{u}}$.

4. Determine the p-value: Compare the LM statistic to a $\chi^2_q$ distribution, where $q$ is the number of restrictions. Calculate the p-value or compare the LM statistic to a critical value from the $\chi^2_q$ distribution to make a decision.

Example 5.3: Economic Model of Crime¶

We will use the crime1 dataset from the wooldridge package to illustrate the LM test. The example considers an economic model of crime where the number of arrests (narr86) is modeled as a function of several factors.

The unrestricted model is:

We want to test the null hypothesis that avgsen (average sentence length) and tottime (total time served) have no effect on narr86, i.e., $H_0: \beta_2 = 0 \text{ and } \beta_3 = 0$. Thus, we have $q=2$ restrictions.

The restricted model under $H_0$ is:

Interpretation of Example 5.3:

• LM Test Statistic: The calculated LM test statistic is 4.071.

• Critical Value: The critical value from the $\chi^2_2$ distribution at the 10% significance level is 4.605.

• P-value: The p-value for the LM test is 0.1306.

Since the LM test statistic (4.071) is less than the critical value (4.605), or equivalently, the p-value (0.1306) is greater than the significance level (0.10), we fail to reject the null hypothesis $H_0: \beta_2 = 0 \text{ and } \beta_3 = 0$. This suggests that avgsen and tottime are jointly statistically insignificant in explaining narr86, given the other variables in the model.

• Comparison with F-test: The F-test directly tests the same hypothesis using the unrestricted model. The F-statistic is 2.034 and the p-value is 0.1310.

The results from the LM test and the F-test are very similar in this case, leading to the same conclusion: we fail to reject the null hypothesis. In linear regression models, under homoskedasticity, the LM test, Wald test, and F-test are asymptotically equivalent for testing linear restrictions on the coefficients. In practice, especially with reasonably large samples, these tests often provide similar conclusions. The LM test is advantageous when estimating the unrestricted model is more complex or computationally intensive, as it only requires estimating the restricted model.

Chapter Summary¶

This chapter explored the asymptotic properties of OLS estimators, demonstrating that large-sample inference is more robust and widely applicable than the finite-sample methods covered in Chapter 4. Asymptotic theory allows us to make valid inferences even when classical assumptions like normality fail.

Key Concepts:

1. Consistency: OLS estimators converge in probability to true parameter values as $n \to \infty$ under minimal assumptions (MLR.1-MLR.4). Consistency requires correct specification (no omitted variable bias) but NOT normality or homoskedasticity. It’s the minimum we should expect from an estimator.

2. Asymptotic Normality: By the Central Limit Theorem, $\hat{\beta}_j$ follows an approximate normal distribution in large samples, even when errors are NOT normally distributed. This powerful result allows us to use t-tests, confidence intervals, and F-tests with non-normal data, provided the sample is large enough (typically $n \geq 30$).

3. Asymptotic Efficiency: Under the Gauss-Markov assumptions (MLR.1-MLR.5), OLS is asymptotically efficient among linear estimators. When heteroskedasticity is present (MLR.5 fails), OLS remains consistent but loses efficiency to Weighted Least Squares (WLS). However, we can still use OLS with robust standard errors for valid inference.

4. Normality Testing is Not Actionable: Testing whether errors are normally distributed rarely leads to useful actions. If we reject normality, we should rely on asymptotic inference anyway (which doesn’t require normality). If we fail to reject, it doesn’t confirm normality. Focus instead on specification, outliers, and using robust inference.

5. The Lagrange Multiplier (LM) Test: An asymptotic alternative to F-tests and Wald tests for testing parameter restrictions. The LM test only requires estimating the restricted model, making it computationally attractive. Under standard conditions, LM, Wald, and Likelihood Ratio tests are asymptotically equivalent.

Practical Implications:

For Inference:

• With $n \geq 30$, use asymptotic inference (standard normal critical values)

• Always use heteroskedasticity-robust standard errors (Chapter 8) as default

• Don’t worry about normality of errors - worry about specification

• LM tests are useful when unrestricted models are complex

For Consistency:

• Correct specification is crucial - no amount of data fixes omitted variable bias

• Consistency requires MLR.4 (zero conditional mean): $E(u|x_1, \ldots, x_k) = 0$

• Include all relevant variables or use instrumental variables (Chapter 15)

• Check for functional form misspecification (Chapter 6)

For Efficiency:

• OLS is efficient under homoskedasticity (MLR.5)

• Under heteroskedasticity, consider WLS or GLS for improved efficiency

• But OLS + robust SEs is a reasonable default strategy

• Don’t sacrifice consistency for efficiency

Simulation Insights:

Our simulations demonstrated three key asymptotic results:

1. Normal errors (Section 5.5.1): Even with normal errors, small samples deviate from asymptotic theory. Convergence to normality is clear by $n = 1000$.

2. Non-normal errors (Section 5.5.2): With chi-square errors (highly skewed), OLS estimates still converge to normality by $n = 1000$, confirming the CLT’s power.

3. Random regressors (Section 5.5.3): Asymptotic normality holds whether we condition on regressors or treat them as random, reflecting real-world applications where $X$ is random.

Comparison with Finite-Sample Theory (Chapter 4):

Looking Forward:

• Chapter 6 explores functional form, scaling, and model selection - all crucial for correct specification

• Chapter 8 covers heteroskedasticity in detail, including robust standard errors and efficiency considerations

• Chapter 15 introduces instrumental variables for handling endogeneity when MLR.4 fails

The Bottom Line:

Asymptotic theory is the workhorse of modern econometrics. It allows us to:

• Make valid inferences with non-normal, heteroskedastic data

• Justify the use of OLS in a wide variety of settings

• Understand when and why OLS might fail (inconsistency from omitted variables)

• Develop alternative estimators and tests (LM, GMM, IV)

Most importantly: Focus on getting the specification right (avoiding omitted variable bias, choosing correct functional form) rather than obsessing over classical assumptions like normality. With correct specification and large samples, OLS with robust standard errors is a reliable, general-purpose tool for empirical analysis.