Checking model conditions using graphs

regression methods using the model $$\begin{aligned} \hat{y} &= \beta_0 + \beta_1x_1 + \beta_2x_2 + \cdots + \beta_kx_k \end{aligned}$$ generally depend on the following four conditions:

1. the residuals of the model are nearly normal (less important for larger data sets),

2. the variability of the residuals is nearly constant,

3. the residuals are independent, and

4. each variable is linearly related to the outcome.

Diagnostic plots

Diagnostic plots can be used to check each of these conditions. We will consider the model from the Lending Club loans data, and check whether there are any notable concerns: $$\begin{aligned} \widehat{rate} &= \ 1.921 + 0.974 \times \texttt{income\_{}ver}_{\texttt{source\_{}only}} + 2.535 \times \texttt{income\_{}ver}_{\texttt{verified}} \\ &\qquad + 0.021 \times \texttt{debt\_{}to\_{}income} + 4.896 \times \texttt{credit\_{}util} + 0.387 \times \texttt{bankruptcy} \\ &\qquad + 0.154 \times \texttt{term} + 0.228 \times \texttt{credit\_{}check} \end{aligned}$$

Check for outliers.

In theory, the distribution of the residuals should be nearly normal; in practice, normality can be relaxed for most applications. Instead, we examine a histogram of the residuals to check if there are any outliers: Figure 9.8 is a histogram of these outliers. Since this is a very large data set, only particularly extreme observations would be a concern in this particular case. There are no extreme observations that might cause a concern.

If we intended to construct what are called prediction intervals for future observations, we would be more strict and require the residuals to be nearly normal. Prediction intervals are further discussed in an online extra on the OpenIntro website:

www.openintro.org/d?id=stat_extra_linear_regression_supp

Absolute values of residuals against fitted values.

A plot of the absolute value of the residuals against their corresponding fitted values ($\hat{y}_i$) is shown in Figure 9.9. This plot is helpful to check the condition that the variance of the residuals is approximately constant, and a smoothed line has been added to represent the approximate trend in this plot. There is more evident variability for fitted values that are larger, which we’ll discuss further.

Residuals in order of their data collection.

This type of plot can be helpful when observations were collected in a sequence. Such a plot is helpful in identifying any connection between cases that are close to one another. The loans in this data set were issued over a 3 month period, and the month the loan was issued was not found to be important, suggesting this is not a concern for this data set. In cases where a data set does show some pattern for this check, time series methods may be useful.

Residuals against each predictor variable.

We consider a plot of the residuals against each of the predictors in Figure 9.10. For those instances where there are only 2-3 groups, box plots are shown. For the numerical outcomes, a smoothed line has been fit to the data to make it easier to review. Ultimately, we are looking for any notable change in variability between groups or pattern in the data.

Here are the things of importance from these plots:

• There is some minor differences in variability between the verified income groups.

• There is a very clear pattern for the debt-to-income variable. What also stands out is that this variable is very strongly right skewed: there are few observations with very high debt-to-income ratios.

• The downward curve on the right side of the credit utilization and credit check plots suggests some minor misfitting for those larger values.

Having reviewed the diagnostic plots, there are two options. The first option is to, if we’re not concerned about the issues observed, use this as the final model; if going this route, it is important to still note any abnormalities observed in the diagnostics. The second option is to try to improve the model, which is what we’ll try to do with this particular model fit.

Options for improving the model fit

There are several options for improvement of a model, including transforming variables, seeking out additional variables to fill model gaps, or using more advanced methods that would account for challenges around inconsistent variability or nonlinear relationships between predictors and the outcome.

The main concern for the initial model is that there is a notable nonlinear relationship between the debt-to-income variable observed in Figure 9.10. To resolve this issue, we’re going to consider a couple strategies for adjusting the relationship between the predictor variable and the outcome.

Let’s start by taking a look at a histogram of debt_to_income in Figure 9.11. The variable is extremely skewed, and upper values will have a lot of leverage on the fit. Below are several options:

• log transformation ($\log{x}$),

• square root transformation ($\sqrt{x}$),

• inverse transformation ($1 / x$),

• truncation (cap the max value possible)

If we inspected the data more closely, we’d observe some instances where the variable takes a value of 0, and since $\log(0)$ and $1 / x$ are undefined when $x = 0$, we’ll exclude these transformations from further consideration.¹ A square root transformation is valid for all values the variable takes, and truncating some of the larger observations is also a valid approach. We’ll consider both of these approaches.

To try transforming the variable, we make two new variables representing the transformed versions:

Square root.

We create a new variable, sqrt_debt_to_income, where all the values are simply the square roots of the values in debt_to_income, and then refit the model as before. The result is shown in the left panel of Figure 9.12. The square root pulled in the higher values a bit, but the fit still doesn’t look great since the smoothed line is still wavy.

Truncate at 50.

We create a new variable, debt_to_income_50, where any values in debt_to_income that are greater than 50 are shrunk to exactly 50. Refitting the model once more, the diagnostic plot for this new variable is shown in the right panel of Figure 9.12. Here the fit looks much more reasonable, so this appears to be a reasonable approach.

The downside of using transformations is that it reduces the ease of interpreting the results. Fortunately, since the truncation transformation only affects a relatively small number of cases, the interpretation isn’t dramatically impacted.

As a next step, we’d evaluate the new model using the truncated version of debt_to_income, we would complete all the same procedures as before. The other two issues noted while inspecting diagnostics in Section 9.3.1 are still present in the updated model. If we choose to report this model, we would want to also discuss these shortcomings to be transparent in our work. Depending on what the model will be used for, we could either try to bring those under control, or we could stop since those issues aren’t severe. Had the non-constant variance been a little more dramatic, it would be a higher priority. Ultimately we decided that the model was reasonable, and we report its final form here: $$\begin{aligned} \widehat{rate} &= \ 1.562 + 1.002 \times \texttt{income\_{}ver}_{\texttt{source\_{}only}} + 2.436 \times \texttt{income\_{}ver}_{\texttt{verified}} \\ &\qquad + 0.048 \times \texttt{debt\_{}to\_{}income\_{}50} + 4.694 \times \texttt{credit\_{}util} + 0.394 \times \texttt{bankruptcy} \\ &\qquad + 0.153 \times \texttt{term} + 0.223 \times \texttt{credit\_{}check} \end{aligned}$$ A sharp eye would notice that the coefficient for debt_to_income_50 is more than twice as large as what the coefficient had been for the debt_to_income variable in the earlier model. This suggests those larger values not only were points with high leverage, but they were influential points that were dramatically impacting the coefficient.

Don’t report results when conditions are grossly violated. While there is a little leeway in model conditions, don’t go too far. If model conditions are very clearly violated, consider a new model, even if it means learning more statistical methods or hiring someone who can help. To help you get started, we’ve developed a couple additional sections that you may find on OpenIntro’s website. These sections provide a light introduction to what are called interaction terms

and to fitting nonlinear curves to data, respectively:

www.openintro.org/d?file=stat_extra_interaction_effects
www.openintro.org/d?file=stat_extra_nonlinear_relationships

Exercises