Introduction to multiple regression

The principles of simple linear regression lay the foundation for more sophisticated regression models used in a wide range of challenging settings. In Chapter 9, we explore multiple regression, which introduces the possibility of more than one predictor in a linear model, and logistic regression, a technique for predicting categorical outcomes with two levels.

Multiple regression extends simple two-variable regression to the case that still has one response but many predictors (denoted $x_1$, $x_2$, $x_3$, ...). The method is motivated by scenarios where many variables may be simultaneously connected to an output.

We will consider data about loans from the peer-to-peer lender, Lending Club, which is a data set we first encountered in Chapters 1 and 2. The loan data includes terms of the loan as well as information about the borrower. The outcome variable we would like to better understand is the interest rate assigned to the loan. For instance, all other characteristics held constant, does it matter how much debt someone already has? Does it matter if their income has been verified? Multiple regression will help us answer these and other questions.

The data set loans includes results from 10,000 loans, and we’ll be looking at a subset of the available variables, some of which will be new from those we saw in earlier chapters. The first six observations in the data set are shown in Figure 9.1, and descriptions for each variable are shown in Figure 9.2. Notice that the past bankruptcy variable (bankruptcy) is an indicator variable, where it takes the value 1 if the borrower had a past bankruptcy in their record and 0 if not. Using an indicator variable in place of a category name allows for these variables to be directly used in regression. Two of the other variables are categorical (income_ver and issued), each of which can take one of a few different non-numerical values; we’ll discuss how these are handled in the model in Section 9.1.1.

Indicator and categorical variables as predictors

Let’s start by fitting a linear regression model for interest rate with a single predictor indicating whether or not a person has a bankruptcy in their record: $$\begin{aligned} \widehat{rate} &= 12.33 + 0.74{} \times bankruptcy \end{aligned}$$ Results of this model are shown in Figure 9.3.

Suppose we had fit a model using a 3-level categorical variable, such as income_ver. The output from software is shown in Figure 9.4. This regression output provides multiple rows for the income_ver variable. Each row represents the relative difference for each level of income_ver. However, we are missing one of the levels: not (for not verified). The missing level is called the reference level, and it represents the default level that other levels are measured against.

The notation used in Example 9.2 may feel a bit confusing. Let’s figure out how to use the equation for each level of the income_ver variable.

The higher interest rate for borrowers who have verified their income source or amount is surprising. Intuitively, we’d think that a loan would look less risky if the borrower’s income has been verified. However, note that the situation may be more complex, and there may be confounding variables that we didn’t account for. For example, perhaps lender require borrowers with poor credit to verify their income. That is, verifying income in our data set might be a signal of some concerns about the borrower rather than a reassurance that the borrower will pay back the loan. For this reason, the borrower could be deemed higher risk, resulting in a higher interest rate. (What other confounding variables might explain this counter-intuitive relationship suggested by the model?)

Including and assessing many variables in a model

The world is complex, and it can be helpful to consider many factors at once in statistical modeling. For example, we might like to use the full context of borrower to predict the interest rate they receive rather than using a single variable. This is the strategy used in multiple regression. While we remain cautious about making any causal interpretations using multiple regression on observational data, such models are a common first step in gaining insights or providing some evidence of a causal connection.

We want to construct a model that accounts not only for any past bankruptcy or whether the borrower had their income source or amount verified, but simultaneously accounts for all the variables in the data set: income_ver, debt_to_income, credit_util, bankruptcy, term, issued, and credit_checks. $$\begin{aligned} \widehat{\texttt{rate}} &= \beta_0 + \beta_1\times \texttt{income\_{}ver}_{\texttt{source\_{}only}} + \beta_2\times \texttt{income\_{}ver}_{\texttt{verified}} + \beta_3\times \texttt{debt\_{}to\_{}income} \\ &\qquad\ + \beta_4 \times \texttt{credit\_{}util} + \beta_5 \times \texttt{bankruptcy} + \beta_6 \times \texttt{term} \\ &\qquad\ + \beta_7 \times \texttt{issued}_{\texttt{Jan2018}} + \beta_8 \times \texttt{issued}_{\texttt{Mar2018}} + \beta_9 \times \texttt{credit\_{}checks} \end{aligned}$$ This equation represents a holistic approach for modeling all of the variables simultaneously. Notice that there are two coefficients for income_ver and also two coefficients for issued, since both are 3-level categorical variables.

We estimate the parameters $\beta_0$, $\beta_1$, $\beta_2$, ..., $\beta_9$ in the same way as we did in the case of a single predictor. We select $b_0$, $b_1$, $b_2$, ..., $b_9$ that minimize the sum of the squared residuals: $$\begin{aligned} SSE = e_1^2 + e_2^2 + \dots + e_{10000}^2 = \sum_{i=1}^{10000} e_i^2 = \sum_{i=1}^{10000} \left(y_i - \hat{y}_i\right)^2 \end{aligned}$$ where $y_i$ and $\hat{y}_i$ represent the observed interest rates and their estimated values according to the model, respectively. 10,000 residuals are calculated, one for each observation. We typically use a computer to minimize the sum of squares and compute point estimates, as shown in the sample output in Figure 9.5. Using this output, we identify the point estimates $b_i$ of each $\beta_i$, just as we did in the one-predictor case.

Example 9.11 describes a common issue in multiple regression: correlation among predictor variables. We say the two predictor variables are collinear (pronounced as co-linear) when they are correlated, and this collinearity complicates model estimation. While it is impossible to prevent collinearity from arising in observational data, experiments are usually designed to prevent predictors from being collinear.

Adjusted $\boldsymbol{R^2}$ as a better tool for multiple regression

We first used $R^2$ in Section 8.2 to determine the amount of variability in the response that was explained by the model: $$\begin{aligned} R^2 = 1 - \frac{\text{variability in residuals}} {\text{variability in the outcome}} = 1 - \frac{Var(e_i)}{Var(y_i)} \end{aligned}$$ where $e_i$ represents the residuals of the model and $y_i$ the outcomes. This equation remains valid in the multiple regression framework, but a small enhancement can make it even more informative when comparing models.

This strategy for estimating $R^2$ is acceptable when there is just a single variable. However, it becomes less helpful when there are many variables. The regular $R^2$ is a biased estimate of the amount of variability explained by the model when applied to a new sample of data. To get a better estimate, we use the adjusted $R^2$.

Because $k$ is never negative, the adjusted $R^2$ will be smaller – often times just a little smaller – than the unadjusted $R^2$. The reasoning behind the adjusted $R^2$ lies in the degrees of freedom associated with each variance, which is equal to $n - k - 1$ for the multiple regression context. If we were to make predictions for new data using our current model, we would find that the unadjusted $R^2$ would tend to be slightly overly optimistic, while the adjusted $R^2$ formula helps correct this bias.

Adjusted $R^2$ could have been used in Chapter 8. However, when there is only $k = 1$ predictors, adjusted $R^2$ is very close to regular $R^2$, so this nuance isn’t typically important when the model has only one predictor.

Exercises