Multiple Regression Analysis Detailed Example: Delivery Time Multiple Regression

The following paper carries a comprehensive regression analysis designed for modeling and interpreting the determinants that have impacts on delivery time, where of course the two most meaningful predictors are the number of cases delivered (x1) and distance to the delivery location (x2). The analysis fits an initial regression model and then goes on to check its validity through residual diagnostics: checking for normality, independence, constant variance, multicollinearity, and the presence of outliers and influential observations. If assumptions are violated, potential remedies are considered, such as refining the model or adding more variables. Following the two-step approach, this research outlines and addresses issues of positive autocorrelation, poor model fit, and influential observations in an attempt to find a more robust and better-predictive model. The step-by-step procedure used provides a detailed framework under which regression analysis and model improvement should be carried out, as was evidenced in the practical interpretation of diagnostic results.

Analysis Step 1

At the first step, the following model is fitted the data to explain the relationship between the delivery time (y) and the number of cases delivered (x1) and the distance between the center and the delivered place (x2).

(See the APPENDIX 1 for the MINITAB output)

The fitted regression equation is obtained as follows:

Delivery Time (y)= 2,34 + 1,62 Number of Cases (x1) + 0,0144 Distance, x2 (ft).

To be sure that the validity of the model fitted, we check the LINE assumptions. For this purpose, the following residual analysis is conducted:

Check for LINE assumptions related to the above fitted model:

1. Normality of errors:

When we look at the Normal Probability Plot (NPP) of (standardized) residuals, it can be seen that standardized residuals do not follow the reference line closely; also there seems to be an outlier observation, whose standardized residual is greater than 3.

To be sure that there is or not normality problem with the errors, we should conduct the most powerful normality test, called Shapiro-Wilk’s test. Note here that in MINITAB, we select Ryan-Joiner (RJ)which is similar to Shapiro-Wilk’s test. Following are the null and the alternative hypotheses for conducting the test.

H0: Errors are distributed normally

versus (vs.)

H1: Errors are not distributed normally

Since the p-value associated with the RJ test given in the NPP of standardized residuals is 0.044, which is less than the default Type-I error probability, α=0.05, we do reject H0, which states normality of errors. Hence, normality assumption of errors is failed; that is, errors do not follow normal distribution.

2. Constant error variance:

When we examine standardized residuals versus fitted values graphics above, there does not seem to be an error variance heterogeneity problem. In order to be sure, we need to conduct the test for equal variances, called Levene test (because normality assumption is not satisfied, we should not use the parametric Bartlett test here). The relevant hypotheses here are

H0: Errors have equal (or constant) variance

vs.

H1: Errors do not have equal (or constant) variance

However, when we attempt to test, MINITAB issues the following message:

“Test for Equal Variances: SRES1 versus Number of Cases, x1; Distance, x2 (ft)

• ERROR * Completion of computation impossible.”

indicating that since there are no replications in the observations, computation of this test is not possible.

3. Independence of Errors:

When we examine the standardized residuals versus observation order plot, we observe positive autocorrelation problem. To be sure, we need to test for the independence of errors. For this purpose, we use Durbin-Watson (D-W) test, where

H0: There is no autocorrelation between the errors (or ρ=0)

Vs.

H1: There is positive autocorrelation between the errors (or ρ>0)

MINITAB calculates the Durbin-Watson statistic as 1,16957. Here, n=25 and p=3, a

And the critical values at α=0.05 obtained from the D-W table are dL=1.206 and dU=1.550. Because the D-W test statistic is less than dL=1.206, we do reject H0 stating that there is no autocorrelation but there is positive autocorrelation.

4. Linearity:

The plot of standardized residual versus fitted values exhibits a curvature, which indicates a need for inclusion of squared terms and or interaction terms in the model. That is there seems to be lack-of-fit in the model hypothesized. To be sure we need to check also the existence of lack-of-fit analytically. The hypotheses for checking are

H0: There is no lack-of fit in the model.

Vs.

H1: There is lack-of-fit in the model.

However, MINITAB issues the following message:

“There are no replicates.

Minitab cannot do the lack of fit test based on pure error.”

indicating that because there are no replications, MINITAB cannot conduct this test.

5. No multicollinearity:

Because the VIF values associated with the parameter estimates are 3.118 which are less than 5 (or 10); we can say that there seems to be no multicollinearity problem in this model.

6. Outliers & Influential observations:

There is only one observation, the 9th one, which has standardized residual (SRES) = 3,21376 greater than 3. This observation can be considered as a potential outlier. To be sure that it is an outlier we can conduct t-test using Bonferoni approach, that is, take the significance level as α/(2*m). Here m, the number of potential outliers to be tested, is one. So, the critical value is t0.025,25–3–1 = 2.07961, and the externally studentized residual (TRES) = 4.31078 for the 9th observation. Because TRES= 4.31078 is greater than the critical value, which is t0.025,25–3–1 = 2.07961, we can be sure at α=0.05 level of significance that the 9th observation is an outlier.

On the other hand,

· When we compare the leverage values with 2p/n= 23/25= 0.24, the 9th and the 22nd observations seem to be leverage points.

· When the Cook distance, D, values are compared to 1 (one), there is only one observation, the 9th one, whose D=3.419, indicating that it is an influential point.

· When DFFIT values are compared with the cut-of-point value, which is

In this example, the cut-off point is

= 0.8729, and again only the 9th observation, whose DFFIT=4.296 seems to be the influential. Also, since the absolute value of DFFIT=abs (-1.1950) =1.1950 is greater than the cut-off point value, the 22nd observation is an influential point.

CONCLUSION:

Above residual analysis indicates that errors are not distributed normally. There is one outlier observation (the 9th one); besides, both the 9th and 22nd observations are influential observations. There is positive autocorrelation problem between errors. Also, there is a lack-of-fit problem. To overcome these problems, we may proceed as solving nonnormality by putting aside the outlier-influential observations and fitting the same model with the rest of 23 or 24 observations. Or alternatively, we first attempt to solve autocorrelation problem by including the observations’ order, or we may first improve the model by incorporating the second-order and interaction terms of the predictors in the model to overcome the lack-of-fit problem. Because the first approach will lead to data lost, which is not desirable, we first proceed to tackle with the autocorrelation problem.

APPENDIX 1: MINITAB Output of the Analysis Step 1

Regression Analysis: Delivery Tim versus Number of Ca; Distance, x2

Analysis Step 2

At the second step, the following model is fitted to the data to explain the relationship between the delivery time (y) and the number of cases delivered (x1), the distance between the center and the delivered place (x2)., and the observation order (x3) for the purpose of overcoming autocorrelation problem which exists between errors.

(See the APPENDIX 2 for the MINITAB output)

The fitted regression equation is obtained as follows:

Delivery Time (y) = 4,32 + 1,71 Number of Cases (x1) + 0,0130 Distance (x2) (ft)

• 0,174 Observation (x3)

To be sure that the validity of the model fitted, we check the LINE assumptions. For this purpose, the following residual analysis is conducted:

Check for LINE assumptions related to the above fitted model:

1. Normality of errors:

When we look at the Normal Probability Plot (NPP) of (standardized) residuals, it can be seen that standardized residuals do follow the reference line closely. Besides, there seems to be an outlier observation but since its standardized residual (SRES=2.95) is not greater than 3, we may not consider it as an outlier.

To be sure that there is or not normality problem with the errors, we should conduct the most powerful normality test, called Shapiro-Wilk test (or alternatively Ryan-Joiner (RJ) which is similar to Shapiro-Wilk’s test). Following are the null and the alternative hypotheses for conducting this test:

H0: Errors are distributed normally

versus (vs.)

H1: Errors are not distributed normally

Since the p-value associated with the RJ test given in the NPP of standardized residuals is greater than 0.1, which is also greater than the default Type-I error probability, α=0.05, we do not reject H0, and conclude that errors do follow normal distribution. Hence, normality assumption of errors is satisfied.

2. Constant error variance:

When we examine standardized residuals versus fitted values graphics above, there does not seem to be an nonconstant error variance problem. In order to be sure, we need to conduct the test for equal variances, called Bartlett test here, since the normality assumption is satisfied. The relevant hypotheses here are

H0: Errors have equal (or constant) variance

vs.

H1: Errors do not have equal (or constant) variance

However, when we attempt to test, MINITAB issues the following message:

“Test for Equal Variances: SRES1 versus Number of Cases, x1; Distance, x2 (ft)

• ERROR * Completion of computation impossible.”

indicating that due to the nonexistence of replicated observations, computation of this test is not possible.

3. Independence of Errors:

When we examine the standardized residuals versus observation order plot, we do not observe positive autocorrelation problem this time. To be sure, we need to test for the independence of errors using Durbin-Watson (D-W) test. Here,

H0: There is no autocorrelation between the errors (or ρ=0)

vs.

H1: There is positive autocorrelation between the errors (or ρ>0)

MINITAB calculates the Durbin-Watson (D-W) statistic as 1.30504. Here, n=25 and p=4, and the critical values at α=0.05 obtained from the D-W table are dL=1.123 and dU=1.654. Because the D-W test statistic is in between dL and dU, the test is inconclusive. In such cases, we should take a conservative approach and attempt to use the time series models. However, for the sake of this course, in such case, we assume that there is no autocorrelation problem, and thus conclude that by including the observations’ order variable in the model, we overcome the autocorrelation problem.

4. Linearity:

The plot of standardized residual versus fitted values exhibits a curvature, which indicates a need for inclusion of squared terms and or interaction terms in the model. That is, there seems to be lack-of-fit in the model hypothesized. To be sure, we need to check also the existence of lack-of-fit analytically. The hypotheses for checking are:

H0: There is no lack-of fit in the model.

vs.

H1: There is lack-of-fit in the model.

However, MINITAB issues the following message:

“There are no replicates.

Minitab cannot do the lack of fit test based on pure error.”

indicating that because there are no replications, MINITAB cannot conduct this test.

5. No multicollinearity:

Because the VIF values of all the parameter estimates are less than 5 (or 10), we can say that there is no multicollinearity problem in this model.

6. Outliers & Influential observations:

There seems to be an outlier observation but since its standardized residual (SRES=2.95) is not greater than 3, we may not consider it as an outlier. Thus, there is no outlier. To determine if the influential points exist,

· When we compare the leverage values with 2p/n= 2p/25= 0.32, there are two observations (the 9th and the 22nd ones) with high leverage values.

· When the Cook distance, D, values are compared to 1 (one), there is only one observation, the 9th one, whose D=2.63, indicating that it is an influential point.

· When DFFIT values are compared with the cut-of-point value, which is

CONCLUSION:

According to the residual analysis conducted above, there is only lack-of-fit problem with a few influential points (1st, 9thand 22th observations) in the above model developed. To overcome this problem, we may proceed by putting aside the two influential observations with respect to two of the measures (the 9th and 22nd ones) and fit the same model with the rest of 23 observations. Because strategically we are against loosing data (at least it not our first choice); therefore we may first improve the model by incorporating second-order and interaction terms of the predictors in the model to overcome the lack-of-fit problem.

APPENDIX 2: MINITAB Output of the Analysis Step 2

Regression Analysis: Delivery Tim versus Number of Ca; Distance, x2; …