Linear regressions

A basic introduction to Model I and Model II linear regressions:

• What they are,
• How they are different,
• Why they are different,
• And when to use them.

A brief history of Model II regressions.

• Who led the intellectual development of these regression techniques.
• Plus, a list of their seminal papers.

An index of downloadable files for use with MATLAB®.

• Model I regressions: normal (Y-on-X), reversed (X-on-Y) and weighted (wY-on-X).
• Model II regressions: major axis, geometric mean and least-squares-cubic.
• Summary of modifications made to these files.

Some rules of thumb to help decide which model regression to use:

• When to use Model I vs Model II.
• Within each type, which of the various models to use.

Testing Model I and Model II regressions:

• Evaluate the Model I linear regressions using data from Bevington and Robinson (2003)
  • Examine the results for standard and weighted regressions.
  • Download the data files – Table 6.1 and Table 6.2. ASCII text file format
• Compare both linear regression models.
  • Download the data file. ASCII text file format
  • Examine the results from the various regression models.
  • View graphs of the regression models.
• How to know which regression you are using.

For further reading regarding Model I and II regressions, see:

• Ricker (1973). Linear regressions in Fishery Research. J. Fish. Res. Board Can. 30: 409-434.
• Laws and Archie (1981). Appropriate use of regression analysis in marine biology. Marine Biology 65: 13-16.
• Bevington & Robinson (1992). Data Reduction and Error Analysis for the Physical Sciences, Second Edition, McGraw-Hill, Inc., New York.
• Sokal and Rohlf (1995). Biometry, 3rd edition. W.H. Freeman and Company, San Francisco, CA.
• Laws (1997). Mathematical Methods for Oceanographers. John Wiley and Sons, Inc., New York, NY.
• Bevington & Robinson (2003). Data Reduction and Error Analysis for the Physical Sciences, Third Edition, McGraw-Hill, Inc., New York.

What are linear regressions?

• Linear regression is a statistical method for determining the slope and intercept parameters for the equation of a line that “best fits” a set of data.
• The most common method for determining the “best fit” is to run a line through the centroid of the data (see below) and adjust the slope of the line such that the sum of the squares of the offsets between the line and the data is a minimum. Thus, this is called the “least squares” technique.
• The centroid is the point determined by the mean of the x-values and the mean of the y-values. Regardless of the model chosen, the line will pass through the centroid of the data. For weighted data sets, the line will pass through the weighted centroid.
• Among the various models, there are different methods for calculating the offsets between the line and the data points. Since many of these methods minimize the sum of the squares of the offsets, they are all called “least squares” techniques; because of this, the term “least squares” does not designate a specific nor a unique method.

How are Model I and Model II regressions different?

• In the case of Model I regressions, the offsets are measured parallel to one of the axes. For example, in the regression of Y-on-X (the most common regression technique), this would be parallel to the Y-axis. So, we fit the line by minimizing the sum of the squares of the y-offsets. For the X-on-Y regression, we would use the x-offsets measured parallel to the X-axis.
• For Model II regressions, the offsets are measured along a line perpendicular (or normal) to the regression line. Thus, to use Pearson’s term, the line is fit by minimizing the sum of the squares of the normal deviates.

Why are Model I and Model II regressions different?

• In the case of Model I regressions, X is the INDEPENDENT variable and Y is the DEPENDENT variable: X is frequently controlled by the experimenter (or known very precisely) and Y varies in response to the changes in X. One assumes little or no error in X and all regression error is attributed to measurement or other error in Y. The equation tells us how Y varies in response to changes in X.
• For Model II regressions, neither X nor Y is an INDEPENDENT variable but both are assumed to be DEPENDENT on some other parameter which is often unknown. Neither are “controlled”, both are measured, and both include some error. We do not seek an equation of how Y varies in response to a change in X, but rather we look for how they both co-vary in time or space in response to some other variable or process. There are several possible Model II regressions. Which one is used depends upon the specifics of the case. See Ricker (1973) or Sokal and Rohlf (1995, pp. 541-549) for a discussion of which may apply. For convenience, I have also compiled some rules of thumb.

March 2016 — revision 4.0

It has now been 8+ years since any edits were made to these scripts: time to check to see if there are any problems or issues due to the many updates to the Matlab code.

• Each of the 8 lsqfit scripts was checked for compatibility with Matlab 2014b and no errors were found.
• However, some of the scripts had calculated variables which were not needed for the results. The code for these unused parameters was deleted from the scripts to improve computational efficiency.
• All of the scripts were then error checked using the test data file and were found to produce the correct results.
• All of the lsqfit script headers were revised with the new revision date & info.
• The revised and updated scripts were then used to produce new graphs of the regression lines.

September 2007 — revision 3.1

I corrected an error in the y-intercept uncertainty calculation for the major axis algorithm:

• The equation was initially derived correctly in Kermack & Haldane (1950), but they introduced a typo when the equation was reprinted in a table of summary formulae at the end of their text. York (1966) copied this typo from the summary formulae and I coded my equations using York’s notation. The algorithm now gives the correct uncertainty for the intercept in all four quadrants.
• The Results Table was corrected as required. The single line data plots for the major axis calculations were corrected as well.
• Minor typos in the header information for the lsqcubic were corrected and the revision date was updated.

April 2007 — revision 3

Additional minor changes were made to the lsqfit algorithms:

• Minor typos in the header information in some of the files were corrected.
• Revision dates on each file were updated.
• Replaced GIF images of line plots with higher resolution JPG files.

January 2003 — revision 2

A minor change was made to the names of the Model I regressions:

• Lsqfitxi.m was renamed lsqfitx.m to make it more consistent with the names of the other algorithms.
• Minor changes were made to lsqbisec.m and lsqfitgm so that they are now compatible with this change.
• The headers in each file were expanded to provide more information about each algorithm.

January 2000 — revision 1

The m-files for the Model I regressions were renamed to better reflect their applications:

• Lsqfit1.m was renamed lsqfity.m since this regression minimizes the sum of the squares of the deviations in y.
• Lsqfit1i.m was renamed lsqfitxi.m since this regression minimizes the sum of the squares of the deviations in x and is inverted so that the slope is directly comparable to the slopes from the other regression algorithms.
• Lsqfit1w.m was renamed lsqfityw.m since this regression minimizes the weighted sum of the squares of the deviations in y.

At this time, it was also noticed that the standard deviations for the slope and intercept of the weighted model I regression, lsqfityw.m (formerly known as lsqfit1w.m), were considerably smaller than the standard deviations calculated by the other regression algorithms (see: results).

• The source code was checked against the original and found to be correct.
• The algorithm calculates the correct values according to the example given in Table 6.2 in Bevington and Robinson (1992).
• Until this situation is resolved, an alternate calculation is presented: lsqfityz.m. This algorithm uses the same equations for slope and intercept as used by lsqfit1w.m and lsqfityw.m, but calculates the standard deviations for these terms based upon the more general equations derived by York (1966). The revised values are more in line with the standard deviations for these terms calculated by the other methods.

The algorithms for the Model II regressions were not changed although the header information for these files was revised for the purposes of clarity and completeness.

A new Model II regression algorithm was added, known here as the “least squares bisector:”lsqbisec.m.

• Sprent and Dolby (1980) have objected to the ad hoc use of the geometric mean regression, because it is not a true unbiased expression for the functional relationship between X and Y.
• They have suggested that an equally strong case can be made for using the line that bisects the minor angle between the two model I regressions.
• While the difference in slope between the geometric mean regression and the “least squares bisector” are small and probably not statistically significant, this new “regression” line is included here for completeness.

And lastly, the test data file, data.txt, was edited and new results were calculated:

• A simple plot of the original raw data revealed that it was not a bivariate normal distribution. It contained numerous outliers and data clusters. These have been removed to give a cleaner looking dataset.
• The modified file is smaller, 110 points vs 139, and one can easily see the shape of the data ellipse.
• Graphs of the regression lines are now available as well.

Here are the results from the MATLAB® shell-script routines for fitting a line to the example datasets from Bevington and Robinson (2003):

Here are the results from my MATLAB® shell-script routines for fitting a line to the example dataset (data.txt) using the various linear regression models:

Footnotes to table:

1. All parameters were calculated with equations from Bevington and Robinson (1992).
2. The standard deviations for the slope and intercept of the weighted regression (lsqfityw.m) seem small when compared to the uncertainties calculated with the other regression equations.
3. Revised weighted model I regression: equations for the slope and intercept were again taken from Bevington and Robinson (1992), however, the standard deviations in these parameters are calculated according to equations derived by York (1966). The uncertainties calculated this way seem more in line with those calculated by the other regressions.
4. The least-squares-bisector algorithm follows the suggestion of Sprent and Dolby (1980) that in the case of model II regressions, an equally strong case can be made for the line that bisects the minor angle between the two model I regressions: Y-on-X and X-on-Y. The uncertainties in the slope and intercept are calculated according to the equations derived by York (1966) in a manner analogous to that used for the geometric mean regression.

Also, please note that:

• The answers in the table have not been rounded-off to the proper number of significant figures. Instead, I have chosen to list the results obtained with Matlab in standard precision mode so that one can compare these results to those from other programs without an undue emphasis on data truncation and round-off errors.
• Equations for the Y-on-X, X-on-Y, major axis, geometric mean and least-squares-bisector regressions all run through the centroid: (mean-x,mean-y).
• Likewise, the correlation coefficients for the Y-on-X, X-on-Y, major axis and geometric mean regressions are all the same. Thus, the correlation coefficient is properly interpreted as a measure of the linearity of the data and not how well the line fits the data.
• For the weighted linear regressions and the least squares cubic the correlation coefficients are different owing to the varying weights applied to each data point.

References:

• Bevington & Robinson (1992). Data Reduction and Error Analysis for the Physical Sciences, Second Edition, McGraw-Hill, Inc., New York.
• Sprent and Dolby (1980). The geometric mean functional relationship. Biometrics 36: 547-550.
• York (1966). Least-squares fitting of a straight line. Canad. J. Phys. 44: 1079-1086.
• Bevington & Robinson (2003). Data Reduction and Error Analysis for the Physical Sciences, Third Edition, McGraw-Hill, Inc., New York.