10g Machine Learning: Isotonic Regression

Introduction to Isotonic Regression

In this section, Michael Perch introduces isotonic regression as a more flexible variant of linear regression that allows us to build models in a piecewise linear manner and encode monotonic constraints.

Linear Regression Recap

• Linear regression is a basic equation for multi-linear regression where we do a linear weighting of all predictor features plus an intercept term.

• The loss function for linear regression is the summation of the squared difference between the values at our training data and the values estimated by our model.

Motivation for Isotonic Regression

• The motivation behind isotonic regression is to cover a more flexible variant of linear regression that allows us to build models in a piecewise linear manner.

• Encoding monotonic constraints into isotonic regression is interesting because it allows us to incorporate physics or other types of information sources into our machines.

Isotonic Approach

• Isotonic regression breaks up the problem into many or at least a few linear model segments, allowing us to establish isotonic constraints (thresholds).

• Our problem becomes estimating a linear model between each one of the thresholds such that we minimize the error with regard to the training data.

Monotonic Constraint on Solution

• We can impose a monotonic constraint on the solution by setting estimates of response at each one of the isotonic constraints such that they go in order from lowest to highest. This means that slope must always be non-negative.

Solving for Isotonic Regression

In this section, the speaker discusses how to solve for the estimates of the response at each one of the isotonic constraints. They explain that they can move up and down and pick other values, but their loss function is the same as before with linear regression. They also discuss how they can minimize this loss function under the constraint.

Loss Function

• The loss function is the sum of squares of differences between actual values of response and estimates made along these lines.

• The model is flexible enough to fit and model those nonlinear behaviors.

Monotonic Increase

• Due to monotonic increase, we lose our closed form analytical solution.

• We must solve problems using some type of iterative manner in which we can minimize this loss function.

Downsides

• For high dimensional problems, there are a significant number of isotonic constraints across each one of the features.

• The number of parameters goes up K to M.

• There is a significant risk of overfitting for large K.

Hyperparameter Tuning

• K is our hyperparameter for this model.

• We can apply test train cross-validation workflow in order to assess how well we do with different numbers of K.

Building a Model

Test Data vs Training Data

• Test data is in blue (25%).

• Gray represents training data.

Results

• Model shown has Tonically increasing slope with no negative slope.

• Good fit to data with no systematic bias.

• Flexible enough to fit nonlinear behaviors.

Isotonic Regression

In this section, Michael Perch discusses isotonic regression and its application in machine learning. He explains how it extends linear regression to be more flexible and produces a piecewise model.

High Parameter Tuning Laws

• Michael sets up the model with training and testing data using simple cross-validation.

• Due to limited testing data, cyclicity occurs when everything between 2 and 35 isotonic constraints are used.

• Good performance is achieved with fewer isotonic constraints (around five).

Conclusion

• A full k-fold cross-validation would provide stronger statements about the results.

• Isotonic regression is a nice extension of linear regression that allows for producing a piecewise model.

• Having a hyperparameter related to the size of segments or number of segments is interesting from a machine learning perspective.

• Examples of workflows shown in the lecture are available on Github for further exploration.

Overall, this section provides an introduction to isotonic regression and its potential applications in machine learning.