# 08_Regression

Linear regression Bayesian solution Non-linear regression Kernelization and Gaussian processes Sparse linear regression Dual linear regression Relevance vector regression Applications

1.Computer vision: models, learning and inference Chapter 8 Regression

2.Structure 2 2 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Linear regression Bayesian solution Non-linear regression Kernelization and Gaussian processes Sparse linear regression Dual linear regression Relevance vector regression Applications

3.Models for machine vision 3 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

4.Body Pose Regression Encode silhouette as 100x1 vector, encode body pose as 55 x1 vector. Learn relationship 4 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

5.Type 1: Model Pr( w | x ) - Discriminative How to model Pr( w | x )? Choose an appropriate form for Pr( w ) Make parameters a function of x Function takes parameters q that define its shape Learning algorithm : learn parameters q from training data x , w Inference algorithm : just evaluate Pr( w|x ) 5 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

6.Linear Regression For simplicity we will assume that each dimension of world is predicted separately. Concentrate on predicting a univariate world state w. Choose normal distribution over world w Make Mean a linear function of data x Variance constant 6 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

7.Linear Regression 7 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

8.Neater Notation To make notation easier to handle, we Attach a 1 to the start of every data vector Attach the offset to the start of the gradient vector f New model: 8 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

9.Combining Equations We have one equation for each x,w pair: The likelihood of the whole dataset is the product of these individual distributions and can be written as where 9 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

10.Learning Maximum likelihood Substituting in Take derivative, set result to zero and re-arrange: 10 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

11.11 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

12.Regression Models 12 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

13.Structure 13 13 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Linear regression Bayesian solution Non-linear regression Kernelization and Gaussian processes Sparse linear regression Dual linear regression Relevance vector regression Applications

14.Bayesian Regression Likelihood Prior (We concentrate on f – come back to s 2 later!) Bayes rule’ 14 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

15.Posterior Dist. over Parameters where 15 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

16.Inference 16 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

17.Practical Issue Problem: In high dimensions, the matrix A may be too big to invert Solution: Re-express using Matrix Inversion Lemma 17 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Final expression: inverses are (I x I) , not (D x D)

18.Fitting Variance We’ll fit the variance with maximum likelihood Optimize the marginal likelihood (likelihood after gradients have been integrated out) 18 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

19.Structure 19 19 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Linear regression Bayesian solution Non-linear regression Kernelization and Gaussian processes Sparse linear regression Dual linear regression Relevance vector regression Applications

20.Regression Models 20 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

21.Non-Linear Regression GOAL: Keep the math of linear regression, but extend to more general functions KEY IDEA: You can make a non-linear function from a linear weighted sum of non-linear basis functions 21 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

22.Non-linear regression Linear regression: Non-Linear regression: where In other words, create z by evaluating x against basis functions, then linearly regress against z. 22 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

23.Example: polynomial regression 23 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince A special case of Where

24.Radial basis functions 24 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

25.Arc Tan Functions 25 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

26.Non-linear regression Linear regression: Non-Linear regression: where In other words, create z by evaluating x against basis functions, then linearly regress against z. 26 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

27.Maximum Likelihood 27 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Same as linear regression, but substitute in Z for X :

28.Structure 28 28 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince Linear regression Bayesian solution Non-linear regression Kernelization and Gaussian processes Sparse linear regression Dual linear regression Relevance vector regression Applications

29.Regression Models 29 Computer vision: models, learning and inference. ©2011 Simon J.D. Prince