1. Lecture  4:     Pixels  and  Filters   Professor  Fei-­‐Fei  Li   Stanford  Vision  Lab   Fei-Fei Li! Lecture 4- ! 1   30-­‐Sep-­‐14

2. What  we  will  learn  today?   •  Images  as  funcEons   •  Linear  systems  (ﬁlters)   •  ConvoluEon  and  correlaEon   Some  background  reading:   Forsyth  and  Ponce,  Computer  Vision,  Chapter  7   Fei-Fei Li! Lecture 4- ! 2   30-­‐Sep-­‐14

3. Images  as  func,ons   •  An  image  contains  discrete  number  of  pixels   –  A  simple  example   75   –  Pixel  value:   •  “grayscale”     (or  “intensity”):  [0,255]   231   148   Fei-Fei Li! Lecture 4- ! 3   2010.12.18

4. Images  as  func,ons   •  An  image  contains  discrete  number  of  pixels   –  A  simple  example   [90,  0,  53]   –  Pixel  value:   •  “grayscale”     (or  “intensity”):  [0,255]   •  “color”   –  RGB:  [R,  G,  B]   [249,  215,  203]   –  Lab:  [L,  a,  b]   –  HSV:  [H,  S,  V]   [213,  60,  67]   Fei-Fei Li! Lecture 4- ! 4   2010.12.18

5. Images  as  func,ons   •  An  Image  as  a  funcEon  f  from  R2  to  RM: •  f( x, y ) gives  the  intensity  at  posiEon  ( x, y ) •  Deﬁned  over  a  rectangle,  with  a  ﬁnite  range:   f: [a,b] x [c,d ] à [0,255] Domain   range   support   Fei-Fei Li! Lecture 4- ! 5   30-­‐Sep-­‐14

6. Images  as  func,ons   •  An  Image  as  a  funcEon  f  from  R2  to  RM: •  f( x, y ) gives  the  intensity  at  posiEon  ( x, y ) •  Deﬁned  over  a  rectangle,  with  a  ﬁnite  range:   f: [a,b] x [c,d ] à [0,255] Domain   range   support   ⎡ r ( x, y ) ⎤ •  A  color  image:     f ( x, y ) = ⎢⎢ g ( x, y ) ⎥⎥ ⎢⎣ b( x, y ) ⎥⎦ Fei-Fei Li! Lecture 4- ! 6   30-­‐Sep-­‐14

7. Images  as  discrete  func,ons   •  Images  are  usually  digital  (discrete):   –  Sample  the  2D  space  on  a  regular  grid   •  Represented  as  a  matrix  of  integer  values   pixel   Fei-Fei Li! Lecture 4- ! 7   30-­‐Sep-­‐14

8. Images  as  discrete  func,ons   Cartesian  coordinates   NotaEon  for     discrete     funcEons   Fei-Fei Li! Lecture 4- ! 8   30-­‐Sep-­‐14

9. What  we  will  learn  today?   •  Images  as  funcEons   •  Linear  systems  (ﬁlters)   •  ConvoluEon  and  correlaEon   Some  background  reading:   Forsyth  and  Ponce,  Computer  Vision,  Chapter  7   Fei-Fei Li! Lecture 4- ! 9   30-­‐Sep-­‐14

10. Systems  and  Filters   •  Filtering:   –  Form  a  new  image  whose  pixels  are  a   combinaEon  original  pixel  values   Goals:     -­‐ Extract  useful  informaEon  from  the  images   •   Features  (edges,  corners,  blobs…)   -­‐  Modify  or  enhance  image  properEes:     •     super-­‐resoluEon;  in-­‐painEng;  de-­‐noising   Fei-Fei Li! Lecture 4- ! 10   30-­‐Sep-­‐14

11. Super-­‐resoluEon   De-­‐noising   In-­‐painEng   Bertamio  et  al     Fei-Fei Li! Lecture 4- ! 11   30-­‐Sep-­‐14

12. 2D  discrete-­‐space  systems  (ﬁlters)   Fei-Fei Li! Lecture 4- ! 12   30-­‐Sep-­‐14

13. Filter  example  #1:  Moving  Average   •  2D  DS  moving  average  over  a  3  ×  3  window  of   neighborhood   h   1   1   1   1   1   1   1   1   1   1 ( f ∗ h)[m, n] = ∑ f [k , l ] h[m − k , n − l ] 9 k ,l Fei-Fei Li! Lecture 4- ! 13   30-­‐Sep-­‐14

14. Filter  example  #1:  Moving  Average   0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 90 90 0 0 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Courtesy  of  S.  Seitz   0 0 0 0 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ( f ∗ h)[m, n] = ∑ f [k, l] h[m − k, n − l] k,l Fei-Fei Li! Lecture 4- ! 14   30-­‐Sep-­‐14

15. Filter  example  #1:  Moving  Average   0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ( f ∗ h)[m, n] = ∑ f [k, l] h[m − k, n − l] k,l 15   Fei-Fei Li! Lecture 4- ! 15   1-­‐Oct-­‐14

16. Filter  example  #1:  Moving  Average   0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ( f ∗ h)[m, n] = ∑ f [k, l] h[m − k, n − l] k,l Fei-Fei Li! Lecture 4- ! 16   1-­‐Oct-­‐14

17. Filter  example  #1:  Moving  Average   0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 30 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ( f ∗ h)[m, n] = ∑ f [k, l] h[m − k, n − l] k,l Fei-Fei Li! Lecture 4- ! 17   1-­‐Oct-­‐14

18. Filter  example  #1:  Moving  Average   0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 30 30 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ( f ∗ h)[m, n] = ∑ f [k, l] h[m − k, n − l] k,l Fei-Fei Li! Lecture 4- ! 18   1-­‐Oct-­‐14

19. Filter  example  #1:  Moving  Average   0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 20 10 0 0 0 90 90 90 90 90 0 0 0 20 40 60 60 60 40 20 0 0 0 90 90 90 90 90 0 0 0 30 60 90 90 90 60 30 0 0 0 90 90 90 90 90 0 0 0 30 50 80 80 90 60 30 0 0 0 90 0 90 90 90 0 0 0 30 50 80 80 90 60 30 0 0 0 90 90 90 90 90 0 0 0 20 30 50 50 60 40 20 0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 30 20 10 0 0 90 0 0 0 0 0 0 0 10 10 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ( f ∗ h)[m, n] = ∑ f [k, l] h[m − k, n − l] k,l Source: S. Seitz Fei-Fei Li! Lecture 4- ! 19   1-­‐Oct-­‐14

20. Filter  example  #1:  Moving  Average   h[⋅ ,⋅ ] In  summary:   •  Replaces  each  pixel   1   1   1   with  an  average  of  its   1   1   1   neighborhood.   1   1   1   •  Achieve  smoothing   eﬀect  (remove  sharp   features)   Fei-Fei Li! Lecture 4- ! 20   30-­‐Sep-­‐14

21. Filter  example  #1:  Moving  Average   Fei-Fei Li! Lecture 4- ! 21   30-­‐Sep-­‐14

22. Filter  example  #2:  Image  SegmentaEon   •  Image  segmentaEon  based  on  a  simple   threshold:   255, Fei-Fei Li! Lecture 4- ! 22   30-­‐Sep-­‐14

23. ClassiﬁcaEon  of  systems   •  Amplitude  properEes   •  Linearity   •  Stability   •  InverEbility     •  SpaEal  properEes   •  Causality   •  Separability   •  Memory   •  Shii  invariance   •  RotaEon  invariance   Fei-Fei Li! Lecture 4- ! 23   30-­‐Sep-­‐14

24. Shii-­‐invariance   If                                                                                          then   for  every  input  image  f[n,m]  and  shiis  n0,m0       Fei-Fei Li! Lecture 4- ! 24   30-­‐Sep-­‐14

25. Is  the  moving  average  system  is  shii  invariant?       0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 20 10 0 0 0 90 90 90 90 90 0 0 0 20 40 60 60 60 40 20 0 0 0 90 90 90 90 90 0 0 0 30 60 90 90 90 60 30 0 0 0 90 90 90 90 90 0 0 0 30 50 80 80 90 60 30 0 0 0 90 0 90 90 90 0 0 0 30 50 80 80 90 60 30 0 0 0 90 90 90 90 90 0 0 0 20 30 50 50 60 40 20 0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 30 20 10 0 0 90 0 0 0 0 0 0 0 10 10 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Fei-Fei Li! Lecture 4- ! 25   30-­‐Sep-­‐14

26. Is  the  moving  average  system  is  shii  invariant?     Yes!   Fei-Fei Li! Lecture 4- ! 26   30-­‐Sep-­‐14

27. Linear  Systems  (ﬁlters)   •  Linear  ﬁltering:   –  Form  a  new  image  whose  pixels  are  a  weighted  sum  of   original  pixel  values   –   Use  the  same  set  of  weights  at  each  point     •  S  is  a  linear  system  (funcEon)  iﬀ  it  S  sa&sﬁes   superposiEon  property   Fei-Fei Li! Lecture 4- ! 27   30-­‐Sep-­‐14

28. Linear  Systems  (ﬁlters)   •  Is  the  moving  average  a  linear  system?   •  Is  thresholding  a  linear  system?   –  f1[n,m]  +  f2[n,m]  >  T   –  f1[n,m]  <  T   –  f2[n,m]<T   No!   Fei-Fei Li! Lecture 4- ! 28   30-­‐Sep-­‐14

29. LSI  (linear  shi\$  invariant)  systems   Impulse  response                     Fei-Fei Li!   Lecture 4- ! 29   30-­‐Sep-­‐14