本套slides来自全美最强四大理工之一的弗吉尼亚理工大学，为计算机视觉课程授课所用。在有了对光与色彩了解的基础上，本章首先举例说明物体表面与光强的关系，并给出了光度立体的数学计算方法。从而介绍了空域滤波、频域滤波。滤波是一种将模板与图像匹配的方法，并引出模板和影像金字塔（Image Pyramids）的概念。以及像素与图像滤波，其应用范围:代表纹理，去噪和非线性图像滤波的相关知识的讲述。
- 134
- 0
- 0
注脚
展开查看详情
1.Image Filtering in Spatial domain Computer Vision Jia-Bin Huang, Virginia Tech
2.Administrative stuffs Any questions? Office hours - Jia-Bin (440 Whittemore Hall ) Friday at 11 AM - 12 PM (this week only) Friday at 3:00 – 4:00 PM (lectures, HW discussions) Friday at 4:00 – 5:00 PM (final project discussions) Office hours - Akrit (264 Whittemore Hall) Wed at 10:30 AM – 11:30 AM MATLAB t utorial session by Akrit Friday 3-4 PM, Whittemore Hall 340A Bring your laptop with MATLAB installed
3.Previous class: Light and Color Reflection models diffuse/specular reflectance, albedo Surface orientation and light intensity Color vision physics of light, trichromacy , color consistency, color spaces (RGB, HSV, Lab ) Object cast light and shadows to each other Interpret images from local differences
4.Reflection models Albedo: fraction of light that is reflected Determines color (amount reflected at each wavelength) Very low albedo (hard to see shape) Higher albedo Slide credit: Derek Hoiem
5.Reflection models Specular reflection: mirror-like Light reflects at incident angle Reflection color = incoming light color Slide credit: Derek Hoiem
6.Reflection models Diffuse reflection Light scatters in all directions (proportional to cosine with surface normal) Observed intensity is independent of viewing direction Reflection color depends on light color and albedo Slide credit: Derek Hoiem
7.Surface orientation and light intensity Amount of light that hits surface from distant point source depends on angle between surface normal and source 1 2 prop to cosine of relative angle Slide credit: Derek Hoiem
8.Application: Photometric S tereo Assume: A Lambertian object A local shading model (each point on a surface receives light only from sources visible at that point) A set of known light source directions A set of pictures of an object, obtained in exactly the same camera/object configuration but using different sources Orthographic projection Goal: reconstruct object shape and albedo S n ??? S 1 S 2 F&P 2 nd ed., sec. 2.2.4 Slide credit: S. Lazebnik
9.Example … Input Recovered normal field x y z Recovered surface model Recovered Albedo Slide credit: S. Lazebnik
10.Photometric Stereo N L 1 L 2 V L 3 Can write this as a matrix equation: Slide credit: N. Snavely
11.Solving the equations Intensity (Known) Light direction (Known ) Albedo x Surface normal (Unknown) How do we get the albedo and surface normal from G? Slide credit: N. Snavely
12.More than three lights Get better results by using more lights What’s the size of L T L ? Least squares solution: Solve for N, k d as before In MATLAB G = L\I; Slide credit: N. Snavely
13.Next three classes: three views of filtering Image filters in spatial domain Filter is a mathematical operation on values of each patch Smoothing, sharpening, measuring texture Image filters in the frequency domain Filtering is a way to modify the frequencies of images Denoising , sampling, image compression Templates and Image Pyramids Filtering is a way to match a template to the image Detection, coarse-to-fine registration Slide credit: Derek Hoiem
14.Today’s class Pixels and image filtering Application: representing textures Denoising and non-linear image filtering
15.Smoothing Sharpening Input https://en.wikipedia.org/wiki/Albert_Einstein_in_popular_culture#/media/File:Einstein_tongue.jpg Why should we care?
16.Why should we care? Image Pyramid Image interpolation/resampling Source: D Forsyth Source : N Snavely
17.Why should we care? Representing textures with filter banks LM filter bank. Code here
18.The raster image (pixel matrix) 0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99 0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91 0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92 0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95 0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85 0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33 0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74 0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93 0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99 0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97 0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
19.Image filtering Image filtering: for each pixel, compute function of local neighborhood and output a new value Same function applied at each position Output and input image are typically the same size 5 1 4 1 7 1 5 3 10 Local image data 7 Modified image data Some function Slide Credit: L. Zhang
20.Image filtering Linear filtering function is a weighted sum/difference of pixel values Really important! Enhance images Denoise , smooth, increase contrast, etc. Extract information from images Texture, edges, distinctive points, etc. Detect patterns Template matching Slide credit: Derek Hoiem 0.5 0.5 0 0 1 0 0 0 0 kernel 8 Modified image data Local image data 6 1 4 1 8 1 5 3 10
21.Given a camera and a still scene, how can you reduce noise? Take lots of images and average them! What’s the next best thing? Source: S. Seitz Question: Noise reduction
22.First attempt at a solution Let’s replace each pixel with an average of all the values in its neighborhood Assumptions: Expect pixels to be like their neighbors Expect noise processes to be independent from pixel to pixel 22 Slide credit: Kristen Grauman
23.1 1 1 1 1 1 1 1 1 Example: box f ilter Slide credit: David Lowe
24.0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 Image filtering Slide credit: S. Seitz
25.0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 Image filtering Slide credit: S. Seitz
26.0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 Image filtering Slide credit: S. Seitz
27.0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 Image filtering Slide credit: S. Seitz
28.0 10 20 30 30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 Image filtering Slide credit: S. Seitz
29.0 10 20 30 30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 ? Image filtering Slide credit: S. Seitz
30.0 10 20 30 30 50 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 ? Image filtering Slide credit: S. Seitz
31.0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 0 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 20 10 0 20 40 60 60 60 40 20 0 30 60 90 90 90 60 30 0 30 50 80 80 90 60 30 0 30 50 80 80 90 60 30 0 20 30 50 50 60 40 20 10 20 30 30 30 30 20 10 10 10 10 0 0 0 0 0 1 1 1 1 1 1 1 1 1 Image filtering Slide credit: S. Seitz
32.What does it do? Replaces each pixel with an average of its neighborhood Achieve smoothing effect (remove sharp features) 1 1 1 1 1 1 1 1 1 Box Filter Slide credit: David Lowe
33.Smoothing with box filter
34.Properties of smoothing filters Smoothing Values positive Sum to 1 constant regions same as input Amount of smoothing proportional to mask size Remove “high-frequency” components; “low-pass” filter 34 Slide credit: Kristen Grauman
35.Say the averaging window size is 2k+1 x 2k+1: Loop over all pixels in neighborhood around image pixel F[i,j] Attribute uniform weight to each pixel Now generalize to allow different weights depending on neighboring pixel’s relative position: Non-uniform weights Slide credit: Kristen Grauman Correlation filtering
36.Filtering an image: replace each pixel with a linear combination of its neighbors. The filter “kernel” or “mask” H [ u,v ] is the prescription for the weights in the linear combination. This is called cross-correlation, denoted Slide credit: Kristen Grauman Correlation filtering
37.0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a b c d e f g h i What is the result of filtering the impulse signal (image) F with the arbitrary kernel H ? ? Slide credit: Kristen Grauman Filtering an impulse signal
38.Convolution: Flip the filter in both dimensions (bottom to top, right to left) Then apply cross-correlation Notation for convolution operator F H Slide credit: Kristen Grauman Convolution
39.Convolution Cross-correlation For a Gaussian or box filter, how will the outputs differ? If the input is an impulse signal, how will the outputs differ? Slide credit: Kristen Grauman Convolution vs. correlation G=filter2(H,F); or G= imfilter (F,H); G=conv2(H,F);
40.Practice with linear filters 0 0 0 0 1 0 0 0 0 Original ? Slide credit: David Lowe (UBC)
41.Practice with linear filters 0 0 0 0 1 0 0 0 0 Original Filtered (no change) Slide credit: David Lowe (UBC)
42.Practice with linear filters 0 0 0 1 0 0 0 0 0 Original ? Slide credit: David Lowe (UBC)
43.Practice with linear filters 0 0 0 1 0 0 0 0 0 Original Shifted left By 1 pixel Slide credit: David Lowe (UBC)
44.Practice with linear filters Original 1 1 1 1 1 1 1 1 1 0 0 0 0 2 0 0 0 0 - ? (Note that filter sums to 1) Slide credit: David Lowe (UBC)
45.Practice with linear filters Original 1 1 1 1 1 1 1 1 1 0 0 0 0 2 0 0 0 0 - Sharpening filter Accentuates differences with local average Slide credit: David Lowe (UBC)
46.Sharpening Slide credit: David Lowe (UBC)
47.Other filters -1 0 1 -2 0 2 -1 0 1 Vertical Edge (absolute value) Sobel
48.Other filters -1 -2 -1 0 0 0 1 2 1 Horizontal Edge (absolute value) Sobel
49.Basic gradient filters 0 0 0 1 0 -1 0 0 0 0 1 0 0 0 0 0 -1 0 1 0 -1 or Horizontal Gradient Vertical Gradient 1 0 -1 or
50.Questions Write as filtering operations, plus some pointwise operations: +, -, .*,> Sum of four adjacent neighbors plus 1 Sum of squared values of 3x3 windows around each pixel: Center pixel value is larger than the average of the pixel values to the left and right:
51.Key properties of linear filters Linearity: filter(f 1 + f 2 ) = filter(f 1 ) + filter(f 2 ) Shift invariance: same behavior regardless of pixel location filter(shift(f)) = shift(filter(f)) Any linear, shift-invariant operator can be represented as a convolution Source: S. Lazebnik
52.More properties Commutative: a * b = b * a Conceptually no difference between filter and signal Associative: a * ( b * c ) = ( a * b ) * c Often apply several filters one after another: ((( a * b 1 ) * b 2 ) * b 3 ) This is equivalent to applying one filter: a * ( b 1 * b 2 * b 3 ) Distributes over addition: a * ( b + c ) = ( a * b ) + ( a * c ) Scalars factor out: ka * b = a * kb = k ( a * b ) Identity: unit impulse e = [0, 0, 1, 0, 0], a * e = a Source: S. Lazebnik
53.Spatially-weighted average 0.003 0.013 0.022 0.013 0.003 0.013 0.059 0.097 0.059 0.013 0.022 0.097 0.159 0.097 0.022 0.013 0.059 0.097 0.059 0.013 0.003 0.013 0.022 0.013 0.003 5 x 5, = 1 Slide credit: Christopher Rasmussen Important filter: Gaussian
54.Smoothing with Gaussian filter
55.Smoothing with box filter
56.Gaussian filters Remove “high-frequency” components from the image (low-pass filter) Images become more smooth Convolution with self is another Gaussian So can smooth with small-width kernel, repeat, and get same result as larger-width kernel would have Convolving two times with Gaussian kernel of width σ is same as convolving once with kernel of width Separable kernel Factors into product of two 1D Gaussians Slide credit: Kristen Grauman
57.What parameters matter here? Size of kernel or mask Note, Gaussian function has infinite support, but discrete filters use finite kernels σ = 5 with 10 x 10 kernel σ = 5 with 30 x 30 kernel Slide credit: Kristen Grauman Gaussian filters
58.What parameters matter here? Variance of Gaussian: determines extent of smoothing σ = 2 with 30 x 30 kernel σ = 5 with 30 x 30 kernel Gaussian filters Slide credit: Kristen Grauman
59.Separability of the Gaussian filter Source: D. Lowe
60.Separability example * * = = 2D filtering (center location only) Source: K. Grauman The filter factors into a product of 1D filters: Perform filtering along rows: Followed by filtering along the remaining column:
61.Separability Why is separability useful in practice? Separability means that a 2D convolution can be reduced to two 1D convolutions (one among rows and one among columns) What is the complexity of filtering an n × n image with an m × m kernel? O(n 2 m 2 ) What if the kernel is separable? O(n 2 m)
62.Some practical matters
63.How big should the filter be? Values at edges should be near zero important! Rule of thumb for Gaussian: set filter half-width to about 3 σ Practical matters
64.Practical matters What about near the edge? the filter window falls off the edge of the image need to extrapolate methods: clip filter (black) wrap around copy edge reflect across edge Source: S. Marschner
65.Practical matters methods (MATLAB): clip filter (black): imfilter (f, g, 0) wrap around: imfilter (f, g, ‘circular’) copy edge: imfilter (f, g, ‘replicate’) reflect across edge: imfilter (f, g, ‘symmetric’) Source: S. Marschner
66.Practical matters What is the size of the output? MATLAB: filter2(g, f, shape ) shape = ‘full’: output size is sum of sizes of f and g shape = ‘same’: output size is same as f shape = ‘valid’: output size is difference of sizes of f and g f g g g g f g g g g f g g g g full same valid Source: S. Lazebnik
67.Application: Representing Texture Source: Forsyth
68.Texture and Material http://www-cvr.ai.uiuc.edu/ponce_grp/data/texture_database/samples/
69.Texture and Orientation http://www-cvr.ai.uiuc.edu/ponce_grp/data/texture_database/samples/
70.Texture and Scale http://www-cvr.ai.uiuc.edu/ponce_grp/data/texture_database/samples/
71.What is texture? Regular or stochastic patterns caused by bumps, grooves, and/or markings
72.How can we represent texture? Compute responses of blobs and edges at various orientations and scales
73.Overcomplete representation: filter banks Code for filter banks: www.robots.ox.ac.uk/~vgg/research/texclass/filters.html scales orientations “Edges” “Bars” “Spots”
74.Filter banks Process image with each filter and keep responses (or squared/abs responses)
75.How can we represent texture? Measure responses of blobs and edges at various orientations and scales Idea 1: Record simple statistics (e.g., mean, std.) of absolute filter responses
76.Can you match the texture to the response? Mean abs responses Filters A B C 1 2 3
77.Representing texture by mean abs response Mean abs responses Filters
78.Representing texture Idea 2: take vectors of filter responses at each pixel and cluster them, then take histograms (more on this in coming weeks)
79.Denoising and Nonlinear Image F iltering Salt and pepper noise : contains random occurrences of black and white pixels Impulse noise: contains random occurrences of white pixels Gaussian noise : variations in intensity drawn from a Gaussian normal distribution Source: S. Seitz
80.Gaussian noise Mathematical model: sum of many independent factors Good for small standard deviations Assumption: independent, zero-mean noise Source: M. Hebert
81.Smoothing with larger standard deviations suppresses noise, but also blurs the image Reducing Gaussian noise
82.Reducing salt-and-pepper noise What’s wrong with the results? 3x3 5x5 7x7
83.Alternative idea: Median filtering A median filter operates over a window by selecting the median intensity in the window Is median filtering linear? Source: K. Grauman
84.Median filter Is median filtering linear? Let’s try filtering
85.Median filter What advantage does median filtering have over Gaussian filtering? Robustness to outliers Source: K. Grauman
86.Median filter Salt-and-pepper noise Median filtered Source: M. Hebert MATLAB: medfilt2(image, [h w])
87.Gaussian vs. median filtering 3x3 5x5 7x7 Gaussian Median
88.Other non-linear filters Weighted median (pixels further from center count less) Clipped mean (average, ignoring few brightest and darkest pixels) Bilateral filtering (weight by spatial distance and intensity difference) http://vision.ai.uiuc.edu/?p=1455 Image: Bilateral filtering
89.Bilateral Filters Edge preserving: weights similar pixels more Carlo Tomasi , Roberto Manduchi , Bilateral Filtering for Gray and Color Images , ICCV, 1998. Original Gaussian Bilateral spatial similarity (e.g., intensity)
90.Guided Image Filters Bilateral filters Guided filters Kaiming He, Jian Sun, Xiaou Tang, Guided Image Filtering. PAMI 2013 B = imguidedfilter (A,G );
91.Take-home messages Linear filtering is sum of dot product at each position Can smooth, sharpen, translate (among many other uses ) Gaussian filters Low pass filters , separability , variance Attend to details: filter size, extrapolation, cropping Application: representing textures Noise models and nonlinear image filters 1 1 1 1 1 1 1 1 1
92.Thank you Next class: Image Filters in Frequency Domain