特征子空间:主成分分析(PCA)和LDA,在图像特征和分类中如何提取可视化表示?在对象检测中如何确定物体在图像中的什么位置?在对象跟踪中如何确定下一帧的对象在哪里?本章将解答一系列问题。本章将介绍两种方法:“特征面”和“渔人面”;在特征子空间部分介绍PCA和FLD,并且查看最近供应商测试的结果,介绍最新方法:深脸法和小面法。

注脚

展开查看详情

1.Face Recognition and Feature Subspaces Computer Vision Jia-Bin Huang, Virginia Tech Many slides from Lana Lazebnik , Silvio Savarese , Fei-Fei Li, and D. Hoiem

2.Administrative stuffs Final project Proposal due Oct 27 (Thursday ) Submit via CANVAS Send a copy to Jia-Bin and Akrit via email HW 4 Due 11:59pm on Wed, November 2nd, 2016

3.The fifth module Face recognition Feature subspace: PCA and LDA Image features and categorization How to extract visual representation? Machine learning crash course What similarities are important ? Object detection Where is the object in the image? Object tracking Where is the object in the next frame?

4.This class: face recognition Two methods: “Eigenfaces” and “ Fisherfaces ” Feature subspaces: PCA and FLD Look at results from recent vendor test Recent method: DeepFace and FaceNet Look at interesting findings about human face recognition

5.Applications of Face Recognition Surveillance

6.Applications of Face Recognition Album organization: iPhoto 2009 http://www.apple.com/ilife/iphoto/

7.Can be trained to recognize pets! http://www.maclife.com/article/news/iphotos_faces_recognizes_cats

8.Facebook friend-tagging with auto-suggest

9.Face recognition: once you’ve detected and cropped a face, try to recognize it Detection Recognition “Sally”

10.Face recognition: overview Typical scenario: few examples per face, identify or verify test example Why it’s hard? changes in expression, lighting, age, occlusion , viewpoint Basic approaches (all nearest neighbor) Project into a new subspace (or kernel space) (e.g., “ Eigenfaces ”=PCA) Measure face features Make 3d face model, compare shape+appearance , e.g., Active Appearance Model (AAM)

11.Typical face recognition scenarios Verification : a person is claiming a particular identity; verify whether that is true E.g., security Closed-world identification : assign a face to one person from among a known set General identification : assign a face to a known person or to “unknown”

12.What makes face recognition hard? Expression

13.What makes face recognition hard? Lighting

14.What makes face recognition hard? Occlusion

15.What makes face recognition hard? Viewpoint

16.Simple idea for face recognition Treat face image as a vector of intensities Recognize face by nearest neighbor in database

17.The space of all face images When viewed as vectors of pixel values, face images are extremely high-dimensional 100x100 image = 10,000 dimensions Slow and lots of storage But very few 10,000-dimensional vectors are valid face images We want to effectively model the subspace of face images

18.The space of all face images Eigenface idea: construct a low-dimensional linear subspace that best explains the variation in the set of face images

19.Principal Component Analysis (PCA) Given: N data points x 1 , … , x N in R d We want to find a new set of features that are linear combinations of original ones: u ( x i ) = u T ( x i – µ ) ( µ : mean of data points) Choose unit vector u in R d that captures the most data variance Forsyth & Ponce, Sec. 22.3.1, 22.3.2

20.Principal Component Analysis Direction that maximizes the variance of the projected data: Projection of data point Covariance matrix of data The direction that maximizes the variance is the eigenvector associated with the largest eigenvalue of Σ (can be derived using Raleigh’s quotient or Lagrange multiplier) N N 1/N Maximize subject to || u ||=1

21.Implementation issue Covariance matrix is huge (M 2 for M pixels) But typically # examples << M Simple trick X is Mx N matrix of normalized training data Solve for eigenvectors u of X T X instead of XX T Then Xu is eigenvector of covariance XX T Need to normalize each vector of Xu into unit length

22.Eigenfaces (PCA on face images) Compute the principal components (“ eigenfaces ”) of the covariance matrix Keep K eigenvectors with largest eigenvalues Represent all face images in the dataset as linear combinations of eigenfaces Perform nearest neighbor on these coefficients M. Turk and A. Pentland , Face Recognition using Eigenfaces , CVPR 1991      

23.Eigenfaces example Training images x 1 ,…, x N

24.Eigenfaces example Top eigenvectors: u 1 ,…u k Mean: μ

25.Visualization of eigenfaces Principal component (eigenvector) u k μ + 3 σ k u k μ – 3 σ k u k

26.Representation and reconstruction Face x in “face space” coordinates: =

27.Representation and reconstruction Face x in “face space” coordinates: Reconstruction: = + µ + w 1 u 1 +w 2 u 2 +w 3 u 3 +w 4 u 4 + … = ^ x =

28.P = 4 P = 200 P = 400 Reconstruction After computing eigenfaces using 400 face images from ORL face database

29.Eigenvalues (variance along eigenvectors)

30.Note Preserving variance (minimizing MSE) does not necessarily lead to qualitatively good reconstruction. P = 200

31.Recognition with eigenfaces Process labeled training images (training) Find mean µ and covariance matrix Σ Find k principal components (eigenvectors of Σ ) u 1 ,… u k Project each training image x i onto subspace spanned by principal components: (w i1 ,…, w ik ) = ( u 1 T ( x i – µ ), … , u k T ( x i – µ )) Given novel image x (testing) Project onto subspace: (w 1 ,…, w k ) = ( u 1 T ( x – µ ), … , u k T ( x – µ )) Optional: check reconstruction error x – x to determine whether image is really a face Classify as closest training face in k-dimensional subspace ^ M. Turk and A. Pentland , Face Recognition using Eigenfaces , CVPR 1991

32.PCA General dimensionality reduction technique Preserves most of variance with a much more compact representation Lower storage requirements (eigenvectors + a few numbers per face) Faster matching What are the problems for face recognition?

33.Limitations Global appearance method: not robust to misalignment, background variation

34.Question Would PCA on image pixels work well as a general compression technique? P = 200

35.Limitations The direction of maximum variance is not always good for classification

36.A more discriminative subspace: FLD Fisher Linear Discriminants  “Fisher Faces” PCA preserves maximum variance FLD preserves discrimination Find projection that maximizes scatter between classes and minimizes scatter within classes Reference: Eigenfaces vs. Fisherfaces , Belheumer et al., PAMI 1997

37.Illustration of the Projection Poor Projection x1 x2 x1 x2 Using two classes as example: Good

38.Comparing with PCA

39.Variables N Sample images: c classes: Average of each class: Average of all data:

40.Scatter Matrices Scatter of class i: Within class scatter: Between class scatter:

41.Illustration x1 x2 Within class scatter Between class scatter

42.Mathematical Formulation After projection Between class scatter Within class scatter Objective: Solution: Generalized Eigenvectors Rank of W opt is limited Rank(S B ) <= |C|-1 Rank(S W ) <= N-C

43.Illustration x1 x2

44.Recognition with FLD Use PCA to reduce dimensions to N-C dim PCA space Compute within-class and between-class scatter matrices for PCA coefficients Solve generalized eigenvector problem Project to FLD subspace (c-1 dimensions) Classify by nearest neighbor   Note: x in step 2 refers to PCA coef ; x in step 4 refers to original data

45.Results: Eigenface vs. Fisherface Variation in Facial Expression, Eyewear, and Lighting Input: 160 images of 16 people Train: 159 images Test: 1 image With glasses Without glasses 3 Lighting conditions 5 expressions Reference: Eigenfaces vs. Fisherfaces , Belheumer et al., PAMI 1997

46.Eigenfaces vs. Fisherfaces Reference: Eigenfaces vs. Fisherfaces , Belheumer et al., PAMI 1997

47.1. Identify a Specific Instance Frontal faces Typical scenario: few examples per face, identify or verify test example What’s hard: changes in expression, lighting, age, occlusion , viewpoint Basic approaches (all nearest neighbor) Project into a new subspace (or kernel space) (e.g., “Eigenfaces”=PCA) Measure face features Make 3d face model, compare shape+appearance (e.g., AAM)

48.Large scale comparison of methods FRVT 2006 Report Not much (or any) information available about methods, but gives idea of what is doable

49.FVRT Challenge: interesting findings Left: Major progress since Eigenfaces Right: Computers outperformed humans in controlled settings (cropped frontal face, known lighting, aligned) Humans outperform greatly in less controlled settings (viewpoint variation, no crop, no alignment, change in age, etc.) False Rejection Rate at False Acceptance Rate = 0.001

50.FVRT Challenge Frontal faces FVRT2006 evaluation: computers win!

51.FVRT Challenge Frontal faces FVRT2006 evaluation: controlled illumination

52.FVRT Challenge Frontal faces FVRT2006 evaluation: uncontrolled illumination

53.Most recent research focuses on “faces in the wild”, recognizing faces in normal photos Classification: assign identity to face Verification: say whether two people are the same Important steps Detect Align Represent Classify State-of-the-art Face Recognizers

54.DeepFace : Closing the Gap to Human-Level Performance in Face Verification Taigman , Yang, Ranzato , & Wolf (Facebook, Tel Aviv), CVPR 2014 Following slides adapted from Daphne Tsatsoulis

55.Face Alignment 1. Detect a face and 6 fiducial markers using a support vector regressor (SVR) 2 . Iteratively scale, rotate, and translate image until it aligns with a target face 3 . Localize 67 fiducial points in the 2D aligned crop 4 . Create a generic 3D shape model by taking the average of 3D scans from the USF Human-ID database and manually annotate the 67 anchor points 5.Fit an affine 3D-to-2D camera and use it to direct the warping of the face

56.Train DNN classifier on aligned faces Architecture (deep neural network classifier) Two convolutional layers (with one pooling layer) 3 locally connected and 2 fully connected layers > 120 million parameters Train on dataset with 4400 individuals, ~1000 images each Train to identify face among set of possible people Verification is done by comparing features at last layer for two faces

57.DNN Architecture: Training Social Face Classification (SFC) dataset 4.4 million labeled faces of 4030 people from Facebook Each person has 800-1200 faces Testing images are the most recent 5% of a person’s images (using time-stamp) Train DNN using SFC Train it as a multi-class classification problem Test using the 5% of held-out data to find optimal parameter settings and network architecture

58.Face Verification: compare features at last layer of network for two faces Do two input instances belong to the same class (person)? Unsupervised Protocol Inner product Restricted Protocol Weighted χ 2 distance Unrestricted Protocol Siamese Network

59.Experiments: Datasets LFW Dataset 13,323 webphotos of 5749 celebrities divided into 6000 face pairs in 10 splits Performance based on mean recognition accuracy using A) Restricted Protocol: Only same and not-same labels are used in training B) Unrestricted Protocol: Given some identity information so many more training pairs can be added to the training set C) Unsupervised Protocol: No training whatsoever is performed on LFW images YTF Dataset 3425 YouTube videos of 1595 people, 5000 video pairs and 10 splits

60.Results: Labeled Faces in the Wild Dataset Performs similarly to humans! (note: humans would do better with uncropped faces) Experiments show that alignment is crucial (0.97 vs 0.88) and that deep features help (0.97 vs. 0.91)

61.OpenFace ( FaceNet ) FaceNet : A Unified Embedding for Face Recognition and Clustering, CVPR 2015 http://cmusatyalab.github.io/openface /

62.MegaFace Benchmark The MegaFace Benchmark: 1 Million Faces for Recognition at Scale , CVPR 2016

63.Face recognition by humans Face recognition by humans: 20 results (2005) Slides by Jianchao Yang

64.

65.

66.

67.

68.

69.

70.

71.

72.Result 17: Vision progresses from piecemeal to holistic

73.Result 17: Vision progresses from piecemeal to holistic

74.Things to remember PCA is a generally useful dimensionality reduction technique But not ideal for discrimination FLD better for discrimination, though only ideal under Gaussian data assumptions Computer face recognition works very well under controlled environments (since 2006) Also starting to perform at human level in uncontrolled settings (recent progress: better alignment, features, more data)

75.Next class Image categorization