展开查看详情
1 . Shanyan Guan* Jingwei Xu* Yunbo Wang†
Bilevel Online Adaptation for Bingbing Ni† Xiaokang Yang
Out-of-Domain Human Mesh MoE Key Lab of Artificial Intelligence,
AI Institute, Shanghai Jiao Tong
Reconstruction University, China
2 . Introduction - SMPL
A parametric model defined by shape and pose.
+ SMPL(𝛽, 𝜃)
Shape 𝛽 Joint rotation matrix 𝜃
Loper M et al. SMPL: A skinned multi-person linear model. (TOG), 2015
3 . Introduction - SMPL
Obtaining 3D joints from the SMPL model.
J=AxM
M(𝛽, 𝜃) 3D joints J
Loper M et al. SMPL: A skinned multi-person linear model. (TOG), 2015
4 .Human Mesh Reconstruction
-Applications
5 . Applications
Pose Transfer
Yining et al.
Yining et al. Dense Intrinsic Appearance Flow for Human Pose Transfer, CVPR 2019
6 . Applications
Digital Human
Vladimir M et al. Human POSEitioning System, CVPR 2021
7 . Applications
Motion Capture
Pavlakos et al. Expressive body capture: 3D hands, face, and body from a single image, CVPR 2019
9 . Related Work
In-the-wild human mesh reconstruction
Kanazawa et al., HMR, CVPR 2018
10 . Related Work
In-the-wild human mesh reconstruction
Nikos et al., SPIN, CVPR 2018
11 .Related Work
Training set: Human 3.6M + COCO + MPI-INF-3DHP + LSP + HR-LSPET + MPII +3DPW…
Human 3.6M COCO MPI-INF-3DHP LSP HR-LSPET MPII 3DPW
How about out-of-domain data?
12 .How about out-of-domain data?
SPIN (Trained on Human 3.6M, Tested on 3DPW)
13 .Why?
The reason is some crucial characteristics of test data are different
from training data.
Bone Length (m) Focal Length (pixel) Camera Height (m)
Human 3.6M 3DPW MPI-INF-3DHP
14 .Goal
Reconstructing 3D human model from out-of-domain data in an
online fashion.
Training on Test on
Human3.6M 3DPW Mesh
15 .Straightforward Solution
Online adapting the model 𝜙 at test-time
1. First adapting the model weights 2. Then estimating 3D model
Weight Update
𝑥! 𝜙!"# 𝜙! 𝑥! 𝜙! 𝑦!
𝐿!"#!$% = ||𝑗 − 𝚥||
̂ &
priors
16 .The problems of straightforward Solution
Problem 1: Lacking of 3D ground-truth leads to various
estimation ambiguities.
e.g., depth ambiguity
𝐿!"#!$% = ||𝑗 − 𝚥||
̂ &
Reference image GT Mesh Estimated Mesh
17 .The problems of straightforward Solution
Problem 2: Sequentially available images cannot access global
knowledge of test domain.
: learned distribution
: GT distribution
: seen samples
: unseen samples
Time step
18 .Our Method
Cooperating with temporal constraints to alleviate ambiguities.
• Short-term motion constraint
ℒ# = ||𝑚 0 % − 𝑚% ||)) ,
where 𝑚 0 % = 𝚥%̂ − 𝚥%&*
̂ , 𝑚% = 𝑗% − 𝑗%&* , 𝚥%̂ and 𝑗% are
estimated and target 2D keypoints at step 𝑖.
• Long-term teacher constraint
ℒ#$ = ||𝑇%&' 𝑥 − ℳ(!"# (𝑥)||)) ,
𝑇%&' is a teacher model at step 𝑖 − 1:
𝑇%&' = 𝛼 𝑇%&) + (1 − 𝛼)ℳ(!"# .
19 . Our Method
Bilevel Online adapting the model 𝜙 to prevent overfitting.
BOA: 2. Weight update
Weight(Line
Update
7-8)
𝑥! 𝜙!$
𝜙!"#Straightforward online 𝜙!
𝑥! 𝜙!"#adaptation 𝜙!
1. Weight probe
(Line 5-6)
20 .Quantitative evaluation on 3DPW
PA-MPJPE (#PS) PA-MPJPE (#PH)
59.2 58.9
60 150
55.7
55 106.1
100
49.5 76.7 73.6
50 58.8
50
45
40 0
SPIN [26] Pose2Mesh [7] EFT [20] BOA SMPLify [5] HMR [21] HMMR [22] BOA
Two evaluation protocols commonly used by previous works
21 .Comparison with VIBE
BOA VIBE
22 .Ablation Studies
Ablations on the number of optimization steps
PA-MPJPE MPJPE
23 .Ablation Studies
Ablations on loss-metric correlations
B1
B2
BOA (Ours)
Temporal
Bilevel
Constraint
B1 × ×
B2 × √
BOA(Ours) √ √
26 .Thanks for listening!
https://sites.google.com/view/humanmeshboa
热门城市
