申请试用
HOT
登录
注册
 
在域外数据(out-of-domain data)上的3D人体重建
0 点赞
0 收藏
0下载
白玉兰开源
/
发布于
/
20
人观看

议题简介

本次报告主要介绍当测试数据和训练数据出现domain gap较大时,如何精确的重建3D人体。当测试数据在相机参数、肢体比例、环境以及动作种类与训练数据差别很大时,现有的方法很难估计出准确的3D人体。针对此问题,我们的总体思路是在附加时序约束的情况下,在测试数据上对模型进行单次动态微调,以此减小domain gap。其中,通过引入时序约束可以有效防止模型过拟合。
此外,为了更好的结合单帧约束和时序约束,我们提出双层在线适配算法(BOA,Bilevel Online Adaptation),将多目标优化过程分解为权重探测和权重更新两个步骤。 我们以Human 3.6M训练集,以3DPW和MPI-INF-3DHP为测试集,验证了BOA能准确重建出域外数据的3D人体。

讲师简介

官善琰,上海交通大学人工智能研究院在读博士生。主要研究兴趣集中于单目3D人体重建。目前以第一作者身份在CVPR、AAAI发表会议论文2篇。

展开查看详情

1. Shanyan Guan* Jingwei Xu* Yunbo Wang† Bilevel Online Adaptation for Bingbing Ni† Xiaokang Yang Out-of-Domain Human Mesh MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong Reconstruction University, China

2. Introduction - SMPL A parametric model defined by shape and pose. + SMPL(𝛽, 𝜃) Shape 𝛽 Joint rotation matrix 𝜃 Loper M et al. SMPL: A skinned multi-person linear model. (TOG), 2015

3. Introduction - SMPL Obtaining 3D joints from the SMPL model. J=AxM M(𝛽, 𝜃) 3D joints J Loper M et al. SMPL: A skinned multi-person linear model. (TOG), 2015

4.Human Mesh Reconstruction -Applications

5. Applications Pose Transfer Yining et al. Yining et al. Dense Intrinsic Appearance Flow for Human Pose Transfer, CVPR 2019

6. Applications Digital Human Vladimir M et al. Human POSEitioning System, CVPR 2021

7. Applications Motion Capture Pavlakos et al. Expressive body capture: 3D hands, face, and body from a single image, CVPR 2019

8.Related Work

9. Related Work In-the-wild human mesh reconstruction Kanazawa et al., HMR, CVPR 2018

10. Related Work In-the-wild human mesh reconstruction Nikos et al., SPIN, CVPR 2018

11.Related Work Training set: Human 3.6M + COCO + MPI-INF-3DHP + LSP + HR-LSPET + MPII +3DPW… Human 3.6M COCO MPI-INF-3DHP LSP HR-LSPET MPII 3DPW How about out-of-domain data?

12.How about out-of-domain data? SPIN (Trained on Human 3.6M, Tested on 3DPW)

13.Why? The reason is some crucial characteristics of test data are different from training data. Bone Length (m) Focal Length (pixel) Camera Height (m) Human 3.6M 3DPW MPI-INF-3DHP

14.Goal Reconstructing 3D human model from out-of-domain data in an online fashion. Training on Test on Human3.6M 3DPW Mesh

15.Straightforward Solution Online adapting the model 𝜙 at test-time 1. First adapting the model weights 2. Then estimating 3D model Weight Update 𝑥! 𝜙!"# 𝜙! 𝑥! 𝜙! 𝑦! 𝐿!"#!$% = ||𝑗 − 𝚥|| ̂ & priors

16.The problems of straightforward Solution Problem 1: Lacking of 3D ground-truth leads to various estimation ambiguities. e.g., depth ambiguity 𝐿!"#!$% = ||𝑗 − 𝚥|| ̂ & Reference image GT Mesh Estimated Mesh

17.The problems of straightforward Solution Problem 2: Sequentially available images cannot access global knowledge of test domain. : learned distribution : GT distribution : seen samples : unseen samples Time step

18.Our Method Cooperating with temporal constraints to alleviate ambiguities. • Short-term motion constraint ℒ# = ||𝑚 0 % − 𝑚% ||)) , where 𝑚 0 % = 𝚥%̂ − 𝚥%&* ̂ , 𝑚% = 𝑗% − 𝑗%&* , 𝚥%̂ and 𝑗% are estimated and target 2D keypoints at step 𝑖. • Long-term teacher constraint ℒ#$ = ||𝑇%&' 𝑥 − ℳ(!"# (𝑥)||)) , 𝑇%&' is a teacher model at step 𝑖 − 1: 𝑇%&' = 𝛼 𝑇%&) + (1 − 𝛼)ℳ(!"# .

19. Our Method Bilevel Online adapting the model 𝜙 to prevent overfitting. BOA: 2. Weight update Weight(Line Update 7-8) 𝑥! 𝜙!$ 𝜙!"#Straightforward online 𝜙! 𝑥! 𝜙!"#adaptation 𝜙! 1. Weight probe (Line 5-6)

20.Quantitative evaluation on 3DPW PA-MPJPE (#PS) PA-MPJPE (#PH) 59.2 58.9 60 150 55.7 55 106.1 100 49.5 76.7 73.6 50 58.8 50 45 40 0 SPIN [26] Pose2Mesh [7] EFT [20] BOA SMPLify [5] HMR [21] HMMR [22] BOA Two evaluation protocols commonly used by previous works

21.Comparison with VIBE BOA VIBE

22.Ablation Studies Ablations on the number of optimization steps PA-MPJPE MPJPE

23.Ablation Studies Ablations on loss-metric correlations B1 B2 BOA (Ours) Temporal Bilevel Constraint B1 × × B2 × √ BOA(Ours) √ √

24.More Results

25.More Results

26.Thanks for listening! https://sites.google.com/view/humanmeshboa

0 点赞
0 收藏
0下载