申请试用
HOT
登录
注册
 
动态人体的三维建模与渲染技术
白玉兰开源
/
发布于
/
100
人观看
活动介绍

动态人体的三维重建与视角合成旨在从RGB视频中重建每一视频帧的三维人体模型,从而实现自由视角地观看视频。该技术在体育直播、虚拟主播和远程虚拟会议等任务中有广泛应用。本次报告将介绍两种基于神经辐射场从稀疏视角视频进行动态人体重建的方法。第一种方法在参数化人体模型上定义结构化的隐变量,从同一组隐变量中生成不同帧的神经辐射场,用于表示动态人体。第二种方法在标准坐标系上定义模版神经辐射场,通过蒙皮算法将各视频帧空间与标准空间建立对应关系。这两种方法整合了各视频帧的图片信息,从而实现了稀疏视角的三维重建。

彭思达,浙江大学CAD&CG国家重点实验室四年级博士研究生,导师为周晓巍研究员。研究方向为三维视觉,主要研究三维重建与视角合成。博士至今以一作身份在TPAMI、CVPR、ICCV等会议或期刊发表5篇论文,google scholar引用超过470次。在2020年获得CCF-CV学术新锐奖。在2021年一作论文入围CVPR Best Paper Candidates。发表论文均已开源,在GitHub上Star数超过2000次。

展开查看详情

1 .Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang Qing Shuai, Hujun Bao, Xiaowei Zhou Input: a sparse multi-view video Output: 3D geometry and appearance Frame 1 Frame 150 Frame 300 Novel view synthesis 3D reconstruction

2 .Problem statement: 什什么是novel view synthesis Input views Novel view synthesis Mildenhall, Ben, et al. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.

3 .Problem statement: 什什么是novel view synthesis Input views Novel view synthesis Mildenhall, Ben, et al. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.

4 .Novel view synthesis的应⽤用:Sports broadcasting 4DREPLAY. https://www.4dreplay.com/

5 .Novel view synthesis的应⽤用:Telepresence https://www.youtube.com/watch?v=Q13CishCKXY

6 .Novel view synthesis的应⽤用:Telepresence https://www.youtube.com/watch?v=Q13CishCKXY

7 . Related work Light field interpolation Image-based rendering Gortler, Davis, Levoy, Hanrahan, et al. Kalantari, Hedman, Choi, Wang, et al. Neural 3D representation NeRF-like works Sitzmann, Lombardi, Wu, Aliev, Thies, et al. Mildenhall,Yu, Trevithick, Liu, Reiser, et al.

8 .Related work: 2D CNN-based rendering Multi-view Neural Human Rendering. In CVPR, 2020.

9 . 当3d feature投影时,不不同视⻆角下, 同⼀一个feature的邻近feature可能会 不不同。 ➔ 2D CNN⽆无法保证渲染结果视⻆角 之间的连续性。 ➔ 在3D空间回归⽬目标物体,渲染 直接得到图⽚片 ! Multi-view Neural Human Rendering. In CVPR, 2020.

10 . Related work: RGB-alpha volume Multi-view images Encoder-decoder Volume rendering Neural Volumes: Learning Dynamic Renderable Volumes from Images. In SIGGRAPH, 2019.

11 . 回归3D volume,需要⼤大量量显存。 ➔ ⽆无法得到⾼高分辨率volume,⽆无法 渲染⾼高分辨率图⽚片。 ➔ ⽤用implicit function表示连续的三 维场景 ! Neural Volumes: Learning Dynamic Renderable Volumes from Images. In SIGGRAPH, 2019.

12 . Related work: Neural radiance field Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.

13 . Challenges for NeRF • Cannot handle dynamic scenes. • Require dense input views.

14 . Challenges for NeRF • Cannot handle dynamic scenes. • Require dense input views.

15 .我们的⽬目标: 从稀疏视⻆角的视频恢复⾃自由视⻆角视频 输⼊入:4-view video 输出:free viewpoint video

16 . 我们的⽬目标: 从稀疏视⻆角的视频恢复⾃自由视⻆角视频 Motivation: 整合时序信息来获得⾜足够的3d shape observation 输⼊入:4-view video 输出:free viewpoint video

17 .Key idea: 利利⽤用latent variable model整合时序信息 提出structured latent variables,从同⼀一组latent variables中⽣生成不不同帧的场景

18 . Overview of our method • Human motion capture from multi-view videos. • Structured latent codes. • Generate neural radiance fields from structured latent codes. Recover SMPLs

19 . Overview of our method • Human motion capture from multi-view videos. • Structured latent codes. • Generate neural radiance fields from structured latent codes.

20 . Overview of our method • Human motion capture from multi-view videos. • Structured latent codes. • Generate neural radiance fields from structured latent codes.

21 . Method: 1) Human motion capture 整合时序信息,需要在视频帧之间建⽴立关联。 ➔ 需要建⽴立correspondence。 ➔ 需要proxy geometry。 ➔ SMPL model !

22 .SMPL可以从稀疏视⻆角图⽚片中准确恢复 https://www.youtube.com/watch?v=kuBlUyHeV5U

23 . Method: 1) Human motion capture Capture human motion using https://github.com/zju3dv/EasyMocap Recover SMPLs

24 .Method: 2) Define structured latent codes on SMPL For each SMPL vertex, we assign a learnable latent code

25 . Learnable latent code是什什么 Auto-encoder Auto-decoder DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In CVPR 2019.

26 . Learnable latent code是什什么 Latent code可以被优 化来包含物体shape and appearance的信息 Auto-decoder DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In CVPR 2019.

27 .Method: 2) Define structured latent codes on SMPL Set the code locations according to the SMPL pose Reposing

28 .Method: 3) Generate scenes from structured latent codes

29 .How to generate continuous scenes from discrete latent codes

0 点赞
1 收藏
6下载
确认
3秒后跳转登录页面
去登陆