Gerard Pons-Moll

Affiliated Researcher

Perceiving Systems

Office: N3.009
Max-Planck-Ring 4
72076 Tübingen
Germany

+49 7071 601 1835

NOTE: In September 2017 I moved to MPI-Informatics in Saarbrücken where I am heading the group "Real Virtual Humans".

For recent news, job offers, projects and publications, please visit my research group website.

I continue to collaborate with MPI-PS and I am an affilated researcher. This website however is not up to date.

News:

NEW! Awarded a starting grant (1.25 million euros for 5 years) from the ZD.B. to establish a research group on the topic Real Virtual Humans (RVHu). 7 proposals out 82 were accepted, 8.5%. RVHu was ranked 1st. Declined due to private reasons.
NEW! Günter Enderle Best Paper Award at Eurographics 2017 for the paper Sparse Inertial Poser, see paper
NEW! Two papers accepted at SIGGRAPH 2017! papers are up, videos coming soon!
NEW! I will organize with Jonatahn Taylor the ICCV 2017 Workshop "PeopleCap: capturing and modelling human bodies, faces and hands". Submission deadline: July 19th.
NEW! Lecturer at the Center for Learning Systems (MPI-ETH) research summer school. School is open to all Master students or recently graduated. Application deadline is April 30!
NEW! Two papers accepted at CVPR 2017! 1 oral and 1 spotlight.
NEW! Motion Capture from only 6 IMUs! Check our Eurographics paper
NEW! Keynote speaker at workshop on social virtual interaction
If you are at SIGGRAPH 2016 do not miss our tutorial on human body modelling! Modeling Human Bodies
I will co-organize a tutorial at ICCV 2015 on how to build a digital human together with my colleagues in the body group! Modelling Human Bodies
Press coverage for Dyna, our recent SIGGRAPH 2015 paper: wired, theverge, phys.org
New paper at CVPR 2014, Posebits for Monocular Human Pose Estimation
Best Science Paper Award BMVC 2013 for the paper Metric Regression Forests for Human Pose Estimation
Research intern at Microsoft Research, Cambridge working on Human Pose Estimation from Kinect, July 2012 - September 2012
Research stay at the Vision Group at the University of Toronto with Prof. David Fleet from January until June 2012.
Slides of ICCV tutorial Looking at People : Generative Human Pose Estimation are available for download, pptx_and_videos , PDF_Slides
Book chapter on Model Based Pose Estimation together with Bodo Rosenhahn, to appear in Guide to Visual Analysis of Humans : Looking at People. T. Moeslund, A. Hilton, V. Krueger, L. Sigal (eds.), Springer, 2011

Education:

2015 Sept.- Research scientist at Max Planck Instititute for Intelligent Systems.
2013 Sept. Postodctoral researcher at Max Planck Instititute for Intelligent Systems.
2012 Summer research internship at Microsoft Research Cambridge, hosts: Jamie Shotton and Andrew Fitzgibbon
2012 Visiting researcher at University of Toronto (UofT), Canada, host David J. Fleet
2009-2013 PhD Student, Leibniz University of Hannover, TNT lab, supervisor Bodo Rosenhahn
2007-2008 Master Thesis at Northeastern University, Boston, USA
2007 Telecomunications Engineering at the Technical University of Catalonia, UPC

Teaching:

Lectures take place at the University of Tübingen and can be attended by all Master and PhD students of the University and MPI Tübingen.

Summer term 2016: Computer Vision SS16 (jointly with Andreas Geiger and Javier Romero)

Current Ph.D interns

Sivaram Prasad Muduni

December 2016 - present

Past Ph.D interns

Kim Meekyoung Chao Zhang

June 2016 - December 2016 July 2016 - October 2016

The lab for computer vision at the Max Planck for Informatics in Saarbrücken has 2 Open PhD positions on the areas of

Human and cloth capture from images, videos and 3D scans
Deep learning for generative models of humans in clothing
3D human pose estimation and tracking using deep learning
Motion capture and performance capture
Reconstructing human motion from sparse Inertial Measurement Units (IMU)

Candidates should have:

Strong mathematical background
Strong programming skills one of (C++, Python, matlab)
Experience working with 3D meshes and knowledge on 3D tools is desired but a not must (e.g. OpenGL)
Knowledge/experience in deep learning

PhD students will work on cutting edge research at the intersection between vision, graphics and learning. Research will require knowledge of deep learning, geometry processing and computer vision.

Open positions for research internships on similar areas as above. Research interns are typically enrolled in a PhD program and stay for 6 months.

Contact me for details.

3D Body Shape and Pose from Images

Much of the field has focused on estimating 2D joints, 3D joints, or the skeleton of the body. We focus on estimating the full 3D shape and pose. This is crucial for reasoning about interactions. Having the ability to do so from RGB images enables markerless motion capture and provides the foundation for h...

Federica Bogo Angjoo Kanazawa Christoph Lassner Peter Vincent Gehler Javier Romero Michael Black Gerard Pons-Moll Martin Kiefel Yinghao Huang Ijaz Akhter

3D Pose from Images

The estimation of 3D human pose from 2D images is inherently ambiguous. To that end, we develop inference methods and human pose models that enable prediction of 3D pose from images. Learned models of human pose rely on training data but we find that existing motion capture datasets are too limited to explore the full rang...

Gerard Pons-Moll Michael Black Ijaz Akhter

4D Shape

Human bodies are dynamic; they deform as they move, jiggle due to soft-tissue dynamics, and change shape with respiration. In \cite{Tsoli:SIGGRAPH:2014} we learn a model of body shape deformations due to breathing for different breathing types and provide simple animation controls to render lifelike br...

Gerard Pons-Moll Javier Romero Naureen Mahmood Matthew Loper Aggeliki Tsoli Michael Black

Clothing Capture and Animation

While our detailed models of the body capture important aspects of body shape, they are missing something important -- clothing. Most people appear in images in clothing and to analyze this we seek models of the body and clothing. Modeling clothing is hard, however, because of the variety of garments, vari...

Gerard Pons-Moll Sergi Pujades Christoph Lassner Peter Vincent Gehler Michael Black

Clothing Models (2011-2015)

Clothed virtual characters in varied sizes and shapes are necessary for film, gaming, and on-line fashion applications. Dressing such characters is a significant bottleneck, requiring manual effort to design clothing, position it on the body, and simulate its physical deformation. Our goal is to do for clothing ...

Gerard Pons-Moll Michael Black David Hirshberg Alex Weiss

IMU-based Human Motion Capture Systems

Marker-based optical motion capture (mocap) systems are intrusive and restrict motions to controlled laboratory spaces. Therefore, simple daily activities like biking, or having coffee with friends cannot be recorded with such systems. To address this, and record human motion in everyday natural situations...

Gerard Pons-Moll Michael Black Yinghao Huang

Multi-Camera Capture

While multi-camera video data facilitates markerless motion capture, many challenges remain.

We formulate the problem of 3D human pose estimation and tracking as inference in a graphical model \cite{Sigal:IJCV:11}. The body is modeled as a collection o...

Michael Black Jürgen Gall Gerard Pons-Moll

Physics of Body Shape and Motion

Humans live in a real world governed by the laws of physics; that is, we apply and exploit forces, such as gravity, in our daily interactions with the world. In this project we allow virtual humans to interact with a virtual world subject to the laws of physics. How would one's body shape deform in cas...

Michael Black Sergi Pujades Meekyoung Kim Gerard Pons-Moll Javier Romero Ludovic Righetti

Pose and Motion Priors

A prior over human pose is important for many human tracking and pose estimation problems.

We introduce a sparse Bayesian network model of human pose that is non-parametric with respect to the estimation of both its graph structu...

Peter Vincent Gehler Andreas Lehrmann Ijaz Akhter Michael Black Gerard Pons-Moll Søren Hauberg Jürgen Gall

Virtual Humans (2011-2015)

The human body is certainly central to our lives and is commonly depicted in images and video. We are developing the world's most realistic models of the body by learning their shape and how they move from data. Our goal is to make 3D models of the body look and move in ways that make them indistinguishable from real human...

Michael Black Javier Romero Matthew Loper Gerard Pons-Moll Naureen Mahmood Federica Bogo

29 results (View BibTeX file of all listed publications)

2018

Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time

Huang, Y., Kaufmann, M., Aksan, E., Black, M. J., Hilliges, O., Pons-Moll, G.

ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 37, pages: 185:1-185:15, ACM, November 2018, Two first authors contributed equally (article)

Abstract

We demonstrate a novel deep neural network capable of reconstructing human full body pose in real-time from 6 Inertial Measurement Units (IMUs) worn on the user's body. In doing so, we address several difficult challenges. First, the problem is severely under-constrained as multiple pose parameters produce the same IMU orientations. Second, capturing IMU data in conjunction with ground-truth poses is expensive and difficult to do in many target application scenarios (e.g., outdoors). Third, modeling temporal dependencies through non-linear optimization has proven effective in prior work but makes real-time prediction infeasible. To address this important limitation, we learn the temporal pose priors using deep learning. To learn from sufficient data, we synthesize IMU data from motion capture datasets. A bi-directional RNN architecture leverages past and future information that is available at training time. At test time, we deploy the network in a sliding window fashion, retaining real time capabilities. To evaluate our method, we recorded DIP-IMU, a dataset consisting of 10 subjects wearing 17 IMUs for validation in 64 sequences with 330,000 time instants; this constitutes the largest IMU dataset publicly available. We quantitatively evaluate our approach on multiple datasets and show results from a real-time implementation. DIP-IMU and the code are available for research purposes.

data code pdf preprint video DOI Project Page [BibTex]

2018

ps Huang, Y., Kaufmann, M., Aksan, E., Black, M. J., Hilliges, O., Pons-Moll, G. Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 37, pages: 185:1-185:15, ACM, November 2018, Two first authors contributed equally (article)

data code pdf preprint video DOI Project Page [BibTex]

Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimation

(Best Student Paper Award)

Omran, M., Lassner, C., Pons-Moll, G., Gehler, P. V., Schiele, B.

In 3DV, September 2018 (inproceedings)

Abstract

Direct prediction of 3D body pose and shape remains a challenge even for highly parameterized deep learning models. Mapping from the 2D image space to the prediction space is difficult: perspective ambiguities make the loss function noisy and training data is scarce. In this paper, we propose a novel approach (Neural Body Fitting (NBF)). It integrates a statistical body model within a CNN, leveraging reliable bottom-up semantic body part segmentation and robust top-down body model constraints. NBF is fully differentiable and can be trained using 2D and 3D annotations. In detailed experiments, we analyze how the components of our model affect performance, especially the use of part segmentations as an explicit intermediate representation, and present a robust, efficiently trainable framework for 3D human pose estimation from 2D images with competitive results on standard benchmarks. Code is available at https://github.com/mohomran/neural_body_fitting

arXiv code Project Page [BibTex]

ps Omran, M., Lassner, C., Pons-Moll, G., Gehler, P. V., Schiele, B. Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimation In 3DV, September 2018 (inproceedings)

arXiv code Project Page [BibTex]

Recovering Accurate 3D Human Pose in The Wild Using IMUs and a Moving Camera

Marcard, T. V., Henschel, R., Black, M. J., Rosenhahn, B., Pons-Moll, G.

In European Conference on Computer Vision (ECCV), Lecture Notes in Computer Science, vol 11214, pages: 614-631, Springer, Cham, September 2018 (inproceedings)

Abstract

In this work, we propose a method that combines a single hand-held camera and a set of Inertial Measurement Units (IMUs) attached at the body limbs to estimate accurate 3D poses in the wild. This poses many new challenges: the moving camera, heading drift, cluttered background, occlusions and many people visible in the video. We associate 2D pose detections in each image to the corresponding IMU-equipped persons by solving a novel graph based optimization problem that forces 3D to 2D coherency within a frame and across long range frames. Given associations, we jointly optimize the pose of a statistical body model, the camera pose and heading drift using a continuous optimization framework. We validated our method on the TotalCapture dataset, which provides video and IMU synchronized with ground truth. We obtain an accuracy of 26mm, which makes it accurate enough to serve as a benchmark for image-based 3D pose estimation in the wild. Using our method, we recorded 3D Poses in the Wild (3DPW ), a new dataset consisting of more than 51; 000 frames with accurate 3D pose in challenging sequences, including walking in the city, going up-stairs, having coffee or taking the bus. We make the reconstructed 3D poses, video, IMU and 3D models available for research purposes at http://virtualhumans.mpi-inf.mpg.de/3DPW.

pdf SupMat data project DOI Project Page [BibTex]

ps Marcard, T. V., Henschel, R., Black, M. J., Rosenhahn, B., Pons-Moll, G. Recovering Accurate 3D Human Pose in The Wild Using IMUs and a Moving Camera In European Conference on Computer Vision (ECCV), Lecture Notes in Computer Science, vol 11214, pages: 614-631, Springer, Cham, September 2018 (inproceedings)

pdf SupMat data project DOI Project Page [BibTex]

2017

A Generative Model of People in Clothing

Lassner, C., Pons-Moll, G., Gehler, P. V.

In Proceedings IEEE International Conference on Computer Vision (ICCV), IEEE, Piscataway, NJ, USA, IEEE International Conference on Computer Vision (ICCV), October 2017 (inproceedings)

Abstract

We present the first image-based generative model of people in clothing in a full-body setting. We sidestep the commonly used complex graphics rendering pipeline and the need for high-quality 3D scans of dressed people. Instead, we learn generative models from a large image database. The main challenge is to cope with the high variance in human pose, shape and appearance. For this reason, pure image-based approaches have not been considered so far. We show that this challenge can be overcome by splitting the generating process in two parts. First, we learn to generate a semantic segmentation of the body and clothing. Second, we learn a conditional model on the resulting segments that creates realistic images. The full model is differentiable and can be conditioned on pose, shape or color. The result are samples of people in different clothing items and styles. The proposed model can generate entirely new people with realistic clothing. In several experiments we present encouraging results that suggest an entirely data-driven approach to people generation is possible.

link (url) Project Page [BibTex]

2017

ps Lassner, C., Pons-Moll, G., Gehler, P. V. A Generative Model of People in Clothing In Proceedings IEEE International Conference on Computer Vision (ICCV), IEEE, Piscataway, NJ, USA, IEEE International Conference on Computer Vision (ICCV), October 2017 (inproceedings)

link (url) Project Page [BibTex]

Dynamic FAUST: Registering Human Bodies in Motion

Bogo, F., Romero, J., Pons-Moll, G., Black, M. J.

In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017, IEEE, Piscataway, NJ, USA, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017 (inproceedings)

Abstract

While the ready availability of 3D scan data has influenced research throughout computer vision, less attention has focused on 4D data; that is 3D scans of moving nonrigid objects, captured over time. To be useful for vision research, such 4D scans need to be registered, or aligned, to a common topology. Consequently, extending mesh registration methods to 4D is important. Unfortunately, no ground-truth datasets are available for quantitative evaluation and comparison of 4D registration methods. To address this we create a novel dataset of high-resolution 4D scans of human subjects in motion, captured at 60 fps. We propose a new mesh registration method that uses both 3D geometry and texture information to register all scans in a sequence to a common reference topology. The approach exploits consistency in texture over both short and long time intervals and deals with temporal offsets between shape and texture capture. We show how using geometry alone results in significant errors in alignment when the motions are fast and non-rigid. We evaluate the accuracy of our registration and provide a dataset of 40,000 raw and aligned meshes. Dynamic FAUST extends the popular FAUST dataset to dynamic 4D data, and is available for research purposes at http://dfaust.is.tue.mpg.de.

pdf video Project Page Project Page Project Page [BibTex]

ps Bogo, F., Romero, J., Pons-Moll, G., Black, M. J. Dynamic FAUST: Registering Human Bodies in Motion In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017, IEEE, Piscataway, NJ, USA, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017 (inproceedings)

pdf video Project Page Project Page Project Page [BibTex]

Detailed, accurate, human shape estimation from clothed 3D scan sequences

Zhang, C., Pujades, S., Black, M., Pons-Moll, G.

In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society, Washington, DC, USA, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), July 2017, Spotlight (inproceedings)

Abstract

We address the problem of estimating human body shape from 3D scans over time. Reliable estimation of 3D body shape is necessary for many applications including virtual try-on, health monitoring, and avatar creation for virtual reality. Scanning bodies in minimal clothing, however, presents a practical barrier to these applications. We address this problem by estimating body shape under clothing from a sequence of 3D scans. Previous methods that have exploited statistical models of body shape produce overly smooth shapes lacking personalized details. In this paper we contribute a new approach to recover not only an approximate shape of the person, but also their detailed shape. Our approach allows the estimated shape to deviate from a parametric model to fit the 3D scans. We demonstrate the method using high quality 4D data as well as sequences of visual hulls extracted from multi-view images. We also make available a new high quality 4D dataset that enables quantitative evaluation. Our method outperforms the previous state of the art, both qualitatively and quantitatively.

arxiv_preprint video dataset pdf supplemental Project Page [BibTex]

ps Zhang, C., Pujades, S., Black, M., Pons-Moll, G. Detailed, accurate, human shape estimation from clothed 3D scan sequences In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society, Washington, DC, USA, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), July 2017, Spotlight (inproceedings)

arxiv_preprint video dataset pdf supplemental Project Page [BibTex]

Data-Driven Physics for Human Soft Tissue Animation

Kim, M., Pons-Moll, G., Pujades, S., Bang, S., Kim, J., Black, M. J., Lee, S.

ACM Transactions on Graphics, (Proc. SIGGRAPH), 36(4):54:1-54:12, 2017 (article)

Abstract

Data driven models of human poses and soft-tissue deformations can produce very realistic results, but they only model the visible surface of the human body and cannot create skin deformation due to interactions with the environment. Physical simulations can generalize to external forces, but their parameters are difficult to control. In this paper, we present a layered volumetric human body model learned from data. Our model is composed of a data-driven inner layer and a physics-based external layer. The inner layer is driven with a volumetric statistical body model (VSMPL). The soft tissue layer consists of a tetrahedral mesh that is driven using the finite element method (FEM). Model parameters, namely the segmentation of the body into layers and the soft tissue elasticity, are learned directly from 4D registrations of humans exhibiting soft tissue deformations. The learned two layer model is a realistic full-body avatar that generalizes to novel motions and external forces. Experiments show that the resulting avatars produce realistic results on held out sequences and react to external forces. Moreover, the model supports the retargeting of physical properties from one avatar when they share the same topology.

video paper link (url) Project Page [BibTex]

ps Kim, M., Pons-Moll, G., Pujades, S., Bang, S., Kim, J., Black, M. J., Lee, S. Data-Driven Physics for Human Soft Tissue Animation ACM Transactions on Graphics, (Proc. SIGGRAPH), 36(4):54:1-54:12, 2017 (article)

video paper link (url) Project Page [BibTex]

Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUs

(Best Paper, Eurographics 2017)

Marcard, T. V., Rosenhahn, B., Black, M., Pons-Moll, G.

Computer Graphics Forum 36(2), Proceedings of the 38th Annual Conference of the European Association for Computer Graphics (Eurographics), pages: 349-360 , 2017 (article)

Abstract

We address the problem of making human motion capture in the wild more practical by using a small set of inertial sensors attached to the body. Since the problem is heavily under-constrained, previous methods either use a large number of sensors, which is intrusive, or they require additional video input. We take a different approach and constrain the problem by: (i) making use of a realistic statistical body model that includes anthropometric constraints and (ii) using a joint optimization framework to fit the model to orientation and acceleration measurements over multiple frames. The resulting tracker Sparse Inertial Poser (SIP) enables motion capture using only 6 sensors (attached to the wrists, lower legs, back and head) and works for arbitrary human motions. Experiments on the recently released TNT15 dataset show that, using the same number of sensors, SIP achieves higher accuracy than the dataset baseline without using any video data. We further demonstrate the effectiveness of SIP on newly recorded challenging motions in outdoor scenarios such as climbing or jumping over a wall

video pdf Project Page [BibTex]

ps Marcard, T. V., Rosenhahn, B., Black, M., Pons-Moll, G. Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUs Computer Graphics Forum 36(2), Proceedings of the 38th Annual Conference of the European Association for Computer Graphics (Eurographics), pages: 349-360 , 2017 (article)

video pdf Project Page [BibTex]

ClothCap: Seamless 4D Clothing Capture and Retargeting

Pons-Moll, G., Pujades, S., Hu, S., Black, M.

ACM Transactions on Graphics, (Proc. SIGGRAPH), 36(4):73:1-73:15, ACM, New York, NY, USA, 2017, Two first authors contributed equally (article)

Abstract

Designing and simulating realistic clothing is challenging and, while several methods have addressed the capture of clothing from 3D scans, previous methods have been limited to single garments and simple motions, lack detail, or require specialized texture patterns. Here we address the problem of capturing regular clothing on fully dressed people in motion. People typically wear multiple pieces of clothing at a time. To estimate the shape of such clothing, track it over time, and render it believably, each garment must be segmented from the others and the body. Our ClothCap approach uses a new multi-part 3D model of clothed bodies, automatically segments each piece of clothing, estimates the naked body shape and pose under the clothing, and tracks the 3D deformations of the clothing over time. We estimate the garments and their motion from 4D scans; that is, high-resolution 3D scans of the subject in motion at 60 fps. The model allows us to capture a clothed person in motion, extract their clothing, and retarget the clothing to new body shapes. ClothCap provides a step towards virtual try-on with a technology for capturing, modeling, and analyzing clothing in motion.

video project_page paper link (url) DOI Project Page Project Page [BibTex]

ps Pons-Moll, G., Pujades, S., Hu, S., Black, M. ClothCap: Seamless 4D Clothing Capture and Retargeting ACM Transactions on Graphics, (Proc. SIGGRAPH), 36(4):73:1-73:15, ACM, New York, NY, USA, 2017, Two first authors contributed equally (article)

video project_page paper link (url) DOI Project Page Project Page [BibTex]

2016

Human Pose Estimation from Video and IMUs

Marcard, T. V., Pons-Moll, G., Rosenhahn, B.

Transactions on Pattern Analysis and Machine Intelligence PAMI, 38(8):1533-1547, January 2016 (article)

data pdf dataset_documentation [BibTex]

2016

ps Marcard, T. V., Pons-Moll, G., Rosenhahn, B. Human Pose Estimation from Video and IMUs Transactions on Pattern Analysis and Machine Intelligence PAMI, 38(8):1533-1547, January 2016 (article)

data pdf dataset_documentation [BibTex]

2015

SMPL: A Skinned Multi-Person Linear Model

Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M. J.

ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1-248:16, ACM, New York, NY, October 2015 (article)

Abstract

We present a learned model of human body shape and pose-dependent shape variation that is more accurate than previous models and is compatible with existing graphics pipelines. Our Skinned Multi-Person Linear model (SMPL) is a skinned vertex-based model that accurately represents a wide variety of body shapes in natural human poses. The parameters of the model are learned from data including the rest pose template, blend weights, pose-dependent blend shapes, identity-dependent blend shapes, and a regressor from vertices to joint locations. Unlike previous models, the pose-dependent blend shapes are a linear function of the elements of the pose rotation matrices. This simple formulation enables training the entire model from a relatively large number of aligned 3D meshes of different people in different poses. We quantitatively evaluate variants of SMPL using linear or dual-quaternion blend skinning and show that both are more accurate than a Blend-SCAPE model trained on the same data. We also extend SMPL to realistically model dynamic soft-tissue deformations. Because it is based on blend skinning, SMPL is compatible with existing rendering engines and we make it available for research purposes.

pdf video code/model errata DOI Project Page Project Page [BibTex]

2015

ps Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M. J. SMPL: A Skinned Multi-Person Linear Model ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1-248:16, ACM, New York, NY, October 2015 (article)

pdf video code/model errata DOI Project Page Project Page [BibTex]

Dyna: A Model of Dynamic Human Shape in Motion

Pons-Moll, G., Romero, J., Mahmood, N., Black, M. J.

ACM Transactions on Graphics, (Proc. SIGGRAPH), 34(4):120:1-120:14, ACM, August 2015 (article)

Abstract

To look human, digital full-body avatars need to have soft tissue deformations like those of real people. We learn a model of soft-tissue deformations from examples using a high-resolution 4D capture system and a method that accurately registers a template mesh to sequences of 3D scans. Using over 40,000 scans of ten subjects, we learn how soft tissue motion causes mesh triangles to deform relative to a base 3D body model. Our Dyna model uses a low-dimensional linear subspace to approximate soft-tissue deformation and relates the subspace coefficients to the changing pose of the body. Dyna uses a second-order auto-regressive model that predicts soft-tissue deformations based on previous deformations, the velocity and acceleration of the body, and the angular velocities and accelerations of the limbs. Dyna also models how deformations vary with a person’s body mass index (BMI), producing different deformations for people with different shapes. Dyna realistically represents the dynamics of soft tissue for previously unseen subjects and motions. We provide tools for animators to modify the deformations and apply them to new stylized characters.

pdf preprint video data DOI Project Page Project Page [BibTex]

ps Pons-Moll, G., Romero, J., Mahmood, N., Black, M. J. Dyna: A Model of Dynamic Human Shape in Motion ACM Transactions on Graphics, (Proc. SIGGRAPH), 34(4):120:1-120:14, ACM, August 2015 (article)

pdf preprint video data DOI Project Page Project Page [BibTex]

Metric Regression Forests for Correspondence Estimation

Pons-Moll, G., Taylor, J., Shotton, J., Hertzmann, A., Fitzgibbon, A.

International Journal of Computer Vision, pages: 1-13, 2015 (article)

springer PDF Project Page [BibTex]

ps Pons-Moll, G., Taylor, J., Shotton, J., Hertzmann, A., Fitzgibbon, A. Metric Regression Forests for Correspondence Estimation International Journal of Computer Vision, pages: 1-13, 2015 (article)

springer PDF Project Page [BibTex]

2014

Posebits for Monocular Human Pose Estimation

Pons-Moll, G., Fleet, D. J., Rosenhahn, B.

In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages: 2345-2352, Columbus, Ohio, USA, IEEE International Conference on Computer Vision and Pattern Recognition, June 2014 (inproceedings)

Abstract

We advocate the inference of qualitative information about 3D human pose, called posebits, from images. Posebits represent boolean geometric relationships between body parts (e.g., left-leg in front of right-leg or hands close to each other). The advantages of posebits as a mid-level representation are 1) for many tasks of interest, such qualitative pose information may be sufficient (e.g. , semantic image retrieval), 2) it is relatively easy to annotate large image corpora with posebits, as it simply requires answers to yes/no questions; and 3) they help resolve challenging pose ambiguities and therefore facilitate the difficult talk of image-based 3D pose estimation. We introduce posebits, a posebit database, a method for selecting useful posebits for pose estimation and a structural SVM model for posebit inference. Experiments show the use of posebits for semantic image retrieval and for improving 3D pose estimation.

pdf Project Page Project Page [BibTex]

2014

ps Pons-Moll, G., Fleet, D. J., Rosenhahn, B. Posebits for Monocular Human Pose Estimation In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages: 2345-2352, Columbus, Ohio, USA, IEEE International Conference on Computer Vision and Pattern Recognition, June 2014 (inproceedings)

pdf Project Page Project Page [BibTex]

Human Pose Estimation from Video and Inertial Sensors

Pons-Moll, G.

Ph.D Thesis, -, 2014 (book)

Abstract

The analysis and understanding of human movement is central to many applications such as sports science, medical diagnosis and movie production. The ability to automatically monitor human activity in security sensitive areas such as airports, lobbies or borders is of great practical importance. Furthermore, automatic pose estimation from images leverages the processing and understanding of massive digital libraries available on the Internet. We build upon a model based approach where the human shape is modelled with a surface mesh and the motion is parametrized by a kinematic chain. We then seek for the pose of the model that best explains the available observations coming from different sensors. In a first scenario, we consider a calibrated mult-iview setup in an indoor studio. To obtain very accurate results, we propose a novel tracker that combines information coming from video and a small set of Inertial Measurement Units (IMUs). We do so by locally optimizing a joint energy consisting of a term that measures the likelihood of the video data and a term for the IMU data. This is the first work to successfully combine video and IMUs information for full body pose estimation. When compared to commercial marker based systems the proposed solution is more cost efficient and less intrusive for the user. In a second scenario, we relax the assumption of an indoor studio and we tackle outdoor scenes with background clutter, illumination changes, large recording volumes and difficult motions of people interacting with objects. Again, we combine information from video and IMUs. Here we employ a particle based optimization approach that allows us to be more robust to tracking failures. To satisfy the orientation constraints imposed by the IMUs, we derive an analytic Inverse Kinematics (IK) procedure to sample from the manifold of valid poses. The generated hypothesis come from a lower dimensional manifold and therefore the computational cost can be reduced. Experiments on challenging sequences suggest the proposed tracker can be applied to capture in outdoor scenarios. Furthermore, the proposed IK sampling procedure can be used to integrate any kind of constraints derived from the environment. Finally, we consider the most challenging possible scenario: pose estimation of monocular images. Here, we argue that estimating the pose to the degree of accuracy as in an engineered environment is too ambitious with the current technology. Therefore, we propose to extract meaningful semantic information about the pose directly from image features in a discriminative fashion. In particular, we introduce posebits which are semantic pose descriptors about the geometric relationships between parts in the body. The experiments show that the intermediate step of inferring posebits from images can improve pose estimation from monocular imagery. Furthermore, posebits can be very useful as input feature for many computer vision algorithms.

pdf [BibTex]

ps Pons-Moll, G. Human Pose Estimation from Video and Inertial Sensors Ph.D Thesis, -, 2014 (book)

pdf [BibTex]

2013

Metric Regression Forests for Human Pose Estimation

(Best Science Paper Award)

Pons-Moll, G., Taylor, J., Shotton, J., Hertzmann, A., Fitzgibbon, A.

In British Machine Vision Conference (BMVC) , BMVA Press, September 2013 (inproceedings)

pdf [BibTex]

2013

ps Pons-Moll, G., Taylor, J., Shotton, J., Hertzmann, A., Fitzgibbon, A. Metric Regression Forests for Human Pose Estimation In British Machine Vision Conference (BMVC) , BMVA Press, September 2013 (inproceedings)

pdf [BibTex]

2012

PCA-enhanced stochastic optimization methods

Kuznetsova, A., Pons-Moll, G., Rosenhahn, B.

In German Conference on Pattern Recognition (GCPR), August 2012 (inproceedings)

pdf [BibTex]

2012

ps Kuznetsova, A., Pons-Moll, G., Rosenhahn, B. PCA-enhanced stochastic optimization methods In German Conference on Pattern Recognition (GCPR), August 2012 (inproceedings)

pdf [BibTex]

Branch-and-price global optimization for multi-view multi-object tracking

Leal-Taixé, L., Pons-Moll, G., Rosenhahn, B.

In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2012 (inproceedings)

project page paper poster [BibTex]

ps Leal-Taixé, L., Pons-Moll, G., Rosenhahn, B. Branch-and-price global optimization for multi-view multi-object tracking In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2012 (inproceedings)

project page paper poster [BibTex]

Exploiting pedestrian interaction via global optimization and social behaviors

Leal-Taixé, L., Pons-Moll, G., Rosenhahn, B.

In Theoretic Foundations of Computer Vision: Outdoor and Large-Scale Real-World Scene Analysis, Springer, April 2012 (incollection)

pdf [BibTex]

ps Leal-Taixé, L., Pons-Moll, G., Rosenhahn, B. Exploiting pedestrian interaction via global optimization and social behaviors In Theoretic Foundations of Computer Vision: Outdoor and Large-Scale Real-World Scene Analysis, Springer, April 2012 (incollection)

pdf [BibTex]

Data-driven Manifolds for Outdoor Motion Capture

Pons-Moll, G., Leal-Taix’e, L., Gall, J., Rosenhahn, B.

In Outdoor and Large-Scale Real-World Scene Analysis, 7474, pages: 305-328, LNCS, (Editors: Dellaert, Frank and Frahm, Jan-Michael and Pollefeys, Marc and Rosenhahn, Bodo and Leal-Taix’e, Laura), Springer, 2012 (incollection)

video publisher's site pdf Project Page [BibTex]

ps Pons-Moll, G., Leal-Taix’e, L., Gall, J., Rosenhahn, B. Data-driven Manifolds for Outdoor Motion Capture In Outdoor and Large-Scale Real-World Scene Analysis, 7474, pages: 305-328, LNCS, (Editors: Dellaert, Frank and Frahm, Jan-Michael and Pollefeys, Marc and Rosenhahn, Bodo and Leal-Taix’e, Laura), Springer, 2012 (incollection)

video publisher's site pdf Project Page [BibTex]

2011

Outdoor Human Motion Capture using Inverse Kinematics and von Mises-Fisher Sampling

Pons-Moll, G., Baak, A., Gall, J., Leal-Taixe, L., Mueller, M., Seidel, H., Rosenhahn, B.

In IEEE International Conference on Computer Vision (ICCV), pages: 1243-1250, November 2011 (inproceedings)

project page pdf supplemental [BibTex]

2011

ps Pons-Moll, G., Baak, A., Gall, J., Leal-Taixe, L., Mueller, M., Seidel, H., Rosenhahn, B. Outdoor Human Motion Capture using Inverse Kinematics and von Mises-Fisher Sampling In IEEE International Conference on Computer Vision (ICCV), pages: 1243-1250, November 2011 (inproceedings)

project page pdf supplemental [BibTex]

Everybody needs somebody: modeling social and grouping behavior on a linear programming multiple people tracker

Leal-Taixé, L., Rosenhahn, G. P. A. B.

In IEEE International Conference on Computer Vision Workshops (IICCVW), November 2011 (inproceedings)

project page pdf [BibTex]

ps Leal-Taixé, L., Rosenhahn, G. P. A. B. Everybody needs somebody: modeling social and grouping behavior on a linear programming multiple people tracker In IEEE International Conference on Computer Vision Workshops (IICCVW), November 2011 (inproceedings)

project page pdf [BibTex]

Efficient and Robust Shape Matching for Model Based Human Motion Capture

Pons-Moll, G., Leal-Taixé, L., Truong, T., Rosenhahn, B.

In German Conference on Pattern Recognition (GCPR), pages: 416-425, September 2011 (inproceedings)

project page pdf [BibTex]

ps Pons-Moll, G., Leal-Taixé, L., Truong, T., Rosenhahn, B. Efficient and Robust Shape Matching for Model Based Human Motion Capture In German Conference on Pattern Recognition (GCPR), pages: 416-425, September 2011 (inproceedings)

project page pdf [BibTex]

Model-Based Pose Estimation

Pons-Moll, G., Rosenhahn, B.

In Visual Analysis of Humans: Looking at People, pages: 139-170, 9, (Editors: T. Moeslund, A. Hilton, V. Krueger, L. Sigal), Springer, 2011 (inbook)

book page pdf [BibTex]

ps Pons-Moll, G., Rosenhahn, B. Model-Based Pose Estimation In Visual Analysis of Humans: Looking at People, pages: 139-170, 9, (Editors: T. Moeslund, A. Hilton, V. Krueger, L. Sigal), Springer, 2011 (inbook)

book page pdf [BibTex]

2010

Analyzing and Evaluating Markerless Motion Tracking Using Inertial Sensors

Baak, A., Helten, T., Müller, M., Pons-Moll, G., Rosenhahn, B., Seidel, H.

In European Conference on Computer Vision (ECCV Workshops), September 2010 (inproceedings)

pdf [BibTex]

2010

ps Baak, A., Helten, T., Müller, M., Pons-Moll, G., Rosenhahn, B., Seidel, H. Analyzing and Evaluating Markerless Motion Tracking Using Inertial Sensors In European Conference on Computer Vision (ECCV Workshops), September 2010 (inproceedings)

pdf [BibTex]

Multisensor-Fusion for 3D Full-Body Human Motion Capture

Pons-Moll, G., Baak, A., Helten, T., Müller, M., Seidel, H., Rosenhahn, B.

In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2010 (inproceedings)

project page pdf [BibTex]

ps Pons-Moll, G., Baak, A., Helten, T., Müller, M., Seidel, H., Rosenhahn, B. Multisensor-Fusion for 3D Full-Body Human Motion Capture In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2010 (inproceedings)

project page pdf [BibTex]

2009

Ball Joints for Marker-less Human Motion Capture

Pons-Moll, G., Rosenhahn, B.

In IEEE Workshop on Applications of Computer Vision (WACV),, December 2009 (inproceedings)

pdf [BibTex]

2009

ps Pons-Moll, G., Rosenhahn, B. Ball Joints for Marker-less Human Motion Capture In IEEE Workshop on Applications of Computer Vision (WACV),, December 2009 (inproceedings)

pdf [BibTex]

Parametric Modeling of the Beating Heart with Respiratory Motion Extracted from Magnetic Resonance Images

Pons-Moll, G., Crosas, C., Tadmor, G., MacLeod, R., Rosenhahn, B., Brooks, D.

In IEEE Computers in Cardiology (CINC), September 2009 (inproceedings)

[BibTex]

ps Pons-Moll, G., Crosas, C., Tadmor, G., MacLeod, R., Rosenhahn, B., Brooks, D. Parametric Modeling of the Beating Heart with Respiratory Motion Extracted from Magnetic Resonance Images In IEEE Computers in Cardiology (CINC), September 2009 (inproceedings)

[BibTex]

4D Cardiac Segmentation of the Epicardium and Left Ventricle

Pons-Moll, G., Tadmor, G., MacLeod, R. S., Rosenhahn, B., Brooks, D. H.

In World Congress of Medical Physics and Biomedical Engineering (WC), 2009 (inproceedings)

[BibTex]

ps Pons-Moll, G., Tadmor, G., MacLeod, R. S., Rosenhahn, B., Brooks, D. H. 4D Cardiac Segmentation of the Epicardium and Left Ventricle In World Congress of Medical Physics and Biomedical Engineering (WC), 2009 (inproceedings)

[BibTex]

Gerard Pons-Moll

NOTE: In September 2017 I moved to MPI-Informatics in Saarbrücken where I am heading the group "Real Virtual Humans".

For recent news, job offers, projects and publications, please visit my research group website.

News:

Education:

Teaching:

Current Ph.D interns

Past Ph.D interns

2018

2018

2017

2017

2016

2016

2015

2015

2014

2014

2013

2013

2012

2012

2011

2011

2010

2010

2009

2009

Latest News

Links

Contact Us