the wanderings of odysseus in 3d scenes

The Wanderings of Odysseus in 3D Scenes

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Search code, repositories, users, issues, pull requests...

Provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

• Notifications

official implementation of [Zhang and Tang, The Wanderings of Odysseus in 3D Scenes, CVPR'22]

yz-cnsdqz/GAMMA-release

• Python 99.5%

Powered by:

Sponsored by:

Subscribe to the PwC Newsletter

Join the community, edit social preview.

Add a new code entry for this paper

Remove a code repository from this paper, mark the official implementation from paper authors, add a new evaluation result row, remove a task, add a method.

• AUTOENCODER

Remove a method

• AUTOENCODER -

Edit Datasets

The wanderings of odysseus in 3d scenes.

CVPR 2022  ·  Yan Zhang , Siyu Tang · Edit social preview

Our goal is to populate digital environments, in which digital humans have diverse body shapes, move perpetually, and have plausible body-scene contact. The core challenge is to generate realistic, controllable, and infinitely long motions for diverse 3D bodies. To this end, we propose generative motion primitives via body surface markers, or GAMMA in short. In our solution, we decompose the long-term motion into a time sequence of motion primitives. We exploit body surface markers and conditional variational autoencoder to model each motion primitive, and generate long-term motion by implementing the generative model recursively. To control the motion to reach a goal, we apply a policy network to explore the generative model's latent space and use a tree-based search to preserve the motion quality during testing. Experiments show that our method can produce more realistic and controllable motion than state-of-the-art data-driven methods. With conventional path-finding algorithms, the generated human bodies can realistically move long distances for a long period of time in the scene. Code is released for research purposes at: \url{https://yz-cnsdqz.github.io/eigenmotion/GAMMA/}

Code Edit Add Remove Mark official

Tasks edit add remove, datasets edit.

Results from the Paper Edit Add Remove

Methods edit add remove.

The Wanderings of Odysseus in 3D Scenes

Conference:  conference on computer vision and pattern recognition (cvpr 2022).

Our goal is to populate digital environments, in which the digital humans have diverse body shapes, move perpetually, and have plausible body-scene contact. The core challenge is to generate realistic, controllable, and infinitely long motions for diverse 3D bodies. To this end, we propose generative motion primitives via body surface markers, shortened as GAMMA. In our solution, we decompose the long-term motion into a time sequence of motion primitives. We exploit body surface markers and conditional variational autoencoder to model each motion primitive, and generate long-term motion by implementing the generative model recursively. To control the motion to reach a goal, we apply a policy network to explore the model latent space, and use a tree-based search to preserve the motion quality during testing. Experiments show that our method can produce more realistic and controllable motion than state-of-the-art data-driven method. With conventional path-finding algorithms, the generated human bodies can realistically move in the scene for a long distance and a long time.

Project PDF Source BibTeX

The Wanderings of Odysseus in 3D Scenes

Our goal is to populate digital environments, in which the digital humans have diverse body shapes, move perpetually, and have plausible body-scene contact. The core challenge is to generate realistic, controllable, and infinitely long motions for diverse 3D bodies. To this end, we propose generative motion primitives via body surface markers, shortened as GAMMA. In our solution, we decompose the long-term motion into a time sequence of motion primitives. We exploit body surface markers and conditional variational autoencoder to model each motion primitive, and generate long-term motion by implementing the generative model recursively. To control the motion to reach a goal, we apply a policy network to explore the model latent space, and use a tree-based search to preserve the motion quality during testing. Experiments show that our method can produce more realistic and controllable motion than state-of-the-art data-driven method. With conventional path-finding algorithms, the generated human bodies can realistically move long distances for a long period of time in the scene. Code will be released for research purposes at: <https://yz-cnsdqz.github.io/eigenmotion/GAMMA/>

Related Research

Mime: human-aware 3d scene generation, correspondence-free online human motion retargeting, temos: generating diverse human motions from textual descriptions, learning disentangled representations for controllable human motion prediction, generative tweening: long-term inbetweening of 3d human motions, contextually learnt detection of unusual motion-based behaviour in crowded public spaces, narrator: towards natural control of human-scene interaction generation via relationship reasoning.

Please sign up or login with your details

Generation Overview

AI Generator calls

AI Chat messages

Genius Mode messages

Genius Mode images

AD-free experience

Private images

• Includes 500 AI Image generations, 1750 AI Chat Messages, 60 Genius Mode Messages and 60 Genius Mode Images per month. If you go over any of these limits, you will be charged an extra $5 for that group.
• For example: if you go over 500 AI images, but stay within the limits for AI Chat and Genius Mode, you'll be charged $5 per additional 500 AI Image generations.
• Includes 100 AI Image generations and 300 AI Chat Messages. If you go over any of these limits, you will have to pay as you go.
• For example: if you go over 100 AI images, but stay within the limits for AI Chat, you'll have to reload on credits to generate more images. Choose from $5 - $1000. You'll only pay for what you use.

Out of credits

Refill your membership to continue using DeepAI

Synthesizing Diverse Human Motions in 3D Indoor Scenes

In this work, we propose a method to generate a sequence of natural human-scene interaction events in real-world complex scenes as illustrated in this figure. The human first walks to sit on a stool ( yellow to red ), then walk to another chair to sit down ( red to magenta ), and finally walk to and lie on the sofa ( magenta to blue ).

We formulate synthesizing human behaviors in 3D scenes as a Markov decision process with a latent action space, which is learned from motion capture datasets. We train scene-aware and goal-driven agent policies to synthesize various human behaviors in indoor scenes including wandering in the room, sitting or lying on an object, and sequential combinations of these actions.

Illustration of the motion primitive generative model that generates future motions conditioned on the history motions.

Illustration of generating random but perpetual motions by recursively sampling the learned motion primitive model.

Illustration of our proposed human-scene interaction synthesis framework, which consists of learned motion primitive (actions), alongside locomotion and interaction policies generating latent actions conditioned on scenes and interaction goals. By integrating navigation mesh-based path-finding and static human-scene interaction generation methods, we can synthesize realistic motion sequences for virtual humans with fine-grained controls.

Illustration of the scene-aware locomotion policy network. The locomotion policy state consists of the body markers, the goal-reaching feature of normalized direction vectors from markers to the goal pelvis, and the interaction feature of a 2D binary map indicating the walkability ( red : non-walkable area, blue : walkable area) of the local 1.6m * 1.6m square area. The locomotion policy network employs the actor-critic architecture and shares the state encoder.

Illustration of the object interaction policy network. The interaction policy state consists of the body markers, the goal-reaching features of both distance and direction from current markers to the goal markers, and the interaction features of the signed distances from each marker to the object surfaces and the signed distance gradient at each marker location. Such interaction features encode the human-object proximity relationship.

Our method generalize to novel objects and real world scenes.

Sitting on a novel chair with unique shape and low height.

Inhabiting reconstructed scenes.

Walking to sit on a bus stop bench in a Paris street point cloud from Paris-CARLA-3D .

Walking to sit on a bed in a reconstructed room from PROX .

Interaction with unusual-shaped objects generated by text-to-3D method Shap-E .

Sit on avacado-shaped chairs.

Sit on dog-shaped chairs.

Policy learning process.

Limitations.

As a kinematics-based method, human-scene penetration remains observable in synthesis results.

Insufficient training motion data for lying leads to degraded motion results with unnatural poses.

Comparison with more related works.

Related Projects

This project is developed on top of prior works, please also consider citing the following projects:

We are More than Our Joints

Predicting how 3d bodies move.

"We are more than our joints", or MOJO for short, is a solution to stochastic motion prediction of expressive 3D bodies. Given a short motion from the past, MOJO generates diverse plausible motions in the near future.

• A body surface marker-based representation. Compared to joint locations, body surface markers contain richer information of the body shape, and provide more body degree-of-freedom constraints. Compared to joint rotations, markers are located in the Euclidean space, which are easier for neural networks to learn.
• A conditional VAE with latent frequencies. With latent frequencies, the generated motion has more high-frequency components and hence looks more realistic. Boosted by DLow as an advanced sampler in the latent space, MOJO produces highly diverse future motions based on the same motion seed.
• A recursive marker reprojection scheme. This scheme is to recover the body meshes from the generated markers during testing. After reprojecting the markers to the mesh template at each time step, it always keeps the markers in the valid body space, and hence can eliminate error accumulation of the recurrent network.
• The Wanderings of Odysseus in 3D Scenes (GAMMA)
• Learning Motion Priors for 4D Human Body Capture in 3D Scenes (LEMO)
• SAGA: Stochastic Whole-Body Grasping with Contact
• EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices
• Synthesizing Diverse Human Motions in 3D Indoor Scenes (DIMOS)

Coming soon ...

Michael J. Black