Researchers from Max Plank propose a generative AI model - MIME

2023-06-21

Through the interaction between people and their surrounding environment, such as moving in a space, touching things, sitting on a chair, or sleeping on a bed, one can infer how the scene is set up and where objects are located. MIME creates a rich, imaginative, and three-dimensional environment by analyzing people's body movements. Many fields, including architecture, gaming, virtual reality, and synthetic data synthesis, can benefit from this technology. For example, AMASS has a large dataset of human 3D motions, but these datasets rarely include details of the three-dimensional environments in which people are situated. Can they use AMASS to create credible three-dimensional scenes for all movements? If so, they can use AMASS to generate training data for real human-scene interactions. They have developed a new technology called MIME (Mining Interaction and Movement to infer 3D Environments) to address this problem. What makes it possible? The basic premise is that human movements in space define areas without furniture in the images. Additionally, this also determines the types and positions of three-dimensional objects when they come into contact with the scene; for example, a sitting person must be sitting on a chair, sofa, or bed. Researchers from the Max Planck Institute for Intelligent Systems in Germany and Adobe have created MIME, a Transformer-based autoregressive 3D scene generation technology that transforms intuition into concrete form. Given an empty floor plan and a sequence of actions by a person, MIME can predict the furniture that will come into contact with the person. Furthermore, it can anticipate objects that do not come into contact with the person but interact with other objects and adhere to the spatial constraints imposed by the person's movements. They classify movements into contact and non-contact categories to provide conditions for creating three-dimensional scenes of human movements. They use POSA to estimate potential contact poses. Non-contact poses project the foot vertices onto the ground plane to establish the room's free space, which they record as a two-dimensional floor map. Contact vertices predicted by POSA create three-dimensional bounding boxes that reflect contact poses and related 3D human models. Objects that meet the criteria for contact and free space are automatically used as input for the transformer. They extended the large-scale synthetic scene dataset 3D-FRONT and created a new dataset called 3D-FRONT HUMAN to train MIME. They automatically add people to 3D scenes, including non-contact people (a series of walking actions and standing people) and contact people (sitting, touching, and lying people). To do this, they use static contact poses scanned from RenderPeople and motion sequences from AMASS. MIME creates a realistic 3D scene layout for input actions during inference. They select 3D models from the 3D-FRONT collection based on this arrangement; then, they fine-tune their 3D positions based on the geometric constraints between human positions and the scene. Their technique produces a collection of 3D objects that can support human touch and movement while placing real objects in free space, unlike pure 3D scene creation systems like ATISS. Their approach can develop objects that do not come into contact with people and can predict complete scenes rather than individual objects, in contrast to recent work like Pose2Room. They demonstrate that their method requires no adjustments on recorded real motion sequences, such as PROX-D. Source: https://www.marktechpost.com/2023/06/20/researchers-from-max-plank-propose-mime-a-generative-ai-model-that-takes-3d-human-motion-capture-and-generates-plausible-3d-scenes-that-are-consistent-with-the-motion/

RECENT AI NEWS

RECENT AI TOOLS