Newly Developed AI System 3D-GPT: Generating 3D Models Based on Text Descriptions

Researchers from the Australian National University, the University of Oxford, and the Beijing Academy of Artificial Intelligence have developed a new artificial intelligence system called "3D-GPT" that can generate 3D models based on textual descriptions provided by users. Described in a paper published on arXiv, the system offers a more efficient and intuitive way to create 3D resources compared to traditional 3D modeling workflows. According to the paper, 3D-GPT is able to "divide procedural 3D modeling tasks into actionable segments and assign appropriate agents to each task." It utilizes multiple AI agents, with each agent focusing on understanding different parts of the textual prompts and performing modeling functions. The researchers state, "3D-GPT positions LLM (large language model) as a skilled problem solver, dividing procedural 3D modeling tasks into accessible segments and assigning appropriate agents to each task." The main agents include the "task assignment agent" for parsing textual instructions, the "conceptualization agent" for adding missing details in the initial description, and the "modeling agent" for setting parameters and generating code to drive 3D software like Blender. By decomposing the modeling process and assigning specialized AI agents, 3D-GPT is able to interpret textual prompts, enhance the details of the descriptions, and ultimately generate 3D resources that align with the user's imagination. The paper explains, "It can elevate concise initial scene descriptions into detailed forms and dynamically adjust the text based on subsequent instructions." The system was tested on prompts such as "a misty spring morning with dew moistening the lush green grass and budding trees." 3D-GPT was able to generate complete 3D scenes with realistic graphics that accurately reflected the elements described in the text. While the graphic quality has not yet reached a photorealistic level, preliminary results indicate the potential of this agent-based approach in simplifying 3D content creation. The modular architecture also allows for independent improvements of each agent component. The researchers write, "Our empirical investigation confirms that 3D-GPT not only interprets and executes instructions to provide reliable results but also effectively collaborates with human designers." By generating code to control existing 3D software instead of building models from scratch, 3D-GPT provides a flexible foundation for further development in the evolving field of modeling techniques. The researchers conclude that their system "highlights the potential of LLM in 3D modeling, providing a fundamental framework for the future advancement of scene generation and animation." This research has the potential to revolutionize the 3D modeling industry, making the process more efficient and user-friendly. As we venture further into the metaverse era, tools like 3D-GPT can be invaluable for creators and decision-makers in various industries such as gaming, virtual reality, film, and multimedia experiences. The 3D-GPT framework is currently in its early stages and has some limitations, but its development marks an important step forward in AI-based 3D modeling and opens up more possibilities for future advancements.