Generative AI can quickly create 3D models, but visually convincing geometry does not always translate into functional objects. Researchers at MIT’s Computer Science and Artificial Intelligence Laboratory, Google, and Northeastern University have developed InstructMesh, an AI-driven design tool that helps users identify and repair flaws in generated models before fabrication, tells Tech Xplore.
InstructMesh combines Microsoft’s TRELLIS 3D-generation system with GPT-4. TRELLIS generates models from text or images, while the language model helps interpret what users want to change. Users can select a problematic region and describe the required modification in natural language. The system then alters the geometry while preserving unaffected areas.
The researchers demonstrated the approach with functional and personalized objects, including a dragon-themed mug, shell-shaped whistle, butterfly-inspired glasses, and multi-spout drink dispenser. They also created a denim-like knee brace and a shrimp-shaped bristle robot with a concealed motor.
A key goal is making 3D design accessible to people without modeling expertise. To test this, researchers asked TRELLIS to recreate popular models from Thingiverse. Nearly 80% of the generated models contained structural flaws. Expert evaluation found that novice users working with InstructMesh identified and repaired problems about 90% of the time.
The software performs edits within the generative model’s latent space, allowing users to describe problems rather than manually manipulate complex geometry. Sliders provide additional control for adjustments such as enlarging or extruding specific features.
Future versions could extend beyond geometry editing. Researchers envision integrating InstructMesh with augmented reality so users can generate objects suited to their surroundings. Physics simulation could also predict whether designs will withstand real-world use and help select appropriate materials. Support for newer 3D-generation models could improve smaller geometric details.
InstructMesh demonstrates an important direction for generative design: giving users practical tools to turn AI-generated shapes into editable, functional objects that can actually be manufactured.
