A bigger screen unlocks the magic
This story has large visuals and videos. For the full experience, please open on desktop 💻
A bigger screen unlocks the magic
This story has large visuals and videos. For the full experience, please open on desktop 💻
A bigger screen
unlocks the magic
This story has large visuals and videos. For the full experience, please open on desktop 💻
Wishcraft
Wish It to Create It
AI-powered XR application that enables users to create and interact with 3D environments using natural gestures and voice commands.
Timeline
Jul-Aug 2024 (6 weeks)
Role
XR Design & Prototype, Video Production
Tool
Unity, Meta XR SDK, TripoSR, Wit.ai, Meta Quest 3
Teammate
Minzhe Zou
Instructor
Yasaman Sheri, Miller Klitsner
Technical Support
Hongming Li, Weide Zhang
Wishcraft
Wish It to Create It
AI-powered XR application that enables users to create and interact with 3D environments using natural gestures and voice commands.
Timeline
Jul-Aug 2024 (6 weeks)
Role
XR Design & Prototype, Video Production
Tool
Unity, Meta XR SDK, TripoSR, Wit.ai, Meta Quest 3
Teammate
Minzhe Zou
Instructor
Yasaman Sheri, Miller Klitsner
Technical Support
Hongming Li, Weide Zhang
Wishcraft
Wish It to Create It
AI-powered XR application that enables users to create and interact with 3D environments using natural gestures and voice commands.
Timeline
Jul-Aug 2024 (6 weeks)
Teammate
Minzhe Zou
Role
XR Design & Prototype, Video Production
Instructor
Yasaman Sheri, Miller Klitsner
Tool
Unity, Meta XR SDK, TripoSR, Wit.ai, Meta Quest 3
Technical Support
Hongming Li, Weide Zhang
People struggle to learn and
create with existing 3D software.
😩
❌ Non-Intuitive 3D modeling
Traditional mouse inputs are limiting and not natural when interacting with 3D objects.
What is we could use natural inputs like gestures and voice commands to make creation feel easier?
❌ Lack of Spatial Awareness
Designing on a flat 2D screen makes it hard to understand space and scale.
What if we could leverage XR technology to design directly in the 3D environment?
❌ Difficult to Visualize Ideas
Not everyone can visualize their ideas through drawing.
What if we could use AI to help us with visualization?
What if we could simply use
our voice and gestures in an immersive space with the assistance from AI to build our 3D world?
Introducing
Wishcraft
AI
XR
Multimodal Input
Revolutionize
3D environment
creation
Leverage voice commands and natural gestures with AI to design and interact with 3D environments, offering a seamless alternative to traditional control methods for a more intuitive creation experience.
Wishcraft is tailored for people who are not 3d modelers — like product managers and creative directors —who have innovative ideas but struggle to visualize them.
By using natural, everyday conversation—much like making a wish—they can see their concepts come to life without the need for professional 3D design skills.
Wishcraft
Immersive
Intelligent
With AI training, Wishcraft transforms ideas into 3D objects and environments.
Wit.ai
Users say what they want using natural language. Then the system convert their speech into text and extracts a generation prompt.
This interface is TripoSR turning text to 3d object:
“I want a delicate blue flower with petals that have a soft gradient."
To make it easier to view, this generation video has been speed up.
The original duration is around 3 minutes.
This interface is TripoSR turning text to 3d object:
“I want a delicate blue flower with petals that have a soft gradient."
To make it easier to view, this generation video has been speed up.
The original duration is around 3 minutes.
TripoSR on ComfyUI
Users say what they want using natural language. Then the system convert their speech into text and extracts a generation prompt.
Using this prompt, the system generate an image and then transformed into a 3D object.
It is not just a tool;
it's a versatile platform applicable to numerous use case
for early stage rapid prototyping and planning
Micro
Macro
Interior
Design
Film & Stage
Set Design
Event
Planning
Accessibility
Testing
Game
Development
Urban
Planning
Theme Park
Design
Designing individual rooms or spaces within a building.
Creating sets for movies and theater productions.
Designing spatial layouts for events like weddings or conferences.
Ensuring spaces are accessible and meet diverse user needs.
Creating immersive environments for video games.
Designing larger scale developments like neighborhoods or public spaces.
Planning extensive theme park layouts including rides and attractions.
Design Process
Behind The Scene
Design Process
Behind The Scene
World IA Day 2025
Public Talk & Live Demo
Introduced Wishcraft during a public talk in the 950 Building lobby at ArtCenter, and invited attendees to try the demo in the Immersion Lab.
Reflection
What I Learned
Reflection
What I Learned
Exploring multimodal inputs with the Meta XR SDK enhanced my understanding of natural interactions in XR environments
Exploring multimodal inputs with the Meta XR SDK enhanced my understanding of natural interactions in XR environments
Quick prototyping and user testing really help refine the design through quick iterations.
Quick prototyping and user testing really help refine the design through quick iterations.
Do not to abandon ideas due to technical challenges but to find alternative ways to demonstrate the concept
Do not to abandon ideas due to technical challenges but to find alternative ways to demonstrate the concept
Reflection
Next Step
Reflection
Next Step
Overview mode for better scene management.
Overview mode for better scene management.
Intuitive “delete” action using a gesture
Intuitive “delete” action using a gesture
Button to turn off all UI elements
Button to turn off all UI elements
Advanced features like recording animation or adding triggers
Advanced features like recording animation or adding triggers
Allow users to define where objects generate (via eye-tracking or pointing)
Allow users to define where objects generate (via eye-tracking or pointing)
