The World
as a Canvas

Materializing Imagination
through Prompting the Physical World
Materializing Imagination through Prompting the Physical World

November 2, 2026  ·  Detroit Marriott at the Renaissance Center

Workshop Overview

This workshop explores how generative AI can reshape the way people create in XR. While it is rapidly lowering barriers to creation, XR introduces fundamentally different challenges as creation moves beyond flat screens into the physical world. XR experiences must account for real spaces, embodied and multimodal interaction, collaboration across heterogeneous devices, and content that unfolds through space and time. These characteristics raise new questions about how creators express intent, preview and refine imagined outcomes, evaluate generated content, and share or transform what they create. The World as a Canvas brings together researchers and practitioners to examine these emerging challenges and develop a shared vision for more accessible, expressive, and collaborative XR creation.

Six Challenges

Each challenge framed open questions we refined together and carried into the afternoon prototyping session.

1In-situ Creation within Physical Space

How can generative AI create content and behaviors that fit the user’s physical surroundings, from lighting and scale to material and weight?

2Previsualization and Mental Imagery

How can tools help creators preview imagined outcomes without constraining their own mental imagery?

3Expressing and Interpreting Creative Intent

How should generative AI interpret creative intent expressed through gesture, gaze, movement, and other signals beyond words?

4Co-Creation across Hardware Heterogeneity

How can people co-create seamlessly across headsets, tablets, and phones with different input, display, and compute capabilities?

5Evaluating Generated Content

How can creators detect and correct generative errors that emerge through interaction, across viewpoints, or over time?

6Cross-Media Creation, Transformation, and Provenance

As creations move across images, videos, XR, and physical artifacts, what should be preserved, and how can their provenance and ownership remain traceable?

Organizers

Program

09:00 – 09:45

Opening & Speed Introductions

09:45 – 10:30

Keynote I — Academic Perspectiveby Ken Perlin (NYU)

10:30 – 11:00

Coffee Break

11:00 – 12:15

Prototyping I — Toolkit and Ideation

12:15 – 13:45

Lunch

13:45 – 14:30

Keynote II — Industry Perspectiveby Andy Wilson (MSR)

14:30 – 15:00

Prototyping II — Build Sprint

15:00 – 15:30

Coffee Break

15:30 – 16:15

Prototyping II — Build Sprint

16:15 – 17:00

Showcase and Demo

Photos

See the day in 360°, too! Photos by Ruofei Du.

More photos! If you have photos of your own, please add them.

Open the Google Photos album ↗

Contact Us

Anything you would like to know about the workshop, just ask.

Email the organizers ↗

UIST '26  ·  November 2, 2026  ·  Detroit, MI