GOAL-DRIVEN WEB INTERFACES

Traditional Web applications require users to navigate menus, pages,
filters, and forms, even when they already know the goal they want to
achieve. This project investigates a different interaction paradigm in
which users express their intentions in natural language and an AI agent
dynamically presents the graphical components required to complete the
task.

The project explores how Large Language Models can orchestrate existing
Web interface components while preserving the precision, reliability,
and direct-manipulation capabilities of graphical user interfaces.
Rather than generating arbitrary interfaces or autonomously clicking
through websites, developers expose reusable components through
semantic descriptions and typed contracts.

This research resulted in Mirage, a framework for
building goal-driven Web applications from declarative and reusable
components. The AI interprets the user’s intention, selects the
appropriate components, and configures them as a simplified and
interactive interface.

Publications

DRIP

Workplace coordination is often fragmented across emails, messages,
and separate task-management systems. Responsibilities, deadlines,
decisions, and pending actions remain buried in conversations and must
be continuously reconstructed by the people involved.

This project investigates state generation, a
Generative UI paradigm in which an AI system does not simply generate
another message or summary, but creates and maintains a persistent,
inspectable, and actionable representation of collaborative work.

The project resulted in
DRIP — Dynamic Responsibility and Intent Platform,
a research prototype that transforms conversational input into a
structured shared workspace. The system represents work objects,
participants, owners, deadlines, decisions, next steps, contextual
actions, and an inspectable history of changes.

The research examines how people understand and act through
AI-maintained coordination state, highlighting the importance of
legibility, provenance, action visibility, correction mechanisms,
human override, and the boundary between private and shared information.

Publications

LLYMX

Extended Reality environments can provide immersive support for
industrial training, maintenance, and operational activities.
However, interacting with complex virtual environments often requires
menus, controllers, or predefined commands that do not reflect how
people naturally refer to objects and situations in physical space.

This project investigates the integration of Large Language Models
into Extended Reality environments to enable natural, multimodal, and
context-aware interaction. Users can communicate through speech, text,
and gestures, for example by pointing at or selecting a virtual object
and asking questions about it.

The project resulted in LlymX, a prototype that
connects 3D objects with validated external knowledge sources.
Structured descriptions of virtual objects, semantic embeddings,
vector databases, and Retrieval-Augmented Generation are combined to
identify the object referenced by the user and retrieve relevant
information from technical documents and knowledge bases.

LlymX has also been evaluated through a comparative user study
involving menu-based, infographic-based, and LLM-driven multimodal
interaction. The results showed that the LLM modality was perceived
as significantly more usable than the infographic-based interface,
while differences in task-completion time and cognitive workload were
not statistically significant.

The broader objective is to develop
Extended Intelligent Reality environments in which
AI understands spatial context and provides accurate assistance
directly within immersive experiences.

Publications