Carnegie Mellon University’s Human-Computer Interaction Institute (HCII) is developing a system that lets ordinary objects sense, infer, and act on human needs. Instead of waiting for instructions, these objects become “unobtrusive” agents; they observe, predict, and move autonomously to assist, tells Tech Xplore.
Here’s how it works. A ceiling camera monitors the space, capturing object positions and human activity. That visual feed is translated into a textual scene description, which is fed into a large language model (LLM). The LLM infers what the user is likely to do or needs next (e.g., needing the stapler), then sends a command to the object’s wheeled robotic base to reposition itself. The system blends perception, reasoning, and actuation.
Their stapler prototype glides across the desk toward a user just as the person reaches for it. Other envisioned behaviors include knives slipping safely aside, mugs rolling forward when someone extends a hand, or shelf panels sliding out to hold a bag. The key is that users never request these actions explicitly; the system anticipates them.
This work was presented at the 2025 ACM Symposium on User Interface Software and Technology. It’s part of HCII’s broader agenda to integrate Physical AI into daily environments, such as homes, offices, and hospitals, where smart objects act without becoming intrusive. Alexandra Ion, leader of the Interactive Structures Lab, calls this approach “adaptive systems for physical interaction” that blend into life while dynamically helping.
The team acknowledges challenges ahead: safety, reliability, privacy, and trust. But their prototype points toward a future where many items around us gain agency, not replacing human control, but subtly improving convenience.