MIT and the MIT-IBM Watson AI Lab have introduced a technique that helps vision-language models go beyond recognizing generic object classes and instead localize personalized objects in new scenes, tells MIT News. Typical models excel at identifying a “dog,” “chair,” or “mug,” but struggle when asked to find this particular dog or your mug in a new image.
The team’s approach harnesses video tracking data in which a specific object is followed across multiple frames. This forces the model to rely on contextual and visual cues rather than memorizing generic object categories. The training encourages the model to focus on stable features, such as shape, texture, and spatial relationships, to distinguish that same object in a fresh image.
Given a few “in-context” images of the target object, the retrained model successfully pinpoints it in a query image. In tests, this new model version outperformed previous systems on the task of in-context localization, while retaining all other recognition and generation capabilities.
This method opens doors for AI systems that can track user-specific items over time, say, your pet, a favorite mug, or a unique tool. Potential uses include assistive technologies (help visually impaired users find things), ecological monitoring (tracking a tagged animal), or personalized surveillance and robotics.
One key insight is that vision-language models don’t natively inherit the same contextual learning abilities that language models do. The team addressed this by carefully curating the training data so that the model cannot rely on memorization.
In all, this work moves us closer to AI systems that understand the world more like humans do—learning from a few examples and generalizing flexibly. As applications demand more fine-grained, user-specific perception, this technique may become an essential component in augmented vision and smart assistants.