Google AI Mode Adding Multimodal Google Lens Search 

AI Mode entered testing a month ago and Google is now expanding access to free Labs users, as well as adding Lens visual search.

At launch, it was just available for Google One AI Premium subscribers. As we previously spotted, Google confirmed today that it’s coming to “millions more Labs users in the U.S.” who are not paying.

Meanwhile, AI Mode is adding multimodal input and understanding that lets you take a new photo with Google Lens or upload an existing image. This lets you “easily ask complex questions about what you see.”

Behind-the-scenes, Google is leveraging Gemini’s multimodal capabilities to “understand the entire scene in an image, including the context of how objects relate to one another and their unique materials, colors, shapes and arrangements.”

Lens precisely identifies each object in the image. Using our query fan-out technique, AI Mode then issues multiple queries about the image as a whole and the objects within the image, accessing more breadth and depth of information than a traditional search on Google. The result is a response that’s incredibly nuanced and contextually relevant, so you take the next step.

In the example above, AI Mode “precisely identifies each book on the shelf and issues queries to learn about the books and similar recommendations that are highly rated.” The end result is a “list of recommended books with links to learn more and purchase,” while you can ask follow-up questions.

Rolling out starting today, Google Lens in AI Mode is available on Android and iOS. Go to the AI Mode homepage for a new Lens icon in the bottom search field. This takes you to the usual Google Lens UI. As users long-press on the shutter button, they can speak their query.

Google today also shared some usage patterns after a month of public testing. People say they like the “clean design, fast response time and ability to understand complex and nuanced questions.” AI Mode queries are said to be “twice as long as traditional Search queries on Google” (on average). It’s being used for exploratory, open-ended questions, and “more complicated tasks — like comparing two products, exploring how-tos and planning a trip.”

Add 9to5Google to your Google News feed.