A method includes receiving, from an image capture device in communication with the data processing hardware, image data for an area of interest of a user. The method also includes receiving a query from the user referring to one or more objects detected within the image data and requesting a digital assistant to discern insights associated with the one or more objects referred to by the query. The method also includes processing the query and the image data to: identify, based on context data extracted from the image data, the one or more objects referred to by the query; and determine the insights associated with the identified one or more objects for the digital assistant to discern. The method also includes generating, for output from a user device associated with the user, content indicating the discerned insights associated with the identified one or more objects.

A contextual query system is configured to perform operations that include: causing display of a graphical user interface at a client device, the graphical user interface including a display of image data that comprises a set of image features; generating a query based on the set of image features of the image data; accessing media content based on the query at a repository, the repository comprising a collection of media content; and causing display of a presentation of the media content within the graphical user interface at the client device.

Systems and methods are disclosed herein for generating supplemental content for media content. One disclosed technique herein generates for display a page of an electronic book. A noun, and a word contextually related to the noun, are identified from the displayed page of the electronic book. Content structures are searched for a content structure that includes a matching object having an object name matching the noun. The content structure includes objects, where each object has attribute table entries. Upon finding an identified attribute table entry of the matching object that matches the related word, a new content structure is generated. The new content structure includes the matching object and the identified attribute table entry. A content segment is generated for output (e.g., for display on the electronic book) based on the new content structure.

Systems and methods are disclosed herein for generating supplemental content for media content. One disclosed technique herein generates for display a page of an electronic book. A noun, and a word contextually related to the noun, are identified from the displayed page of the electronic book. Content structures are searched for a content structure that includes a matching object having an object name matching the noun. The content structure includes objects, where each object has attribute table entries. Upon finding an identified attribute table entry of the matching object that matches the related word, a new content structure is generated. The new content structure includes the matching object and the identified attribute table entry. A content segment is generated for output (e.g., for display on the electronic book) based on the new content structure.

A computing system accesses an image-based document and a text document having text extracted from the image-based document and provides a user interface displaying at least a portion of the image-based document. In response to selection of a text portion of the image-based document, the system determines an occurrence of the text portion within at least a portion of the image-based document and then applies a search model on the text document to identify the same occurrence of the text portion. Once matched, alignment data indicating a relationship between a selected tag and both the text portion of the image-based document and the text portion of the text document is stored.

A system for identifying a person approaching an access way of a sporting event. The is identified using pass-through facial recognition technology in which one or more images of the person are taken while the person is in motion. The person is identified by comparing one of the images taken while the person approaches the access way of the sporting event with a profile image of the person.

This disclosure relates generally to visio-linguistic understanding. Conventional methods use contextual visio-linguistic reasoner for visio-linguistic understanding which requires more compute power and large amount of pre-training data. Embodiments of the present disclosure provide a method for visio-linguistic understanding using contextual language model reasoner. The method converts the visual information of an input image into a format that the contextual language model reasoner understands and accepts for a downstream task. The method utilizes the image captions and confidence score associated with the image captions along with a knowledge graph to obtain a combined input in a format compatible with the contextual language model reasoner. Contextual embeddings corresponding to the downstream task is obtained using the combined input. The disclosed method is used to solve several downstream tasks such as scene understanding, visual question answering, visual common-sense reasoning and so on.

Methods, apparatus, systems, and articles of manufacture are disclosed for processing an image using visual and textual information. An example apparatus includes at least one memory, instructions in the apparatus, and processor circuitry to execute the instructions to detect regions of interest corresponding to a product promotion of an input digital leaflet, extract textual features from the product promotion by applying an optical character recognition (OCR) algorithm to the product promotion and associating output text data with corresponding ones of the regions of interest, determine a search attribute corresponding to the product promotion, generate a first dataset of candidate products corresponding to the product in the product promotion by comparing the search attribute against a second dataset of products, and select a product from the first dataset of candidate products to associate with the product promotion, the product selected based on a match determination.

A system for selecting an image to accompany text from a user in connection with a social media post. The system includes receiving text from the user; identifying one or more search terms based on the text; identifying candidate images from images in one or more image databases using the search terms, where the candidate images comprise a sponsored image; presenting one or more candidate images to the user, where the sponsored image is presented preferentially compared to other candidate images; receiving from the user a selected image from the one or more candidate images; generating the social media post comprising the selected image and the user-submitted text; and transmitting the social media post for display.

A method performed by a computing device includes obtaining a set of image segment identigens for image segments of an image to produce sets of image segment identigens. A set of image segment identigens is a set of possible interpretations of a first image segment of the image segments. The method further includes identifying a subset of valid image segment identigens of each set of image segment identigens by applying identigen rules to the sets of image segment identigens to produce subsets of valid image segment identigens. Each valid image segment identigen of a subset of valid image segment identigens represents a most likely interpretation of a corresponding image segment. The method further includes generating an image entigen group utilizing the subsets of valid image segment identigens, where the image entigen group represents a most likely interpretation of the image.