APPARATUS AND METHOD FOR EVENT TRIGGERING FROM AUDIO CONTENT DIGITAL ID

Abstract

The present document describes an apparatus and method for monitoring and interacting with listeners of audio-video content. There are provided audio signals from which audio-video content can be identified by a monitoring peripheral paired with a user's electronic device. The monitoring peripheral monitors audio signals from its surroundings and communicates to the paired user's electronic device, decoded and/or generated digital IDs from these audio signals. A master service application or registered service application miming on the electronic device could transmit the digital ID to a remote server and retrieve additional information from these digital IDs with or without the use of a backend system. Events could be triggered and relevant audio-video content be presented to the user of the master service application or a registered service provider when digital IDs are received from the monitoring peripheral.

Claims

1. A method of interacting with users of audio-video content, the method comprising: providing a monitoring peripheral for detecting audio signals and an electronic device having a master service application installed thereon; detecting a proximity between the monitoring peripheral and the electronic device; upon detecting the proximity, activating the master service application; providing audio signals containing the audio-video content to be played back, the audio-video content comprising digital IDs; detecting, by the monitoring peripheral, the digital IDs and sending a signal containing the digital IDs to the electronic device having the master service application thereon.

2. The method of claim 1, wherein the audio-video content is generated by a remote content server in connection with an authorization server to authorize the remote content server to broadcast the audio-video content to at least one of the users, further comprising pairing the master service application to the monitoring peripheral by providing an authorization communication between the electronic device and the authorization server.

3. The method of claim 1, further comprising retrieving, by the master service application, additional information for the digital IDs on a local application installed on the electronic device other than the master service application.

4. The method of claim 1, further comprising retrieving, by the master service application, additional information for the digital IDs from a remote backend system.

5. The method of claim 4, further comprising sending by the remote backend system to the electronic device a communication related to the digital IDs.

6. The method of claim 4, wherein a selected portion of the audio-video content is identified by a selected digital ID.

7. The method of claim 6, wherein the selected portion of the audio-video content identified by the selected digital ID comprises a portion of the audio-video content that advertises a selected product and retrieving, by the master service application, additional information for the selected digital ID comprises retrieving information related to the selected product.

8. The method of claim 1, further comprising monitoring the digital IDs to detect when the audio-video content relates to a registered service provider.

9. The method of claim 8, further comprising notifying the master service application in response to a detection of a digital ID that relates to the registered service provider in the step of detecting digital IDs step.

10. The method of claim 9, further comprising notifying, by the master service application, the registered service provider, triggering the registered service provider to generate a communication related to the audio-video content identified by the digital ID that relates to the registered service provider, the communication being directed to the electronic device or to a remote server.

11. The method of claim 10, wherein a selected portion of the audio-video content is identified by a selected digital ID and further wherein if the communication is directed to the electronic device, the communication comprises a call-to-action message associated with the selected portion of the audio-video content identified by the digital ID.

12. The method of claim 8, further comprising receiving, by the master service application, a deactivation signal for a given registered service provider on the electronic device, and in response to the deactivation signal the monitoring the digital IDs is not performed for the given registered service provider.

13. The method of claim 1, wherein each of the digital IDs is at least one of a fingerprint, a watermark, embedded data inside the audio-video content, and embedded data in a transmitting envelope of the audio signals.

14. An apparatus for monitoring audio-video content, the apparatus comprising: an input for detecting audio signals of the audio-video content which comprises a digital ID; a processing unit for decoding the digital ID; a beacon to broadcast a presence of the apparatus to an electronic device; and a short range wireless device to transmit the digital ID to the electronic device if the electronic device is nearby; and wherein the apparatus is adapted to communicate with a master service application on the electronic device, the master service application being adapted to communicate with a remote backend system to retrieve additional information related to the digital ID.

15. The apparatus of claim 14, wherein the electronic device comprises an input and an application, the short range wireless device being adapted to communicate with the input of the electronic device to send the digital ID detected in the audio-video content for use by the application.

16. The apparatus of claim 14, wherein the apparatus is provided substantially as a USB peripheral.

17. The apparatus of claim 16, wherein the USB peripheral is connectable to a set-top box which manages the audio-video content in which the audio signals are detected.

18. The apparatus of claim 14, wherein the apparatus is provided as an internal part of a set-top box which manages the audio-video content in which the audio signals are detected.

19. The apparatus of claim 14, wherein the input for detecting audio signals is a microphone.

20. The apparatus of claim 19, wherein the digital ID is at least one of: a fingerprint, a watermark, embedded data inside the audio-video content, and embedded data in a transmitting envelope of the audio signals.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Further features and advantages of the present disclosure will become apparent from the following detailed description, taken in combination with the appended drawings, in which:

[0057] FIG. 1 is a diagram illustrating a system comprising a backend system of a service provider, in relation with a monitoring peripheral and an electronic device comprising a master service application and registered service providers' applications, according to an embodiment;

[0058] FIG. 2 is a diagram illustrating a system comprising a backend system of a service provider, in relation with a monitoring peripheral and an electronic device comprising a master service application for registered service providers' applications, according to an embodiment;

[0059] FIG. 3 is a flowchart illustrating a method of using a monitoring peripheral and an electronic device comprising a master service application, according to an embodiment; and

[0060] FIG. 4 is a top view illustrating a monitoring peripheral, according to an embodiment.

[0061] It will be noted that throughout the appended drawings, like features are identified by like reference numerals.

DETAILED DESCRIPTION

[0062] The emergence of mobile applications (also known as apps) makes it possible for the viewer to interact and engage with TV and radio content such as programming and advertising. A direct coordination between a mobile app and a TV or radio content is made possible using a monitoring peripheral combined with digital IDs and Bluetooth technology integrated into this monitoring peripheral. Of course, Bluetooth technology is only an example of a type of short range wireless communication technology. Other types of short range wireless devices and technologies are used according to other embodiments.

[0063] The monitoring peripheral could be a device chosen, without limitation, from the following: a custom peripheral, a mobile device, a connected TV or a set-top box. According to an embodiment, the monitoring peripheral uses Bluetooth and beacon technologies to broadcast its availability to devices such as mobile phones and tablets and uses wireless protocols to communicate detected digital ID part of an audio/video content to a master service application installed on that mobile or tablet device. Upon receiving the digital ID, the master service application can communicate with the event notification backend system (backend system) or locally with other applications installed on the mobile or tablet device. After receiving a specific digital ID, the master service application can react according to predefined business rules (i.e., steps known as business logic performed by the system to offer a service to a customer). The definition of these business rules is not part of the scope of this document but could be to have a message (e.g., a push message) be displayed on the TV, or create events on the user electronic device (e.g., smartphone, tablet, or any handheld connected devices), directly inside the app itself, or through messages to a third party backend service. The use of Bluetooth and beacon technologies allows the master service application to be launched only when needed and to activate the communication with the monitoring peripheral, thus allowing the master service application to provide additional information based on digital IDs found in the audio only when the user is in the presence of that audio-video content and nearby the monitoring peripheral.

[0064] The problem with existing solutions is that they offer no reliable relationship between the audio-video content displayed on the television or heard on the radio and the user's proximity to the TV or radio and live interest when exposed to the audio content. Existing solutions also offer no explicit privacy protection to the viewer. With the use of audio-video content recognition (ACR), audio fingerprinting or audio watermark inside smartphones or tablets, existing solutions are limited to one application actively running on the device and using its microphone to receive the audio signal. Existing ACR solutions require significant cloud server infrastructure to support the amount of live triggers on broadcast content, with limited viewer control over the activation and deactivation of monitoring and personalized graphic overlays.

[0065] The term user is used herein to describe a radio listener, TV viewer/consumer, or subscriber of a service provider. The terms monitoring peripheral are used herein to describe a device that detects digital IDs from audio signals, and make uses of wireless and beacon technologies to filter and communicate these digital IDs based on the user's proximity to that device. The terms master service application used herein correspond to a mobile application using the mobile SDK and therefore capable of communicating with a monitoring peripheral and receiving the digital IDs. The terms mobile SDK used herein correspond to the set of libraries used to communicate with the backend system, the monitoring peripheral and the mobile application. The term audio/video device is used herein to describe a playback device of broadcast, on-demand audio or video content, such as a television, radio, mobile, tablet, monitor, or any combination thereof. The terms registered service provider are used herein to describe mobile applications from retailers, brand manufacturers, advertisers, agencies, TV programmers, producers and network operators or any combination thereof for which the application has been registered on the backend system to be notified of detected digital IDs.

[0066] The term electronic device is used herein to describe a mobile device, tablet, smartphone or any handheld connected devices.

[0067] The term beacon used below refers to device enabling the iBeacon technology, commercialized by Apple Inc., the Eddystone technology commercialized by Google Inc., or an equivalent thereof. The beacon thus refers to these devices (or transmitters) that work using Bluetooth low energy technology to broadcast an identifier to nearby portable electronic devices such as smartphones. The beacon uses Bluetooth low energy proximity sensing to broadcast its identifier around it; the signal that is transmitted can be picked up by an app or by the operating system (OS) of the electronic device. Events can thus be triggered on a portable electronic device even though no action is performed on the portable electronic device by the user. Moreover, the portable electronic device does not have to be listening to (or monitoring) its environment; the beacon actively communicate its presence to the portable electronic device.

[0068] The digital ID embedded in the audio signal can be a sound signal provided in the audible spectrum (i.e., in the 20 Hz to 20 kHz range), or in the inaudible spectrum, as long as the sound detector (i.e., microphone) provided in the apparatus (i.e., the monitoring peripheral) can detect the sound frequency in which the digital ID is embedded.

[0069] There is a need for software running on a monitoring peripheral in order to support the different monitoring technologies and the peripheral interactions with the user.

[0070] Registered service providers can make use of the mobile software development kit (mobile SDK) to enable their mobile application to use monitoring peripherals, thus eliminating additional steps to their users to activate the service.

[0071] One embodiment described herein has three components: a monitoring peripheral, an electronic device having a mobile SDK installed thereon, and a backend system. The monitoring peripheral uses wireless communication and beacon technologies in order to be discovered by the mobile SDK (and therefore be discovered by a master service application too) and to notify of detected digital IDs. More precisely, the proximity of the electronic device to the monitoring peripheral will have the beacon provided in the monitoring peripheral wake up a master service application on the electronic device. The electronic device can then connect to the monitoring peripheral and start receiving digital IDs decoded information from the audio signal to the electronic device, either spontaneously or after having been requested by the electronic device. More generally, once a user's mobile application on the electronic device is connected to a monitoring peripheral, the mobile SDK may generate local event notifications, monitor different activities and send event notifications to the backend system based on detected digital IDs.

[0072] In FIG. 1, the relationship between illustrated components shows how the backend system operates, according to an embodiment. Content owners 101 communicate with the backend system 102 for content ingestion, digital ID, content management, delivery and monitoring, and deliver to the user their content integrating digital IDs through audio-video playback devices 109. The backend system 102 communicates with the mobile SDK 111 used by the master service application 107 for content instructions and management and user pairing to the monitoring peripheral 110. The pairing process to create the link between the master service application and monitoring peripherals 110 is done using short range wireless pairing protocols such as Bluetooth and mobile OSes specifics. Once the pairing process completed, the mobile SDK 111 sends back the monitoring peripheral ID to the backend system 102. At this point, all paired monitoring peripherals 110 can start detecting digital IDs from the audio signal of audio-video playback devices 109 and the monitoring peripheral SDK 108 will be notifying of these digital IDs to their paired mobile device's mobile SDK 111. The mobile SDK 111 will then execute content instructions and notify registered service provider applications 103 and the backend system 102 accordingly.

[0073] In FIG. 2, the plurality of relationships between the mobile SDK 111, the registered service provider mobile applications 103 and content owners 101 is shown in relation to the backend system 102. The backend system 102 connects to the mobile SDK 111 through the content and authentication servers. Registered service provider mobile applications 103 connect to the backend system 102 through the content server 213. The monitoring peripheral SDK 108 is integrated in the monitoring peripheral devices 110. Monitoring peripherals 110 receiving, using a microphone or line-in 209, content owners' digital IDs from audio-video playback devices 109 and transmitting these digital IDs to the master service application 107 through the monitoring peripheral SDK 108 and mobile SDK 111. The mobile SDK 111 provides: pairing 211 to a monitoring peripheral, user authentication 205 to the backend system, digital IDs data receiver 202, monitoring peripheral activation triggered by proximity detection using Bluetooth short range wireless device 210 and beacon protocol, manage audio-video content associated with received digital IDs and manage monitoring peripheral OTA software updates 204. The backend system 102 can associate one or more service from registered service provider mobile apps and web services 103. The backend system 102 can provision and manage audio-video contents for one or more registered service provider applications and web services. The monitoring peripheral SDK provides: detection of digital ID client 201 and reception 206, digital ID management 207, transmission over wireless connection 208 and over-the-air (OTA) software update 212.

[0074] In FIG. 3, an example of using the backend system from the perspective of a user monitoring peripheral 110 who has the ability to run the monitoring peripheral SDK 108 software. The process begins with the user providing a new monitoring peripheral or installing a software update on their existing monitoring peripheral (step 401). The user then sets up the new monitoring peripheral 110 or executes the software update on their existing monitoring peripheral (step 402). In step 403, the user device connects the master service application to that monitoring peripheral 110 using short range wireless device 210 or an equivalent wireless connection. In step 404, the monitoring peripheral 110 receives all digital IDs from audio-video playback devices in the living room, for example. The monitoring peripheral 110 then transmits digital IDs to the master service application 107 over the Bluetooth connection 210 (step 405). The master service application eventually receives all digital IDs and sends notifications to targeted registered service providers and to the backend system based on content instructions (step 406).

[0075] FIG. 4 shows an embodiment of a monitoring peripheral. There is shown a USB device 504 that comprises a microphone 501, a status LED 502 and a button 503, used primarily for pairing, and a Bluetooth connection 210 that is beacon-enabled. This device makes use of the monitoring peripheral SDK 108 and is capable of detecting and accumulating digital IDs received from audio-video playback devices 109, to transmit the digital IDs over wireless connection 208 to a master service application 107 integrating the mobile SDK 111.

[0076] Based on content instructions, and according to an embodiment, registered service providers receive event notifications, triggered by digital ID detections, from the master service application or the backend system, the master service application or registered service providers may react from notification with a variety of call-to-action signals, such as sending a notification to the user's mobile hosting the registered service application or computer-based service, or updating the user's shopping basket or shopping list or service data or social media actions, engagement or taking an action relating to a request for more information about a product or service by email, SMS, etc., or for the download of another mobile app or file including but not limited to coupons, offers, music, video, podcasts, or any events can be fulfilled by an action that is taking through a mobile app or web service. The master service application 107 may also just monitor service provider's related digital IDs and report playback activities to backend system 102 who prepares information reports to the service provider's backend system.

[0077] Content owners 101 audio-video assets are assigned digital IDs (identifications such as watermarking, fingerprinting or custom ID value) and those digital IDs distributed to the user using audio-video playback devices 109 and accessible by the monitoring peripheral SDK 108 to be detected by digital ID client 201. Content owners 101 without mechanism to assigned digital IDs can use backend system 102 custom ID system to identify their audio-video assets. The monitoring peripheral 110 will then be able to receive such digital ID using standard data transmission demodulation techniques in the audible spectrum, process and package the digital IDs and transmit such packaged digital IDs to the master service application 107 over the monitoring peripheral 110 wireless connection. The mobile SDK 111 detects a monitoring peripheral 110 using Bluetooth and beacon protocols identification is triggering the user mobile device operating system to start the master service application 107 digital ID data receiver 202 which begins receiving digital IDs over wireless. The received digital IDs over wireless are analyzed by the Content Manager 203. The content manager 203 notifies the master service application 107 and the backend system 102 of that digital ID. The master service application 107 or the backend system 102 content server 213 (aka the remote content server) notifies the registered service provider backend who could generate its call-to-action signal and send an appropriate notification to the user through its mobile application or web service. In other embodiments, the master service application 107 or the backend system 102 may take responsibility and send the notification directly to the registered service provider application 103. According to an embodiment, the backend system 102 is not remote; it is then considered as a local system.

[0078] Although the algorithms described above including those with reference to the foregoing flow charts have been described separately, it should be understood that any two or more of the algorithms disclosed herein can be combined in any combination. Any of the methods, algorithms, implementations, or procedures described herein can include machine-readable instructions for execution by: (a) a processor, (b) a controller, and/or (c) any other suitable processing device. Any algorithm, software, or method disclosed herein can be embodied in software stored on a non-transitory tangible medium such as, for example, a flash memory, a CD-ROM, a floppy disk, a hard drive, a digital versatile disk (DVD), or other memory devices, but persons of ordinary skill in the art will readily appreciate that the entire algorithm and/or parts thereof could alternatively be executed by a device other than a controller and/or embodied in firmware or dedicated hardware in a well-known manner (e.g., it may be implemented by an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable logic device (FPLD), discrete logic, etc.). Further, although specific algorithms are described with reference to flowcharts depicted herein, persons of ordinary skill in the art will readily appreciate that many other methods of implementing the example machine readable instructions may alternatively be used. For example, the order of execution of the blocks may be changed, and/or some of the blocks described may be changed, eliminated, or combined.

[0079] While preferred embodiments have been described above and illustrated in the accompanying drawings, it will be evident to those skilled in the art that modifications may be made without departing from this disclosure. Such modifications are considered as possible variants comprised in the scope of the disclosure.