Augmented mirror

Abstract

An augmented mirror comprising: a partially silvered mirrored surface; a screen underneath the mirrored surface; a camera; and a computer module for receiving data from the camera and for providing graphical images to the screen, wherein the field of view over which visual information is processed is reduced to visual information from within a visual corridor, the visual corridor corresponding to a subset of the field of view of the camera.

Claims

1. An augmented mirror comprising: a partially silvered mirror surface; a screen disposed underneath the mirror surface; a camera having a field of view; external lighting; and a computation module configured to: receive visual information from the camera; reduce the visual information from the field of view to a visual corridor, the visual corridor based on at least one of a shape of the mirror surface or a size of the mirror surface and sized to match a region over which a user can see their reflection in the mirror surface; process the visual information over the visual corridor; detect that the user has entered the field of view of the camera; determine a perpendicular distance of the user relative to the mirror surface; compare the perpendicular distance to an optimal distance to determine a difference; after detecting that the user has entered the visual corridor, control the external lighting to gradually increase illumination of the external lighting as the difference decreases; display one or more images on the screen based upon the processed visual information; initiate a timer based on the visual information; compare the timer to a time threshold; and vary illumination of the external lighting when the timer exceeds the time threshold.

2. The augmented mirror of claim 1, wherein the camera is an RGB-D device which includes a depth scanner for capturing an image and associated depth information of the user.

3. The augmented mirror of claim 2, wherein the RGB-D device includes input software to extract skeletal data points of each detected person within the field of view.

4. The augmented mirror of claim 3, wherein the computation module includes elimination software configured to: process the skeletal data points extracted by the RGB-D device; extract a position of a specific skeletal region for each detected person detected by the RGB-D device; and discard data corresponding to any detected person whose specific skeletal region is located outside of the visual corridor.

5. The augmented mirror of claim 4, wherein the elimination software is further configured to: discard data corresponding to any detected person other than a selected detected person, where the selected detected person has a specific skeletal region that is closest to the RGB-D device.

6. The augmented mirror of claim 5, wherein the computation module further comprises: execution software which receives non-discarded data from the elimination software and processes the non-discarded data.

7. The augmented mirror of claim 2, wherein the visual corridor is provided by way of maximum x, y, and z inputs at the RGB-D device.

8. The augmented mirror of claim 1, wherein the computation module is configured to perform a linear progression of illumination of the external lighting as the illumination of the external lighting is gradually increased.

9. The augmented mirror of claim 8, wherein the computation module is configured to perform two linear progressions of illumination of the external lighting as the illumination of the external lighting is gradually increased, the computation module being configured such that the two linear progressions meet when the user is located at the optimal distance.

10. The augmented mirror of claim 1 in combination with a portable electronic device.

11. The augmented mirror of claim 1, wherein the computation module is further configured to start an interaction session when the user is at the optimal distance.

12. A mirror system comprising: a mirror surface; a cam era; a light panel comprising a first lighting element, a second lighting element, and a plurality of third lighting elements positioned between the first lighting element and the second lighting element; and a computation module configured to: receive visual information from the camera; reduce the visual information to a visual corridor, the visual corridor based on at least one of a shape of the mirror surface or a size of the mirror surface and sized to match a region over which a user can see their reflection in the mirror surface; determine, based on the visual information, a position of the user relative to the mirror surface; initiate a timer based on the visual information; compare the timer to a time threshold; vary illumination of at least one of the first lighting element and the second lighting element when the timer exceeds the time threshold; compare the position to an optimal position to determine a difference; and while the first lighting element and the second lighting element are illuminated, gradually increase illumination of the plurality of third lighting elements as the difference decreases.

13. The mirror system of claim 12, wherein the computation module is further configured to: detect that a user has entered a field of view of the camera; and after detecting that the user has entered the field of view of the camera, illuminate the first lighting element and the second lighting element.

14. The mirror system of claim 12, wherein the computation module is further configured to start an interaction session when the user is in the optimal position.

15. A mirror system comprising: a partially silvered mirror surface; a screen disposed underneath the mirror surface; a camera having a field of view; external lighting; and a computation module configured to: receive visual information from the camera over the field of view; reduce the field of view to a visual corridor, the visual corridor corresponding to a subset of the field of view and being based on at least one of a shape of the mirror surface or a size of the mirror surface and sized to match a region over which a user can see their reflection in the mirror surface; process the visual information over the visual corridor; detect that the user has entered the visual corridor; after detecting that the user has entered the visual corridor, obtain a distance between the user and the mirror surface; compare the distance to an optimal distance to determine a difference; control the external lighting to gradually increase illumination of the external lighting as the difference decreases; display one or more images on the screen based upon the processed visual information; initiate a timer based on the visual information; compare the timer to a time threshold; and vary illumination of at least the external lighting when the timer exceeds the time threshold.

16. The mirror system of claim 15, wherein the computation module is further configured to start an interaction session when the user is at the optimal distance or close to the optimal distance.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) Embodiments of the invention will now be described by way of example with reference to the accompanying drawings in which:

(2) FIG. 1 shows a schematic diagram of an augmented mirror;

(3) FIG. 2 shows a schematic diagram of an alternative augmented mirror;

(4) FIG. 3 shows a schematic diagram of software for an augmented mirror;

(5) FIG. 4 shows a schematic diagram of housing for an augmented mirror;

(6) FIG. 5 shows a schematic diagram of two users using an augmented mirror;

(7) FIG. 6 shows a schematic diagram of a user using an augmented mirror;

(8) FIG. 7 shows a schematic diagram of co-ordinates of an augmented mirror;

(9) FIG. 8 shows a schematic diagram of a method of determining 2D pixels at which graphics should be positioned for a specific user;

(10) FIG. 9 shows an example of a FMCG container for use with an augmented mirror.

(11) FIG. 10 shows a schematic diagram of a method of lexicology analysis;

(12) FIG. 11 shows a schematic diagram of a second method of lexicology analysis;

(13) FIG. 12 shows a schematic diagram of a walled system including an augmented mirror;

(14) FIG. 13 shows a schematic diagram of an alternative walled system including an augmented mirror;

(15) FIG. 14 shows a schematic diagram of an alternative augmented mirror with a visual corridor functionality;

(16) FIG. 15 shows a schematic diagram of a method carried out by the augmented mirror of FIG. 14; and

(17) FIG. 16 shows a schematic diagram of a method of operating external lighting and drawing/displaying an overlay of an object.

DETAILED DESCRIPTION AND FURTHER OPTIONAL FEATURES OF THE INVENTION

(18) An example of an augmented mirror is described below in relation to FIGS. 1 and 4. The augmented mirror 1 comprises a mirrored surface 2 under which a display screen 20 is located, and a camera 3.

(19) Both the display screen 20 and the camera 3 are communicably connected to a computer module 4. The display screen is connected to the computer module 4 via a first connection 21 such as an HDMI cable, and the camera is connected to the computer module by a second connection 22 such as a USB connection.

(20) The camera forms part of a camera module such as an RGB-D device which includes a visual camera and a depth scanner for capturing an image and associated depth information of a user of the mirror. The RGB-D device may be purpose built, or may be a pre-existing product such as a Microsoft Kinect 2 motion sensing input device.

(21) The mirrored surface is partially silvered so that the display screen can be seen through the mirrored surface. In the embodiment shown in FIG. 1, the partially silvered surface is a surface of the display screen itself onto which the partially reflective material has been coated directly. However, it is envisaged that a partially silvered surface could be provided as a separate component from the display screen and then positioned directly in front of the display screen.

(22) The partially silvered mirrored coating should be chosen to have a reflectivity/transparency ratio which is optimised to give maximum reflectivity but still allow a high level of image brightness to be transmitted through the mirror surface. An example of a suitable transmittance value is 66% and an example of a suitable reflectance value is 31%. However, the skilled person would understand that other values would be acceptable.

(23) The computer module comprises one or more processors including a central processing unit (CPU) in communication with a memory and various other components.

(24) These other components may include a power supply, a network interface (in some cases not desired), an input/output interface, an audio interface, and user controls. The mirror may be voice activated, other audio triggers may include turning on of hairdryer.

(25) The computer module is programmed to receive data from the camera 3 and to provide graphical images to the display screen so that they are visible through the (partially silvered) mirrored surface 2. A routine learning algorithm is used for capturing and analysing data from the camera and providing information on the user based on that analysis.

(26) An (optional) housing unit 25 encloses the mirrored surface, display screen and camera. The housing would be advantageous in a bathroom setting to prevent against water damage.

(27) An augmented mirror with additional features is described in relation to FIG. 2 where like components are given the same numerals as in FIG. 1.

(28) In addition to the components found in FIG. 1, the augmented mirror of FIG. 2 includes a plurality of light panels in the form of 2 elongate light panels, in this case LED panels, one panel located on either side of the mirror. A lighting controller 8, such as an Arduino™ microprocessor provides an interface between the plurality of lighting panels and the computer module. Connections between the lighting controller 8 and the computer may be USB connections or any other suitable connectors. An additional power source supplies power to the lighting panels.

(29) The computer module may control the light emitted by the lighting panels in response to inputs received by the camera so that the lighting acts as a feedback mechanism. The lighting may be triggered by a threshold having been reached. Such thresholds may include time thresholds (e.g. a certain amount of time during which a particular action such as tooth brushing has taken place). Alternatively, or in addition, a threshold which triggers a certain lighting event may be a distance trigger (such as approaching the mirror).

(30) Each lighting panel may be made up of a plurality of lighting segments, the segments being individually controllable so that certain parts of the lighting panel can be controlled individually. For example, as a user approaches the mirror, the individual segments of the lighting panels may sequentially light up from one and of the panel to another. When all segments within the lighting panels are lit up, this indicates to the user that they are standing in an optimal position.

(31) By way of example, the mirror may include a pair lighting panels, each made up of a plurality of lighting segments. Each panel may be generally linear, and arranged vertically along a respective side of the mirror, for example. When a user is identified as having entered in to the field of view of the mirror, or alternatively within a visual corridor of the mirror (described below), the illumination of the lighting panels may be controlled to indicate to the user that they are approaching an optimal position for interaction with the mirror.

(32) For example, when a user enters the field of view of the mirror, or the visual corridor, then the uppermost and lowermost lighting segment of each lighting panel is illuminated. As the user moves towards an optimal position for interaction with the mirror (for example, the centre of the field of view or visual corridor) the remaining lighting segments of each panel are sequentially illuminated. The sequence of illumination may begin with the uppermost and lowermost segment (as described), with illumination of the lighting panel progressing gradually towards the centre of the panel from the uppermost and lowermost segments, respectively. In other words, as the user moves from their initial position closer towards the optimal position, the uppermost-but-one segment and the lowermost-but-one lighting segment may both be illuminated. As the user continues to move towards the optimal position, the remaining lighting segments are sequentially illuminated. As will be appreciated, as the user moves towards the optimal position the effect of each lighting panel appears to the user as a “pinch” effect, where the lighting segments are sequentially illuminated from the top and bottom of the panel to eventually “meet” at the centre of the lighting panel. The lighting controller may be configured to illuminate the segments of the panel so that the illumination sequence from the top of the panel and the illumination from the bottom of the panel meet at the centre of the panel when the user in the optimal position. The closer the user is to the optimal position, the more the LEDs converge at the vertical centre of the mirror. When the user is successfully located in the optimal position, all, or a portion, of the lighting segments may be switched off.

(33) If a user moves away from the optimal position in front of the mirror, the “pinch” illumination of the lighting may progress in reverse. That is to say that the segments in the centre of the lighting panel are extinguished first, followed by progressive extinguishing of the segments in an upwards and downwards direction towards the uppermost and lowermost segment, respectively.

(34) Such a lighting configuration provides an intuitive method of indicating to a user when they are in the optimal position for interaction with the mirror. Such a lighting configuration can be used to indicate to the user the optimal angular position relative to the mirror, and/or optimal distance for the user from the mirror. The lighting configuration described above forms a feedback mechanism for the user, in so far as the consequences of the user's movement for the optimal operation of the mirror are ergonomically conveyed to the user. In this way, the user is able to place themselves in the optimal position for the mirror without needing to understand that there is an optimal position, or where that optimal position is located. This configuration of the lighting abstracts away from the optimal position concept for the user, and ergonomically and simply allows a user to put themselves in the optimal position.

(35) A use of the lighting described above is shown in the schematic shown in FIG. 11.

(36) Lighting panels may provide specific lighting levels to stimulate a reproducible or specific environment. A light sensor such as a photodiode (not shown) can detect the intensity of light and/or wavelengths of light reflected from a user to provide a measurement of the actual levels and enable specific intensities or hues to be achieved by varying the intensity and/or wavelength of the light emitted by the lighting panels. The lighting panels used for this may be separate from those lighting panels used for feedback and/or data transfer so that the different lighting functions (atmospheric and feedback) can be carried out simultaneously. By creating reproducible conditions for image capture the user can more accurately assess product performance within certain lightfields (office/home/sunny day/restaurant etc.

(37) The lighting controller can be configured to flash or pulse and therefore to transmit information from the computer module 4 to FMCG objects/containers/devices 10, 110 optically via high frequency flashes, the control if such information being achieved by the computer module.

(38) One or more elements of the light panel may be directional and/or focusable. The lighting panels may be highly controllable, adaptable to different scenarios and may interact with applications/programs. For example, the lighting could be used as a timer indicator, with a lighting signal such as full lights or no lights indicating that an activity such as brushing of teeth should be stopped.

(39) In addition to visible light provided by the lighting panels, an additional light source (not shown) may be present to provide IR and/or UV light. Such additional light source(s) provide a data transference mechanism which is not visible to the user.

(40) The embodiment shown in FIG. 2 also includes an audio interface made up of a microphone (not shown) for receiving audio signals such as voice commands from a user, and a speaker 6 for relaying audio information to the user. Both the speaker and the microphone are communicably connected to the computer module 4, for example via a USB connection.

(41) The embodiment of FIG. 2 also included a communication module 7 which enables the augmented mirror to communicate wirelessly with a user device 10. The user device could take the form of a mobile device such as a cell phone, tablet or laptop. Alternatively, it may take the form of a smart FMCG such as a Bluetooth enabled toothbrush. In addition to its own communication module, the smart FMCG may have other smart features such as an accelerometer, sound sensors, light sensors, a camera, a gyroscope and a chemical analyser. Information from one or more of these smart features can be communicated to the computer module of the augmented mirror 101 via the communication module 7. In addition to Bluetooth, other suitable methods of wireless communication can be supported by the communications module 7 such as WiFi, Near field communication (NFC), and Bluetooth LE (low energy).

(42) It is envisaged that any one of or combination of the extra features described in relation to FIG. 2 could be applied to the embodiment of FIG. 1.

(43) The Augmented Mirror allows the use of hand-held devices with built-in cameras and Bluetooth or WiFi data transmission capability to let users see parts of their body that they would not normally be able to see given the location of their eyes. This interaction is easy to use as the user can see in the Mirror Screen both the location of the hand-held device as well as the image captured by this device. This allows the user to easily control and navigate the hand-held device.

(44) Users of the Augmented Mirror can install programs from online repositories. Some programs can interact with the mirror from online servers. To allow this whilst preserving security and privacy, any interaction between such online servers and the Mirror Operating System (OS) has to be done by Mirror OS api calls.

(45) FIG. 3 shows an overview of the Mirror OS software components of the embodiment shown in FIG. 2 where like numbers correspond to those features found in FIGS. 1 and 2. Processing of visual and depth information received by the camera module 3 is achieved in two main stages. At a first stage, software located at the actual camera module itself performs analysis of the data. For example, software at the camera module may include one or more of: Face recognition; Lexicology analysis (one example of which would be a “Thumbs up recogniser”; and a glyph recogniser. In a second stage, further applications may be run within the computer module 4 of the augmented mirror 1, 101. Objects such as FMCG containers can function as 1st class interaction characters in the context of screen-based interactions. The FMCG's presence can be detected by vision sensing; for example by RGB or RGB-D cameras. Alternatively, they could be detected by wireless communication techniques such as Bluetooth, WiFi or NFC. An example of an FMCG container used as a controller is depicted in FIG. 9.

(46) When using RGB-D, location, and orientation of the FMCG container can also be determined.

(47) In some embodiments, the detection of an FMCG container will trigger loading of product-specific programs or online content. It could also enable limited user input to programs displayed on the screen by using the presence or absence of the FMCG as a binary input.

(48) Vision-based sensing would allow the FMCG to be used as a multi-modal controller, where its orientation and location can serve as fine-grained inputs to the programs.

(49) Finally, combining the vision-based sensing of the FMCG's location with visual inputs received from the user (e.g. sensing the user's (head) pose, gestures and facial expressions) allows a third set of inputs to the program.

(50) The operation of the augmented mirror is described in more detail in relation to FIGS. 5 to 8.

(51) In the embodiment shown in FIGS. 5 to 8, the camera 3 of the augmented mirror is an RGB-D scanner with integrated face alignment. As described in more detail below, the computer module, which is communicably connected to the display screen, 20 has been fitted with an algorithm that will allow any user to see digital content related to a real-world object at the correct position relative to this object.

(52) This is a non-trivial problem because, as shown in FIG. 5, the projection of the object will be in a different location on the Mirrored Screen depending on the perspective of the user. User A will “see” the reflection of the object at point RA on the mirror, whereas user B will “see” the reflection of the object at point RB of the mirror.

(53) It is the location RA/RB at which the reflection appears for that specific user (i.e. the point at which the path of the light from the object to the user's eyes reflects at the mirrored surface) that must be determined in order to overlay graphics at the correct location. In this way, graphics such as an arrow pointing to the reflection of an object, highlighting a specific part of an object or surrounding a reflection of an object can be reliably produced.

(54) This enables all users to interact with graphics on the screen positioned relative to their reflections with a high degree of accuracy.

(55) The perspective of the user depends on the location of a person's eyes relative to the mirror as well as the location of the object. Thus, to accurately place graphics on the display screen 20 both the eyes and the object must be located in the real-world space, i.e. in terms of 3D real-world coordinates, where the coordinate system is relative to the mirrored surface (MS) the origin is in the centre of the MS, the x-axis parallel to the horizontal side of the MS (in use), the y-axis parallel to the vertical side of the MS (in use), and the z-axis perpendicular to the MS plane.

(56) To detect the eyes of a user (s2), the RGB-D camera's face alignment algorithm is used. This returns the 3D coordinates of the eyes of any user (s3) in view of the camera in a coordinate system centred on the camera. A program running on the computer module 4 of the augmented mirror transforms this measurement by the camera in the camera coordinate system to MS real-world coordinates. This transformation requires a one-off measurement of the relative position of the camera with respect to the MS coordinate system origin. In the embodiments described herein, the origin of the real-world coordinate system is the centre of the mirror's surface. However, another suitable point such as the corner of the mirror could be used as the origin. With this measurement of the camera position with respect to the mirror, the eye positions (as measured by the camera) can be rotated and translated into the real world coordinate system.

(57) Bespoke object recognition software (located in the computer module 4) locates one or more objects (s4), in this case a consumer products (FMCG container). In order for the object recognition software to work with new objects, new object recognition software must be added or existing software must be updated to include (to “learn”) the new objects.

(58) Once the location of the user's eyes and the object has been determined, projective geometry is used to calculate the point x° on the mirror/screen where the object is projected (s5). The scene is observed from a RGB-D camera, which provides the 3D coordinates in a reference system relative to the camera (3D Camera World Coordinates) of the head and of the object of interest. The task is to infer the 2D coordinates in the 2D mirror coordinates, where the reference system is in correspondence with or can be mapped on to the pixel locations of the screen.

(59) Given a 3D point on an object as observed by the RGB-D camera, the distance d to the mirror is computed first. Then, the point is translated perpendicularly to the mirror and in its direction by twice this distance (see FIG. 6). The point in the mirror as observed by the user is then computed by finding the intersection between the mirror plane and the line joining the virtual 3D point just computed and the location of the eyes of the user. The resulting point is however still in 3D Camera World coordinates. Its translation to the 2D mirror coordinates (Mirror Space, in pixels) can be directly computed by knowing the location of the centre of the Mirror Screen in terms of 3D Camera Space coordinates and resolution of the Mirror Screen in terms of pixels per inch of the screen (as shown in FIG. 7).

(60) In particular, the coordinates of the mirror plane in the 3D Camera World coordinates are given by a point in the plane and the vector normal to the plane:
P.sub.m={p,v}, where ∥v∥.sub.2=1

(61) The specular location of 3D point x is given as:
x′=x−2((x−p).sup.tv)v

(62) The point in the mirror onto which the reflected object is perceived, x.sup.0 is computed as the intersection of the plane Pm and a line:

(63) $L_1=\left\{x^{\prime },\stackrel{.fwdarw.}{r_1}\right\},\mathrm{where}\stackrel{.fwdarw.}{r_1}=\frac{x^{\prime }-s}{{.Math.x^{\prime }-s.Math.}_2}$

(64) Alternatively, the same point can be computed as the intersection between the two following lines:

(65) $L_1=\left\{x^{\prime },\stackrel{.fwdarw.}{r_1}\right\},\mathrm{where}\stackrel{.fwdarw.}{r_1}=\frac{x^{\prime }-s}{{.Math.x^{\prime }-s.Math.}_2}$$L_2=\left\{s^{\prime },\stackrel{.fwdarw.}{r_2}\right\},\mathrm{where}\stackrel{.fwdarw.}{r_2}=\frac{x-s^{\prime }}{{.Math.x-s^{\prime }.Math.}_2}$

(66) Finally, it is necessary to translate (s6) from the 3D as shown in FIG. 6:
(v.sub.x.sup.tx.sup.0−p.sub.0,v.sub.y.sup.tx.sup.0−p.sub.0)

(67) For optimal accuracy, the angle of an object may be taken into account. The conversion into pixels on the display screen 20 therefore involves the determination of the following physical variables: RGBDF axis angle: The number of degrees the RGBD sensor is tilted down RGBDF offset Y: The distance in meters that the RGBD sensor is above the top left corner of the mirror RGBDF offset X: The distance in meters that the RGBD sensor is offset from the top left corner of the mirror ScreenHeight: the X resolution of the screen in pixels ScreenWidth: the Y resolution of the screen in pixels screenDotsPerMeter: the number of pixels per meter square

(68) Once these physical variables are known, the following operational steps can be carried out:

(69) The application receives two pieces of information as 3D vectors in meters from the RGBD sensor: 1. The user's eye position 2. The position of the object to overlay

(70) Once these information inputs have been established, the six transformation steps are carried out:

(71) 1. Rotation and translation of eye positions around RGBDF axis angle to correct for the angle and offset (using RGBDF axis angle, RGBDF offset Y, RGBDF offset X)

(72) 2. Rotation and translation of object positions around RGBDF axis angle to correct for the angle and offset (using RGBDF axis angle, RGBDF offset Y, RGBDF offset X)

(73) 3. Calculation of the reflected position of the object

(74) 4. Calculation of the vector between the new eye position and the reflected position of the object

(75) 5. Calculation of the intersection of this vector with the mirror in real world coordinates

(76) 6. Conversion of the intersection point to pixel coordinates (using Screen Height, ScreenWidth, screenDotsPerMeter)

(77) These six steps return the x,y coordinates in pixels on the display screen 20 at which a specific user sees the reflection of the object.

(78) The lexicology analysis may include the recognition of a number of gestures on the part of a user. Such gestures may include, for example, a thumbs up gesture (to indicate acceptance, or a positive answer), and a thumbs down gesture (to indicate denial, or a negative answer). It will be appreciated that other gestures may also be recognised by the lexicology analysis. The gestures may not be limited to hand gestures, for example, the lexicology analysis may recognise facial/head gestures, for example, nodding or shaking of the users head to indicate positive and negative responses. The ability to extract lexicological information allows a user to interact with the mirror in an extremely intuitive and ergonomic way. In particular, little teaching is required before a user can use such a mirror because the lexicology analysis is able to recognise actions with which the user may already be familiar from their day-to-day life. In other words, a user does not have to learn a special set of commands in to order to interact with the mirror including the lexicology analysis.

(79) Two particular lexicological interactions are described in detail below. Both interactions require the identification of the position of a user's hand, as seen by the user, in the image reflected from the mirror, and perceived interaction between the user's hand an image displayed on the screen that is coincident with the user's hand (as perceived by the user from their viewing position).

(80) As a first calculation, the real-world 3D position of the user's eyes is determined, according to the method described above. In a similar manner, the real-world 3D position of a user's hand is also determined. In one example, the position of the centre of the user's hand is determined. In another example, the position of the centre of the user's palm is determined.

(81) In accordance with the methodology described above, the position of the user's hand or palm in reflected space is calculated. In turn, the 2D position of the hand/palm's reflection on the surface of the mirror is calculated. The 2D position on screen generally corresponds to the coordinates on the screen at which the user sees the reflection of the hand (from their perspective from the real-world 3D position of the user's eyes).

(82) The 2D position of the hand on the screen may then be used a point of interaction. On the screen of the mirror, an interface component may be displayed. Such an interface component may generally correspond to a region of the graphical display with which a user may interact. Examples of interface components include buttons, toggles, handles (which may be turned), knobs (which may be turned). The skilled person will appreciated that there is a range of interface components that are possible. Each interface component has a position on the screen, and the user can see the interface component at that position. For example, an interface component has an interaction object centre, defined by 2D screen coordinates (Ix, Iy). The interface component is also displayed with a finite size on the screen: a rectangular interface component, for example, has an interaction object width, Iw, and an interaction object height, Ih. It will be appreciated that other interface component shapes are equally possible, for example circular interface components.

(83) Two example modes of interaction with an interface object are described, which each mode utilising the point of interaction. The first mode interaction is “grabbing”, which is illustrated in the schematic shown in FIG. 10. In the grabbing mode, the user aligns the reflection of their hand (as seen from the user's eye position) with an interface component with which they wish to interact. When the user perceives alignment between the reflection of his/her hand and the interface component, he/she can close their hand into a first to interact with the interface component. For example, closing the first may cause the interface component to be “grabbed”. Conversely, when the user opens his/her hand, the interface component is “released”. The grabbing and releasing functions effectively implement a drag-and-drop functionality for the mirror. The closing of the user's hand may also cause a different interaction with the interface component, for example closing the hand may cause a button press of the interface component.

(84) An example methodology of implementing the hovering mode of interaction is shown in FIG. 11. The method begins with face and eye detection and position measurement, and concludes measuring the time that the user's hand is in-position hovering over the interface component to determine if that period exceeds the predetermined trigger period, before triggering selection of the interface component.

(85) The second mode of interaction is “hovering”, which is illustrated in the schematic shown in FIG. 11. In the hovering mode, the user aligns the reflection of their hand (as seen from the user's eye position) with an interface component with which they wish to interact. When the user perceives alignment between the reflection of his/her hand and the interface component, he/she can hold their hand still over the interface component. The mirror is configured to measure the period of time during which the user holds their hand over the interface component. When the user has held his/her hand over the interface component for at least a predetermined trigger period an interaction with the interface object is triggered. For example, the object may be “grabbed” or selected. The hovering function implements simple interaction with the mirror using only the user's reflection as perceived by the user.

(86) The method shown in FIG. 11 begins with face and eye detection and position measurement, and concludes measuring the time that the user's hand is in-position hovering over the interface component to determine if that period exceeds the predetermined trigger period, before triggering selection of the interface component.

(87) For both the grabbing and hovering modes of interaction, a degree coincidence of the reflection of the user's hand and the interface component is calculated.

(88) Assuming the interaction point is rectangular, the following are known: Reflected hand centre 2D coordinate=(Hx,Hy) Interaction component centre 2D coordinate=(Ix,Iy) Interaction component width=Iw Interaction component height=Ih

(89) For example, the interaction between hand and interface component is triggered when the follow condition is true:
(Hx+(Iw/2))<(Ix+(Iw/2))AND(Hx−(Iw/2))>(Ix−(Iw/2))AND(Hy+(Ih/2))<(Iy+(Ih/2))AND(Hy−(Ih/2))>(Iy−(Ih/2))

(90) The above condition essentially amounts to determining if the reflected hand 2D coordinate is within the rectangular interaction component, as perceived by the user.

(91) In the case of the hovering functionality, there may also be predetermined trigger period condition implemented, as described above.

(92) Both the grabbing and hovering modes of interaction are highly ergonomic and intuitive for the user, whom is able to interact easily with the mirror without the need to learn specific commands.

(93) The interface component may be a part of the user displayed on the screen.

(94) In some embodiments, where security is an important consideration, the augmented mirror forms part of a walled system. Examples of such systems are described below with reference to FIGS. 12 and 13.

(95) The system of FIG. 12 includes an augmented mirror (such as any of those described above in relation to FIGS. 1 to 4) where a display screen 20 is located underneath a mirrored surface and a camera module including a depth scanner for capturing an image and associated depth information of a user of the mirror. The augmented mirror 1 will comprise a computer module 4 which itself includes one or more processors configured to process images received from the camera module and control images displayed on the display screen.

(96) Where the augmented mirror 1 forms part of a walled system, the sole means of communication between the outside world and the computer module is a connection to the one or more user objects 10. In this case, the user objects 10 are physical products (such as FMCGs) sold with data already embedded.

(97) As described elsewhere within this application, the connection between the augmented mirror 1 and the object 10 could, for example be an optical connection or a wireless data connection. In the embodiment shown in FIG. 12, data from an object 10 can be received by a communication module 7 on the augmented mirror i.e., the traffic of data is one way, with only the mirror receiving data from the product. This embodiment may be preferred for users for whom security is a concern since no data is sent to the product.

(98) By avoiding a network connection between the computer module 4 of the augmented mirror 1 and any other devices, security risks are minimised because the data stays in home. That is to say, increased privacy is provided since there is no need to connect to internet of things.

(99) This mechanism of data transfer is most feasible where the one or more user objects include FMCG containers since the containers of the products are regularly purchased and can therefore provide constant and regular updates. In other words, the present invention is not concerned with “big data”; instead it aims to help a user (who is also a consumer of FMCG), to get the most out of the FMCG products.

(100) The embodiment shown in FIG. 13 differs from the embodiment shown in FIG. 12 in that, the augmented mirror is capable of transmitting data 72 to the user and to the object 10.

(101) In the embodiments shown in FIGS. 12 and 13 computer module 4 comprises a built in database. The database may include personal information of user which may be entered into the database by the system actively or passively (“cold start”).

(102) The database may have pre-compiled software templates which can be modified and activated by data from products (as described in more detail below). The processor may utilise user information to optimise/personalise software updates. In addition, the database may consist of a range of applications related to different use cases (tooth brushing, skin cream application, hair styling).

(103) In some embodiments, the product can be passive, data transfer triggered/managed by mirror.

(104) Data located on the product may be encrypted and may contain an ID key associated with the data. In this way it is possible to avoid the risk of hacked/false products tampering with the mirror. In some embodiments, a one-way (i.e. a receiving only) network connection can be used to update ID keys at the mirror.

(105) In some embodiments, the augmented mirror may comprise hardware updates (e.g. diagnostic sensors) which are activated upon the receipt of data from FMCG product.

(106) In some embodiments, certain algorithms/variables/attributes of these applications running from a memory of the augmented mirror would have a generic base setting. If data is received from an FMCG object, this would then update and “personalise” that particular application. The variable/new algorithm/attribute information would be contained directly in the FMCG object. The transfer of information could be achieved via wireless data transmission (e.g. Bluetooth) or visually (e.g. encoded in a barcode or similar marking on the container of an FMCG.

(107) Alternatively, or in addition, the range of applications stored in a memory of the augmented mirror is complete. Each application may be “unlocked” by way of a digital “key”. The digital key may take the form of a data transmission from a specific FMCG object and may be encoded. An advantage of such an embodiment is a much greater range of potential changes, with only limited amounts of data being transferred from the FMCG to the mirror. However, a disadvantage is the fact that the choice of applications is limited to what is pre-programmed into the mirror.

(108) A connection between the computer module of the augmented mirror and the user objects could be achieved via any one or more of: light, sound, local electronic means, vibration, object form, object movement and object graphics. Where graphic on a FMCG container or other object acts as either a digital key or a source of data, it may be printed in an ink which is only visible in a non-visible part of the electromagnetic spectrum (i.e. under UV or IR light).

(109) In particular, the FMCG object may be a “dumb FMCG”. The dumb FMCG does not include a means of actively outputting information or data from itself. Nor is the dumb FMCG capable of carrying out a two-way interaction. In general, the total information content of dumb FMCG is fixed, and the dumb FMCG allows that information to be detected by a system that observes the dumb FMCG by inspection with a camera. The total information content of a dumb FMCG may therefore be conveyed visually. In other words, the information from the dumb FMCG may be determined only from the appearance of the dumb FMCG. The specific markings may be invisible to the human eye under normal circumstances.

(110) The specific markings on the packaging of the dumb FMCG may be especially designed to be easily distinguished and measured by the camera and computer module of the augmented mirror. The specific markings may include, for example, a glyph, a barcode, a 2-dimensional barcode, or a QR code.

(111) A glyph may be a simple geometric shape displayed (for example, printed) on the dumb FMCG packaging. The glyph may be included on the packaging so that it has a high visual contrast relative to the remainder of the dumb FMCG packaging, making it easier for computer module of the augmented mirror to identify the glyph. For example, the glyph may be a simple geometric black shape on a white background. The glyph may not be rotationally symmetric. If the glyph is not rotationally symmetric, then a system observing the glyph with camera is able to identify an angular orientation of the glyph. If the system can also identify the dumb FMCG from the glyph, then the orientation of the glyph can be used to infer the orientation of the dumb FMCG to which it is applied.

(112) The specific markings may enable the mirror to identify the FMCG, for example from a database of glyphs and corresponding FMCGs. The database may be stored locally to the mirror or may be stored remotely from the mirror. If the database is stored remotely, then the mirror may query the database over a network, for example, the internet.

(113) Specific protocols may be used to prevent data transmitted by the mirror from being picked up by any device other than the FMCG where the FMCG is capable of receiving data. This may be achieved by encoding the data so that it can only be read by the objects concerned.

(114) Furthermore, a transfer system such as NFC or Bluetooth which is short range can be used which advantageously means that only devices within a certain distance can interact with the mirror.

(115) Finally, the transfer system chosen may be keyed to a specific recognition parameter, for example visual recognition of a physical FMCG object. In this way, an extra check is provided which checks the visual data, determines the shape of an object, checks the shape against a rule stored in the memory of the mirror, and only performs the transfer of information if the shape of the FMCG object matches the shape as defined by the rule in the memory.

(116) Where there is a network connection, this may be a one way connection so that incoming data only is permitted. In this way security risks are minimised but the mirror can still receive updates and can also interact with external devices e.g. weather forecasts. Where security is not an issue at all, a two way connection may exist so that data can be transferred in and out of the system via the mirror.

(117) A further embodiment of the present invention is described below in relation to FIGS. 14 and 15.

(118) As per previous embodiments, the computation module 4 of the augmented mirror 201 is configured to receive visual information from the camera; to process the information; and display one or more images on the display screen based upon the processed information. However, this embodiment differs from those described above in relation to FIGS. 1 and 2 in that the field of view over which visual information is processed by the computational module is reduced to visual information from within a visual corridor 202, 203 the visual corridor corresponding to a subset of the field of view of the camera.

(119) Accordingly, the height and width of the visual corridor may be determined on the basis on the physical dimensions of the mirror. For example, the visual corridor may correspond to locations at which a user can see themselves reflected in the mirror. In this way, the mirror only interacts with a user who can see themselves in the mirror. Implementing such a visual corridor prevents inadvertent interaction of the mirror with users outside the visual corridor.

(120) In some embodiments, the extent of the visual corridor is defined for the particular mirror, and is applied to all applications and/or functions implemented by that mirror. Furthermore, the same visual corridor may be used with a number of hardware configurations of a particular mirror.

(121) FIG. 14 shows two possible options for a visual corridor; one which is cone shaped, and one which is rectangular shaped. Note that, in either case, all four users C, D, E, and F fall outside of the visual corridor despite that fact that they fall within the field of view 204 of the camera.

(122) For mirror applications, the user is only interested in a range of view over which the user's reflection is visible. This is significantly less than the overall field of view of a standard camera e.g. for use with a video game console.

(123) Since data outside the visual corridor is thrown away before calculations are performed, the visual corridor results in simplification of calculations as well as a reduction in the memory space required.

(124) In the embodiment shown in FIG. 14, the camera is an RGB-D device which includes a depth scanner for capturing an image and associated depth information of a user of the mirror. The RGB-D device includes input software to extract the skeletal data points of each detected person within the field of view of the camera.

(125) The process carried out by the augmentation mirror of FIG. 14 is shown in a flowchart in FIG. 15.

(126) A full frame (i.e. an image over the possible field of view of the camera) is captured by the RGB-D camera. The visual corridor is provided by way of maximum x, y and z inputs at the RGB-D device which represent a subset of the full frame.

(127) The computational module includes elimination software, the elimination software configured to: processes the skeletal points extracted by the RGB-D device; extract the position of a specific skeletal region for each user detected by the RGB-D device; discard data corresponding to any user whose specific skeletal region is located outside the visual corridor.

(128) The skeletal region may be the spine of the user, or may be another body part or area such as a head, arm, or leg. A central region on the body such as the spine or head may be advantageous.

(129) The elimination software may carry out the step of locating the real world Z and X coordinates of the specific skeletal point for each detected skeleton and discarding visual data of users relating for whom the real world X and Z coordinates lay outside of the visual corridor.

(130) The elimination software is further configured to discard data corresponding to any user other than the one whose specific skeletal region is closest to the RGB-D device. That is to say, the user for which the z-coordinate of the specific skeletal region (e.g. the spine) has the lowest value.

(131) In addition to the elimination software, the computational module further comprises: execution software which receives the non-discarded data from the elimination software and processes this data.

(132) The augmented mirror of the present embodiment can include software to interact with portable user devices. In such examples, the portable electronic device includes features which activate only when they appear in the corridor. For example, an interactive smart tooth brush will become activated once it is moved into the visual corridor.

(133) This can be achieved by portable electronic device including an LED and a photo receiver. When light from the LED is reflected by the mirror and received by the photo receiver, at least one feature on the portable electronic device will switch from an “off” configuration to an “on” configuration when it “sees” itself in the mirror.

(134) The maximum x, y and z co-ordinates (or polar co-ordinates if a more complicated shaped corridor is chosen) corresponds to the furthest “left” or “right” a person could be before their information will not be processed by the computational module. These limits can be set to correspond to the furthest positions at which a user can stand in front of the mirror and still see their reflection (i.e. the “outer limits” of the corridor”).

(135) In some embodiments, additional software located at the computational module may be programmed to “lock on” to the location of a user's head.

(136) As the user moves in, their skeleton is picked up by the RGB-D device. This outputs coordinates for the position of their head. For example, the coordinates may correspond to the centre each user's head or the topmost point of each user's head. These coordinates are then used to create a dynamic “bounding box”; an area which would roughly encompass their head. The coordinates of this bounding box will move as the user moves.

(137) These bounding box coordinates are then applied to the camera image. In this way, the camera image shows the head of the user, and is “locked on” to their head, regardless of where they move it within the sightlines of the RGB-D device.

(138) One application of the bounding box is to finesse the face recognition. The facial recognition software is only applied to the visual data within the bounding box, which reduces the size of the image area where a face has to be matched with one in a database located within a memory of the computational module.

(139) FIG. 16 illustrates a methodology and associated steps for drawing on the screen an overlay that appears to the user to be on an object reflected in the mirror, from the perspective of a user of the mirror. The object may be a part of the user or may be an object held by the user or located on the user.

(140) The method illustrated begins by detecting all skeletons in the visual corridor (the visual corridor is described above). In the second step, the system identifies which of the skeletons within the visual corridor is the closest to the mirror. This is an important step because there may be multiple people in the visual corridor. The closest user is identified as “active”, and is the user whose actions are monitored by the mirror, and who will interact with the mirror.

(141) Whilst the active user is within the visual corridor, he/she may not be at the optimal position for interaction with the mirror. In order to have the active user move to the optimal position, the lighting surrounding the mirror can be controlled as described above to direct the user to the optimal user in an intuitive manner.

(142) When the active user is in the optimal position, or close to it, the system identifies the face of the active user and starts an interaction session. The system also identifies an object that can be seen by the camera (for example, a dumb FMCG) and the location of that object. The object is to be overlaid with graphics, as perceived by the active user.

(143) Next, the active user's face is detected, and subsequently the position of the eyes of the active user is identified (the 3D coordinates of the active user's eyes). Object detection of the object is also performed.

(144) From the 3D coordinates of the object, the 3D position virtual position of the object in the reflected space is calculated. To calculate the point on the mirror surface where the object appears to the user, the computational module of the mirror: measures the perpendicular distance from the mirror surface to the object, computes a 3D virtual position at the same perpendicular distance from the mirror on the opposite side of the mirrored surface to the object; calculates a line of sight from the 3D virtual position to the 3D coordinates of the user's eyes; and determine the point at which the plane of the mirror surface intersects this line of sight. This point of intersection corresponds to the point on the mirror surface where the object appears to the user in terms of the 2D coordinates on the mirror surface.

(145) An overlay for the object is then drawn on the display screen at the point of intersection on the mirror surface. This overlay coincides with the position of the reflection of the object, as perceived by the user, and so the overlap appears to be coincident with the object. The object may be part of the user themselves. For example, the overlay may comprise an arrow that appears to point to the object. The overlay may be an encirclement, so the overlap appears to surround the object. In this way, objects may be highlighted to a user of the mirror.

(146) In connection with any of the embodiments described hereon, the augmented mirror may include software programmed to produce two screen outputs. The screen itself may be a dual view screen wherein the screen provides two separate visual outputs and therefore receives two visual outputs from the computer module; a first visual output viewable over a first range of angles; and a second visual output viewable over a second range of angles, the second range of angles chosen so that there is no overlap with the first range of angles.

(147) Such a feature may be used in a salon so that the customer sees something different to hairdresser/salon worker.

(148) In any of the embodiments described herein, the augmented mirror may be triggered by an audio or visual input. Visual triggers could include specific actions in relation to a product such as taking a lid off container. In this case, software in the computer module could be taught or pre-programmed to recognise such an event using object learning.

(149) An example of an object learning process for use with any one of the aspects or embodiments described herein is set out below:

(150) 1. The user presents an object to the mirror, and uses voice commands to initiate object learning, including giving it a unique id;

(151) 2. The mirror takes a still image using its camera, and displays it on the mirror. An initial segmentation of the object is done based on the user's hand location, colour, and depth information;

(152) 3. The user fine-tunes the segmentation using voice commands'

(153) 4. Once complete, the RGB-D representation of the object is stored as a positive example of the object;

(154) 5. A number of negative examples are also sampled from other parts of the image;

(155) 6. A statistical machine learning model is updated with the new training data collected in steps 4 and 5;

(156) 7. The user presents the object to the mirror in a different position, and the mirror segments the object's location using a weighted scoring of the initial segmentation procedure of step 2 and the machine learning prediction of the model trained in step 6. It also shows the models' prediction for the user to evaluate for correctness; and

(157) 8. Steps 3-7 are repeated until the user is satisfied with the automatic recognition results of the system.

(158) A further embodiment of the present invention is described below. This embodiment concerns a “cold start” mechanism that can be used in combination with any of the aspects and/or embodiments described above.

(159) In this embodiment, the computer module includes “cold start software” configured to extract information about a user.

(160) The software obtains inputs from one or more sensors of the augmented mirror system and extracts pertinent information about an individual.

(161) In this way, a degree of personalised information about a user can be obtained and stored passively without the user having to actively answer questions. This provides a quicker, more seamless user interaction.

(162) A cold start mechanism is particularly useful in relation to use of the augmented mirror for product recommendation. Traditionally, product recommendation includes the completion of a questionnaire to describe hair and other attributes relevant to making the recommendation; for example: Hair colour Hair length Hair texture Hair movement Preferred fashion style Current Hair care regime

(163) Once the answers have been inputted, a traditional questionnaire system can make a recommendation based on the information inputted by the user. The process of entering data can be lengthy and is often frustrating for a user.

(164) The following gives an example of how the “cold start” mechanism could be used to improve the recommendation process: User approaches mirror and a visual image is captured; cold start software applies an algorithm for detect hair region; cold start software applies an algorithm to detect face region Information about hair and face are extracted from the relevant regions, for example: 1. Hair colour 2. Hair length 3. Hair texture 4. Hair movement 5. Preferred fashion style The extracted information is used alone or alongside a questionnaire that is reduced in length from that in the traditional method to make a recommendation.

(165) The cold start software, when applied to product recommendation, enables personalised product or regime recommendations faster, by using visual information assessed from a consumer's Smart Mirror reflection to reduce the need for data collected by traditional questionnaire methods. Business benefits may also exist such as reduced ‘user drop-out’ during the personalisation process. This may be relevant in-store or in-home.

(166) The software could be adapted to capture information over an extended amount of time to identify a regime or trends in behaviour by the user.

(167) While the invention has been described in conjunction with the exemplary embodiments described above, many equivalent modifications and variations will be apparent to those skilled in the art when given this disclosure. Accordingly, the exemplary embodiments of the invention set forth above are considered to be illustrative and not limiting. Various changes to the described embodiments may be made without departing from the spirit and scope of the invention.

(168) The embodiments described herein are all relate to augmented mirrors with a partially silvered surface, it is envisaged that the concepts described herein, particularly the concept of the visual corridor could also be applied to a simple display screen without an extra partially silvered coating.

(169) All references referred to above are hereby incorporated by reference.