Hands-Free Pedestrian Navigation System and Method

Abstract

A hands-free pedestrian navigation method includes mounting on a user's head (i) a display for projecting a visual image in front of the user's gaze, and (ii) an IMU, obtaining from a GPS unit carried by the user an approximate user location for locating the user in a computerized map. Confirmation is obtained from the user that the user's gaze is directed to a specified landmark in sight of the user and azimuth is computed between the user location and the landmark location extracted from the computerized map. Vocal prompts are provided and ancillary visual prompts are projected on the display to navigate the pedestrian. In a system, the user wears a head-mounted device containing the IMU and display and carries a GPS unit and a portable computing device coupled to the device and GPS unit and programmed to carry out the method.

Claims

1. A hands-free pedestrian navigation method for assisting a user to reach a destination, the method comprising: (a) mounting on the user's head (i) a display for projecting a visual image in front of the user's gaze, and (ii) an IMU; (b) obtaining from a GPS unit carried by the user an approximate user location; (c) using the approximate user location to locate the user in a computerized map of an area containing the user; (d) determining in said area a landmark having a known landmark location within view of the user; (e) obtaining confirmation from the user that the user's gaze is directed to the landmark; (f) determining a computed azimuth between the user location and the landmark location extracted from the computerized map; (g) using the computing azimuth to determine an angular orientation between the users' gaze and the destination; and (h) navigating the user by providing vocal prompts and projecting ancillary visual prompts via the display.

2. The method according to claim 1, further including: (a) obtaining a measured azimuth from a head-mounted magnetometer in the IMU when the user's gaze is directed to the landmark; and (b) orienting the user by computing an offset between the measured azimuth and the computed azimuth.

3. The method according to claim 1, wherein at least one of the following is conveyed vocally: (i) the confirmation from the user that the user's gaze is directed to the landmark; (ii) identification of the landmark; and (iii) identification of the destination.

4. The method according to claim 1, wherein the landmark is identified by displaying a panoramic photo of an area of sight on a smartphone carried by the user and identifying an object in said panoramic photo selected by the user.

5. The method according to claim 1, including projecting via the display a marker on to a field of view of the user and obtaining confirmation from the user that the user's gaze is directed to the landmark when the marker is aligned with the landmark and optionally verifying that the user is following a correct route by displaying visual indications in his field of view via the display and optionally alerting the user that he is deviating from a correct route by displaying visual indications in his field of view via the display.

6. The method according to claim 1, wherein the user conveys information vocally via a software application having an interface to a third-party virtual assistant coupled to a navigation server.

7. The method according to claim 6, wherein the third-party virtual assistant is a cloud application supporting Amazon™ Alexa and Skills and is coupled to the navigation server over the Internet.

8. A computer program product comprising a computer-readable memory storing program code instructions, which when executed by a computer processing unit carries out the method according to claim 1.

9. A pedestrian navigation system for directing the user to a destination, the system comprising: a GPS unit carried by a user, a head-mountable device for wearing by the user and including an IMU and a display for projecting a visual image in front of the user's gaze, and a portable computing device carried by the user operatively coupled to the GPS unit and the head-mountable device and having a memory and a processing unit programmed to: (a) obtaining from a GPS unit carried by the user an approximate user location; (b) using the approximate user location to locate the user in a computerized map of an area containing the user; (c) determining in said area a landmark having a known landmark location within view of the user; (d) obtaining vocal confirmation from the user that the user's gaze is directed to the landmark; (e) determining a computed azimuth between the user location and the landmark location extracted from the computerized map; (f) using the computing azimuth to determine an angular orientation between the users' gaze and the destination; and (g) navigating the user by providing vocal prompts and projecting ancillary visual prompts via the display.

10. The system according to claim 9, further including: (g) obtaining a measured azimuth from a magnetometer in the head-mounted device when the user's gaze is directed to the landmark; and (h) orienting the pedestrian by computing an offset between the measured azimuth and the computed azimuth.

11. The system according to claim 9, further including a remote navigation server storing map data in communication with the portable computing device for conveying the computerized map to the portable computing device.

12. The system according to claim 9, wherein the portable computing device is a smartphone that includes the GPS unit built-in.

13. The system according to claim 9, wherein the head-mountable device is detachably mounted on a spectacle frame and the display contains a micro-display for displaying an image and optics for projecting the image on to a scene viewed by the user and optionally a microphone and earphone are built-in the spectacle frame and are configured for coupling to the portable computing device for conveying vocal information thereto and for conveying vocal instructions for directing the user to a required destination.

14. The system according to claim 13, wherein the image is a marker that is projected on to the landmark when the user's gaze is aligned with the landmark.

15. The system according to claim 13, wherein the processing unit is programmed to convey a navigation aid to the micro-display.

16. The system according to claim 9, further including a microphone and earphone coupled to the portable computing device for conveying vocal information thereto and for conveying vocal instructions for directing the user to a required destination.

17. The system according to claim 9, wherein: the portable computing device executes a software application having an interface to a third-party virtual assistant coupled to a navigation server, and optionally the microphone and earphone connect via Bluetooth™ to a host application in the portable computing device, and the host application is configured to communicate over the Internet with a cloud to application supporting Amazon™ Alexa and Skills.

18. The system according to claim 17, wherein: the spectacle frame is an Amazon™ Echo Frame, the portable computing device executes a software application having an interface to a third-party virtual assistant coupled to the navigation server, the spectacle frame connects via Bluetooth™ to a host application in the portable computing device, and the host application is configured to communicate over the Internet with a cloud application supporting Amazon™ Alexa and Skills.

19. The system according to claim 17, wherein: the host smartphone application is configured to perform voice/text conversion and interfaces to a navigation application cloud over the Internet; and Amazon™ Sharing Alexa Skills are used to transfer data between the host cloud application and the navigation application cloud to enable activation of navigation functions in the navigation application cloud and to convey navigation instructions over the Internet back to the host smartphone application.

20. The system according to claim 9, wherein the display is a see-through display.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to understand the invention and to see how it may be carried out in practice, embodiments will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which:

[0063] FIG. 1a is a pictorial representation of a user wearing a device according to the invention;

[0064] FIG. 1b is a pictorial representation of the device showing typical dimensions;

[0065] FIG. 2 is a block diagram showing a system according to the invention;

[0066] FIGS. 3a and 3b are pictorial representations of the detachable device when seen from the rear and front respectively;

[0067] FIG. 3c is a partially cut-away view of the detachable device when seen from the rear showing partial details of the optics contained therein;

[0068] FIG. 3d is a schematic representation of the optics;

[0069] FIGS. 4a and 4b are block diagrams of the device in wireless and wired configurations, respectively;

[0070] FIG. 5 shows the principal operations associated with methods for calibrating the IMU magnetometer and providing navigation instructions to the user;

[0071] FIG. 6 shows the principal operations associated with methods for verifying that the user is following the correct route; and

[0072] FIG. 7 shows pictorially an alternative embodiment for calibrating the magnetometer when no discernible landmarks can be identified.

DETAILED DESCRIPTION OF EMBODIMENTS

[0073] FIG. 1a shows device 10 detachably clip-mounted on to the front of a spectacles frame 11 worn by a user 12. By such means the device 10 is mounted directly in front of the user's eyes, allowing him to observe a scene through an exit window 13. Also, the user can see graphical annotations projected to his eye by the detachable device 10 and overlaid on the real scene. Preferably, the spectacle frame 11 is a so-called “smart frame” supporting a microphone and earphone, but these can be retrofitted to a conventional spectacle frame with additional, separate, microphone and earphone or be mounted discretely as independent accessories. Preferably, the device 10 is an integral accessory configured for attaching to the front of the spectacle frame with none of its components being mounted on the temple pieces (side arms) of the user's spectacles, thus allowing the device to be mounted on a conventional spectacle frame. However, the invention does not preclude the possibility to mount or build some of the components into the side arms of a custom frame.

[0074] FIG. 1b shows typical dimensions of the device 10 according to a preferred embodiment whose minimal components are an IMU, a see-through display comprising a micro-display, semi-transparent mirror and optics as described below with reference to FIGS. 2 and 3, and communication means. In order to permit such miniaturization, the FOV of the see-through display is small, and lateral adjustment of the device allows the optical exit window to be moved into alignment with the user's eye. When wireless communication is used, the device includes a small battery. To reduce power consumption, only the most essential components are used or even included. Most significantly a camera is not required and may be omitted, partly in order to reduce size but mainly to avoid excessive power consumption, additional electronics for image compression, high bandwidth communication (such as WiFi) for transmitting the video to the smartphone, all of which add bulk to the device and drains the battery.

[0075] FIG. 2 is a block diagram of a system 20 that integrates a smartphone 21 having a GPS unit 22 and a host smartphone application 23 with a voice-controlled navigation system depicted as a host cloud application 24 and the clip-on device 10. The clip-on device 10 contains an IMU 25 and a micro-display 26 and is connected via BLE (Bluetooth™ Low Energy) to a smartphone control application 27. The smart frame 11 connects via Bluetooth™ to the host smartphone application 23. The smart frame may be the Amazon Echo Frame having a microphone and earphones 28 as integral components and the host cloud application 24 may be the Amazon Alexa and Skills. Such an implementation avoids the need to incorporate the audio components in the device 10 although this option is obviously also possible, albeit at the expense of some additional bulk and increased battery consumption. For ease and consistency of description, we will refer to software applications that are loaded on to the smartphone as smartphone applications, while software applications that are operated over the Internet will be referred to as cloud applications.

[0076] The smartphone control application 27 connects to a navigation application cloud 29, in which all navigation tasks are performed and conveyed to and from the control application over the Internet. The host smartphone application 23 connects to the host cloud application 24 via the Internet. In the figure, short-range Bluetooth™ communication is depicted by a black arrow, while Internet communication is depicted by a white arrow. The host smartphone application 23 performs voice/text conversion and interfaces to the navigation application cloud 29 over the Internet. Amazon Sharing Alexa Skills are used to transfer data between the host cloud application 24 and the navigation application cloud 29 to also enable activation of necessary functions, such as navigation, in the navigation application cloud 29. More specifically, Amazon allows device manufacturers to integrate Alexa voice capabilities into their own connected products by using the Alexa Voice Service (AVS), a cloud-based service that provides APIs to interface with Alexa. This requires that communication between the host smartphone application 23 and the smartphone control application 27 be mediated via the clouds 24 and 29. However, other platforms may accommodate direct internal communication between the two smartphone applications 23 and 27.

[0077] FIG. 3a is a pictorial representation showing a rear view of the device 10. The user 12 sees the real scene through an exit window 30 and also sees a graphical annotation overlaid on this scene. These annotations may include marks, text, graphical shapes etc. The detachable device has a housing 31 to which there are fixed mounting brackets 32 for supporting a clip-on unit (not shown) such as described in WO 2019/215739 by means of which the device 10 is clipped on to the front of the user's eyeglasses. The device 10 may alternatively be attached magnetically to the user's spectacles or it may be any other kind of head-mounted device. Optionally, a USB connector 34 is provided for coupling the device 10 to the smartphone 19.

[0078] FIG. 3b shows a front view of the detachable device 10 as seen from the front, i.e. looking towards the user 12. In some embodiments, an optional window 36 is provided through which an optional built-in camera 37 located inside the device and shown schematically in dotted outline in FIG. 3c images the external scene. Also shown is an entrance window 30′ through which light from scene passes to the exit window 30, to be observed by the user. The camera when provided can be used as a supplementary navigation aid as described in above-mentioned WO 2019/215739 and can also be used to calibrate the IMU in the same way as Google's Live View but with the advantage that the user does not need to hand-hold his or her smartphone.

[0079] FIG. 3c shows in more detail the inner structure of the device 10. A printed circuit board (PCB) 38 supports an electronic circuit for the electronic operation of the detachable device. Also connected to the PCB 38 are the IMU 25, USB connector 34 (needed only if wired communication is used) a micro-display 26 and an optional built-in camera 37.

[0080] The micro-display 26 is a high-brightness monochrome display having the following main characteristics:

[0081] Number of pixels: 304×256

[0082] Pixel size: 12 μm×12 μm

[0083] Active area: 3.648 mm×3.972 mm

[0084] The device 10 houses optics 40 shown in FIG. 3d, which includes two doublets 41, 41′ and a combined prism 42. The doublets 41, 41′ create a combined objective with equivalent focal length of 21 mm. The light emitting plane of the micro-display 26 is located at the focal plane of this combined objective so that an image of the display is created at infinity, whereby the user 12 sees the image on the display projected on the real scene. A typical image of the screen includes a cross-shaped marker or reticle, used by the user to designate an object in his field of view. The image is further folded, as illustrated by the arrows, first by a mirror surface 45 of the combined prism 42 and then is directed to the eye of the user 12 by a partial mirror surface 46, which allows light from the display 26 to reach the user's eye, and at the same time allows the user to see the real scene transmitted through the partial mirror surface 46 via the exit window 30 in FIG. 3b. Typically, the partial mirror surface 46 has a reflection of ˜50% and transmission of ˜50%. Since the focal length of the combined doublets is 21 mm, the display screen captures a Field of View of H: 10×V: 8.3 degrees (13 degrees diagonal). The optical design allows the user an eye motion box of 4.5 mm diameter.

[0085] Such an arrangement allows for a very compact device to be mounted on the user's spectacle frame in a manner that allows for lateral adjustment so that the exit window 30 is aligned with the user's eye thereby obviating the need for a large eye motion box, which would mandate use of more bulky optics. Many conventional see-through AR systems require a large field of view for displaying graphical annotations on reality. Such systems are heavy, cumbersome, and expensive. Furthermore, many use built-in cameras to image landmarks for allowing a navigation system to determine the user's location based on the landmark image. This further adds to the bulk of the AR device and adds significant overhead to the communications bandwidth.

[0086] In the device according to the invention, small field of view, with minimal AR annotation for markers and directions only and the lack of reliance on a camera enable a compact device. By combining this and voice, complex annotations (i.e. landmark's names, complex directions) are given verbally, enabling use of small field of view. This results in a compact AR, hands-free on-the-go pedestrian navigation.

[0087] FIGS. 4a and 4b are block diagrams of the device 10, in wireless and wired configurations, respectively. Thus, as shown in FIG. 4a, optics 40 as described above project to the eye of the user 12 the micro-display 26 whose screen image is thus overlaid on the external scene. The device 10 accommodates the Inertial Magnetic Unit (IMU) 25 for tracking motion of the device and thereby head motion of the user. The micro-display 26 and the IMU 25 are connected to a microcontroller 72. The microcontroller 72 also processes the data for two-way communication with a Low Energy (LE) Bluetooth™ unit 74 via an RF antenna 75. The Bluetooth™ unit 74 communicates with a Bluetooth™ unit 76 in the smartphone 21. A built-in battery 77 provides power to the device components.

[0088] FIG. 4b shows an alternative embodiment using USB wired communication. The microcontroller 72 contains SPI (Serial Peripheral Interface) drivers and USB to SPI converters 78, 78′ which connect the display 26 and the IMU 25, respectively to a USB hub 79. The USB hub 79 connects to the smartphone 21 via a USB cable.

[0089] FIG. 5 shows the principal operations associated with methods for calibrating the IMU magnetometer and providing navigation instructions to the user. For the sake of abundant clarity, the magnetometer is a component of the IMU 25 and is not shown discretely in the drawings. The user 12 first attaches the device 10 on to the spectacle frame 11 and switches it on. In a first step, the magnetometer calibration is performed in order to establish the orientation of the device 10 relative to Earth coordinates when the user is gazing at a known landmark.

[0090] This magnetometer calibration (80) is done as follows. [0091] The smartphone control application 27 gets the GPS data from the smartphone GPS unit 22. [0092] The smartphone control application determines ‘area of sight’ and based on GPS location and estimated observed area, it gets relevant landmarks from GPS/Map database, i.e. Google Maps. The map is stored in the navigation application cloud 29 from which the relevant portion of the map corresponding to the ‘area of sight’ is downloaded to the smartphone. The smartphone control application identifies a visible landmark in the vicinity of the user (e.g. a known Department Store), draws a marker on the display 26 of device 10 and tells the user to point his view towards this landmark until the marker is overlaid on this landmark. The user verbally confirms that his gaze is directed toward the landmark. [0093] The smartphone control application reads the azimuth measured by the magnetometer, calculates the azimuth between the user location and the landmark location based on the map, and then calibrates the magnetometer, marks a “V” on the display, and the user may start navigating.

[0094] Although the landmark may be suggested by the smartphone application 27, it may also be suggested by the user. In this case the user may say “I see Starbucks coffee on my right”, the system will recognize it, and the process will continue as before. This corresponds to the conventional use of Alexa, in which the user vocalizes a request. Alexa converts the speech to text and conveys it to the Alexa cloud where it is parsed and processed to derive a suitable response that is conveyed to the user as a text string, which is then converted locally to voice and vocalized.

[0095] Once the magnetometer is calibrated, the smartphone application 27 can indicate the computed direction of travel graphically by displaying on the micro-display 26 ancillary visual prompts such as an arrow whose direction points to magnetic north and/or textually such as North-North West. The display image is superimposed on the external scene by the optics 40. However, in a simplified device having no see-through display the IMU can be head-mounted without projection of any marker or with projection of a marker using a non-see-through display such as Google Glass. Alignment with a named landmark is simply confirmed verbally and the user's orientation is determined based on the IMU measurements conveyed to smartphone application 27. This is likely to be less accurate but still permits the user to gaze toward a landmark that allows the smartphone application 27 to orientate the user and provide comprehensive navigation instructions. Also, although verbal commands are preferred, the invention contemplates use of a pushbutton micro-switch on the clip-on device 10 and which may be pressed by the user to confirm to the smartphone application when his or her gaze is directed to the landmark. While, of course, this is not a completely hands-free operation it still does not require manual interaction with the smartphone. Moreover, since calibration is executed only when initializing the navigation application and during verification, such a variation may still, to all intents and purposes, be regarded as a vocal navigation system.

[0096] Once calibration is done, navigation (81) may commence. [0097] The user tells the system where he wants to go. [0098] The smartphone application calculates the route and compiles landmarks along the route, obtains the IMU and GPS data, and once ready, displays on the display 26 graphical navigation aids such as an arrow showing the direction the user should head, and/or conveys verbal instructions to assist in navigation. [0099] During the navigation, the smartphone application provides the user with verbal instructions and graphic annotations. [0100] During the navigation, either the user or the smartphone application may initiate a verification mode to ensure that the user is still on track.

[0101] FIG. 6 shows the principal operations associated with methods for verifying that the user is following the correct route. As shown, verification can be initiated by the user as shown in (82), for example when in doubt or by the smartphone application as shown in (83). When user-initiated, the user vocally identifies a landmark that he is passing, e.g. “I see Starbucks on my right”. The smartphone application checks if this is correct, and if so responds by “Great, keep on walking”. Otherwise, the smartphone application will direct the user to a nearby landmark and will direct him to perform a calibration routine, similar to that shown in FIG. 5 as 80 and then, if necessary, will update the navigation instructions.

[0102] Smartphone verification (83) operates in a similar manner to initial calibration except that the smartphone vocally prompts the user to confirm that a landmark that should be in view is correctly seen by the user. Regardless of whether verification is initiated by the user of by the smartphone application, the landmark may be selected by either the user or by the smartphone application. For example, the user can simply say: “Help” and the smartphone application will carry out the verification protocol (83).

[0103] As noted above, an aspect of the invention can be implemented without a display but only with a smart glass frame or earphone containing an IMU, a microphone and earphones.

[0104] In this case the user points his head to the coarse direction of a landmark when a calibration or verification is needed. Although the orientation determined this way is less accurate than that achieved using a projected marker, it is still more accurate than that available by present magnetometers. Nevertheless, it is clear that in this case graphical annotations are not possible and only verbal directions are available.

[0105] FIG. 7 shows pictorially an alternative embodiment for calibrating the IMU when no discernible landmarks can be identified. This can happen in a situation in which the elements surrounding the user lack any special attributes, e.g. when the buildings lack any signs, so that neither the user nor the system can verbally describe any object around the user. In such cases, the smartphone application, based on the user's GPS position, will provide him with a panoramic photo of his Area of Sight to his phone. Such photo can be retrieved from the Street View cloud database or similar. The user locates the object he is looking at in the displayed panoramic photo and clicks on it. Based on the selected area in the panoramic photo, the smartphone application will be able to correlate the selected image to a real object the user is looking at and identify it thereby establishing the user's direction of gaze.

[0106] It will be appreciated that modifications can be made without departing from the scope of the invention as claimed. For example, the microphone and the earphone need not be integrated into the spectacle frame. The microphone may be clipped on to the user's jacket and coupled to the smartphone as may be the earphone. In such case, either or both may be wirelessly coupled to the smartphone, typically using the smartphone Bluetooth™ interface or may be connected via suitable cables.

[0107] It should also be noted that while embodiments have been described with particular reference to the calibration of an IMU employing a magnetometer, as noted above IMUs may employ gyroscopes that provide a relative angular displacement. In such case, calibration of the IMU determines an initial azimuth of the pedestrian relative to a known landmark, and the gyroscope indicates relative rotation of the IMU and hence the user relative to the initial azimuth.

[0108] The invention can be implemented using a display that is not see-through but which provides visual navigation signals. For example, LEDs having different colors or geometrical forms may be mounted in spatial association with the head-mountable device so that the user sees them when looking toward in the distance. An illuminated LED will be visible to the user, albeit not in sharp focus, and may indicate direction. The user may direct his or her gaze via a specific one of these LEDS or may point his head to the coarse direction of a landmark when a calibration or verification is needed as described previously. The LEDs can be built into or mounted in front of a spectacle lens.

[0109] It should be noted that features that are described with reference to one or more embodiments are described by way of example rather than by way of limitation to those embodiments. Thus, unless stated otherwise or unless particular combinations are clearly inadmissible, optional features that are described with reference to only some embodiments are assumed to be likewise applicable to all other embodiments also.

[0110] It will also be understood that the software according to the invention may be implemented by a computer program being readable by a computer processor for executing the method of the invention. The invention further contemplates a machine-readable memory tangibly embodying a program of instructions executable by the machine for executing the method of the invention.