Method and system for smart navigation for the visually impaired

Abstract

In 2019, the World Health Organization stated that globally, approximately 2.2 billion people live with some form of vision impairment. Visual impairment limits the ability to perform everyday tasks and adversely affects the ability to interact with the surrounding world, thus discouraging individuals navigating unpredictable and unknown environments. The present invention is a method and a system to define and develop a smart navigation intelligent cane (i-Cane) that enables a visually impaired person to navigate his or her environment. The method and the system detects objects along the path of the visually impaired person, measures the distance of the objects from the person, identifies the objects, uses speech to alert the person of the approaching objects, the type of objects obstructing the path, and the distance between the objects and the person.

Claims

1. A method to define and develop a smart navigation intelligent cane (i-Cane) that aids a visually impaired person to move around the surroundings, the method comprising of: first, detecting the approaching objects along the path of the visually impaired person carrying the i-Cane using an ultrasonic sensor and then calculate the distance of the objects from the person carrying the i-Cane (Detect Object) next, identifying the objects, if the distance between the approaching objects and the visually impaired person carrying the i-Cane meets a certain distance threshold (Identify Object): by capturing an image of the approaching objects (Capture Image) and by labeling and classifying the image of the approaching objects using computer vision technology (Classify Image) finally, generating a voice alert using speech synthesis technology to indicate the type of the object and the distance between the object and the visually impaired person carrying the i-Cane, forewarning the person of the approaching object in a natural language (Generate Voice Alert) and continuing the flow and repeating steps of object detection, object identification (image capture and classification), voice alert generation, as the virtually impaired person continues with his/her path and as the objects appear in the path.

2. A system for implementing and demonstrating the method, as described above, to define and develop a smart navigation intelligent cane (i-Cane) that enables a visually impaired person to navigate the environment, the system comprising of: a computing runtime that consists of: a single-board mini portable computing platform such as the Raspberry Pi 3, mounted on an intelligent cane, i-Cane, providing an execution environment for the software components implementing the method described above an ultrasonic sensor such as HC-SR04 connected to single-board mini portable computing platform using a circuitry a camera such as the Pi Camera connected to the camera port on the single-board mini portable computing platform using a camera cable a software program implementing multiple software components that detects the approaching object in the path of the visually impaired person carrying the i-Cane using an ultrasonic sensor and calculates the distance of the object from the person carrying the i-Cane triggers signals to the ultrasonic sensor to measure the distance of the obstacle and then waits to receive the echo back from the sensor calculates the distance between the ultrasonic sensor on the i-Cane and the approaching object using the formula
S=2D/t, therefore, D=(S×t)/2 where, S is Speed of sound, so S=34030 cm/s D is Distance between the approaching object and sensor t is Time taken for the sensor to receive the echo back continues to detect the subsequent approaching objects and doesn't attempt to identify the approaching object or generate a voice alert, if the distance between the approaching object and the visually impaired person carrying the i-Cane is greater than a distance threshold value that is configurable for a given person captures an image of the approaching objects by interfacing with a camera such as the Pi Camera, if the distance between the approaching object and the visually impaired person carrying the i-Cane is less than the distance threshold value identifies the objects by calling the computer vision software passing the captured image classifying the image based on the label annotations and the corresponding relevancy scores returned back from the computer vision software generates an audio alert using speech synthesis software indicating the type of the approaching object and the distance between the approaching object and the visually impaired person carrying the i-Cane, thereby alerting the person of the approaching object.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0017] FIG. 1 illustrates the main process flow and steps for the method defined in this invention.

[0018] FIG. 2 depicts the system behind i-Cane and the underlying building blocks of computing runtime and software components.

[0019] FIG. 3 illustrates the connections between a mini portable computing platform such as the single-board computer Raspberry Pi 3, an ultrasonic sensor such as HC-SR04 and the circuitry to connect the sensor to the Raspberry Pi.

DETAILED DESCRIPTION OF THE INVENTION

[0020] Visual impairment has a severe impact on the course of daily living, discouraging individuals from moving freely in an unknown environment. The world is full of dangers and wonders which are avoided or appreciated with our vision. The physical world poses the greatest challenge for the visually impaired person. How does one know what and where things are and how to obtain them? How does one understand where he/she wants to go without the danger of colliding with things around them?

[0021] Blind individuals may be discouraged from moving freely and comfortably. What can help them to identify the approaching objects in their path of navigation and determine the distance of the objects from the individuals, when they are moving in a house or walking in a mall or strolling through isles in a grocery store?

[0022] The purpose of this invention is to define a method and a system to develop a simple but affordable way to assist visually impaired persons to navigate around their environment. The method defines an approach to develop a smart navigation intelligent cane (i-Cane) that aids a visually impaired person to move around the surroundings: [0023] by first detecting the approaching objects in the path of the visually impaired person carrying the i-Cane, finding the distance between the approaching objects and the person, and then identifying the objects leveraging ultrasonic sensor, camera, and computer vision technology and [0024] finally, by generating a voice/speech alert for the visually impaired person in natural language using speech synthesis technology

[0025] In FIG. 1, the Flow Diagram 100 shows the method developed in this invention and its overall flow and key steps. The key steps in this method are: [0026] Detect Object (as shown by 101 in FIG. 1)—As the visually impaired person carrying the i-Cane travels through a path, first, detect the approaching object in the path using an ultrasonic sensor and then calculate the distance between the object and the person carrying the i-Cane [0027] Identify Object (as shown by 102 in FIG. 1)—Next, identify the object, if the distance between the approaching object and the visually impaired person carrying the i-Cane meets a certain distance threshold: [0028] by capturing an image (as shown by 103 in FIG. 1) of the approaching object and [0029] by classifying and labeling the image of the approaching object (as shown by 104 in FIG. 1) using computer vision technology [0030] Generate Voice Alert (as shown by 105 in FIG. 1)—Finally, generate voice alert using speech synthesis technology to indicate the type of the object and the distance of the object from the visually impaired person carrying the i-Cane, forewarning the person about the approaching object in a natural language so that the person can take some corrective actions to avoid the potential collision with the object [0031] Continue with the flow, as shown by 106 in FIG. 1, as the virtually impaired person continues with his/her path and as the objects appear in the path

[0032] As part of this invention, a system is also defined to demonstrate the method developed in this invention. FIG. 2 illustrates the Component Diagram 200 for the system that implements the method and its flow as depicted by 100 in FIG. 1. The system for designing and developing the i-Cane is composed of Computing Runtime (as shown by 201 in FIG. 2) and Software Components (as shown by 205 in FIG. 2). The Computing Runtime (as shown by 201 in FIG. 2) includes a single-board Mini Portable Computing Platform (as shown by 202 in FIG. 2) providing an execution environment for the software components implementing the method described above. The Computing Runtime (as shown by 201 in FIG. 2) also enables the single-board Mini Portable Computing Platform (as shown by 202 in FIG. 2) to interface with an Ultrasonic Sensor (as shown by 203 in FIG. 2) and a Camera (as shown by 204 in FIG. 2). A representative computing runtime for the defined system can be made of a Raspberry Pi 3 as the single-board mini portable computing platform, an HC-SR04 as the ultrasonic sensor, and a Pi Camera as the camera.

[0033] The Software Components (as shown by 205 in FIG. 2) includes: [0034] Object Detection (as shown by 206 in FIG. 2) detects the approaching object in the path of the visually impaired person carrying the i-Cane using an ultrasonic sensor and calculates the distance of the object from the person carrying the i-Cane. [0035] Object Identification (as shown by 207 in FIG. 2) identifies the object if the distance between the approaching object and the visually impaired person carrying the i-Cane is less than a certain distance threshold. The Image Capture sub-component (as shown by 208 in FIG. 2) within the Object Identification component takes a picture of the approaching object using the camera. And then, the Image Classification sub-component (as shown by 209 in FIG. 2) labels and classifies the image using the Computer Vision Software (as shown by 210 in FIG. 2) running on the cloud. [0036] Voice Alert Generation (as shown by 211 in FIG. 2) generates a voice alert using the Speech Synthesis Software (as shown by 212 in FIG. 2) to inform the visually impaired person carrying the i-Cane of the approaching object, its type and the distance between the object and the person, forewarning the person of the approaching object.

[0037] In FIG. 3, the Connection Diagram 300 illustrates the connection between the Raspberry Pi 3 (the single-board mini portable computing platform) and HC-SR04 (the ultrasonic sensor) and the circuitry in between connecting the two hardware components. The Pi Camera is directly connected to the camera port on the Raspberry Pi 3 using a camera cable. The system consisting of the Raspberry Pi 3 mini portable computing platform connected with the HC-SR04 ultrasonic sensor and the Pi Camera is mounted on the i-Cane.

[0038] 301 in FIG. 3, Illustrates the single-board mini portable computing platform Raspberry Pi 3 and its pin layout. 302 in FIG. 3 shows the ultrasonic sensor HC-SR04 and its four pins, namely, 5V Power, TRIGGER (TRIG), ECHO, GROUND (GND).

[0039] Connecting Ultrasonic Sensor to Raspberry Pi 3 [0040] The 5V Power pin of the ultrasonic sensor is connected to the GPIO 5V pin (Pin number 2) of the Raspberry Pi 3 as shown by 305 in FIG. 3. [0041] The TRIG pin of the ultrasonic sensor is connected to the GPIO 23 pin (Pin number 16) of the Raspberry Pi 3 as shown by 306 in FIG. 3. [0042] The ECHO pin of the ultrasonic sensor is connected to the resistor R1 (330Ω, represented by 303 in FIG. 3) as shown by 307 in FIG. 3. The other end of the resistor R1 is connected to the resistor R2 (470Ω, represented by 304 in FIG. 3) as shown by 308 in FIG. 3. [0043] The other end of resistor R2 is connected to GND pin of the ultrasonic sensor as shown by 309 in FIG. 3. And the common point of the resistor R2 and GND pin of the ultrasonic sensor is connected to GPIO GND pin (Pin number 6) of the Raspberry Pi 3 as shown by 311 in FIG. 3. [0044] The common point of R1 and R2 resistors is connected to the GPIO 24 pin (Pin number 18) of the Raspberry Pi 3 as shown by 311 in FIG. 3. The GPIO 24 pin sits between the resistors R1 and R2 thereby forming a parallel circuit and hence, reducing the voltage to approximately 3V from 5V. Mathematically, V.sub.out=V.sub.in×R2/(R1+R2) where V.sub.in=5V, R1+R2=800 Ω and R2=470 Ω and substituting the values V.sub.out is approximately equal to 3V.

[0045] A software program using Python programming language is run on the Raspberry Pi 3 mini portable computing platform, as shown by 202 in FIG. 2 and as shown by 301 in FIG. 3, connected to the ultrasonic sensor, as shown by 203 in FIG. 2 and as shown by 302 in FIG. 3 through the circuitry as shown in FIG. 3 and connected to the Pi Camera, as shown in 204 in FIG. 2. [0046] Object Detection component triggers a signal to the ultrasonic sensor then waits to receive the echo back from the sensor and calculates the distance between the ultrasonic sensor on the i-Cane and the approaching object using the formula:

S=2D/t, therefore, D=(S×t)/2 [0047] where, [0048] S is Speed of sound, so S=34030 cm/s [0049] D is Distance between the approaching object and sensor [0050] t is Time taken for the sensor to receive the echo back [0051] lithe distance between the approaching object and the visually impaired person carrying the i-Cane is greater than a distance threshold value (e.g. 150 cm) the system does not attempt to identify the approaching object or generate a voice alert, continuing to detect the subsequent approaching object. The distance threshold value is configurable by visually impaired person. [0052] Object Identification is composed of Image Capture and Image Classification sub-components. The Image Capture sub-component takes the picture of the approaching object using the Pi Camera. The Image Classification sub-component calls the Computer Vision Software component on a cloud platform to determine the label annotation of the image and classifies the image based on the labels with top relevancy scores. [0053] Voice Alert Generation component generates an audio alert using the Speech Synthesis software indicating the type of the approaching object and the distance between the approaching object and the visually impaired person carrying the i-Cane, thereby alerting the person of the approaching object so that the person can take some corrective actions to avoid the potential collision with the object.

NON-PATENT CITATIONS

[0054] WHO. World report on vision. World Health Organization, 2019. [0055] Blackwell, Debra L, Lucas, Jacqueline W, and Clarke, Tainya C. “Summary Health Statistics for U.S. Adults: National Health Interview Survey, 2012”. National Center for Health Statistics. Vital and Health Statistics 10(260), 2014. [0056] Upton, Eben, and Gareth Halfacree. Raspberry Pi: User Guide. John Wiley & Sons, 2013. [0057] Monk, Simon. Programming the Raspberry Pi, Second Edition: Getting Started with Python. McGraw-Hill Education, 2015. [0058] McManus, Sean, and Mike Cook. Raspberry Pi for Dummies. John Wiley & Sons, 2013.