Information processing device and information processing method

Abstract

An information processing device calculates an occupancy rate of a pedestrian crossing in an image obtained by capturing an image of a traveling direction of a target person, determines a crossing status of the target person for the pedestrian crossing based on the calculated occupancy rate, and generates support information for supporting the target person crossing the pedestrian crossing based on the crossing status.

Claims

1. An information processing device on a cane, comprising: a camera in a grip portion of the cane; a calculation unit that calculates an occupancy rate of a pedestrian crossing in an image obtained by capturing an image of a traveling direction of a target person with the camera; a crossing status determination unit that determines a crossing status of the target person for the pedestrian crossing based on the calculated occupancy rate; and a support information generation unit that generates support information for supporting the target person crossing the pedestrian crossing based on the crossing status of the target person.

2. The information processing device according to claim 1, wherein when the occupancy rate is smaller than a predetermined threshold value, the crossing status determination unit determines that there is a possibility that the target person deviates from the pedestrian crossing.

3. The information processing device according to claim 1, wherein: the occupancy rate is an occupancy rate of the pedestrian crossing in a lateral direction of the image; and the calculation unit calculates the occupancy rate based on a ratio of a lateral dimension of a pedestrian crossing strip to a lateral dimension of the image.

4. The information processing device according to claim 1, wherein: for a first pedestrian crossing strip located at a bottom of the image and a second pedestrian crossing strip located adjacent to and above the first pedestrian crossing strip, the calculation unit calculates a traveling direction appropriateness rate based on a ratio of a lateral dimension of the second pedestrian crossing strip to a lateral dimension of the first pedestrian crossing strip; and when the traveling direction appropriateness rate is smaller than a predetermined threshold value, the crossing status determination unit determines that there is a possibility that the target person deviates from the pedestrian crossing.

5. The information processing device according to claim 1, wherein: for a first pedestrian crossing strip located at a bottom of the image, the calculation unit obtains a height dimension in a vertical direction from a lower end of the image to the first pedestrian crossing strip, and the calculation unit calculates a starting point distance securing rate based on a ratio of the height dimension of the first pedestrian crossing strip to a vertical dimension of the image; and when the starting point distance securing rate is equal to or larger than a predetermined threshold value, the crossing status determination unit determines that the target person is located at a point in front of the pedestrian crossing.

6. The information processing device according to claim 1, wherein: for a first pedestrian crossing strip located at a bottom of the image and a second pedestrian crossing strip located adjacent to and above the first pedestrian crossing strip, the calculation unit obtains a height dimension in a vertical direction from a lower end of the image to each of the first pedestrian crossing strip and the second pedestrian crossing strip, and calculates a pedestrian crossing entrance rate based on a ratio of the height dimension of the second pedestrian crossing strip to the height dimension of the first pedestrian crossing strip; and when the pedestrian crossing entrance rate is smaller than a predetermined threshold value, the crossing status determination unit determines that the target person is located at a point in front of the pedestrian crossing.

7. The information processing device according to claim 1, wherein the calculation unit inputs a captured image to a learned model to estimate a shape of a pedestrian crossing strip of the pedestrian crossing, the learned model being generated using an image of a pedestrian crossing that does not include a pedestrian and an image of a pedestrian crossing that includes a pedestrian as training data, and calculates an occupancy rate of the pedestrian crossing in the captured image based on the estimated shape of the pedestrian crossing strip.

8. The information processing device according to claim 1, further comprising a signal detection unit, wherein: the signal detection unit detects a signal state of a traffic light installed at the pedestrian crossing based on the image; and the support information generation unit generates the support information based on the signal state.

9. The information processing device according to claim 1, further comprising an obstacle detection unit, wherein: the obstacle detection unit detects an obstacle approaching in a direction of crossing the pedestrian crossing based on a captured image; and the support information generation unit generates the support information when an obstacle is recognized.

10. The information processing device according to claim 1, further comprising: an output unit that outputs the support information.

11. An information processing method, comprising: calculating, by a calculation unit, an occupancy rate of a pedestrian crossing in an image obtained by capturing, with a camera in a grip portion of a cane, an image of a traveling direction of a target person; determining, by a crossing status determination unit, a crossing status of the target person for the pedestrian crossing based on the calculated occupancy rate; and generating, by a support information generation unit, support information for supporting the target person crossing the pedestrian crossing based on the crossing status of the target person.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) Features, advantages, and technical and industrial significance of exemplary embodiments of the disclosure will be described below with reference to the accompanying drawings, in which like signs denote like elements, and wherein:

(2) FIG. 1 is a diagram showing an example of use of an information processing device according to an embodiment by a visually impaired person when the information processing device is provided on a white cane;

(3) FIG. 2 is a block diagram showing a hardware configuration of the information processing device according to the embodiment;

(4) FIG. 3 is a block diagram showing a functional configuration of the information processing device according to the embodiment;

(5) FIG. 4 is a block diagram illustrating a learning phase according to the embodiment;

(6) FIG. 5A is a diagram showing an example of an image obtained by capturing an image of a traveling direction of a target person;

(7) FIG. 5B is an example showing shapes of pedestrian crossing strips in an image that are estimated by a calculation unit;

(8) FIG. 6 is a diagram illustrating a lateral dimension of the image and lateral dimensions of the pedestrian crossing strips;

(9) FIG. 7 is a diagram illustrating a vertical dimension of the image and height dimensions of the pedestrian crossing strips;

(10) FIG. 8A is a schematic plan view schematically showing a position of the target person and the traveling direction of the target person with respect to the pedestrian crossing;

(11) FIG. 8B is a schematic plan view schematically showing the position of the target person and the traveling direction of the target person with respect to the pedestrian crossing;

(12) FIG. 8C is a schematic plan view schematically showing the position of the target person and the traveling direction of the target person with respect to the pedestrian crossing;

(13) FIG. 8D is a schematic plan view schematically showing the position of the target person and the traveling direction of the target person with respect to the pedestrian crossing;

(14) FIG. 9 is a flowchart showing an example of a flow of a support process in the embodiment;

(15) FIG. 10 is a flowchart showing an example of a flow of a start position support process in the embodiment; and

(16) FIG. 11 is a flowchart showing an example of a flow of a crossing position support process in the embodiment.

DETAILED DESCRIPTION OF EMBODIMENTS

(17) Hereinafter, an information processing device 10 according to the present embodiment will be described with reference to FIGS. 1 to 11. In the present embodiment, a configuration in which the information processing device 10 is applied to a white cane 1 used by a visually impaired person will be described as an example.

(18) As shown in FIG. 1, the information processing device 10 provides a target person P carrying the information processing device 10 with support information for safely crossing a pedestrian crossing. In the present embodiment, as an example, the information processing device 10 is applied to the white cane 1 used by a visually impaired person, and provides various kinds of support information when the target person P who is visually impaired crosses a pedestrian crossing.

(19) Hardware Configuration of Information Processing Device 10

(20) FIG. 2 is a block diagram showing a hardware configuration of the information processing device 10. As shown in FIG. 2, the information processing device 10 includes a central processing unit (CPU: processor) 12, a read-only memory (ROM) 14, a random access memory (RAM) 16, a storage 18, a communication interface 20, and an input-output interface 22. The components above are connected via a bus (for which reference characters are omitted) so as to be able to communicate with each other.

(21) The CPU 12 is a central arithmetic processing unit that executes various programs and controls various units. That is, the CPU 12 reads the program from the ROM 14 or the storage 18, and executes the program using the RAM 16 as a work area. The CPU 12 controls the above components and performs various arithmetic processes in accordance with the programs stored in the ROM 14 or the storage 18.

(22) The ROM 14 stores various programs and various data. The RAM 16 temporarily stores a program or data as a work area. The storage 18 is composed of a hard disk drive (HDD) or a solid state drive (SSD), and stores various programs including an operating system, and various data. In the present embodiment, the ROM 14 or the storage 18 stores a program for performing a silencing process, a learned model M (see FIG. 4), and the like.

(23) The communication interface 20 is an interface for the information processing device 10 to communicate with a server and other devices (not shown), and standards such as Ethernet (registered trademark), long-term evolution (LTE), fiber-distributed data interface (FDDI), and Wi-Fi (registered trademark) are used therefor.

(24) A camera 24 and an output device 26 are connected to the input-output interface 22. The camera 24 is disposed, for example, in a grip portion 1A that is held by the target person P in the white cane 1. The camera 24 is directed forward of the target person P, and captures an image of the traveling direction of the target person P. The output device 26 includes, for example, at least one of a speaker and a vibrator. In addition to this, an operation unit (not shown) capable of starting and stopping the camera 24 is connected to the input-output interface 22. Further, a GPS device, a gyro sensor, a light detection and ranging (Lidar), and the like may be connected to the input-output interface 22 as needed.

(25) Functional Configuration of Information Processing Device 10

(26) The information processing device 10 realizes various functions using hardware resources mentioned above. The functional configuration realized by the information processing device 10 will be described with reference to FIG. 3.

(27) As shown in FIG. 3, the information processing device 10 includes an imaging unit 30, a learning unit 32, a calculation unit 34, a crossing status determination unit 36, a signal detection unit 38, an obstacle detection unit 40, a support information generation unit 42, and an output unit 44 as functional configurations. Each functional configuration is realized as the CPU 12 serving as a control unit reads and executes the program stored in the ROM 14 or the storage 18.

(28) The imaging unit 30 has a function of capturing an image of the traveling direction of the target person P with the camera 24 when the camera 24 is activated through the operation of the operation unit by the target person P.

(29) As shown in FIG. 4, the learning unit 32 generates a learned model M by performing machine learning using typical image data obtained by capturing images of pedestrian crossings as training data. Specifically, the typical image data represents images of pedestrian crossings that do not include pedestrians and images of pedestrian crossings that include pedestrians. Images showing pedestrian crossings and pedestrians are acquired from the typical image data, and machine learning is performed by the learning unit 32 using the image data of pedestrian crossing strips and pedestrians as teacher data. Thereby, the learning unit 32 generates the learned model M in which parameters of the feature points having the highest image recognition rate (object detection rate) from the viewpoint of recognizing pedestrian crossing strips and the viewpoint of recognizing pedestrians are stored for each class. The class means a category of an object recognized with image recognition. To the learned model M, for example, a deep neural network is applied. As an example, a long short term memory (LSTM) that is a kind of recursive neural network (RNN) is applied to the learned model M of the present embodiment.

(30) The calculation unit 34 estimates the shapes of the pedestrian crossing strips of the pedestrian crossing by inputting the image captured by the imaging unit 30 to the learned model M generated by the learning unit 32. Specifically, the calculation unit 34 performs object detection by inputting the captured image to the learned model M. The calculation unit 34 classifies the class of the object based on the feature points of the object that are acquired from the image input to the learned model, and detects that the object is a pedestrian crossing strip or a pedestrian.

(31) In addition, the calculation unit 34 performs a recognition process of recognizing an area of the detected object. Then, the calculation unit 34 generates a bounding box represented by a rectangular box, for an approximate area where the object is located in the image. The approximate area where the object is located is, for example, an area of a size necessary to enclose the object. The bounding box indicates an area larger than the area where the object is actually located in the detection of an object such as a person having an outer shape containing a curved line. However, in the detection of the object having a rectangular outer shape like the pedestrian crossing strip, the area enclosed by the bounding box substantially coincides with the outer shape of the object. Therefore, in the calculation of an occupancy rate and the like (R.sub.1 to R.sub.4) described later, dimensional errors with respect to the actual pedestrian crossing strips can be made small.

(32) For example, FIG. 5A shows an image 24A captured by the camera 24 of the white cane 1 from a position in front of the pedestrian crossing. The image 24A includes a pedestrian crossing 2 on the front side in the traveling direction and a traffic light 3 installed at the pedestrian crossing 2. The pedestrian crossing 2 includes a plurality of pedestrian crossing strips 2A extending in the width direction of the pedestrian crossing 2, and the pedestrian crossing strips 2A are arranged at predetermined intervals along a straightforward direction of the pedestrian crossing 2 so as to be parallel to each other. When the image 24A shown in FIG. 5A is input to the learned model M, bounding boxes B indicating the areas of the pedestrian crossing strips 2A are generated as shown in FIG. 5B.

(33) The calculation unit 34 also calculates the occupancy rate R.sub.1, a traveling direction appropriateness rate R.sub.2, a starting point distance securing rate R.sub.3, and a pedestrian crossing entrance rate R.sub.4, based on the sizes of the generated bounding boxes B. Definition of each rate will be described in order.

(34) The occupancy rate R.sub.1 is an occupancy rate of the pedestrian crossing 2 with respect to the image 24A in the lateral direction, and is a value serving as an index indicating the size that the pedestrian crossing occupies in the image 24A assuming a view ahead of the target person P. For the occupancy rate R.sub.1 of the present embodiment, the occupancy rate of the pedestrian crossing with respect to the image in the lateral direction is calculated. The relative positional relationship between the target person P and the pedestrian crossing 2 in the lateral direction can be determined based on the magnitude of the occupancy rate R.sub.1. As shown in the following equation (1), the occupancy rate R.sub.1 is represented by the ratio of the lateral dimension W.sub.x of the pedestrian crossing strip 2A to the lateral dimension W.sub.img of the image 24A.
Equation 1
R.sub.1=W.sub.x/W.sub.img  (1)

(35) As shown in FIG. 6, the lateral dimension W.sub.img of the image 24A is the width dimension of the image 24A. The lateral dimension of the pedestrian crossing strip 2A is the lateral dimension W.sub.x1 of the bounding box B. From the viewpoint of making the image 24A closer to the view ahead of the target person P that would be seen if the target person P recognizes using his/her vision, the lateral dimension W.sub.x1 may be used to calculate the occupancy rate R.sub.1 based on the bounding box B of the pedestrian crossing strip 2A located at the bottom of the image 24A that is closest to the target person P. The occupancy rate R.sub.1 may be a value obtained by correcting the value of the above equation (1) with a correction value corresponding to the physique and walking speed of the target person P.

(36) As shown in FIGS. 8A and 8B, when the target person P captures an image of the traveling direction from the center position of the pedestrian crossing 2 in the width direction, the lateral dimension W.sub.x1 of the pedestrian crossing strip 2A approaches the lateral dimension W.sub.img of the image 24A. Therefore, the occupancy rate R.sub.1 of the equation (1) approaches 1. On the other hand, as shown in FIG. 8C, when the target person P captures an image of the traveling direction from the position deviating to the right from the center of the pedestrian crossing 2, the lateral dimension W.sub.x1 of the pedestrian crossing strip 2A becomes smaller relative to the lateral dimension W.sub.img of the image 24A, so the occupancy rate R.sub.1 accordingly decreases.

(37) The traveling direction appropriateness rate R.sub.2 is an index indicating the degree of inclination of the traveling direction of the target person P with respect to the straightforward direction of the pedestrian crossing 2. As shown in the following equation (2), for the pedestrian crossing strip 2A located at the bottom of the image and the pedestrian crossing strip 2A located adjacent to and above the pedestrian crossing strip 2A at the bottom, the traveling direction appropriateness rate R.sub.2 is represented by the ratio of the lateral dimension W.sub.x(i+1) of the upper pedestrian crossing strip 2A to the lateral dimension W.sub.xi of the pedestrian crossing strip 2A at the bottom.
Equation 2
R.sub.2=W.sub.x(i+1)/W.sub.xi  (2)

(38) As shown in FIG. 6, the lateral dimension W.sub.x2 of the pedestrian crossing strip 2A located adjacent to and above the pedestrian crossing strip 2A at the bottom of the image 24A is a lateral dimension of the corresponding bounding box B. The traveling direction appropriateness rate R.sub.2 may be a value obtained by correcting the value of the above equation (2) with a correction value corresponding to the physique and walking speed of the target person P.

(39) As shown in FIGS. 8A and 8B, when the target person P is traveling along the straightforward direction of the pedestrian crossing 2 from the center position of the pedestrian crossing 2 in the width direction, a difference between the lateral dimensions W.sub.x1, W.sub.x2 of the adjacent pedestrian crossing strips 2A is small. Therefore, the traveling direction appropriateness rate R.sub.2 approaches 1. On the other hand, as shown in FIG. 8D, when the target person P is facing in a direction that is greatly inclined to the right with respect to the straightforward direction inside the pedestrian crossing 2, the lateral dimension W.sub.x2 of the pedestrian crossing strip 2A located adjacent to and above the pedestrian crossing strip 2A at the bottom becomes smaller relative to the lateral dimension W.sub.x1 of the pedestrian crossing strip at the bottom. Therefore, the traveling direction appropriateness rate R.sub.2 decreases. That is, it can be seen that as the traveling direction appropriateness rate R.sub.2 decreases, the amount of inclination of the traveling direction of the target person P with respect to the straightforward direction of the pedestrian crossing 2 increases. Thus, in such a case, it can be determined that the target person P is likely to deviate from the pedestrian crossing if the target person P continues to travel in this direction.

(40) The starting point distance securing rate R.sub.3 is an index indicating the distance between the starting point of the pedestrian crossing 2 and the target person P. The starting point of the pedestrian crossing 2 is a position of the pedestrian crossing strip 2A that is closest to the target person P when viewed from the position in front of the pedestrian crossing 2. As shown in the following equation (3), the starting point distance securing rate R.sub.3 is represented by the ratio of a height dimension H.sub.x of the pedestrian crossing strip 2A to the vertical dimension H.sub.img of the image 24A.
Equation 3
R.sub.3=H.sub.x/H.sub.img  (3)

(41) As shown in FIG. 7, the vertical dimension H.sub.img of the image 24A is the dimension of the image 24A in the up-down direction. The height dimension of the pedestrian crossing strip 2A is the height dimension H.sub.x1 in the vertical direction from the lower end of the image 24A to the centerline of the pedestrian crossing strip 2A. From the viewpoint of making the image 24A closer to the view ahead of the target person P that would be seen if the target person P recognizes using his/her vision, the height dimension H.sub.x1 may be used to calculate the starting point distance securing rate R.sub.3 based on the bounding box B of the pedestrian crossing strip 2A located at the bottom of the image 24A that is closest to the target person P. The starting point distance securing rate R.sub.3 may be a value obtained by correcting the value of the above equation (3) with a correction value corresponding to the physique and walking speed of the target person P.

(42) As shown in FIG. 8A, when the target person P travels from a standing position in front of the pedestrian crossing 2 with a predetermined distance secured from the starting point, to a position within the pedestrian crossing 2 in FIG. 8B, the height dimension H.sub.x1 of the pedestrian crossing strip 2A at the bottom of the image 24A becomes smaller relative to the vertical dimension H.sub.img of the image 24A as the target person P approaches the starting point. Therefore, the starting point distance securing rate R.sub.3 decreases.

(43) The pedestrian crossing entrance rate R.sub.4 is an index indicating the degree of entrance of the target person P into the pedestrian crossing 2. As shown in the following equation (4), the pedestrian crossing entrance rate R.sub.4 is represented by the ratio of the height dimension H.sub.x(i+1) of the pedestrian crossing strip 2A that is located adjacent to and above the pedestrian crossing strip 2A at the bottom of the image 24A to the height dimension H.sub.xi of the pedestrian crossing strip 2A at the bottom.
Equation 4
R.sub.4=H.sub.x(i+1)/H.sub.xi  (4)

(44) As shown in FIG. 7, the height dimension H.sub.x2 of the pedestrian crossing strip 2A located adjacent to and above the pedestrian crossing strip 2A at the bottom of the image 24A is the height dimension H.sub.x2 in the vertical direction from the lower end of the image 24A to the centerline of the corresponding bounding box B. The pedestrian crossing entrance rate R.sub.4 may be a value obtained by correcting the value of the above equation (4) with a correction value corresponding to the physique and walking speed of the target person P.

(45) As shown in FIG. 8A, when the target person P travels from a standing position in front of the pedestrian crossing 2 with a predetermined distance secured from the starting point, to a position within the pedestrian crossing 2 in FIG. 8B, the height dimension H.sub.xi of the pedestrian crossing strip 2A at the bottom of the image 24A becomes smaller relative to the height dimension H.sub.X2 of the pedestrian crossing strip 2A located adjacent to and above the pedestrian crossing strip 2A at the bottom. Therefore, as the target person P approaches the starting point, the pedestrian crossing entrance rate R.sub.4 increases.

(46) The crossing status determination unit 36 determines the crossing status of the target person for the pedestrian crossing, based on the occupancy rate R.sub.1, the traveling direction appropriateness rate R.sub.2, the starting point distance securing rate R.sub.3, and the pedestrian crossing entrance rate R.sub.4 calculated by the calculation unit 34. Specifically, the crossing status determination unit 36 determines a status as to whether the target person P can deviate to the outside of the pedestrian crossing 2. The crossing status determination unit 36 also determines the status as to whether the safety is ensured at the position in front of the pedestrian crossing 2 before the target person P crosses the pedestrian crossing 2.

(47) When the traffic light 3 is installed at the pedestrian crossing 2, the signal detection unit 38 detects a signal state of the traffic light 3 based on the image 24A captured by the camera 24. The signal state is, for example, a crossable state (green light) that allows crossing of the pedestrian crossing 2 and a non-crossable state (red light) that does not allow crossing of the pedestrian crossing 2.

(48) When the traffic light 3 is not installed at the pedestrian crossing 2, the obstacle detection unit 40 detects an obstacle approaching in the direction of crossing the pedestrian crossing 2 based on the image 24A captured by the camera 24.

(49) The support information generation unit 42 generates the support information for supporting the target person crossing the pedestrian crossing based on various crossing statuses determined by the crossing status determination unit 36, the detection result of the signal detection unit 38, and the detection result of the obstacle detection unit.

(50) The output unit 44 outputs various kinds of support information generated by the support information generation unit 42 with the output device 26 to notify the target person P. Specifically, the output unit 44 may notify by operating the vibrator at the timing when the various kinds of information are generated. In this case, the vibration pattern of the vibrator may be changed in accordance with the output information. Further, the output unit 44 may output an audio signal or a message from the speaker to notify the target person P of the various kinds of information.

(51) Operations

(52) Next, operations of the present embodiment will be described.

(53) Support Process

(54) An example of the support process for providing the support information to the target person P will be described using the flowchart shown in FIG. 9. The support process is executed as the CPU 12 reads a support program from the ROM 14 or the storage 18, deploys the program to the RAM 16, and executes the program.

(55) As shown in FIG. 9, the CPU 12 activates the camera 24 in step S100. The camera 24 is activated by operating the operation unit at the timing when it is determined that the target person P has reached the vicinity of the pedestrian crossing 2.

(56) Next, the CPU 12 acquires a captured image in step S101. Specifically, the CPU 12 acquires, with the function of the imaging unit 30, the image 24A of the traveling direction of the target person P that is captured by the camera 24.

(57) The CPU 12 estimates the shape of the pedestrian crossing strip 2A included in the image 24A in step S102. Specifically, the CPU 12 generates the bounding box B corresponding to the area of the pedestrian crossing strip 2A with the function of the calculation unit 34, and estimates the shape of the pedestrian crossing strip 2A. The learned model M is used to generate the bounding box B at this time.

(58) The CPU 12 determines in step S103 whether there is the shape of the pedestrian crossing strip 2A. When the CPU 12 determines that there is a pedestrian crossing strip, the CPU 12 proceeds to step S104 and executes the start position support process. The start position support process can provide support for causing the target person P to travel to a safe position such that the target person P does not enter the pedestrian crossing before crossing the pedestrian crossing.

(59) On the other hand, for example, when the pedestrian crossing 2 is congested and thus the pedestrian crossing strip is not detected even with the learned model M, or when the captured image 24A is an image that does not include the pedestrian crossing, the CPU 12 determines that there is no pedestrian crossing strip, and proceeds to step S105. In step S105, the CPU 12 generates the support information indicating that the pedestrian crossing strip cannot be recognized with the function of the support information generation unit 42, and notifies the target person P of the support information with the function of the output unit 44. When the CPU 12 determines that there is no pedestrian crossing strip because the pedestrian crossing is congested, the support information may be notified as a voice message “The pedestrian crossing is congested”. Alternatively, the support information may be notified as a voice message “Please stop” or the like. When the CPU 12 completes the process of step S105, the CPU 12 returns to step S102.

(60) Subsequently, the CPU 12 determines in step S106 whether there is a traffic light. Specifically, the CPU 12 determines presence or absence of a traffic light from the image 24A captured with the function of the signal detection unit 38. When the CPU 12 determines that there is a traffic light, the CPU 12 proceeds to step S107, and determines whether the signal state of the traffic light is a green light.

(61) When the CPU 12 determines in step S107 that the traffic light is a green light, the CPU 12 proceeds to step S110. When the CPU 12 determines in step S107 that the traffic light is a red light, the CPU 12 repeats the process of step S107.

(62) On the other hand, when the CPU 12 determines in step S106 that there is no traffic light, the CPU 12 proceeds to step S108, and determines whether there is an obstacle. Specifically, the CPU 12 analyzes the image 24A with the function of the obstacle detection unit 40, and determines presence or absence of an obstacle approaching in the direction of crossing the pedestrian crossing 2. When the CPU 12 determines in step S108 that there is an obstacle, the CPU 12 proceeds to step S109, and provides a warning to the target person P. Specifically, the CPU 12 generates the support information for notifying that there is an obstacle with the function of the support information generation unit 42, and notifies the target person P of the support information with the function of the output unit 44. Thereby, it is possible to suppress a collision between the target person P and an obstacle. When the CPU 12 completes the process of step S109, the CPU 12 returns to step S108.

(63) On the other hand, when the CPU 12 determines in step S108 that there is no obstacle, the CPU 12 proceeds to step S110.

(64) The CPU 12 notifies the target person of the fact that it is possible to cross the pedestrian crossing 2 in step S110. Specifically, the CPU 12 generates support information for prompting for crossing of the pedestrian crossing with the function of the support information generation unit 42, and outputs the support information from the output device 26 with the function of the output unit 44.

(65) The CPU 12 executes a crossing position support process in step S111. Through the crossing position support process, it is possible to provide support information for supporting the target person P not to deviate from the pedestrian crossing when crossing the pedestrian crossing.

(66) Subsequently, the CPU 12 determines whether the camera 24 has stopped in step S112. The camera 24 is stopped by operating the operation unit of the white cane 1 at the timing when it is determined that the target person P has crossed the pedestrian crossing. When the CPU 12 determines that the camera 24 has stopped, the CPU 12 ends the process. When the CPU 12 determines that the camera 24 has not stopped, the CPU 12 returns to step S106 and repeats the process.

(67) Start Position Support Process

(68) Next, the start position support process will be described with reference to FIG. 10. This process is executed, for example, when the shape of the pedestrian crossing strip 2A is estimated by the calculation unit 34.

(69) The CPU 12 calculates the starting point distance securing rate R.sub.3 in step S200. After that, the CPU 12 proceeds to step S201 and determines whether the starting point distance securing rate R.sub.3 is equal to or larger than a predetermined threshold value. When the CPU 12 determines that the starting point distance securing rate R.sub.3 is equal to or larger than the predetermined threshold value, the CPU 12 determines that the target person P is in a safe position in front of the pedestrian crossing, and proceeds to step S203. When the CPU 12 determines that the starting point distance securing rate R.sub.3 is smaller than the predetermined threshold value, the CPU 12 determines that the target person P has entered the pedestrian crossing or will soon enter the pedestrian crossing, and proceeds to step S202.

(70) The CPU 12 notifies that the target person P has entered the pedestrian crossing in step S202. Specifically, the CPU 12 generates the corresponding support information with the function of the support information generation unit 42, and outputs the support information with the function of the output unit 44. The support information may be provided by directly indicating the direction in which the target person P should travel with a voice message. When the CPU 12 completes the process of step S202, the CPU 12 returns to step S200.

(71) The CPU 12 calculates the pedestrian crossing entrance rate R.sub.4 in step S203. After that, the CPU 12 proceeds to step S204 and determines whether the pedestrian crossing entrance rate R.sub.4 is smaller than a predetermined threshold value. When the CPU 12 determines that the pedestrian crossing entrance rate R.sub.4 is smaller than the predetermined threshold value, the CPU 12 determines that the target person P is in a safe position in front of the pedestrian crossing, and proceeds to step S206.

(72) When the CPU 12 determines in step S104 that the pedestrian crossing entrance rate R.sub.4 is equal to or larger than the predetermined threshold value, the CPU 12 determines that the target person P has entered the pedestrian crossing or will soon enter the pedestrian crossing, and proceeds to step S205. The CPU 12 notifies that the target person P has entered the pedestrian crossing in step S205. Specifically, the CPU 12 generates the corresponding support information with the function of the support information generation unit 42, and outputs the support information with the function of the output unit 44. The support information may be provided by directly indicating the direction in which the target person P should travel with a voice message. When the CPU 12 completes the process of step S205, the CPU 12 returns to step S203.

(73) In step S206, the CPU 12 provides the support information indicating that the target person P is in an appropriate start position (standby position), and returns to the step of the support process.

(74) Crossing Position Support Process

(75) Next, the crossing position support process will be described with reference to FIG. 11. This process is executed, for example, when the output unit 44 notifies that the target person P can cross.

(76) The CPU 12 calculates the occupancy rate R.sub.1 in step S300, and determines in step S301 whether the occupancy rate R.sub.1 is equal to or larger than the predetermined threshold value. When the occupancy rate R.sub.1 is smaller than the predetermined threshold value, the CPU 12 determines that the position of the target person P is deviated in the lateral direction with respect to the pedestrian crossing 2, so the target person P is likely to deviate from the pedestrian crossing 2 if the target person P continues to travel in this direction (see the position in FIG. 8C). In this case, the CPU 12 proceeds to step S302 and provides support information indicating that the target person P is outside the pedestrian crossing. When the CPU 12 completes the process of step S302, the CPU 12 returns to step S300.

(77) When the occupancy rate R.sub.1 is equal to or larger than the predetermined threshold value, the CPU 12 determines that the position of the target person P is inside the pedestrian crossing, and proceeds to step S303.

(78) The CPU 12 calculates the traveling direction appropriateness rate R.sub.2 in step S303, and determines in step S304 whether the traveling direction appropriateness rate R.sub.2 is equal to or larger than the predetermined threshold value. As described above, when the traveling direction appropriateness rate R.sub.2 is smaller than the predetermined threshold value, it can be seen that the traveling direction of the target person P is greatly inclined with respect to the straightforward direction of the pedestrian crossing 2 (see the traveling direction in FIG. 8D). Thus, the CPU 12 determines that the target person P is likely to deviate from the pedestrian crossing 2 if the target person P continues to travel in this direction with the function of the crossing status determination unit 36. Then, when the CPU 12 determines that the traveling direction appropriateness rate R.sub.2 is smaller than the predetermined threshold value, the CPU 12 proceeds to step S305 and provides the support information indicating that the traveling direction is inappropriate. The support information may be provided by directly indicating the direction to which the target person P should turn with a voice message. When the CPU 12 completes the process of step S305, the CPU 12 returns to step S303.

(79) When the CPU 12 determines that the traveling direction appropriateness rate R.sub.2 is equal to or larger than the predetermined threshold value, the CPU 12 provides, in step S306, the support information indicating that the target person P is inside the pedestrian crossing and ends the process.

(80) As described above, in the information processing device 10 of the present embodiment, the calculation unit 34 calculates the occupancy rate R.sub.1 of the pedestrian crossing 2 in the image 24A. The captured image 24A of the traveling direction of the target person P is an image close to the view ahead of the target person P. Therefore, the crossing status determination unit 36 can determine the crossing status of the target person P for the pedestrian crossing 2 based on the occupancy rate R.sub.1. The support information generation unit 42 generates the support information for supporting the target person P crossing the pedestrian crossing 2 based on the crossing status of the target person P. In this way, the information processing device 10 can determine the crossing status of the target person P based on the captured image 24A of the traveling direction and support the target person P crossing the pedestrian crossing 2. Further, since a configuration is adopted in which the crossing status is determined based on only the image captured when the target person P crosses the pedestrian crossing, the target person P can easily use the information processing device 10.

(81) Specifically, when the occupancy rate R.sub.1 of the pedestrian crossing 2 in the captured image 24A is smaller than the predetermined threshold value, that is, when the presence of the pedestrian crossing 2 in the view ahead of the target person P is small, it can be seen that the target person P is located at a position deviated from the pedestrian crossing. Thus, when the occupancy rate R.sub.1 is smaller than the predetermined threshold value, the crossing status determination unit 36 of the information processing device 10 determines that the target person P may deviate from the pedestrian crossing 2.

(82) Further, in the present embodiment, as shown in the equation (1), the occupancy rate R.sub.1 of the pedestrian crossing 2 in the lateral direction of the image 24A is calculated based on the lateral dimension W.sub.img of the image 24A and the lateral dimension W.sub.x of the pedestrian crossing strip 2A in the image 24A. Thus, the lateral positional deviation of the target person P with respect to the pedestrian crossing 2 can be effectively estimated with the occupancy rate R.sub.1.

(83) Further, in the present embodiment, the traveling direction appropriateness rate R.sub.2 is calculated based on the lateral dimensions W.sub.xi, W.sub.x(i+1) of the pedestrian crossing strip 2A at the bottom of the image 24A and the pedestrian crossing strip 2A located adjacent to and above the pedestrian crossing strip 2A at the bottom.

(84) The pedestrian crossing strips 2A are arranged in parallel with each other. Therefore, when the target person P is traveling straightforward on the pedestrian crossing 2, that is, when the traveling direction is appropriate, the difference between the lateral dimensions of the pedestrian crossing strip 2A at the bottom and the pedestrian crossing strip 2A located adjacent to and above the pedestrian crossing strip 2A at the bottom becomes a small value. Thus, the traveling direction appropriateness rate R.sub.2 approaches 1. When the target person P is traveling in a direction that is greatly inclined with respect to the straightforward direction of the pedestrian crossing 2, the lateral dimension W.sub.x2 of the pedestrian crossing strip 2A on the upper side in the image in FIG. 6 becomes smaller relative to the lateral dimension W.sub.x1 of the pedestrian crossing strip 2A at the bottom, so the traveling direction appropriateness rate R.sub.2 decreases. Thus, when the traveling direction appropriateness rate R.sub.2 is smaller than the predetermined threshold value, the crossing status determination unit 36 can determine that the target person P may deviate from the pedestrian crossing 2.

(85) Further, in the present embodiment, the starting point distance securing rate R.sub.3 of the equation (3) is calculated based on the vertical dimension H.sub.img of the image shown in FIG. 7 and the height dimension H.sub.x1 of the pedestrian crossing strip 2A at the bottom. The pedestrian crossing strip 2A located at the bottom of the image indicates the starting point of the pedestrian crossing 2 ahead of the target person P in the traveling direction. Thus, the relative positional relationship between the starting point of the pedestrian crossing 2 and the target person P can be estimated with the starting point distance securing rate R.sub.3. Further, as the target person P approaches the starting point of the pedestrian crossing 2, the starting point distance securing rate R.sub.3 decreases. Thus, when the starting point distance securing rate R.sub.3 is equal to or larger than the predetermined threshold value, the crossing status determination unit 36 can determine that the target person P is located at a point in front of the pedestrian crossing 2.

(86) Further, in the present embodiment, the pedestrian crossing entrance rate R.sub.4 is calculated based on the height dimensions H.sub.x1, H.sub.x2 of the pedestrian crossing strip 2A at the bottom of the image 24A and the pedestrian crossing strip 2A located adjacent to and above the pedestrian crossing strip 2A at the bottom shown in FIG. 7. That is, as the target person P approaches the pedestrian crossing strip 2A at the bottom, the height dimension H.sub.x1 of the pedestrian crossing strip 2A at the bottom becomes smaller relative to the height dimension H.sub.x2 of the pedestrian crossing strip 2A located adjacent to and above the pedestrian crossing strip 2A at the bottom. Therefore, the pedestrian crossing entrance rate R.sub.4 becomes high. Thus, when the pedestrian crossing entrance rate R.sub.4 is smaller than the predetermined threshold value, the crossing status determination unit 36 can determine that the target person P is located at a point in front of the pedestrian crossing 2.

(87) Further, in the present embodiment, the calculation unit 34 inputs the captured image 24A to the learned model M, so as to be able to accurately estimate the shape of the pedestrian crossing strip 2A and calculate the occupancy rate R.sub.1 even when the image 24A includes another pedestrian using the pedestrian crossing 2. Thus, even when there are other pedestrians around the target person, the pedestrian crossing 2 can be accurately recognized and the support information can be provided.

(88) Further, in the present embodiment, it is possible to support the travel of the target person P by detecting the signal state of the traffic light 3. Thus, when the target person P crosses the pedestrian crossing 2 at which the traffic light 3 is installed, it is possible to provide more safety-conscious support information.

(89) Further, in the present embodiment, it is possible to support the target person P by detecting an obstacle approaching in the direction of crossing the pedestrian crossing 2. Thus, even in the pedestrian crossing 2 at which the traffic light is not installed, the support information can be provided so as to allow the safe crossing of the target person P.

(90) Further, in the present embodiment, the information processing device 10 is provided on the white cane 1, and the information processing device 10 includes the imaging unit 30 for capturing the image 24A and the output unit 44 for outputting the support information. Thus, when the target person P is a visually impaired person, the support information can be provided using the white cane 1 that is carried on a daily basis, which can improve the convenience.

(91) Supplementary Description

(92) The information processing device 10 and the information processing method executed by the information processing device 10 according to the present embodiment have been described above. It goes without saying that the present disclosure can be implemented in various modes without departing from the scope of the present disclosure. For example, the information processing device 10 is provided on the white cane 1 in the above embodiment, but the present disclosure is not limited to this. All or part of the information processing device may be composed of another device such as a smartphone that can be carried by the target person.

(93) For example, the output device 26 is provided on the white cane 1 in the above embodiment, but the present disclosure is not limited to this. The output device 26 may be provided in a device such as a smartphone or earphones that can be carried by the target person P. In this case, the device equipped with an output device such as a smartphone may receive output signals from the output unit of the white cane 1 by short-range wireless communication such as Bluetooth (registered trademark) and output as a sound or vibration pattern.

(94) Further, the target person for support is a visually impaired person in the above embodiment, but the present disclosure is not limited to this. For example, the support information for crossing a pedestrian crossing may be provided for elderly people who have difficulty in turning their bodies quickly in a predetermined direction for visual checking or people with physical disabilities who act in wheelchairs.

(95) In the above embodiment, the camera 24 is activated through the operation of the operation unit by the target person P and the information processing device does not have map data that requires a GPS device or a large-scale storage device so that the entire device is simplified and downsized. However, the present disclosure is not limited to this. For example, a configuration may be adopted in which the current position of the target person is acquired by a GPS device and the camera 24 is automatically activated when it is detected that the target person is in the vicinity of a pedestrian crossing with reference to the map data based on the current position. Alternatively, a configuration may be adopted in which the information processing device can communicate with an external device such as a smartphone, the current position of the target person P (current position of the information processing device) is acquired based on the GPS function and the map data reference function of the smartphone or the like, and the camera 24 is activated when it is detected that the current position is in the vicinity of the pedestrian crossing.

(96) Further, in the above embodiment, the occupancy rate in the image in the lateral direction is calculated based on the lateral dimension of the captured image and the lateral dimension of the pedestrian crossing strip 2A. However, the present disclosure is not limited to this. For example, the occupancy rate of area of the pedestrian crossing with respect to the image may be defined as the occupancy rate of the pedestrian crossing in the present disclosure, based on the area of the captured image and the total area of the captured pedestrian crossing strips.

(97) Further, various processors other than the CPU 12 may execute the support process, the start position support process, and the crossing position support process that are executed as the CPU 12 reads the software (program) in the above embodiment. Examples of the processor in this case include a programmable logic device (PLD) of which circuit configuration can be changed after being manufactured, such as a field-programmable gate array (FPGA), and a dedicated electric circuit that is a processor including a circuit configuration that has been exclusively designed for executing a specific process, such as an application specific integrated circuit (ASIC). In addition, the support process, the start position support process, and the crossing position support process may be executed by one of these various processors, or a combination of two or more processors of the same type or different types (for example, a combination of FPGAs and a combination of a CPU and an FPGA, and the like). Further, the hardware configuration of the various processors is, more specifically, an electric circuit in which circuit elements such as semiconductor devices are combined.

(98) Further, in the above embodiment, the storage 18 is configured to store various data. However, the disclosure is not limited to this. For example, a recording medium such as a compact disk (CD), a digital versatile disk (DVD), or a universal serial bus (USB) memory may be used as a storage unit. In this case, various programs, data, and the like are stored in the recording medium mentioned above.