Display control device, display control system, display control method, display control program, and recording medium

Abstract

A display control device (10) comprises an image acquisition component (11), an orientation estimation component (12), and a controller (13). The image acquisition component (11) acquires an infrared image (G1) sensed by infrared array sensors (21a and 21b) installed in a room. The orientation estimation component (12) estimates the orientation of a care receiver (P1) in a room (30) on the basis of the infrared image (G1) acquired by the image acquisition component (11). The controller (13) controls a display device (22) so that a dummy image (D1), which shows a simplified view of the orientation of the care receiver (P1) estimated by the orientation estimation component (12), is displayed superimposed over the infrared image (G1).

Claims

1. A display control device comprising: an image acquisition component configured to acquire image data sensed by an image sensor installed in a room; an orientation estimation component configured to estimate the orientation of a person in the room on the basis of the image data acquired by the image acquisition component; and a controller configured to control a display device so that a first dummy image, which shows a simplified view of the orientation of the person estimated by the orientation estimation component, is displayed superimposed over the image data, and a determination component configured to determine a care urgency level according to an amount of flickering of an image near the person in the image data by referring to the plurality of pieces of image data continuously acquired by the image acquisition component, and the controller superimposes and displays the head or face portion of the first dummy image, using as a reference the position of the head or face of the person included in the image data.

2. The display control device according to claim 1, further comprising: a storage component configured to store a plurality of the first dummy images in a state of being associated with orientations of the person.

3. The display control device according to claim 1, wherein the image sensor is an infrared sensor configured to acquire a thermal image, and the determination component makes a determination according to the degree of fluctuation in the position of the thermal center of gravity in an area near the person in the thermal image continuously acquired by the infrared sensor.

4. The display control device according to claim 1, wherein the image sensor is an infrared sensor configured to acquire a thermal image, and the determination component makes a determination by detecting enlargement of a heat source in an area near the person in the thermal image continuously acquired by the infrared sensor.

5. The display control device according to claim 1, further comprising a count estimator configured to estimate the number of people included in the image data acquired by the image acquisition component.

6. The display control device according to claim 5, wherein the controller performs display control of the display device when it is determined by the count estimator that the number of people in the image data is one.

7. The display control device according to claim 1, wherein the controller changes a color of the first dummy image on the display device according to an estimation result produced by the orientation estimation component.

8. The display control device according to claim 1, wherein the controller flashes the first dummy image on the display device according to an estimation result produced by the orientation estimation component.

9. The display control device according to claim 1, wherein the controller displays the first dummy image superimposed over the image data and displays an emergency message on the display device, according to the estimation result produced by the orientation estimation component.

10. The display control device according to claim 1, wherein the orientation estimation component estimates the orientation by using detection results produced by the plurality of the image sensors, which detect a person who is the room from different directions.

11. The display control device according to claim 1, wherein the controller displays the first dummy image, corresponding to at least one of a standing position, a seated position, a supine position, a lateral recumbent position, a prone position, and a half-sitting position of the person, superimposed over the image data.

12. The display control device according to claim 1, wherein the controller displays a second dummy image corresponding to furniture and equipment installed in the room so as to be superimposed with the locations of the furniture and the equipment in the image data.

13. A display control system, comprising: the display control device according to claim 1; and an image sensor configured to supply the image data to the image acquisition component.

14. The display control system according to claim 13, further comprising a display device whose display is controlled by the controller of the display control device.

15. The display control system according to claim 14, wherein the display device includes the display device of a host terminal used by the caregiver who cares for the person who is in the room, or a portable terminal owned by the caregiver.

16. The display control system according to claim 1, wherein the image sensor includes: a first image sensor configured to detect the person from above in the room; and a second image sensor configured to detect the person from a side.

17. The display control system according to claim 1, wherein the image sensor is either an infrared array sensor or a distance sensor.

18. A display control method, comprising: acquiring image data sensed by an image sensor installed in a room; estimating the orientation of a person in the room on the basis of the image data acquired in the image acquisition step; controlling a display device so that a first dummy image, which shows a simplified view of the orientation of the person estimated in the orientation estimation step, is displayed superimposed over the image data, and superimposing and displaying the head or face portion of the first dummy image, using as a reference the position of the head or face of the person included in the image data; and determining a care urgency level according to an amount of flickering of an image near the person in the image data by referring to the plurality of pieces of image data continuously acquired.

19. A non-transitory computer-readable storage medium that stores a display control program for causing a computer to execute a display control method, the display control method comprising: acquiring image data sensed by an image sensor installed in a room; estimating the orientation of a person in the room on the basis of the acquired image data; controlling a display device so that a first dummy image, which shows a simplified view of the estimated orientation of the person, is displayed superimposed over the image data, and superimposing and displaying the head or face portion of the first dummy image, using as a reference the position of the head or face of the person included in the image data; and determining a care urgency level according to an amount of flickering of an image near the person in the image data by referring to the plurality of pieces of image data continuously acquired.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is a block diagram of the configuration of a display control system including the display control device according to an embodiment of the present invention;

(2) FIG. 2 is an oblique view of the configuration in a room where an image is acquired that is displayed on the display device with the display control system in FIG. 1;

(3) FIG. 3A is a diagram showing an infrared array sensor attached to the ceiling of the room shown in FIG. 2, and FIG. 3B is a diagram showing an infrared array sensor attached to a shelf in the room shown in FIG. 2;

(4) FIGS. 4A and 4B are diagrams showing an example of a display device that constitutes the display control system in FIG. 1 (a host computer and a portable terminal);

(5) FIG. 5 is a diagram illustrating first image data used in orientation estimation by the orientation estimation component constituting the display control device in FIG. 1, and the examination region thereof;

(6) FIG. 6 is a diagram illustrating second image data used in orientation estimation by the orientation estimation component constituting the display control device in FIG. 1, and the examination region thereof;

(7) FIG. 7 is a diagram illustrating the examination regions of first and second image data used in orientation estimation by the orientation estimation component constituting the display control device in FIG. 1;

(8) FIG. 8 is a diagram illustrating the examination regions of first and second image data used in orientation estimation by the orientation estimation component constituting the display control device in FIG. 1;

(9) FIG. 9 is a diagram illustrating a transition model of orientation estimated by the orientation estimation component constituting the display control device in FIG. 1;

(10) FIG. 10 is a diagram illustrating the transition model information in FIG. 9;

(11) FIG. 11 shows region information indicating the examination regions of the first and second image data for each of the orientations in the transition model in FIG. 9, and reference value information representing reference values for determining whether there has been a transition to each orientation;

(12) FIG. 12 shows an image displayed on the display device constituting the display control system in FIG. 1;

(13) FIG. 13 is a flowchart showing the flow of display control by the display control device in FIG. 1;

(14) FIG. 14 is a flowchart showing the flow of orientation estimation by the display control device in FIG. 1;

(15) FIGS. 15A and 15B show a standing image captured using two infrared array sensors (on the ceiling and to the side) installed in the room shown in FIG. 2;

(16) FIGS. 16A and 16B show a seated image captured using two infrared array sensors (on the ceiling and to the side) installed in the room shown in FIG. 2;

(17) FIGS. 17A and 17B show a recumbent image (with a blanket) captured using two infrared array sensors (on the ceiling and to the side) installed in the room shown in FIG. 2;

(18) FIGS. 18A and 18B show a recumbent image (with the blanket turned down) captured using two infrared array sensors (on the ceiling and to the side) installed in the room shown in FIG. 2;

(19) FIGS. 19A and 19B show a rolled-over recumbent image (emergency) captured using two infrared array sensors (on the ceiling and to the side) installed in the room shown in FIG. 2;

(20) FIGS. 20A and 20B show a rolled-over seated image (emergency) captured using two infrared array sensors (on the ceiling and to the side) installed in the room shown in FIG. 2;

(21) FIG. 21A is a diagram showing a recumbent image (no flickering) captured using an infrared array sensor installed on the ceiling of the room shown in FIG. 2, and FIG. 21B is a diagram showing a recumbent image (heat source expansion around the head) captured using an infrared array sensor installed on the ceiling of the room shown in FIG. 2;

(22) FIG. 22A is a diagram showing a recumbent image (with vigorous flickering) captured using an infrared array sensor installed on the ceiling of the room shown in FIG. 2, and FIG. 22B is a diagram showing a recumbent image (blanket turned down, vigorous flickering) captured using an infrared array sensor installed on the ceiling of the room shown in FIG. 2;

(23) FIG. 23 shows a seated image (vigorous flickering) captured using an infrared array sensor installed on a side wall of the room shown in FIG. 2;

(24) FIG. 24 is a block diagram of the configuration of a display control device according to another embodiment of the present invention, and a display control system comprising this device; and

(25) FIGS. 25A and 25B are diagrams of a display that combines text information with a rolled-over recumbent image (emergency) on the display device whose display is controlled by the display control device according to yet another embodiment of the present invention.

DETAILED DESCRIPTION

(26) A display control device 10 according to an embodiment of the present invention, and a display control system 20 comprising this device, will be described below through reference to FIGS. 1 to 23.

(27) 1. Configuration of Display Control System 20

(28) The display control system 20 according to this embodiment is, for example, a system for monitoring a care receiver in need of nursing care in his everyday activities, such as in a nursing home or a hospital, and allowing a caregiver to accurately recognize the state of the c are receiver. As shown in FIG. 1, the display control system 20 comprises the display control device 10, infrared array sensors 21a and 21b, and a display device 22.

(29) Here, with the display control system 20 in this embodiment, in order to watch over the care receiver (person) P1 in a room 30, as shown in FIG. 2, the system makes use of the two infrared array sensors 21a and 21b, which are capable of capturing images of the care receiver P1 from above and from the side in the room 30.

(30) A bed 31, a wheelchair 32, and other such in-room equipment are installed in the room 30, forming the living space of the care receiver P1. A coordinate system X-Y-Z of the inside of the room is shown in FIG. 2.

(31) The display control device 10 uses an infrared image G1 (see FIG. 12) captured by the infrared array sensors 21a and 21b to determine whether or not the care receiver P1 in the room 30 needs care, and displays the result on the display device 22. The configuration of the display control device 10 will be described below in more detail.

(32) Here, with the display control device 10 in this embodiment, the infrared image G1 (see FIG. 12) is used in order to watch over the care receiver P1 in the room 30. Consequently, compared to when the watching is performed using a surveillance camera image, for example, watching can be performed while protecting the privacy of the care receiver P1.

(33) The infrared array sensors 21a and 21b are designed so that the portion of a person's face in the captured infrared image G1 taken is made up of 2 to 6 pixels. Consequently, an individual cannot be identified even with individual recognition technology using a portion of a face in an image, for example, which allows someone to watch over the care receiver P1 while still protecting the privacy of the care receiver.

(34) As shown in FIG. 2, the infrared array sensors 21a and 21b are attached to the ceiling 30a in the room 30, and capture images of the care receiver P1 in the room 30 from above.

(35) As shown in FIG. 3A, the infrared array sensor (first image sensor) 21a is fixed directly to the ceiling 30a of the chamber 30.

(36) As shown in FIG. 2, the infrared array sensor 21b (second image sensor) is attached to the wall 30b in the room 30, and captures images of the care receiver P1 in the room 30 from the side.

(37) As shown in FIG. 3B, rather than being installed on the wall 30b, the infrared array sensor 21b may be embedded in something within the room, such as a shelf 33.

(38) The infrared image data acquired by the infrared array sensors 21a and 21b is transmitted to the image acquisition component 11 of the display control device 10.

(39) The infrared image data captured by the infrared array sensors 21a and 21b is such that the temperature of the image region increases the higher is the pixel value, and the temperature of the image region decreases the lower is the pixel value. That is, since an area where a person is located will have a higher temperature, the value of pixels in an area where a person was imaged will be higher. Thus, an area where a person is present (a region of higher temperature) can be identified from the infrared image data by finding an area with higher pixel values.

(40) The display device 22 is, for example, a monitor of a host computer 22a installed in a nursing facility (see FIG. 4A), a mobile terminal 22b (see FIG. 4B), or the like, and displays an infrared image G1 obtained by imaging the care receiver P1 (see FIG. 12). The display device 22 is controlled by the display control device 10.

(41) The host computer 22a is installed in a caregiver station in a nursing home, for example. This allows a caregiver to remotely check on a number of care receivers P1.

(42) The portable terminal 22b encompasses a mobile phone, smart phone, tablet terminal, or the like owned by a caregiver.

(43) Consequently, even if the caregiver is in the room of another care receiver P1, or is moving around somewhere, if an image is transmitted to the portable terminal 22b, in the event of an emergency the caregiver can rush straight to the care receiver P1.

(44) 2. Configuration of Display Control Device 10

(45) As shown in FIG. 1, the display control device 10 in this embodiment comprises the image acquisition component 11, an orientation estimation component 12, a controller 13, a storage component 14, a determination component 15, and a count estimation component 16.

(46) The image acquisition component 11 acquires from the infrared array sensors 21a and 21b the infrared image data captured by the infrared array sensors 21a and 21b.

(47) The orientation estimation component 12 uses the infrared image data acquired by the image acquisition component 11 to estimate the orientation of the care receiver P1 in the room 30.

(48) The orientation estimation done the orientation estimation component 12 is accomplished using a known method (see Patent Literature 1) involving a transition model (described below). The orientations of the care receiver P1 estimated by the orientation estimation component 12 include, for example, a standing position, a seated position, a supine position, a lateral recumbent position, a prone position, a half-sitting position, and so forth.

(49) The controller 13 selects a dummy image (first dummy image) D1 (see FIG. 12) corresponding to each orientation stored in the storage component 14, on the basis of the orientation of the care receiver P1 estimated by the orientation estimation component 12. The selected dummy image D1 is then displayed on the display device 22 in a state of being superimposed over the infrared image G1.

(50) Here, the dummy image D1 is an image showing a simplified view of the appearance and shape of the care receiver P1 whose orientation has been estimated, and images corresponding to a plurality of types of orientation are prepared. In this embodiment, dummy images D1 are prepared for the respective directions need to superimpose them over images of the care receiver P1 captured from above and from the side.

(51) The display control of the display device 22 by the controller 13 will be described below in greater detail.

(52) The storage component 14 stores the infrared image data acquired by the image acquisition component 11, in association with a plurality of dummy images D1 corresponding to a plurality of types of orientation of the care receiver P1 estimated by the orientation estimation component 12.

(53) The dummy images D1 corresponding to the orientation of the care receiver P1 that are stored in the storage component 14 include, for example, a standing position, a seated position, a supine position, a lateral recumbent position, a prone position, and a half-sitting position.

(54) These dummy images D1 are stored in the storage component 14 in a number of types corresponding to the orientations of the infrared image G1 in which the room 30 is imaged from above and the infrared image G1 as imaged from the side.

(55) For example, with a standing position, a dummy image D1 to be superimposed over t the infrared image G1 captured from above, and a dummy image D1 to be superimposed over the infrared image G1 captured from the side may be provided. Similarly, with other orientations, dummy images D1 corresponding to the infrared image G1 from above and to the infrared image G1 from the side are stored in the storage component 14.

(56) The determination component 15 determines how urgently the care receiver P1 in the room 30 needs care, on the basis of the infrared image data acquired by the image acquisition component 11.

(57) More specifically, the determination component 15 refers, for example, to a plurality of infrared images G1 that have been acquired continuously (a moving picture), detects changes in flickering in the image, the expansion of a heat source, or the like, and thereby determines how urgently care is needed.

(58) For instance, if the orientation of the care receiver P1 estimated by the orientation estimation component 12 is a seated position, and there is little change in flickering in the plurality of images continuously acquired by the image acquisition component 11, there is the risk that the care receiver P1 has slumped over while in a seated position.

(59) Conversely, if the orientation of the care receiver P1 is a seated position, and there is a large amount of flicker in the plurality of images continuously acquired by the image acquisition component 11, it is presumed that the care receiver P1 is performing some operation, such as changing clothes or operating a portable terminal, while seated.

(60) Thus, if there is almost no flickering over a long period while the same orientation is maintained in the plurality of infrared images G1 that are continuously acquired, the determination component 15 determines that there is an urgent need for care, and the controller 13 causes the display component 22 to give a warning display.

(61) Also, if the orientation of the care receiver P1 estimated by the orientation estimation component 12 is a recumbent position, and a heat source in the plurality of images continuously acquired by the image acquisition component 11 has expanded, there is a high probability that the care receiver P1 is vomiting (see FIG. 21B).

(62) Thus, in this case, the determination component 15 determines that there is an urgent need for care, and the controller 13 causes the display device 22 to give a warning display.

(63) Here, examples of a warning display include flashing of the dummy image D1 on the display device 22, changing the color of the display, and giving a warning that combines text information. When the color is changed, for example, it may be changed from the normal display color of the dummy image D1 to red. Also, a warning display that combines text information may include, for example, a text display such as “care receiver needs nursing care!” A warning sound such as a beep may also be used in conjunction with the display.

(64) The count estimation component 16 uses the infrared image data acquired by the image acquisition component 11 to estimate the number of people in the room 30.

(65) More specifically, the count estimation component 16 senses a high-temperature portion presumed to be a human head included in the infrared image G1, and counts the number of such portions to estimate the number of people in the room 30.

(66) The number of people estimated to be in the room 30 by the count estimation component 16 is sent to the controller 13. If the number of people in the room 30 estimated by the count estimation component 16 is two or more, the controller 13 performs no display control of the display device 22.

(67) That is, with the display control device 10 in this embodiment, if the number of care receivers P1 residing in the room 30 is usually just one, for example, it is assumed that there is less need for watching over the room if there is someone in it other than the care receiver P1 (such as a caregiver, a close relative, or a friend).

(68) Thus, with the display control apparatus 10 in this embodiment, when there is someone other than the care receiver P1 in the room 30, that is, when the value estimated by the count estimation component 16 is two or more, no display control is performed on the display device 22.

(69) 3. Display Control

(70) 3-1. Orientation Estimation

(71) With the display control device 10 in this embodiment, as described above, the image acquisition component 11 acquires infrared image data captured by the infrared array sensors 21a and 21b. The orientation estimation component 12 then uses the infrared image G1 taken of the room 30 from above and the infrared image G1 taken from the side (both acquired by the image acquisition component 11) to estimate the orientation of the care receiver P1.

(72) Estimation of the orientation of the care receiver P1 by the orientation estimation component 12 is performed by the following procedure.

(73) That is, the orientation estimation component 12 performs orientation estimation by using examination information that includes region information representing an inspection region of the infrared image data (from above and from the side) at each orientation in a transition model of orientation of the human body, and reference value information representing a reference value for determining whether a transition has been made to each orientation.

(74) The transition model information and examination information are used to estimate the orientation of the body at the current point in time.

(75) FIGS. 5 and 6 are diagrams illustrating the infrared image data captured from the ceiling 30a (from above) by the infrared array sensor 21a, and the examination region thereof. The infrared image data is made up of 16 pixels in the X direction and 16 pixels in the Y direction.

(76) The region RV1 is the total region of the infrared image data, that is, a region in the range of 0≤X≤15, 0≤Y≤15.

(77) The region RV2 is the region a specific width (3 pixels) from a position adjacent to the boundary in the lengthwise direction of the region corresponding to the bed 31 (see FIG. 2), that is, a region in the range of 7≤X≤9, 3≤Y≤10.

(78) The region RV3 is the region corresponding to the bed 31, that is, a region in the range of 3≤X≤6, 3≤Y≤10.

(79) The region RV4 is the region excluding the region RV3, and is made up of four regions. The first region is a region in the range of 0≤X≤2, 0≤Y≤15. The second region s a region in the range of 7≤X≤15, 0≤Y≤15. The third region is a region in the range of 3≤X≤6, 0≤Y≤2. The fourth region is a region in the range of 3≤X≤6, 11≤Y≤15.

(80) FIGS. 7 and 8 are diagrams illustrating the infrared image data taken from the wall 30b (from the side) by the infrared array sensor 21b, and the examination region thereof. The infrared image data is made up of 16 pixels in the X direction, and 16 pixels in the Z direction.

(81) The region RH1 is the entire region of the second image data, that is, a region in the range of 0≤X≤15, 0≤Z≤15.

(82) The range in the vertical direction (Z direction) of the region RH2 is the total range. The range in the horizontal direction (X direction) of the region RH2 is a range of a specific width (3 pixels) from a position adjacent to the range corresponding to the bed 31 (11≤X≤15). Therefore, the region RH2 is a region in the range of 8≤X≤10, 0≤Z≤15.

(83) The range in the vertical direction (Z direction) of the region RH3 is a range of a specific width (5 pixels) from a position adjacent to the upper boundary of the range corresponding to the bed 31. The range in the horizontal direction (X direction) of the region RH3 is a range the same as the range corresponding to the bed 31. Therefore, the region RH3 is a region in the range of 11≤X≤15, 5≤Z≤9.

(84) The range in the vertical direction (Z direction) of the region RH4 is a range of a specific width (three pixels) from a position adjacent to the upper boundary of the range corresponding to the bed 31. The range in the horizontal direction (X direction) of the region RH4 is a range the same as the range corresponding to the bed 31. Therefore, the region RH4 is a region in the range of 11≤X≤15, 7≤Z≤9.

(85) The range in the horizontal direction of the region RH5 is a range obtained by excluding the range corresponding to the bed 31 from the entire range. The range in the vertical direction of the region RH5 is a range of a specific width (six pixels) upward from the lowest position (Z=15). Therefore, the region RH5 is a region in a range of 0≤X≤10, 10≤Z≤15.

(86) The range in the horizontal direction of the region RH6 is a range obtained by excluding the range corresponding to the bed 31 from the entire range. The range in the vertical direction of the region RH6 is a range of a specific width (three pixels) upward from a specific position (Z=12). Therefore, the region RH6 is a region in a range of 0≤X≤10, 10≤Z≤12.

(87) FIG. 9 is a diagram representing the transition model of the orientation of the care receiver (person) P1. FIG. 10 is a diagram representing transition model information.

(88) The care receiver P1 (the person being monitored) is not in the room 30 in the initial state (A).

(89) The orientation that comes after the orientation (A) is either that the person is in the room (B) or the original orientation (A).

(90) The orientation that comes after the orientation (B) is either standing (C) next to the bed 31, fallen (X), not in the room (A), or the original orientation (B).

(91) The orientation that comes after the orientation (C) is either lying on the bed 31 (D), sitting on the bed 31 (E), sitting on the end of the bed 31 (F), fallen (X), in the room (B), or the original orientation (C).

(92) The orientation that comes after the orientation (D) is either sitting on the bed 31 (E), sitting in end of the bed 31 (F), standing next to the bed 31 (C), fallen from the bed 31 (first pattern) (Y1), or the original orientation (D).

(93) The orientation that comes after the orientation (E) is either lying on the bed 31 (D), sitting on the end of the bed 31 (F), standing next to the bed 31 (C), fallen from the bed 31 (first pattern) (Y1), or the original orientation (E).

(94) The orientation that comes after the orientation (F) is either lying on the bed 31 (D), sitting on the bed 31 (E), standing next to the bed 31 (C), fallen from the bed 31 (second pattern) (Y2), or the original orientation (F).

(95) FIG. 11 is a table of region information representing the examination regions of the infrared image data for each of the orientations in the transition model, and reference value information representing reference values for determining whether there has been a transition to each orientation.

(96) In order to determine whether or not there has been a transition to absence (A) in the room 30, the infrared array sensors 21a and 21b calculate the sum of the statistical quantity of the region RV1 of the infrared image data captured from above, plus the statistical quantity of the region RH1 of the infrared image data captured from the side. When this sum of the statistical quantities is at or above a reference value THA, it is determined that there has been a shift to the orientation (A).

(97) In order to determine whether or not there has been a transition to being in the room 30 (B), the infrared array sensors 21a and 21b calculate the sum of the statistical quantity of the region RV1 of the infrared image data captured from above, plus the statistical quantity of the region RH1 of the infrared image data captured from the side. When this sum of the statistical quantities is at or above a reference value THB, it is determined that there has been a shift to the orientation (B).

(98) In order to determine whether or not there has been a transition to standing next to the bed 31 (C), the infrared array sensors 21a and 21b calculate the sum of the statistical quantity of the region RV2 of the infrared image data captured from above, plus the statistical quantity of the region RH2 of the infrared image data captured from the side. When this sum of the statistical quantities is at or above a reference value THC, it is determined that there has been a shift to the orientation (C).

(99) In order to determine whether or not there has been a transition to lying on the bed 31 (D), the infrared array sensors 21a and 21b calculate the sum of the statistical quantity of the region RV3 of the infrared image data captured from above, plus the statistical quantity of the region RH4 of the infrared image data captured from the side. When this sum of the statistical quantities is at or above a reference value THD, it is determined that there has been a shift to the orientation (D).

(100) In order to determine whether or not there has been a transition to sitting on the bed 31 (E), the infrared array sensors 21a and 21b calculate the sum of the statistical quantity of the region RV3 of the infrared image data captured from above, plus the statistical quantity of the region RH3 of the infrared image data captured from the side. When this sum of the statistical quantities is at or above a reference value THE, it is determined that there has been a shift to the orientation (E).

(101) In order to determine whether or not there has been a transition to sitting on the end of the bed 31 (F), the infrared array sensors 21a and 21b calculate the sum of the statistical quantity of the region RV3 of the infrared image data captured from above, plus the statistical quantity of the region RH2 of the infrared image data captured from the side. When this sum of the statistical quantities is at or above a reference value THF, it is determined that there has been a shift to the orientation (F).

(102) In order to determine whether or not there has been a transition to having fallen (X), the infrared array sensors 21a and 21b calculate the sum of the statistical quantity of the region RV4 of the infrared image data captured from above, plus the statistical quantity of the region RH5 of the infrared image data captured from the side. When this sum of the statistical quantities is at or above a reference value THX, it is determined that there has been a shift to the orientation (X).

(103) Here, in order to determine whether or not there has been a transition to having fallen from the bed 31 (first pattern) (Y1), the infrared array sensors 21a and 21b calculate the sum of the statistical quantity of the region RV2 of the infrared image data captured from above, plus the statistical quantity of the region RH6 of the infrared image data captured from the side. When this sum of the statistical quantities is at or above a reference value THY1, it is determined that there has been a shift to the orientation (Y1).

(104) The reason why the region RH6 is used for this determination is that the location where the infrared array sensor 21b is installed is only a short distance from the location where the care receiver P1 has fallen out of the bed 31, so with the infrared image data captured from the side, the captured portion of the care receiver P1 who has fallen is at a position higher than the lowest line (Y=15).

(105) Also, in order to determine whether or not there has been a transition to having fallen from the bed 31 (second pattern) (Y2), the infrared array sensors 21a and 21b calculate the sum of the statistical quantity of the region RV2 of the infrared image data captured from above, plus the statistical quantity of the region RH6 of the infrared image data captured from the side. When this sum of the statistical quantities is at or above a reference value THY2, it is determined that there has been a shift to the orientation (Y2).

(106) The reason why the region RH6 is used for this determination is the same as with falling out of the bed 31 (first pattern).

(107) 3-2. Display Control

(108) With the display control device 10 in this embodiment, as a result of the above-mentioned orientation estimation, the display device 22 is controlled so that the dummy image D1 corresponding to each orientation stored in the storage component 14 is displayed superimposed over the infrared image G1 on the basis of the estimated orientation of the care receiver P1.

(109) That is, the storage component 14 stores a plurality of the dummy images D1 corresponding to the orientations of the care receiver P1 estimated by the orientation estimation component 12. Therefore, as shown in FIG. 12, the controller 13 checks the result of the attitude estimation by the orientation estimation component 12, reads the appropriate dummy image D1 corresponding orientation of the care receiver P1 in the infrared image G1 from the storage component 14, and displays this image superimposed over the thermal image G1.

(110) Here, when the dummy image D1 corresponding to the estimated orientation is displayed superimposed over the infrared images G1, the superposition is based on the position of the head of the care receiver P1.

(111) More precisely, in the infrared image G1, the head and face portions are usually located higher up on the body, and are usually exposed most of the time, so these portions are represented as having a higher temperature. Thus, the head and face portions of the dummy image D1 are positioned so as to be superimposed in their display on the display device 22, based on the estimated position of the heat and face portions in the infrared image G1.

(112) 3-3. Flow of Display Control

(113) The display control device 10 in this embodiment carries out display control according to the flowchart shown in FIG. 13.

(114) That is, in step S11, the image acquisition component 11 acquires the infrared image G1 captured in the room 30 from above and from the side, using the two infrared array sensors 21a and 21b.

(115) Next, in step S12, the count estimation component 16 uses the infrared image G1 thus acquired to estimate the number of people in the room 30.

(116) Next, in step S13, it is determined whether or not the number of people in the room 30 estimated by the count estimation component 16 just one. If there is only one person, the process proceeds to step S14. On the other hand, if there are two or more people, it is determined that someone other than the care receiver P1 is in the room 30 and can provide care, and the display control process is ended.

(117) Next, in step S14, the orientation estimation component 12 estimates the orientation of the care receiver P1 determined to be the only person in the room 30 in step S13.

(118) Next, in step S15, the controller 13 reads from the storage component 14 the dummy image D1 corresponding to the orientation (such as a standing position) of the care receiver P1 estimated in step S14.

(119) Next, in step S16, the controller 13 superimposes and displays the dummy image D1 read out from the storage component 14, based on the head or face portion of the care receiver P1 included in the infrared image G1.

(120) Consequently, the caregiver can recognize the orientation of the care receiver P1 displayed on the display device 22, and can easily determine whether or not there is any abnormality.

(121) 3-4. Flow of Orientation Estimation

(122) With the display control device 10 in this embodiment, the orientation estimation in step S14 in the flowchart of FIG. 13 is carried out according to the flowchart shown in FIG. 14.

(123) That is, after the infrared image G1 is acquired from the infrared array sensors 21a and 21b in step S11, it is determined in step S21 whether or not the orientation of the care receiver P1 in the infrared image G1 is indicates that the person has fallen down. Here, if it is determined that the person has fallen down, the process proceeds to step S22. On the other hand, if it is determined that the person has not fallen down, that means the orientation is something other than having fallen down, so the process proceeds to step S23.

(124) Next, in step S22, the orientation estimation component 12 determines that the care receiver P1 is in a fallen orientation based on the determination result in step S21.

(125) Next, in step S23, it is determined whether or not the care receiver P1 is at the side of the bed in the infrared image G1. Here, if the care receiver P1 is determined to be at the side of the bed, the process proceeds to step S24. On the other hand, if the care receiver P1 has been determined not to be at the side of the bed, the process proceeds to step S25.

(126) Next, in step S24, it is determined from the result of determination by the orientation estimation component 12 in step S23 that there is someone next to the bed.

(127) Next, in step S25, it is determined whether or not the care receiver P1 in the infrared image G1 is on the bed. If it is determined that the care receiver P1 is on the bed, the process proceeds to step S26. On the other hand, if it is determined that the care receiver P1 is not on the bed, the process proceeds to step S29.

(128) Next, in step S26, it is determined whether or not the care receiver P1 in the infrared images G1 is in a seated position on the bed. Here, if it has been determined that the care receiver P1 is in a seated position on the bed, the process proceeds to step S27. On the other hand, if the care receiver P1 has been determined not to be in a seated position on the bed, the process proceeds to step S28.

(129) Next, in step S27, it is determined from the result of determination by the orientation estimation component 12 in step S26 that the care receiver P1 is in a seated position on the bed.

(130) Next, in step S28, it is determined from the result of determination by the orientation estimation component 12 in step S26 that the care receiver P1 is in a recumbent position on the bed.

(131) Next, in step S29, since the care receiver P1 is neither the beside the bed or on the bed, it is determined whether or not the care receiver P1 is within the measurement region in the infrared image G1. Here, if it is determined that the care receiver P1 is within the measurement region, the process proceeds to step S30. On the other hand, if it is determined that the care receiver P1 is outside the measurement region, the process proceeds to step S31.

(132) Next, in step S30, it is determined that the care receiver P1 is moving somewhere within the room 30 other than next to the bed or on the bed.

(133) Next, in step S31, it is checked once again whether the care receiver P1 is on the bed. Here, if it is determined that the care receiver P1 is on the bed, the process proceeds to step S33. On the other hand, if the person is determined not to be on the bed, the process proceeds to step S32.

(134) Next, in step S32, it is determined from the result of determination using the infrared image G1 that the care receiver P1 has not fallen down and is not within the measurement region next to the bed or on the bed, and is therefore outside the measurement region.

(135) Next, in step S33, since it has been determined in step S32 that the care receiver P1 is on the bed, it is determined that the person is in a recumbent position covered by a blanket, for example.

(136) 4. Display Example of Dummy Image for Each Orientation

(137) 4-1. Standing Position

(138) As shown in FIGS. 15A and 15B when the care receiver P1 is in a standing position, a dummy image D1 showing a state in which the care receiver P1 is standing is displayed superimposed over the infrared image G1.

(139) More specifically, as shown in FIG. 15A, in the infrared image G1 acquired from the infrared array sensor 21a, which captures an image of the room 30 from above, the care receiver P1 is imaged from above his head, so a dummy image D1 is used in which the head and shoulder portions are displayed in plan view.

(140) On the other hand, as shown in FIG. 15B, in the infrared image G1 acquired from the infrared array sensor 21b, which captures an image of the room 30 from the side, the care receiver P1 is imaged from his side, so a dummy image D1 (second dummy image) is used in which the entire body of the care receiver P1 is displayed.

(141) A dummy image (second dummy image) D2 showing the bed 31 as a piece of interior furnishing is displayed on the infrared image G1 shown in FIGS. 15A and 15B.

(142) Consequently, where the care receiver P1 is located within the room 30 can be recognized while referring to the position of the bed 31 or other such interior furnishings.

(143) 4-2. Seated Position

(144) As shown in FIGS. 16A and 16B, if the care receiver P1 is in a seated position, a dummy image D1 showing a state in which the care receiver P1 is sitting in a wheelchair or a chair is displayed superimposed over the infrared image G1.

(145) More specifically, as shown in FIG. 16A, in the infrared image G1 acquired from the infrared array sensor 21a that images the room 30 from above, the care receiver P1 is imaged from above his head, so a dummy image D1 is used in which the legs are displayed in plan view in addition to the head and shoulder portions.

(146) On the other hand, as shown in FIG. 16B, in the infrared image G1 acquired from the infrared array sensor 21b that images the room 30 from the side, the care receiver P1 is imaged from above his side, so a dummy image D1 is used in which the care receiver P1 is bent at the waist.

(147) 4-3. Recumbent Position (with Blanket)

(148) As shown in FIGS. 17A and 17B, if the care receiver P1 is in a recumbent position in which he is sleeping on the bed 31, a dummy image D1 showing a state in which the care receiver P1 is lying on the bed 31 the shown is displayed superimposed over the infrared image G1.

(149) More specifically, as shown in FIG. 17A, in the infrared image G1 acquired from the infrared array sensor 21a that images the room 30 from above, the sleeping care receiver P1 is imaged from above, so a dummy image D1 is used in which the entire body of the care receiver P1 is displayed.

(150) If the care receiver P1 sleeping on the bed 31 is covered by a blanket (dummy image D3), as shown in FIG. 17A, the portion that is covered by the blanket in the infrared image G1 is displayed as a portion of lower temperature, and just the head is displayed as a portion of higher temperature.

(151) Meanwhile, as shown in FIG. 17B, in the infrared image G1 acquired from the infrared array sensor 21b that images the room 30 from the side, the care receiver P1 is imaged from the side, so a dummy image D1 is used in which the care receiver P1 is displayed in a recumbent position in which he is sleeping on the bed 31.

(152) Similarly to FIG. 17A, if the care receiver P1 sleeping on the bed 31 is covered with a blanket, as shown in FIG. 17B, in the infrared image G1 the portion covered with the blanket is displayed as a portion of lower temperature, and just the head is displayed as a portion of higher temperature.

(153) 4-4. Recumbent Position (Blanket Turned Down)

(154) As shown in FIGS. 18A and 18B, if the care receiver P1 is in a recumbent position in which he is sleeping on the bed 31 and has the blanket (dummy image D3) turned down, a dummy image D1 showing a state in which the care receiver P1 is sleeping on the bed 31 is displayed superimposed over the infrared image G1.

(155) More specifically, as shown in FIG. 18A, in the infrared image G1 acquired from the infrared array sensor 21a that images the room 30 from above, the sleeping care receiver P1 is imaged from above, so a dummy image D1 is used in which the upper half of the body of the care receiver P1 is displayed.

(156) If the lower half of the body of the care receiver P1 sleeping on the bed 31 is covered by a blanket (dummy image D3), as shown in FIG. 18A, in the infrared image G1 the portion covered by a blanket is displayed as a portion of lower temperature, and just the upper half of the body is displayed as a portion of higher temperature.

(157) On the other hand, as shown in FIG. 18B, in the infrared image G1 acquired from the infrared array sensor 21b that images the room 30 from the side, the care receiver P1 is imaged from the side, so a dummy image D1 is used in which the care receiver P1 is displayed in a recumbent position in which he is sleeping on the bed 31.

(158) Similarly to FIG. 18A, if the lower half of the body of the care receiver P1 sleeping on the bed 31 is covered by a blanket, as shown in FIG. 18B, in the infrared image G1 the portion covered with the blanket is displayed as a portion of lower temperature, and just the upper half of the body is displayed as a portion of higher temperature.

(159) 4-5. Recumbent Position (Fallen)

(160) As shown in FIGS. 19A and 19B, if the care receiver P1 is in a recumbent position in which he has fallen next to the bed 31, a dummy image D1 showing a state in which the care receiver P1 has fallen next to the bed 31 (dummy image D2) is displayed superimposed over the infrared image G1.

(161) More specifically, as shown in FIG. 19A, in the infrared image G1 acquired from the infrared array sensor 21a that images the room 30 from above, the care receiver P1 lying next to the bed 31 is imaged from above, so a dummy image D1 is used in which the entire body of the care receiver P1 is displayed.

(162) In this case, the care receiver P1 is sleeping in a different location from that in the dummy image D2 showing the bed 31, and is assumed to have fallen, so the dummy image D1 is, for example, shown in red, yellow, or some other color that is different from the normal color, or is flashing.

(163) On the other hand, as shown in FIG. 19B, in the infrared image G1 acquired from the infrared array sensor 21b that images the room 30 from the side, the care receiver P1 is imaged from the side, so a dummy image D1 is used in which the care receiver P1 is displayed in a recumbent position in which he is sleeping below the bed 31.

(164) Similarly to FIG. 19A, if the care receiver P1 is sleeping below the bed 31, it is assumed that he has fallen, so the dummy image D1 is, for example, shown in red, yellow, or some other color that is different from the normal color, or is flashing.

(165) This allows a caregiver to check the infrared image G1 shown in FIGS. 19A and 19B and recognize a state in which care is urgently required.

(166) 4-6. Seated Position (Fallen)

(167) As shown in FIGS. 20A and 20B, when the care receiver P1 is in a seated position next to the bed 31, a dummy image D1 showing a state in which the care receiver P1 has fallen and is slumped over next to the bed 31 (dummy image D2) is displayed superimposed over the infrared image G1.

(168) More specifically, as shown in FIG. 20A, in the infrared image G1 acquired from the infrared array sensor 21a that images the room 30 from above, the care receiver P1 who is sitting next to the bed is imaged from above, so a dummy image D1 is used in which the head, shoulders, and lefts of the care receiver P1 are displayed in plan view.

(169) In this case, since the care receiver P1 is assumed to have fallen away from the dummy image D2 showing the bed 31 and is sitting next to it, the dummy image D1 is, for example, shown in red, yellow, or some other color that is different from the normal color, or is flashing.

(170) On the other hand, as shown in FIG. 20B, in the infrared image G1 acquired from the infrared array sensor 21b that images the room 30 from the side, the care receiver P1 is imaged from the side, so a dummy image D1 is used in which the care receiver P1 is displayed in a seated position in which he is sitting and slumped against the bed 31.

(171) Similarly to FIG. 20A, if the order care receiver P1 is sitting next to the bed 31, it is assumed that he has fallen and is slumped over, so the dummy image D1 is, for example, shown in red, yellow, or some other color that is different from the normal color, or is flashing.

(172) This allows a caregiver to check the infrared image G1 shown in FIGS. 20A and 20B and recognize a state in which care is urgently required.

(173) The urgency of a care receiver P1 who is in a seated position next to the bed 31 can be determined by checking a continuously acquired infrared image G1 and looking for any movement.

(174) 4-7. Recumbent Position (with Flickering)

(175) As shown in FIG. 21A, when the care receiver P1 is a recumbent position and there is little flickering in the continuously acquired infrared image, there is a high probability that the care receiver P1 is asleep.

(176) Thus, in this case, the determination component 15 determines that this is not a state in which the care receiver P1 needs care.

(177) As shown in FIG. 22A, when the care receiver P1 is a recumbent position and a flicker region G1a with a large amount of flickering is detected in the continuously acquired infrared image near the upper half of the body of the care receiver P1, there is a high probability that the care receiver P1 is trying to get up.

(178) Furthermore, as shown in FIG. 22B, when the care receiver P1 is a recumbent position and a flicker region G1a with a large amount of flickering is detected in the continuously acquired infrared image near the upper half of the body of the care receiver P1 who is covered with a blanket, there is a high probability that the care receiver P1 is trying to get up.

(179) In this case, the care receiver P1 needs help getting up, so the determination component 15 determines this to be a state in which care is required.

(180) Here, determination using the flickering of pixels in the infrared image G1 is carried out as follows.

(181) That is, in performing determination based on the flickering of a continuously acquired infrared image, the determination component 15 makes its determination according to the fluctuation of the position of the thermal center of gravity in the image.

(182) More specifically, the position of the thermal center of gravity is found, for example, by cutting out a region in which the temperature is at least a specific amount above room temperature from the infrared image G1, and finding the average position in the XY direction of the pixels of the cut-out region. The position of this thermal center of gravity may be determined as a measure of how urgently care is needed, according to whether or not the position has moved (fluctuated) from a specific threshold.

(183) Consequently, when an infrared sensor is used as the image sensor, the characteristics of an infrared image (a thermal image) can be utilized to determine a state in which the care receiver P1 needs care when the change in the position of the thermal center of gravity is smaller than a predetermined threshold, for example.

(184) Conversely, if the change in the position of the thermal center of gravity is greater than a predetermined threshold, it can be determined, for example, that the care receiver is changing clothes, operating a portable terminal, or in some other such state that does not require care.

(185) As a result, the caregiver can look at the image of the care receiver P1 displayed on the display component 22 and accurately determine whether or not care is needed.

(186) 4-8. Recumbent Position (Heat Source Expansion)

(187) As shown in FIG. 21B, when the care receiver P1 is a recumbent position and a heat source expansion region G1b is detected in which a heat source near the head is expanding in the continuously acquired infrared image, there is a high probability that the care receiver P1 has vomited while sleeping.

(188) Therefore, in this case, the determination component 15 determines that this is a state in which the care receiver P1 needs care urgently.

(189) The controller 13 takes into account the result of estimating the orientation by the orientation estimation component 12 and the result of determining the urgency of care by the determination component 15, and controls the display so that the dummy image D1 is changed to another color, is flashed, etc., as shown in FIG. 21B.

(190) This allows the caregiver to check the screen displayed on the display device 22 and easily recognize that the care receiver P1 is a recumbent position and urgently needs care.

(191) 4-9. Seated Position (with Flickering)

(192) As shown in FIG. 23, when the care receiver P1 is in a seated position in a dummy image D4 of a wheelchair (second dummy image), if a flicker region G1a is detected in which there is a large amount of flickering near the upper half of the body of the care receiver P1 on the dummy image D4 showing the wheelchair 32 in the continuously acquired infrared image, there is a high probability that care receiver P1 is changing clothes.

(193) Thus, in this case, the determination component 15 determines that this is not a state in which the care receiver P1 needs care.

(194) Conversely, when the care receiver P1 is in a seated position in the wheelchair 32 and there is almost no flickering for at least a certain length of time near the upper half of the body of the care receiver P1 in the continuously acquired infrared image, there is the risk that the care receiver P1 is slumped over in the wheelchair.

(195) Thus, in this case, the determination component 15 determines that this is a state in which the care receiver P1 needs care.

(196) In this case, the controller 13 takes into account the result of estimating the orientation by the orientation estimation component 12 and the result of determining the urgency of care by the determination component 15, and controls the display so that the dummy image D1 is changed to another color, is flashed, etc.

(197) Consequently, the caregiver can recognize that the care receiver P1 has been stationary in the wheelchair for a long time, and can rush to the room of the care receiver P1.

Other Embodiments

(198) An embodiment of the present invention was described above, but the present invention is not limited to or by the above embodiment, and various modifications are possible without departing from the gist of the invention.

(199) (A)

(200) In the above embodiment, an example was given in which the present invention was implemented as the display control device 10 for conducting the above-mentioned display control method, and as the display control system 20 comprising said device, but the present invention is not limited thereto.

(201) For instance, the present invention may be implemented as the above-mentioned display control method.

(202) Alternatively, the present invention may be implemented as a display control program for causing a computer to execute the above-mentioned display control method. Furthermore, the present invention may be implemented as a non-transitory computer-readable storage medium for storing this display control program.

(203) Regardless of how the invention is implemented, it allows the same effect as described above to be obtained.

(204) (B)

(205) In the above embodiment, as shown in FIG. 1, the display control device 10 comprising the storage component 14 for storing dummy images D1 corresponding to various orientations of the care receiver P1 was described as an example, but the present invention is not limited thereto.

(206) For example, as shown in FIG. 24, a cloud space 114 provided outside the display control device 110 and the display control system 120, or the like, may be used as a storage component for storing the dummy images.

(207) That is, the display control device of the present invention may be configured not to have an internal storage component, so long as the configuration is one in which dummy images are read from an external storage component (cloud space, server, etc.).

(208) (C)

(209) In the above embodiment, an example was described in which the dummy image D1 was displayed superimposed over the infrared image G1 on the display device 22, but the present invention is not limited to this.

(210) For instance, as shown in FIGS. 25A and 25B, text information such as “abnormality has occurred with Mr. X in room No. Y” may be displayed together with the image information.

(211) In this case, the urgency of the care can be displayed in a way that is more easily understandable to the caregiver.

(212) This text information may consist of a plurality of sets of text information corresponding to the determination result by the determination component that determines the how urgently the care receiver needs care, with this information stored ahead of time in the storage component 14 or the like.

(213) (D)

(214) In the above embodiment, as shown in FIG. 2, an example was described in which the infrared array sensors 21a and 21b were provided on the ceiling and the wall as the first and second image sensors, but the present invention is not limited to this.

(215) For example, an image sensor may be provided on just the ceiling, or just on the wall, an image sensor for imaging the human body from an angle may be used.

(216) However, in order to determine the body orientation correctly, it is preferable to adopt a configuration as in the above embodiment, in which a image sensors are provided on both the ceiling and the wall.

(217) (E)

(218) In the above embodiment, an example was described in which the infrared array sensors 21a and 21b that sensed the temperature distribution were used as image sensors for transmitting images to the image acquisition component 11, but the present invention is not limited to this.

(219) For example, it is also possible to use other image sensors, such as a surveillance camera or a distance sensor, as the image sensors.

(220) (F)

(221) In the above embodiment, an example was described in which the display controller 10 and the display control system 20 of the present invention were applied to monitoring in a care facility or the like where a plurality of care receivers P1 are cared for, but the present invention is not limited to this.

(222) For example, in addition to nursing facilities, the present invention may also be applied to a facility such as a hospital or the like in which disabled people or elderly people live.

(223) Alternatively, the present invention may be applied to the monitoring of elderly people who are living alone.

(224) In this case, mobile terminals own by the children who live apart from the elderly people can be used as a display device, allowing the monitoring to be carried out from a remote location.

INDUSTRIAL APPLICABILITY

(225) The display control device of the present invention has the effect of allowing a situation in which a care receiver is in need of care, for example, to be properly recognized by a caregiver, and therefore can be widely applied to various kinds of device used for monitoring seniors, children, pets, and so forth.

REFERENCE SIGNS LIST

(226) 10 display control device 11 image acquisition component 12 orientation estimation component 13 controller 14 storage component 15 determination component 16 count estimation component 20 display control system 21a infrared array sensor (first image sensor) (ceiling) 21b infrared array sensor (second image sensor) (side) 22 display device 22a host computer (display device) 22b mobile terminal (display device) 30 room 30a ceiling 30b wall 31 bed 32 wheelchair 33 shelf 110 display control device 114 cloud space 120 display control system G1 infrared image (image) G1a flickering region G1b heat source expansion region D1 dummy image (caregiver) (first dummy image) D2 dummy image (bed) (second dummy image) D3 dummy image (blanket) (second dummy image) D4 dummy image (wheelchair) (second dummy image) P1 care receiver (person)