A SYSTEM AND METHOD FOR PROVIDING ASSISTANCE DURING BOTTLE-FEEDING

Abstract

A system provides assistance during bottle-feeding. Video images of a subject bottle-feeding an infant are captured and displayed. Using image analysis, a reorientation of the bottle and/or infant is determined that is required in order to reach a desired bottle orientation and/or infant orientation. Reorientation instructions are provided in combination with the video images to assist the subject in reorienting the bottle and/or the infant to achieve the desired bottle orientation.

Claims

1. An image processor for use in a system for providing assistance during bottle-feeding, comprising: an input for receiving video images of a subject bottle-feeding an infant; an output for providing video images for display, wherein the image processor is adapted to: identify the infant within the video images; determine the orientation of the infant; identify the bottle within the video images; determine the orientation of the bottle; determine a required reorientation of the bottle and/or infant in order to reach a desired bottle orientation; and control a display device to output reorientation instructions overlaid over the video images to assist the subject in reorienting the bottle and/or the infant to achieve the desired bottle and/or infant orientation.

2. The image processor of claim 1, further adapted to identify the orientation of the hand of the subject in the images, and determine the orientation of the long axis of the bottle to be perpendicular to the orientation of the palm of the hand.

3. The image processor of claim 1, wherein the desired bottle orientation corresponds to a tilt angle relative to a body axis of the infant within a predefined range.

4. The image processor of claim 1, further adapted to identify the position of the head of the infant and the position of the feeding hand of the subject in the images.

5. The image processor of claim 1, further adapted to identify the pose of the subject in the images.

6. The image processor of claim 1, further adapted to output reorientation instructions as an indication of the desired orientation on the display output.

7. The image processor of claim 1, further adapted to control an audio output device to output reorientation instructions using audio signals.

8. A system for providing assistance during bottle-feeding, comprising: a camera for capturing video images of a subject bottle-feeding an infant; a display for displaying the video images; and the image processor of claim 1.

9. A method for providing assistance during bottle-feeding, comprising: identifying an infant within video images captured of a subject bottle-feeding an infant; determining the orientation of the infant; identifying the bottle within the video images; determining the orientation of the bottle; determining a required reorientation of the bottle and/or infant in order to reach a desired bottle orientation; and displaying the video images in combination with output reorientation instructions to assist the subject in reorienting the bottle and/or infant to the desired bottle and/or infant orientation.

10. The method of claim 9, comprising identifying the orientation of the hand of the subject in the images, and determining the orientation of the long axis of the bottle to be perpendicular to the orientation of the palm of the hand.

11. The method of claim 9, wherein the desired bottle orientation corresponds to a tilt angle relative to a body axis of the infant within a predefined range.

12. The method of claim 9, comprising identifying the pose of the subject in the images.

13. The method of claim 9, comprising providing output reorientation instructions as an indication of the desired orientation on the display output.

14. A computer program product comprising instructions for implementing the method of claim 9 when said program is run on a processor.

15. A computer readable medium comprising the computer program product of claim 14.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0047] For a better understanding of the invention, and to show more clearly how it may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0048] FIG. 1 shows a sequence of image processing steps employed in the system of the invention;

[0049] FIG. 2 shows the hardware part of a system; and

[0050] FIG. 3 shows a method for providing assistance during bottle-feeding.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0051] The invention will be described with reference to the Figures.

[0052] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the apparatus, systems and methods, are intended for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, systems and methods of the present invention will become better understood from the following description, appended claims, and accompanying drawings. It should be understood that the Figures are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the Figures to indicate the same or similar parts.

[0053] The invention provides a system for providing assistance during bottle-feeding. Video images of a subject bottle-feeding an infant are captured and displayed. Using image analysis, a reorientation of the bottle and/or infant is determined that is required in order to reach a desired bottle orientation and/or infant orientation. Reorientation instructions are provided in combination with the video images to assist the subject in reorienting the bottle and/or the infant to achieve the desired bottle orientation.

[0054] FIG. 1 shows a sequence of image processing steps employed in the system of the invention.

[0055] Video images 10 are captured of the subject 12 (mother or caregiver) bottle-feeding the infant, e.g. baby, 13. The images may be captured by a mobile device such as a mobile phone. The process may start with the caregiver holding the infant in front of the camera of the mobile device to initiate the recognition process, after which the subject may adopt the feeding pose. The image process may then involve tracking the position of the infant and the subject, after the initial identification. However, the image processing may instead identify all required features without needing any initiation steps. The mobile device then starts taking live video data.

[0056] Image preprocessing steps are then carried out, for example based on a grayscale conversion of the images.

[0057] In step 14, frame extraction of the images is carried out, and in step 16 3D pose estimation is carried out to identify at least the pose of the infant. In one example, the pose of the subject is also determined. In particular, the orientation at least of the subject’s hand and the pose of the infant are determined, with respect to the bottle.

[0058] The pose information of the subject also assists in determining the orientation (in three dimensions) of the infant, since the infant orientation can be partly deduced from the pose of the subject.

[0059] However, if the image processing is able to locate and analyze the bottle and infant with sufficient accuracy, the pose information of the subject may not be needed, since it is primarily used to deduce the orientation of the bottle, particularly as much of the bottle may be obscured from view.

[0060] The pose estimation for example involves running the frames through a neural network to extract 2D key points from each frame. From this, key locations such as the face and body of the infant and the actual bottle holding hand of the subject are identified.

[0061] The subject is for example encouraged to hold the bottle in a particular way so that the system can make an assumption about the relationship between the orientation of the bottle-feeding hand of the subject and the long axis of the bottle. For example, the long axis of the bottle may be assumed to be perpendicular to the palm of the hand, i.e. perpendicular to the general plane of the hand.

[0062] The correct bottle orientation relative to the hand (so that the assumptions made by the system are valid) may for example be displayed to the subject while they hold the bottle to the camera, so the user can correctly hold the bottle.

[0063] The key points and poses of the subject and infant can be converted into a map, as represented by image 18. This may include the head, hands, arms, neck etc. of the subject, the body orientation of the infant, and the viewed or assumed bottle orientation. Image 18 schematically shows the long axis of the baby and the line between the heads of the mother and the infant, at right angles.

[0064] Image 20 shows an enlarged are of the region of interest, including the head of the infant, the hand of the subject and hence the bottle.

[0065] Suitable image processing algorithms for identifying and tracking features of interest in video data are well known and outside the scope of this application.

[0066] In step 22 the optimum bottle angle and/or orientation of the subject is determined.

[0067] In a most simple case, the pose of the subject is a suitable feeding pose, so the optimum bottle angle is determined for that particular pose of the subject.

[0068] In step 24, a visualization of the optimum bottle angle orientation is generated. This visualization is shown as image 26, and it represents reorientation instructions for the positioning of the bottle.

[0069] In a more advanced case, the system may first assess if the subject pose (and hence the infant pose) is suitable for bottle feeding. If not, reorientation instructions or training information may be presented to the subject to improve the way they hold the infant during bottle-feeding.

[0070] The instructions for example take the form of arrows and dotted lines to indicate how much the bottle should be shifted from its current location. Similar instructions may be provided for adapting the pose of the subject or the orientation of the infant, or training videos may be provided for this purpose.

[0071] After post processing the reorientation instructions in step 27 so that they are compatible for combination with the video images, the reorientation instructions are combined with the original video images, in real time, to overlay the instructions over the real image. This is shown as image 28.

[0072] The reorientation instructions may be based on 3D visualization using graphics which show, e.g. using a dotted line, exactly how the bottle should be held. If the subject is unable to constantly look at the screen, audio cues may also be provided that reinforce that the bottle should be moved up, down or the angle adapted. The visualization for example uses augmented reality to provide an optimum visualization.

[0073] In professional setting like a hospital, the system may also alert medical staff in a hospital that are remotely monitoring, if the feeding position (of the infant and/or bottle) is not optimized within a specific time frame such that action can be taken timely to support and educate the parents.

[0074] FIG. 2 shows the hardware components of a system for providing assistance during bottle-feeding, for implementing the process explained above. A camera system 30, which may comprise a single camera, or multiple cameras, or a 3D camera, is for capturing video images of a subject bottle-feeding an infant. The camera system may be a fixed camera or it may be the camera of a mobile device such as a tablet or phone, which the user mounts facing them.

[0075] A display 32 is for displaying the video images. The display may be the display of the same mobile device or it may be a separate display to which the images are sent, e.g. by short range communication such as Bluetooth. An audio output system is also provided, for example comprising the speaker of the same mobile device.

[0076] An image processor 34 performs the processing described above. The image processor may simply be the processor of the mobile device used as the camera and optionally also the display. Thus, the invention may be implemented as an App to be loaded on a tablet or phone. However, the processing may instead be performed remotely, for example with a local device using short range communication (e.g. a laptop) or even with communication with a remote device over the internet.

[0077] The key functions of the image processing are to: [0078] identify the infant within the video images; [0079] determine the orientation of the infant; [0080] identify the bottle within the video images (either by direct recognition or based on assumptions from the image information about the subject); [0081] determine the orientation of the bottle.

[0082] Based on the image processing, a required reorientation of the bottle and/or infant, or even a change in posture of the subject, can be determined in order to reach desired feeding conditions.

[0083] The desired bottle angle may for example simply be an angle to the horizontal such as 45 degrees. This angle may however depend on the bottle type, and the system may be tuned to the particular type of bottle being used. The infant needs to be in an orientation which enables drinking from the bottle when at the desired bottle angle.

[0084] Thus. the bottle angle and/or the infant orientation may need to be adjusted. The desired bottle angle degree gives a downward slope relative to the horizontal to maintain a full teat, and also a suitable angle relative to the body orientation of the infant is also desired so that the baby can comfortable latch on to the bottle teat. The infant should be at a suitable inclination to assist with burping, for example to prevent colic.

[0085] The display device is controlled to output reorientation instructions in combination with the video images to assist the subject in reorienting the bottle and/or the infant to achieve the desired bottle orientation. This system keeps the subject engaged by providing positive reinforcement and constant audio and/or visual feedback.

[0086] FIG. 3 shows a method for providing assistance during bottle-feeding, comprising: [0087] in step 40, capturing video images of a subject bottle-feeding an infant; [0088] in step 42, identifying the infant within the video images; [0089] in step 44, determining the orientation of the infant; [0090] in step 46, identifying the bottle within the video images; and [0091] in step 48 determining the orientation of the bottle.

[0092] A required reorientation of the bottle is determined in step 50 in order to reach a desired bottle orientation.

[0093] The video images in combination with output reorientation instructions are displayed in step 52 to assist the subject in reorienting the bottle to the desired bottle orientation.

[0094] The method may also start at step 42, for instance when captured video is already available.

[0095] The system can additionally be used to track infant feeding patterns and habits. The invention can also be used for research on infant feeding behavior and patterns in hospital as well as home application.

[0096] The system may be based entirely on analysis of natural images. However, it is also possible to make use of markers, which can be recognized more easily, such as a marker on the body or face of the infant, and/or a marker on the bottle-holding hand of the subject, and/or markers on the bottle.

[0097] The image recognition system may obtain all required pose information automatically, without any special action of the subject. However, the subject may instead provide input, for example with an initial pose in front of the camera as explained above.

[0098] The desired bottle angle is for example perpendicular to the body axis of the infant, which is determined as part of the 3D pose information. As a minimum, the system makes use of the infant body orientation, the position of the head of the infant, and the bottle orientation (and position) so that the angle between them can be determined. The angle to the horizontal is also known.

[0099] The desired orientation may be obtained as a 3D vector. This may be represented in simplified 2D form, so that for a given pose of the subject and infant, all that is required is alignment of the bottle with a 2D vector on the display, with real time feedback to show when alignment is achieved. 3D directional information may be displayed using more complex representations of the required reorientation.

[0100] A 3D vector of the desired orientation may be transferred to 2D space on the screen using augmented reality. Alternatively a representation in 3D on the screen maybe selected by the user through a viewing option menu in the user interface.

[0101] A more precise estimation of the desired orientation could be created with extra sensors (such as 3D cameras or multiple cameras) giving more detailed bottle position and infant orientation information. Reorientation instructions in the 3D domain could also be given using audio feedback to the user.

[0102] The skilled person would be readily capable of developing a processor for carrying out the described method. Thus, each step of the flow chart of FIG. 3 may represent a different action performed by a processor, and may be performed by a respective module of the processing processor.

[0103] The processor can be implemented in numerous ways, with software and/or hardware, to perform the various functions required. The processor typically employs one or more microprocessors that may be programmed using software (e.g., microcode) to perform the required functions. The processor may be implemented as a combination of dedicated hardware to perform some functions and one or more programmed microprocessors and associated circuitry to perform other functions.

[0104] Examples of circuitry that may be employed in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0105] In various implementations, the processor may be associated with one or more storage media such as volatile and non-volatile computer memory such as RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that, when executed on one or more processors and/or controllers, perform the required functions. Various storage media may be fixed within a processor or controller or may be transportable, such that the one or more programs stored thereon can be loaded into a processor.

[0106] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality.

[0107] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0108] A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0109] If the term “adapted to” is used in the claims or description, it is noted the term “adapted to” is intended to be equivalent to the term “configured to”.

[0110] Any reference signs in the claims should not be construed as limiting the scope.