Surrounding intelligent motion sensor with adaptive recognition

Abstract

A wearable proximity warning device is provided that uses a novel method of processing images from a high frame rate digital camera to detect human threats from behind and determine if there are any approaching threats by using novel pixel counting and threat detection analysis algorithms. The device is worn on the back of the body either by use of a belt clip or with chest straps. The user may use select from a variety of warning options from the device including audible warning tones, device vibration or smartphone SMS/MMS text messaging. Stored video is saved by the device and may be periodically uploaded to secure cloud storage. The device contains a rechargeable battery that may be recharged using a USB port. The device uses adaptive human recognition that switches between facial recognition detection mode and body recognition detection mode depending on distance of the threat from the user.

Claims

1. A wearable device for detection and warning of approaching threats, comprising: a motherboard that uses a threat detection algorithm to detect an approaching human threat; an alert device electronically connected to and controlled by the motherboard; a digital camera electronically connected to the motherboard; the digital camera configured and arranged to detect a distance of the threat from the wearable device; the threat detection algorithm being configured and arranged to detect an entire body of the approaching human threat and a face of the approaching human threat; and the threat detection algorithm being configured and arranged to detect the face of the approaching human threat using facial recognition when the distance of the threat is less than a predetermined threshold amount and being configured and arranged to detect the full body of the approaching human threat using body recognition when the distance of the threat is more the predetermined threshold amount, wherein the distance of the threat is determined using a real time continuous measurement of a pixel count of the approaching human in images captured by the digital camera, and wherein the threat detection algorithm is configured and arranged to determine if the distance of the threat is increasing as a function of an increasing pixel count of the approaching human in images captured by the digital camera and to determine if the distance of the threat is decreasing as a function of a decreasing pixel count of the approaching human in images captured by the digital camera.

2. The wearable device of claim 1, further comprising: a housing including a front portion and a rear portion that mate together; an audio board; a vibration device; a smartphone communications board; a camera control board; and a speaker.

3. The wearable device of claim 2, wherein the threat detection algorithm identifies humans from captured images from the digital camera; a human to pixel count conversion algorithm converts a total number of pixels from humans identified from the human recognition algorithm and represents identified humans as a number of pixels.

4. The wearable device of claim 2, wherein the audio board is configured and arranged to provide an audio warning tone to the speaker when activated by the motherboard.

5. The wearable device of claim 2, wherein the vibration device is configured and arranged to vibrate when activated by the motherboard.

6. The wearable device of claim 2, wherein the smartphone communications board is configured and arranged to send approaching threat warning messages to at least one linked smartphone.

7. A method for detecting and warning of approaching threats, comprising the steps of: providing a device with a motherboard, an alert device, a digital camera electronically connected to each other; the device being wearable by a user; detecting an approaching human threat using a threat detection algorithm; the threat detection algorithm being configured and arranged to detect an entire body of the approaching human threat and a face of the approaching human threat; detecting a distance of the human threat to the device; setting a threshold distance between the device and the human threat; detecting the face of the approaching human threat using the threat detection algorithm using facial recognition when the detected distance between the device and the human threat is less than the threshold distance; detecting the body of the approaching human threat using the threat detection algorithm using body recognition when the detected distance between the device and the human threat is greater than the threshold distance, wherein the detected distance of the threat is determined using a real time continuous measurement of a pixel count of the approaching human in images captured by the digital camera, and wherein the threat detection algorithm is configured and arranged to determine if the distance of the threat is increasing as a function of an increasing pixel count of the approaching human in images captured by the digital camera and to determine if the distance of the threat is decreasing as a function of a decreasing pixel count of the approaching human in images captured by the digital camera.

8. The method of claim 7, further comprising the step of: invoking an alert upon detection of the human threat by either facial recognition or body recognition.

9. The method of claim 8, wherein the step of detecting the approaching human threat using the threat detection algorithm identifies humans from captured images from the digital camera.

10. The method of claim 8, further comprising the steps of: converting a total of number of pixels from humans identified from the human recognition algorithm by a human to pixel count conversion algorithm; converting a total number of pixels; representing identified humans as a number of pixels.

11. The method of 8, wherein the alert is an audio warning tone.

12. The method of claim 8, wherein the alert is a vibration.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is a front view of the SIMS device.

(2) FIG. 2 is a rear view of the SIMS device.

(3) FIG. 3 shows a female user wearing the SIMS device using the chest belt option.

(4) FIG. 4 shows a male user wearing the SIMS device using the belt clip option.

(5) FIG. 5 is a component diagram of the SIMS device.

(6) FIG. 6 is a functional block diagram showing the functional relationships of the various components of the SIMS device.

(7) FIG. 7 is a flowchart showing how an approaching threat determination is made in the SIMS device.

(8) FIG. 8 is a flowchart showing the various warning methods that the SIMS device will use if an approaching threat from behind has been detected.

(9) FIGS. 9(a)-9(c) show a sequence of time elapsed images of running joggers to demonstrate the capability of SIMS to detect humans using a human recognition detection algorithm.

(10) FIGS. 10(a)-10(c) show the capability of a human to pixel count conversion algorithm to count pixels of the detected humans from previous FIGS. 9(a)-9(c).

(11) FIG. 11 is a table showing the results of the threat detection algorithm to detect an approaching threat by an increasing pixel count of detected human T1.

(12) FIG. 12 shows the adaptive human recognition feature of the present invention.

DESCRIPTION OF THE INVENTION

(13) Referring now to the drawings and in particular FIG. 1, the present invention is designated by reference numeral 1000 and shall be referred to hereafter as the SIMS device. The SIMS device has a front housing 10 that is preferably made from a strong plastic such as ABS or nylon. A digital camera lens 70 is located in the center of the housing and has the typical features of modern digital cameras including a focusing capability and an adjustable aperture to adjust the brightness of the image. A USB port 97 is located at the bottom end of the SIMS device and provides a means for the recharging of an internal rechargeable battery 99 (not shown).

(14) Referring again to FIG. 1, there are two belt or strap attachment points 55, one on each side of front housing 10. These attachment points allow for a belt or strap to be secured to them as one method of wearing the SIMS device on the body. A power on/off button 40 is shown located at the upper left hand corner of the SIMS device and provides a means to turn SIMS on or off. A Bluetooth activation button 20 is located below the power button 40 and when this button is pressed it will enable the SIMS device to use Bluetooth cellular phone warning options that shall be explained later in the specification. The SIMS device can provide either an audible tone warning or a vibration warning depending on which position the warning mode selector switch 30 is in. For example, in FIG. 1, the mode selector switch 30 is shown in the audible tone warning position. When the audible tone warning mode is selected, an internal speaker 85 (not shown) will produce sound that will be emanated from speaker holes 87 shown in the top center section of the SIMS device.

(15) Referring next to FIG. 2, the back side of the SIMS device 1000 is shown. A belt clip 50 is shown centered on the rear housing 15 of the SIMS device. The belt clip 50 provides an additional means of wearing the SIMS device on the body.

(16) The two methods of wearing the SIMS device 1000 on the body are shown in FIGS. 3 & 4. FIG. 3 shows a woman wearing the SIMS device using a chest strap that wraps around her chest and is secured to the SIMS device using the body strap attachment points 55 shown previously in FIGS. 1 & 2. While the user wears the SIMS device and when it is powered on, it will provide continuous monitoring for any threats approaching the user from behind. The SIMS device 1000 may also be worn on the body by using the belt clip 50 located on the backside of the SIMS device to secure to a belt as demonstrated by the man wearing the SIMS device as shown in FIG. 4. It is recommended to only wear the SIMS device at waist height or higher in order to maximize the field of view that the camera will have when in use.

(17) Referring next to FIGS. 5 & 6, the functional relationships of the various components that make the SIMS device 1000 shall next be explained. There are 16 essential components that comprise the SIMS device. These components are all shown in FIGS. 5 & 6 and are numbered for reference as follows:

(18) Motherboard96

(19) Camera Control Board65

(20) Camera Lens & Variable Aperture70

(21) Image Sensor72

(22) Image Processing Board75

(23) Power On Button40

(24) Rechargeable Battery99

(25) Battery Recharging Board98

(26) USB Port97

(27) Audio Board95

(28) Speaker85

(29) Vibration Device90

(30) Smartphone Communications Board80

(31) Antenna81

(32) GPS Receiver82

(33) Audio/Vibrate Warning Mode Selector Switch30

(34) Bluetooth Activation Button20

(35) The motherboard 96 is the central control board that manages all of the operations of the SIMS device. The motherboard receives input from the audio/vibrate mode switch 30 and based on the input received will either warn the user of an approaching threat by either activating the audio board 95 or the vibration device 90. The motherboard is powered by a rechargeable battery 99 and also regulates the DC power inputs to all of the separate modules under its control. When the power on button 40 is pressed it allows DC power to flow from the rechargeable battery to the motherboard. When the Bluetooth activation button 20 is pressed it sends a signal to the motherboard to activate the smartphone communications board 80. Once the SIMS device is turned on, the motherboard activates the camera control board 65. A GPS receiver 82 continuously sends the device's location (latitude and longitude coordinates) to the motherboard while the SIMS device is operating. This GPS location data can later be used as part of the text of an SMS text detected threat warning message or included as part of an MMS text message that includes a picture showing the detected threat. In the case of the MMS text message, the GPS location data could be embedded as part of the picture sent so linked smartphone users can access the location of the SIMS user for assistance.

(36) The camera control board 65 controls the operation of the auto focus camera with variable aperture 70. Based on the lighting conditions during use, the camera control board will regulate the aperture size to allow more or less light into the camera in order to obtain a sufficient contrast to process the images. Images are taken at a specified frame rate (images per second) and are focused onto the image sensor 72 which is preferably a charge-coupled device (CCD). The image sensor is composed of thousands or pixels (picture elements). The light from each picture image captured is converted to a digital signal representation by the image sensor and sent to the image processing board 75.

(37) The image processing board 75 is the component of the SIMS device that performs the highly complex tasks of threat detection and monitoring. First, a human recognition detection algorithm is used to identify humans from the images. Once a human is identified, the human recognition detection (HRD) algorithm 500 will continuously track the identified human until it leaves the field of view of the camera. While humans are being tracked by HRD 500, a separate human to pixel count (HPC) image conversion algorithm 600 is used to count the total number of pixels that each detected human represents in each image processed. The pixel counts vary over time based on how far away the detected humans are from the SIMS device. Humans approaching the SIMS device will have increasing pixel counts over time and humans that move further away from the SIMS device will have decreasing pixel counts over time.

(38) The human pixel counts from the image processing board 75 are continuously tabulated and sent to the motherboard 96. These human pixel counts are then supplied as inputs to a threat determination (TD) algorithm 700 that is embedded firmware and is used by the motherboard 96. This novel method of detection of an approaching threat shall be explained in detail later in the specification.

(39) The smartphone communications board 80 provides the capability of sending a warning message of a detected approaching threat to smartphones that have the SIMS device app installed. The smartphone user simply enables the SIMS device app by going into the smartphone settings and enabling the app. Once enabled, the smartphone app will use Bluetooth to connect the SIMS device to the user's smartphone.

(40) The smartphone communications board 80 performs three independent functions. The first function is to send a Multimedia Messaging Service (MMS) picture and text message 200 showing the first image frame that is considered as a detected approaching threat by SIMS plus a short warning message. An example of a MMS text message 200 would be a text message such as WARNING: SIMS DEVICE HAS DETECTED APPROACHING THREAT followed by a picture showing the actual detected approaching threats. The second function is to send a Short Message Service (SMS) text message 300 which is only a text warning message without any pictures. A final function is the capability of sending stored video images from the memory of the motherboard 96 to the cloud. The SIMS user would enable these three functions as options that would displayed in the SIMS device app. Hardware to support the smartphone communications board includes an RF antenna 81 to transmit the data wirelessly to the selected SIMS enabled smartphones and also optionally to the cloud for secure video storage.

(41) Another option in the SIMS device app is to allow the user to select a specific audio warning tone to use when the audio warning mode is enabled. The selected warning tone is then stored as an audio file in the motherboard and sent to the audio board 95 whenever a threat is detected. The audio board then sends the tone signal to the speaker 85 to play the sound. Yet another option in the SIMS device app is to allow the user to select the duration time and intensity of the vibration warning to use when the vibrate mode is enabled. The selected vibration time and intensity is then stored as a file in the motherboard's memory and is used to produce the desired vibration warning via the vibration device 90.

(42) The USB port 97 is used for the recharging of the SIMS device whenever the rechargeable battery 99 level is low. The USB port interfaces with a battery recharging board 98 which is essentially a step down transformer that converts the DC voltage from the USB power to a lower voltage to charge the battery. Once the battery 99 has been fully charged, the battery recharging board 98 will switch off automatically to prevent overcharging.

(43) Referring next to the flowchart shown in FIG. 7, the novel method of detection of an approaching threat using the SIMS device 1000 shall be explained in further detail. The SIMS device uses a novel method of threat detection that is far superior to prior art methods due to its ability to first recognize humans from digital camera images. First a human recognition detection (HRD) algorithm 500 scans the camera image file and identifies any humans. The details of how the HRD algorithm works are beyond the scope of this specification but would be known to one skilled in the art of artificial intelligence software code development. Next, a second algorithm defines a boundary around each detected human in the image and calculates the number of pixels inside each detected human boundary. This second algorithm is defined in this specification as a human to pixel count conversion (HPC) algorithm 600. The HPC algorithm 600 then sends the pixel count outputs of each detected human as inputs to a third threat detection (TD) algorithm 700 that is part of the embedded firmware in the motherboard 96. This TD algorithm 700 is continuously fed pixel counts for each detected human as each image is processed by the image processing board 75.

(44) Referring again to FIG. 7, the TD algorithm 700 is always run whenever humans are detected by the SIMS device using the HRD algorithm 500. Pixel counts at a first time (T1) are then compared to pixel counts at a second later time (T2) and stored frame by frame in the motherboard's memory. The elapsed time (T2T1) between image frames is determined by the frame rate of the camera 70. The SIMS device would preferably use a high frame rate camera (for example 1000 frames per second or 1000 fps) in order to achieve a fast detection time of an approaching threat. The TD algorithm then compares the pixel counts for each detected human over a short time interval (say 1 to 3 seconds for an outdoor exercise scenario) and determines if the pixel counts are increasing or decreasing. An increasing pixel count for any detected human is considered as a detected threat. If the pixel counts are all decreasing then the TD algorithm continues to monitor the next sequence of pixel counts received from the image processing board and the process continues this way in a continuous monitoring mode until the SIMS device is turned off.

(45) Referring next to FIG. 8, once a threat is detected, the SIMS device will take immediate action to warn the user. There are four possible methods of warning that can be used by SIMS. The first method is audio alert which plays a user selected audio warning tone from the speaker 85 driven by an audio board 95. The second method is vibration alert which vibrates the SIMS device using an internal vibration device 90 based on a user selected vibration file stored in the motherboard's memory. Enabling the Bluetooth mode on the SIMS device allows two additional alert methods that send either a Short Message Service (SMS) standard text only warning text message 300 or a Multimedia Messaging Service (MMS) combined picture and text message that displays the same warning text message as SMS but adds a picture taken from the digital camera at the moment of first threat detection.

(46) Referring finally to FIGS. 9 thru 11, an example scenario using the SIMS device to detect and count the pixel sizes of potential threats is shown. FIGS. 9(a), 9(b) and 9(c) shows three separate images taken at three second intervals of three joggers that are running behind a user wearing an activated SIMS device. Initially, all three joggers are running together (FIG. 9(a)) but by the final image (FIG. 9(c)) they have all been separated from each other. The HRD algorithm 500 has detected three humans in the images and for reference these detected humans are referred to as T1, T2 and T3. Next, FIGS. 10(a), 10(b) and 10(c) show the same time elapsed images that show the detected human image areas of T1, T2 and T3 that will be converted to pixel counts. Finally, the results of the HPC algorithm 600 are shown in FIG. 11. Note that the pixel counts only show one detected human (T1) to be an approaching threat as only the pixel counts of the detected human T1 were shown to be increasing over time. Based on this result the SIMS device would have alerted the user of an approaching threat.

(47) Turning now to FIG. 12, an alternative embodiment of the present invention is shown that employs an adaptive human recognition detection feature. In accordance with the present invention, as discussed above, the SIMS detection algorithm uses human recognition (e.g. body) to detect a threat coming toward the user. In accordance with the alternative embodiment shown in FIG. 12, the mode of detection changes depending on how far away the threat is away from the user. Once the threat gets within a certain distance from the user, such as 60 feet, the mode of detection changes from human detection (e.g. body) to facial recognition to pick up the threat.

(48) More specifically, facial recognition software and associated imaging processing is added to engage in the near field located at, for example, 60 ft or less. As the threat approaches the user, the device switches to a facial recognition mode to find faces in near field. This is highly desirable because within a near field distance, such as 60 ft, the body is no longer present or in the full view of the device yet threat detection is still desired. Thus, the device scans in the far field using the algorithm as above. However, as the threat approaches into the near field, where the body/human detection is not available or present, it locates the face using facial recognition detection. FIG. 12 shows the change in detection mode depending on the distance of the threat from the device and user wearing it. For example, within near field 60 feet, facial recognition is employed and from far field 60-500 feet (or more), human (body) recognition is employed to detect the threat.

(49) To determine which mode is employed, the distance of the threat is determined in real time by the continuous measurement of the pixel count captured by the digital camera. Humans approaching the SIMS device will have increasing pixel counts over time and humans that move further away from the SIMS device will have decreasing pixel counts over time. As discussed in detail above, since the pixel counts vary based on how far away the detected humans are from the SIMS device, the distance of the human threat to the user and the device they are wearing can be determined by the algorithm. This distance data is then used to set a predetermined threshold distance at which the mode of detection changes between from facial recognition and body recognition.

(50) In accordance with the present invention, the mode of detection between facial near field detection and body far field is automatic and adaptive. In other words, the detection mode changes from human (body) recognition to facial recognition as the threat gets closer. It is also possible that the detection mode changes from facial recognition to human (body) recognition as the threat gets further away, such as where the threat stops following the user.

(51) The SIMS device 1000 provides a robust threat detection capability using a series of two image processing algorithms and a threat detection algorithm to provide a reliable and fast detection time as compared with prior art examples cited. Although the example shown was for a daytime scenario, other embodiments of the SIMS device would allow the SIMS device to be used for nighttime military patrol applications. In such nighttime applications for a military version of the SIMS device, the main difference in hardware would be the inclusion of a second infrared camera that could be deployed for nighttime approaching threat detection.

(52) It would be appreciated by those skilled in the art that various changes and modifications can be made to the illustrated embodiments without departing from the spirit of the present invention. All such modifications and changes are intended to be covered by the appended claims.