Method and Apparatus of An Automated Safety Response System in a Self-organizing, multi-networked cooperative NvisiLink Mesh with Echo Positioning

Abstract

The present invention teaches the implementation for a system of networked heterogenous signal capture and analysis sensor-enabled devices tethered in a cooperative multi-protocol wireless local area network (WLAN) providing an automated safety monitoring and response services during an active shooter situation. The present invention describes a method to leverage the standard sensors on most smartphones into a real-time swarming of localized tracking, monitoring and guidance networked to direct people to identified safe zones in the covered build and public venues. The system utilizes multi-device real-time two-way positioning/ranging with acoustic based source geo-location algorithm to pinpoint danger regions within the coverage area. The response system described in this invention activates automatically upon detection of discharge of any firearm in the protected area without manual intervention.

Claims

1. A system for cooperative wireless networking of a collection of sensor signal capture devices, including automated simultaneous multi-protocol wireless safety and response transmissions, the system comprising: a. at least three distinct and separately located microphones within a coverage area; b. at least one distinct real-time relative positioning apparatus for computing real-time relative positioning co-located with each of the microphones, wherein the co-located real-time relative positioning apparatus is in communication with the microphone to which the at least one distinct real-time relative positioning apparatus is co-located; c. at least one distinct display unit for displaying images and textual data co-located with each of the microphones, wherein the co-located display unit is in communication with the microphone to which the at least one distinct display unit is co-located; d. at least one local processor running a gunshot detection and classification algorithm on incoming transduced sound wave co-located with each microphone, wherein the at least one of the local processors is in communication with the microphone to which the at least one local processor is co-located; e. at least one wireless transmission device running a first wireless transmission protocol co-located with each distinct microphone, wherein the wireless transmission device is capable of at least four simultaneous connections over a wireless medium without the need to a priori scheduling of the wireless medium, wherein the co-located wireless transmission device is in communication with the microphone to which the wireless transmission device is co-located; f. at least one sensor device capable of measuring device orientation relative to an internal frame of reference co-located with each distinct microphone, wherein the co-located sensor device is in communication with microphone to which the sensor device is co-located; g. at least a second wireless transmission device running a second wireless transmissions protocol co-located with each distinct microphone, wherein the wireless transmission device is capable of communicating with law enforcement personnel or first responders.

2. A system of claim 1, wherein the first wireless transmissions protocol facilitates at least four simultaneous connections over the wireless medium, wherein the four simultaneous connections is communicates with a digital chaos connected mesh network comprising of devices transmitting and receiving Digital Chaos signatures

3. A system of claim 1, wherein the real-time relative positioning apparatus is GPS receiver.

4. A system of claim 1, wherein the real-time relative positioning apparatus is a non-GPS receiver.

5. A system of claim 3, wherein the real-time relative positioning apparatus is one of a two-way ranging UWB or digital chaos enabled devices.

6. A system for having a wireless receiver, wherein the wireless receiver is configured for real-time coordinate transformation from frame of reference derived from time-difference of arrival of acoustic measurements to an equivalent coordinate system and frame of reference derive from time-difference of arrival of rf measurements, wherein the real-time coordinate transformation is computed using an onboard processor system of WLAN AP measuring the time-difference of arrival of rf measurements.

7. A wireless receiver of claim 6, wherein the real-time coordinate transformation is computed using an cloud based processor system using time-difference of arrival of rf measurements from WLAN AP.

8. A wireless receiver of claim 6, wherein the receiver is configured for eliminating multipath false peaks from two-way ranging calculations using real-time shared orientation data and measured wireless channel state information.

9. A wireless receiver of claim 8, where eliminating multipath false peaks is computed with an onboard processor at one of the receiving device participating in the two-way ranging procedure.

10. A wireless receiver of claim 8, wherein the receiver is configured to immediate transmission of a SOS beacon frame containing the device relative position and other situational awareness information to all active devices on the wireless medium when gunfire is detected at the device to preserve the time-difference of arrival information between devices.

11. A wireless receiver of claim 10, wherein other situational awareness information includes measurements from integrated IMU co-located with the microphone.

12. A wireless receiver of claim 10, wherein other situational awareness information includes images from camera at known locations in the coverage area.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0034] A more complete understanding of the present invention may be derived by referring to the various embodiments of the invention described in the detailed descriptions and drawings and figures in which like numerals denote like elements, and in which:

[0035] FIG. 1 illustrates the sequential relationship occurring when a cause event triggers a chain reaction that initiates a generation mechanism (via transmitting transduction device) that traverse a medium as a propagating wave of energy to be received by a receiving transduction device.

[0036] FIG. 2 depicts the key elements of involved in range calculation based on propagating waves and its known speed through the medium between transducing devices.

[0037] FIG. 3 shows the interaction between a Map Reference and fusion of navigational sensor data to provide accurate distributed timing information to communication devices connected to smartphones

[0038] FIG. 4 illustrates multiple two-way ranging between tags of unknown positions with an anchor of known position;

[0039] FIG. 5 is an exemplary diagram for NvisiLink mesh network with cluster heads labeled with numbers.

[0040] FIG. 6 is an exemplary example of non-interfering concurrent signals of the wireless medium in accordance with various embodiments of the invention;

[0041] FIG. 7 is an exemplary example of false peaks due to multipath during two-way ranging in accordance with various embodiments of the invention;

[0042] FIG. 8 is an exemplary implementation of MIMO unit, in accordance with various embodiments of the invention;

[0043] FIG. 9 is an exemplary floorplan of a typical coverage, in accordance with various embodiments of the invention;

[0044] FIG. 10 is an exemplary example of components of a sensor fusion engine that help eliminates false peak triggers due to multipath, in accordance with various embodiments of the invention;

[0045] FIG. 11 is an exemplary process flow of expected sequence and acoustic geolocation procedure during an active scenario, in accordance with various embodiments of the invention;

[0046] FIG. 12 is a deep learning neural network structure for analytics, classification, and training, in accordance with various embodiments of the invention;

[0047] FIG. 13 is an exemplary illustration of swarm movement of smartphone holders, displayed on a downloaded digital map of the type depicted in FIG. 9 being directed to safe zone away from shooter locations.

DETAILED DESCRIPTION

[0048] The detailed description of exemplary embodiments of the invention herein refers to the accompanying drawing and flowchart, which show the exemplary embodiment by way of illustration and its best mode. While these exemplary embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, it should be understood that other embodiments may be realized, and that logical and mechanical changes may be made without departing from the spirit and scope of the invention. Thus, the description herein is presented for purposes of illustration only and not of limitation. For example, the steps recited in any of the method or process descriptions may be executed in any order and are not limited to the order presented.

[0049] The present invention may be described herein in terms of functional block components and various processing steps. It should be appreciated that such functional blocks may be realized by any number of hardware and/or software components configured to perform the specified functions. For example, the present invention may employ various integrated circuit (IC) components (e.g., memory elements, processing elements, logic elements, look-up tables, and the like), which may carry out a variety of functions under the control of one or more microprocessors or other control devices. Similarly, the software elements of the present invention maybe implemented with any programming or scripting language such as C, C++, java, COBOL, assembler, PERL, or the like, with the various algorithms being implemented with any combination of data structures, objects, processes, routines or other programming elements. Further, it should be noted that the present invention may employ any number of conventional techniques for data transmission, signaling, data processing, network control, and the like. Still further, the invention could be used to detect or prevent security issues with a scripting language, such as JavaScript, VBScript or the like. For a basic introduction of cryptography, please review a text written by Bruce Schneider which is entitled Applied Cryptography: Protocols Algorithms, And Source Code In C, published by John Wiley & Sons (second edition, 1996), which is hereby incorporated by reference.

[0050] It should be appreciated that the particular implementations shown and described herein are illustrative of the invention and its best mode and are not intended to otherwise limit the scope of the present invention in any way. Indeed, for the sake of brevity, conventional wireless data transmission, transmitter, receivers, modulators, base station, data transmission concepts and other functional aspects of the systems (and components of the individual operating components of the systems) may not be described in detail herein. Furthermore, the connecting lines shown in the various figures contained herein are intended to represent exemplary functional relationships and/or physical couplings between the various elements. It also should be noted that many alternative or additional functional relationships or physical connections may be present in a practical electronic transaction or file transmission system. Additionally, where elements of the invention are described as communicating with, or in communication with, the invention contemplates direct communication between components or communicating through one or more communicating or connected components.

[0051] As will be appreciated by one of ordinary skill in the art, the present invention may be embodied as a method, a data processing system, a device for data processing, and/or a computer program product. Accordingly, the present invention may take the form of an entirely software embodiment, an entirely hardware embodiment, or an embodiment combining aspects of both software and hardware. Furthermore, the present invention may take the form of a computer program product on a computer-readable storage medium having computer-readable program code means embodied in the storage medium. Any suitable computer-readable storage medium may be utilized, including hard disks, CD-ROM, optical storage devices, magnetic storage devices, and/or the like.

[0052] To simplify the description of the exemplary embodiment, we described for one of possible scenarios to illustrate the sequence of events taught in this invention. Further, it should be appreciated the sequence of events described herein is one just of many possible sequences and should be construed as a limit or all-encompassing operational use of the invention. FIG. 1 details the broad stages that occur during the operation of the invention. The invention is an automatic response system thus requiring a cause 101 or trigger to initiate the start of the process. Discharge of a firearm is one of the targeted causes this invention is intended to detect. Discharging a firearm generates 102 specific sound profiles 102 that are used today in monitoring systems. These monitoring systems utilize one form of transduction 104 to convert the propagating acoustic wave associated sound 103 profiles to a form for ease of detection. After detection, there is a predetermined step or sequence of steps 105. The present invention describes new approaches in the detection process and automatic responses not part of the current state of art

[0053] The present invention teaches a cooperative, distributed methodology for gunshot detection as source location not found in the art. The use of multiple source recording/capturing devices (such microphones) for source location is not new. Microphone arrays have been used for this purpose; however, the position of each microphone is typically known precisely relative to each other. Furthermore, their positions remain permanently fixed or relative fixed. In the preferred embodiment of the invention, the invention is implemented directly as a system on chip (SoC) intellectual property (IP) component within electronics of any commercially available smartphones. In this embodiment, there is a direct and known relationship between measurements from integrated sensors on the smartphone to measurements provided to on-board processor in the SoC implementation of this invention. An alternative embodiment, the SoC IP would be implemented as a separate dongle with its own integrated sensors and attached to the smartphone. Similarly, the relationship between the measurements from integrated sensors on the dongle to the smartphone is known by on-board processor in the SoC. The SoC maintains a swarm-like sharing of environmental and operational conditions amongst all SoC units communicating through a NvisiLink mesh network.

[0054] The NvisiLink mesh nodes of the present invention are able to simultaneously communication in small clusters of members without self-interference and perform inter-cluster communications via designated cluster heads such as depicted in FIG. 5. Member 3 (503) of cluster B is a designated cluster head as it can communicate directly with other members in different clusters within its range. For example, member 3 can communicate with member 1 of cluster C. Member 1 can communicate with member 8 of the same cluster. In this way, members of all clusters are updated with periodic real-time information. Upon detection of gunfire standard means of the state of the art, each member immediately sends out a beacon containing at least orientation information 1020, inertia information 1060, and GPS 1050 to other members and the central access point in the local area network for processing with their respective sensor fusion engines. This joint messaging mechanism represents a significant improvement in the state of the art as it infers the time difference of arrival between SoC devices detections of the gunfire from the access point (AP) perspective. In other words, the first arriving beacon to the access point originates from the device closest to the gunfire since the rf wave propagation speed is over five orders of magnitude faster than acoustic wave propagation speed. In a preferred embodiment, the AP would support dual modes of communications: WiFi 4-7 physical signaling operating cooperatively with NvisiLink mesh network in a local area network (LAN) coverage environment.

[0055] The present invention teaches training of deep learning neural network (DNN) 1200 to eliminate any non-direct line of sight measurements 1070 from concurrent two-way ranging calculation in FIG. 2 based on IM U measurements and orientation sensor data 1020 along with the current wireless channel state information indicating multipath at the receiver.

[0056] Further this invention teaches overlaying the absolute position of mobile SoC devices onto a floorplan containing fixed anchors (emitters 910, 930) strategically place 900 throughout the coverage area for monitoring and evacuation. Moreover, the invention further teaches downloading a digital version floorplan 1300 to all participating during an active event and indicating the area where the estimated source location of the gunfire on downloaded map.

[0057] In exemplary embodiments of the invention, the locations of the NvisiLink are made available to authorized staff, first responders, and law enforcement through the wireless LAN network while instructing via text and visual ques on the digital map of safe zones away from the source of the gunfire during an active threat scenario.

[0058] Further embodiments of the invention allow other external aid during an active threat scenario to track the source of the gunfire when acoustic measurements are insufficient. In a preferred implement, existing security camera feeds 1040 are used to track the location of the firearm and direct smartphone users away from harm.