Wireless threat detection device, system, and methods to detect signals in wideband RF systems and localize related time and frequency information based on deep learning

Abstract

The present invention comprises a novel system and method to detect and estimate the time-frequency span of wireless signals present in a wideband RF spectrum. In preferred embodiments, the Faster RCNN deep learning architecture is used to detect the presence of wireless transmitters from the spectrogram images plotted by searching for rectangular shapes of any size, then localize the time and frequency information from the output of the FRCNN deep learning architecture.

Claims

1. A wireless threat detection device comprising: a processor; a memory unit; a wideband sensor comprising a wideband RF receiver coupled with a wideband RF antenna; and a digital processing unit configured to detect instances of wireless signal transmission, generate a multi-dimensional spectrogram image which includes representations of said one or more detected wireless signal transmissions and estimate location and dimension information of each of said representation in said spectrogram image using a machine learning based object detection algorithm.

2. The wireless threat detection device of claim 1, wherein said device is further configured to convert each of said estimated location and dimension information into a set of detected information including at least one of the transmission start time, frequency channel used, dwell time and bandwidth information.

3. The wireless threat detection device of claim 2, wherein said device is further configured to identify a list of wireless devices responsible for transmitting said detected instances of wireless signal transmission from a predetermined list of wireless device types.

4. The wireless threat detection device of claim 3, wherein said predetermined list of wireless device types includes at least one of Wi-Fi, Bluetooth device, Zigbee and unknown device type.

5. The wireless threat detection device of claim 3, wherein the device is further configured to categorize each of said detected information according to said predetermined list of wireless device types.

6. The wireless threat detection device of claim 3, wherein said device is further configured to determine treatment actions on at least one of said detected wireless devices.

7. The wireless threat detection device of claim 6, wherein said treatment actions include performing narrow-band jamming to block a specific time and frequency resources used by a detected wireless device.

8. The wireless threat detection device of claim 6, wherein said treatment actions include generating alerts.

9. The wireless threat detection device of claim 1, wherein said multi-dimensional spectrogram image is a three-dimensional spectrogram image and is generated utilizing short-time Fourier transform.

10. The wireless threat detection device of claim 1, wherein said digital processing unit uses deep learning to estimate location and dimension information in the spectrogram image.

11. The wireless threat detection device of claim 10, wherein said location information is represented by a pair of (x, y) coordinates on a Cartesian coordinate system.

12. The wireless threat detection device of claim 10, wherein said dimension information includes length and width information.

13. The wireless threat detection device of claim 1, wherein said digital processing unit further converts said location information into transmission start time of a corresponding detected wireless signal transmission.

14. The wireless threat detection device of claim 13, wherein said location information is the value of an x-coordinate.

15. The wireless threat detection device of claim 1, wherein said digital processing unit further converts said location information into frequency channel used by a corresponding detected wireless signal transmission.

16. The wireless threat detection device of claim 15, wherein said location information is the value of a y-coordinate.

17. The wireless threat detection device of claim 1, wherein said digital processing unit further converts said dimension information into dwell time of a corresponding detected wireless signal transmission.

18. The wireless threat detection device of claim 17, wherein said dimension information is length information.

19. The wireless threat detection device of claim 1, wherein said digital processing unit further converts said dimension information into bandwidth of a corresponding detected wireless signal transmission.

20. The wireless threat detection device of claim 19, wherein said dimension information is width information.

21. The wireless threat detection device of claim 1, wherein said representation of wireless signal transmission is a rectangular-shaped box.

22. The wireless threat detection device of claim 1, wherein said multi-dimensional spectrogram image is a three-dimensional spectrogram image, and wherein said digital processing unit comprises logic a deep learning based signal detector module configured to receive time-domain information of at least one wideband radio-frequency signal as input and transform each of said time domain information into said a rectangular-shaped box representation in said three-dimensional spectrogram image.

23. The wireless threat detection device of claim 9, wherein said deep learning based signal detector module further detects at least one rectangular-shaped box present in said spectrogram image; determines said location information and dimension information of each detected rectangular-shaped box by using a Faster region convolutional neural network (FRCNN) deep learning architecture; converts said dimension information of each detected rectangular box reported by the FRCNN deep learning architecture into time and frequency information; and stores said time and frequency information of each of said wideband radio-frequency signal in said memory unit.

24. The wireless threat detection device of claim 1, wherein said digital processing unit further comprises: an analyzer module configured to receive at least one set of detected information as input, wherein each detected information set includes at least one detected spectrogram rectangular box and its corresponding time-frequency information, and wherein said analyzer module converts said detection information into a higher level of abstraction which includes identifying a list of wireless devices responsible for transmitting said detected signals from a predetermined list of wireless device types and categorizing each of said detected information according to the wireless device type.

25. The wireless threat detection device of claim 1, wherein said digital processing unit further comprises a decision-making module configured to determine treatment actions on at least one of said detected wireless devices.

26. A system to detect the presence of wireless devices based upon localization of time and frequency information of wireless signals present in a wideband RF spectrum using deep learning, comprising: a deep learning based signal detector unit configured to: receive time-domain information of at least one wideband radio-frequency signal as input; transform each of said time domain information into a rectangular-shaped box representation in said multi-dimensional spectrogram image; detect at least one rectangular-shaped box present in said spectrogram image; determine said location information and dimension information of each detected rectangular-shaped box by using a machine learning based object detection algorithm; convert said location information of each representation reported by said machine learning based object detection method into transmission start time and frequency channel used information of said corresponding wideband radio-frequency signal; convert said dimension information of each detected rectangular box reported by the said machine learning based object detection algorithm into dwell time information and bandwidth information of said corresponding wideband radio-frequency signal; and store said at least one of said transmission start time information, frequency channel used information, dwell time information and bandwidth information of said at least one wideband radio-frequency signal; an analyzer unit configured to: receive at least one set of detected information as input, wherein each detected information set includes at least one detected spectrogram rectangular box and its corresponding time-frequency information; convert said detection information into a higher level of abstraction which includes identifying a list of wireless devices responsible for transmitting said wideband radio-frequency signals from a predetermined list of wireless device types and categorizing each of said detected information according to the wireless device type; and a decision-making unit configured to determine treatment actions on at least one of said detected wireless devices.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Some embodiments of the present invention are illustrated as an example and are not limited by the figures of the accompanying drawings, in which like references may indicate similar elements and in which:

[0029] FIG. 1 is a block diagram illustrating architecture of the proposed system for signal detection and time-frequency localization.

[0030] FIG. 2 illustrates the time content and spectrogram image of an example base-band complex-valued wideband RF capture, when the capture duration is 100 ms, wideband bandwidth is 56 MHz, and sampling rate is 56 MHz.

[0031] FIG. 3 is a block diagram illustrating FRCNN architecture.

[0032] FIG. 4 illustrates example outputs of a trained FRCNN model in which the detections and the estimated dimensions of the signals are overlaid with the input spectrogram.

[0033] FIG. 5 is a block diagram illustrating a Wireless threat detector device.

[0034] FIG. 6 illustrates sample output of Deep learning based signal detector block at different times. Deep learning based signal detector is used in Wireless threat detector device shown in FIG. 6.

[0035] FIG. 7 illustrates sample output of Analyzer block at different times. Analyzer block is used in Wireless threat detector device shown in FIG. 6.

DETAILED DESCRIPTION OF THE INVENTION

[0036] The present disclosure is to be considered as an exemplification of the invention and is not intended to limit the invention to the specific embodiments illustrated by the figures or description below.

[0037] The present invention will now be described by referencing the appended figures representing preferred embodiments.

[0038] The present invention provides a system and method to detect and estimate the time-frequency information of all wireless signals present in a wideband RF spectrum. The time-frequency information of each RF signal is composed of starting time, dwell time, frequency channel, and bandwidth. The proposed framework takes the wideband RF time-series data as the input and provides the detected signals along with the time and frequency information of each detected signal as the output.

[0039] FIG. 1 depicts the 4-stage architecture of the proposed system for signal detection and time-frequency localization. The four stages in the proposed system are RF time series capture, Spectrogram creation and preprocessing, box detection in spectrograms, time-frequency information extraction.

[0040] In one embodiment, FRCNN is applied. FIG. 3 depicts the architecture of FRCNN. FRCNN is a deep learning object detection framework. As in FIG. 3. FRCNN is composed of three main modules. The three main modules in FRCNN architecture are the base network (BN), the region proposal network (RPN), and the detector network. Once FRCNN is trained in a supervised manner, the modules inside will operate as follows. BN is the input interface where the image is fed. BN extracts the features of the input image and summarizes them into a scaled down feature map. In our case, the input images are spectrogram images that are obtained from the raw data containing RF signals. Next, RPN receives the scaled down feature map as an input. RPN is also fed with a set of anchor boxes (ABs) that serve as guidelines for finding approximate regions that may contain the object of interest. The objects of interest are RF signals which appear as rectangular boxes inside the spectrogram image. In the end, the detector network will tighten the approximate proposed regions around the rectangular shaped objects. In summary, FRCNN is the module that localizes the RF signals in the form of rectangular boxes inside spectrogram images.

[0041] Localization performed on a spectrogram image refers to detecting a rectangular box in the spectrogram image and estimating its location in the image and its dimensions. Each rectangular box is equivalent to an RF transmission from a wireless device. The location of the rectangular box inside the spectrogram corresponds to the starting time and frequency channel of that RF transmission, and the dimensions of the rectangular box corresponds to the dwell time and bandwidth of that RF transmission. Localization can be achieved with various types of machine learning based object detection algorithms.

[0042] Details on the four stages of the proposed signal detection and time-frequency localization system are given below.

[0043] 1) RF time-series capture: In the first stage, consider that we employ a wideband RF receiver with center frequency f.sub.c, bandwidth W, and sampling rate f.sub.s to record time-series RF data in fragments of T milliseconds each. The time and frequency content of an example synthetically generated baseband complex-valued wideband capture with, f.sub.s=56 MHz, W=56 MHz, and T=100 ms is given in FIG. 2. FIG. 2a plots the signal amplitude as a function of time, whereas FIG. 2b plots the corresponding spectrogram image.

[0044] 2) Spectrogram creation and pre-processing: For a compact representation of the wideband signal in terms of time and frequency, we apply short-time Fourier transform (STFT) on the RF time-series captures and obtain the Power Spectral Density (PSD) as a function of time and frequency. STFT is a representation of how the frequency content of a signal changes with elapse of time. In order to obtain STFT of a time domain signal, the signal is divided into short and optionally overlapped windows, followed by calculating the discrete Fourier transform (DFT) of these windows. Each window corresponds to a point in time, and DFT of that window corresponds to the frequency content of the signal at that point in time. The product of applying STFT on a signal in time domain is generally referred to as the spectrogram of that signal. STFT values are complex numbers, consisting of real and imaginary parts. Three-dimensional spectrogram images are then created by plotting the PSD values along the time and frequency axes, wherein time is laid over x-axis, frequency is laid-over y-axis, and amplitude of the STFT values for a given time and frequency produces the z-axis values. FIG. 2b illustrates the spectrogram image created for the RF capture in FIG. 2a, when the STFT parameters are chosen as follows: number of points in a window is 4096, number of point in DFT is 4096, a Hann-type smoothing filter is applied to mitigate the effects of DFT artifacts on the corners of the window, and the window overlap is set to 50% at 2048 points. As may be noted from FIG. 2b, the signals to be detected appear in the form of rectangular boxes in the spectrogram image. From the spectrogram, each rectangular box is equivalent to an RF transmission from a wireless device, the location of the rectangular box inside the spectrogram is equivalent to the starting time and frequency channel of that RF transmission, and the dimensions of the rectangular box is equivalent to the dwell time and bandwidth of that RF transmission. Thus, the location and dimensions of each rectangular box within the spectrogram give us the time and frequency information of the corresponding wireless signal. The problem of signal detection and time-frequency localization therefore boils down to the problem of detecting and estimating the dimensions of rectangular-shaped boxes present in the spectrogram. Before attempting box-detection in the spectrogram image, we may employ some pre-processing steps to improve the detection performance in low SNR regimes. For example, we may employ denoising methods, such as wavelet denoising, to improve the signal-to-noise ratio (SNR) of the spectrogram.

[0045] Alternatively, one could add the phase information to the amplitude information and have a 4-dimensional image. In this case, the amplitude of the STFT values composes the 3rd dimension, and the phase of the STFT values constructs the 4th dimension.

[0046] Another alternative is to work directly with real and imaginary parts of the STFT values. This case also provides a 4-dimensional image, in which the real part of STFT values composes the 3rd dimension, and the imaginary part of the STFT values gives the 4th dimension.

[0047] In all cases, time and frequency are the first two dimensions.

[0048] 3) Box detection in spectrograms: To detect the rectangular-shaped boxes present in the spectrograms, we take a supervised machine learning approach, wherein, we train a Faster RCNN (FRCNN) model with several labelled spectrogram images. Like any other supervised machine learning method, the machine infers the operation it is expected to perform by seeing several examples of the input-output pairs., i.e., training data. In our case, the operation that is expected to be performed by the machine is to detect rectangular boxes in images. Therefore, the input-output pairs, i.e., training data, are labeled spectrogram images. Labeled spectrogram images are spectrogram images along with the time frequency information of all the rectangular boxes that are supposed to be detected inside a given image. The labeled spectrogram images can be produced artificially through simulations or can be obtained by processing raw data obtained from actual wideband receivers. The labels can be crafted manually, or by developing detectors for devices that are expected to be detected by the machine. The trained FRCNN model, when input with a test spectrogram image, detects the rectangular-shaped boxes present in the image and reports their dimensions.

[0049] FIG. 3 depicts the architecture of FRCNN. FRCNN is a deep learning object detection framework. As in FIG. 3. FRCNN is composed of three main modules. The three main modules in FRCNN architecture are the base network (BN), the region proposal network (RPN), and the detector network. Once FRCNN is trained in a supervised manner, the modules inside will operate as follows. BN is the input interface where the image is fed. BN extracts the features of the input image and summarizes them into a scaled down feature map. In our case, the input images are spectrogram images that are obtained from the raw data containing RF signals. Next, RPN receives the scaled down feature map as an input. RPN is also fed with a set of anchor boxes (ABs) that serve as guidelines for finding approximate regions that may contain the object of interest. The objects of interest are RF signals which appear as rectangular boxes inside the spectrogram image. In the end, the detector network will tighten the approximate proposed regions around the rectangular shaped objects. In summary, FRCNN is the module that localizes the RF signals in the form of rectangular boxes inside spectrogram images.

[0050] Localization performed on a spectrogram image refers to detecting a rectangular box in the spectrogram image and estimating its location in the image and its dimensions. Each rectangular box is equivalent to an RF transmission from a wireless device. The location of the rectangular box inside the spectrogram corresponds to the starting time and frequency channel of that RF transmission, and the dimensions of the rectangular box corresponds to the dwell time and bandwidth of that RF transmission. Localization can be achieved with various types of machine learning based object detection algorithms.

[0051] To illustrate the output of our method, FIG. 4 shows sample predictions from a trained FRCNN model in which the detections and estimated dimensions of signals are overlaid with the input spectrogram.

[0052] 4) Time frequency information extraction: As the final step, dimensions of each rectangular box reported by the FRCNN model is converted into time and frequency information. For example, using the STFT parameters employed in the spectrogram creation stage, we may scale the x and y dimensions of each box into the time and frequency span of the corresponding signal. The same approach is followed to obtain transmission starting time of the signal, as well as its channel center frequency, by converting the location of the rectangular box within the spectrogram into time and frequency information.

[0053] Wireless Threat Detector

[0054] The described invention may be used in a Wireless threat detector device that is a wireless spectrum monitoring tool that detects the presence of unexpected wireless devices in a given protected geographical area. The protected geographical area can be an office environment, campus of a research facility, airport runway, correctional facility, etc. The Wireless threat detector device shall be deployed at a central location where the wireless signal is not blocked by large physical object. If a finding a single central location with enough coverage is not possible, multiple Wireless threat detector devices shall be installed in a grid so that the whole area is covered.

[0055] FIG. 5 shows the block diagram of Wireless threat detector device. In this figure, the described invention in this patent is encapsulated in the second building block: Deep learning based signal detector. The first component in this device is a wideband sensor. The wideband sensor is a wideband RF receiver paired with a wideband antenna. This sensor provides the raw data pertaining to the wireless environment that is being monitored. This raw data is in the form of a baseband complex-valued time domain signal.

[0056] The raw data is fed into the Deep learning based signal detector block. Deep learning based signal detector block has the architecture depicted in FIG. 1. This block converts the raw data into spectrogram images, conducts rectangular box detection, and extracts time-frequency information for the detected rectangular boxes. As described before, each rectangular box is equivalent to an RF transmission from a wireless device, the location of the rectangular box inside the spectrogram is equivalent to the starting time and frequency channel of that RF transmission, and the dimensions of the rectangular box is equivalent to the dwell time and bandwidth of that RF transmission.

[0057] FIGS. 6 and 7 show sample outputs from intermediate building blocks inside Wireless threat detector device.

[0058] FIG. 6 shows a sample output of Deep learning based signal detector block at different times. In this figure, time N is later than time 1 (To illustrate, time 1 can be 1:00 μm, and time N can be 2:00 pm). In this sample output, one can see that at time 1 there is one set of spectrogram rectangular boxes in the environment, while with time progressing, another group of rectangular boxes appears.

[0059] The detected spectrogram rectangular boxes (i.e. RF transmissions) along with their time-frequency information are fed into an Analyzer block. The time and frequency information includes start time, frequency channel, dwell time, and bandwidth. The Analyzer block may use separate algorithms to convert the detection information into a higher level of abstraction. For instance, the Analyzer block may categorize the detection information into possible wireless devices that are transmitting those detections. The analyzer block clusters the detections into several clusters based on the time frequency information of the detections. Detections that are mapped to the same cluster have similar time frequency information. This common time frequency information can be referred to as the profile of the cluster. FIG. 7 shows a sample output of the Analyzer block at different times. Note that as compared to output of Deep learning based signal detector block in FIG. 6, now the detected RF transmissions are labeled with their own groups, i.e., wireless devices. For the sake of illustration, in FIG. 7, Analyzer block is detecting a single device at time 1, and two devices at time N.

[0060] The analyzer may have a library of known devices profiles, such as Wi-Fi devices, to which it can match the profile of a given cluster and determine that the cluster is indeed of that type. The profile of the cluster/device can be, for example, a certain time-frequency information that is obtained by averaging over many observations. In case of comparing with a library of profiles of known devices, the analyzer block may conclude that one or more clusters do not match to any of the known devices. In this case, the analyzer may only announce presence of devices with unknown types.

[0061] Note that the clustering step can be done without consulting with a library of known device profiles. If no library is available, all the clusters will be of unknown type, but still the analyzer block can distinguish between the devices.

[0062] In the end, a Decision maker block may make decisions regarding how to treat the detected devices. As an example, the decision maker block may deem device 1 at time 1 a friendly device, and device 2 that has appeared at time N an unexpected device.

[0063] The present invention can be commercialized into products for wireless monitoring and security, and wireless spectrum management. Wireless monitoring and security products can be built to make ad-hoc security decisions such as sending emergency alerts when unusual spectrum activity, i.e., an unexpected wireless device, while monitoring a protected environment. In more sensitive environments, the security product may perform narrow-band jamming to mitigate rogue devices by blocking the specific time and frequency resources used by the rogue device without affecting other friendly devices. Such security measures can be of interest particularly in commonly used frequency bands such as the Industrial Scientific and Medical (ISM) band, while they are also of interest in licensed frequency bands. Wireless spectrum management products can be built to dynamically monitor and manage sharing the spectrum among the vast number and variety of heterogeneous devices in the IoT space to improve the user experience. For example, knowledge of which time and frequency resources are under-utilized and which ones have minimum interference can aid in smart spectrum allocations wherever the wireless traffic is dense.

[0064] Persons skilled in the art will appreciate that numerous variations and modifications will become apparent. All such variations and modifications which become apparent to persons skilled in the art, should be considered to fall within the spirit and scope that the invention broadly appearing before described.