INERTIAL MEASUREMENT UNITS AS VIBROACOUSTIC DATA RECEIVERS
20220011113 · 2022-01-13
Assignee
Inventors
Cpc classification
H04W4/80
ELECTRICITY
H04B11/00
ELECTRICITY
G01C19/5719
PHYSICS
International classification
Abstract
Inertial measurement units with gyroscopic sensors are standard in mobile computers. The present invention shows that these sensors can be co-opted for vibroacoustic data reception. The present invention illustrates a new capability for an old sensor utilizing the commodity gyroscope sensor found in most average smartphones and a low-cost transducer to the present invention can transmit error-corrected data at 2028 bits per sec with the expectation that 95% of packets will be successfully received.
Claims
1. A method of transmitting data, through physical touching, from an object to a smart device comprising a gyroscope sensor, the method comprising: Configuring the object with a transducer that is capable of producing vibrations; and Configuring the transducer to transmit data through the vibrations, Whereby, when the smart device is physically touching the object, the gyroscope sensor detects the vibrations and the transmitted data.
2. The method of claim 1, wherein the smart device is selected from the group consisting of a smart phone, a tablet, a mobile phone, a personal digital assistant and a mobile computing device.
3. The method of claim 1, wherein the smart device is configured to receive and translate the data to information usable by the smart device.
4. The method of claim 1, wherein the smart device also comprises an inertial measurement unit, which contains the gyroscope sensor.
5. The method of claim 1, wherein the vibrations are ultrasonic vibrations.
6. The method of claim 1, wherein the data is transmitted by using more than one frequency.
7. The method of claim 1, wherein the gyroscope sensor is of a vibrating-comb design.
8. The method of claim 1, wherein the transducer is selected from the group consisting of a speaker, a piezoelectric element, a piezoelectric disc, a voice coil, and a linear resonance actuator.
9. The method of claim 1, wherein the data is transmitted by outputting at least one frequency and the data is converted into a stream of symbols.
10. The method of claim 9, wherein the stream of symbols are modulated using at least one approach selected from the group consisting of amplitude-shift keying, phase-shift keying, differential phase-shift keying, frequency-shift keying, quadrature-amplitude modulation and on-off keying.
11. A method of receiving data from an object that produces vibrations, comprising: Selecting a smart device comprising a gyroscope to receive data that is in the form of vibrations; and Touching the smart device to the object configured with a transducer that is configured to produce vibrations and to transmit data through the vibrations, whereby the data can be transmitted from the object to the smart device.
12. The method of claim 11, also comprising configuring the smart device to translate the data.
13. A system for transmitting data through physical touching, comprising: An object; A transducer that is associated with the object and capable of producing structured vibrations for transmission of data; and A smart device equipped with a gyroscope configured to receive the data from the transducer when the smart device touches the object.
14. The system of claim 13, wherein the smart device is selected from the group consisting of a smart phone, a tablet, a mobile phone, a personal digital assistant, and a mobile computing device.
15. The system of claim 13, wherein the smart device is configured to translate the data.
16. The system of claim 13, wherein the smart device further comprises an inertial measurement unit, which contains the gyroscope.
17. The system of claim 13, wherein the vibrations are ultrasonic vibrations.
18. The system of claim 13, wherein the data is transmitted by using more than one frequency.
19. The system of claim 13, wherein the smart device also comprises a driver that is configured to sample the gyroscope, and whereby the smart device is configured to pick-up the vibrations and demodulate them into binary data.
20. The system of claim 13, wherein only one axis of the gyroscope is utilized.
21. The system of claim 13, wherein the gyroscope is of a vibrating-comb design.
22. The system of claim 13, wherein the transducer is selected from the group consisting of a speaker, a piezoelectric element, a piezoelectric disc, a voice coil, and a linear resonance actuator.
23. The system of claim 13, wherein the data is transmitted by outputting at least one frequency and the data is converted into a stream of symbols.
24. The system of claim 23, wherein the stream of symbols is modulated using at least one approach selected from the group consisting of amplitude-shift keying, phase-shift keying, differential phase-shift keying, frequency-shift keying, quadrature-amplitude modulation, and on-off keying.
25. A secure payment method comprising touching a smart device comprising a gyroscope, which smart device is configured to receive and translate data, to an object having a transducer, which transducer is capable of producing vibrations that transmit data, Whereby the touching of the smart device to the object enables the data from the object to be transmitted securely to the gyroscope.
26. A method of transmitting data, through acoustic coupling, from an object to a smart device comprising a gyroscope sensor, the method comprising: Configuring the object with a transducer that is capable of producing vibrations; and Configuring the transducer to transmit data through the vibrations, Whereby, when the smart device is acoustically coupled to the object, the gyroscope sensor detects the vibrations and the smart device converts the vibrations and data to information usable by the smart device.
Description
BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS
[0022] For the purpose of facilitating understanding of the invention, the accompanying drawings and description illustrate preferred embodiments thereof, from which the invention, various embodiments of its structures, construction, method of operation and many advantages may be understood and appreciated.
[0023]
[0024]
[0025]
[0026]
[0027]
[0028]
[0029]
[0030]
[0031]
[0032]
[0033]
[0034]
[0035]
[0036] and
[0037]
DETAILED DESCRIPTION OF THE INVENTION
[0038] The present invention will be understood more readily by reference to the following detailed description of the invention and the accompanying figures, which form a part of this disclosure. This invention is not limited to the specific devices, methods, processes, elements or parameters described and/or shown herein and the terminology used herein is for the purpose of describing particular embodiments and is by way of example only and not intended to be limiting of the claimed invention. Any and all patents and other publications identified in this specification are incorporated by reference as though fully set forth herein. The following detailed descriptions should not be taken in a limiting sense. The accompanying figures and drawings are hereby incorporated by reference.
[0039] Also, as used in the specification including the appended claims, the singular forms “a,” “an,” and “the” include the plural, and reference to a particular numerical value includes at least that particular value, unless the context clearly dictates otherwise. Ranges may be expressed herein as from “about” or “approximately” one particular value and/or to “about” or “approximately” another particular value. When such a range is expressed, another embodiment includes from the one particular value and/or to the other particular value. Similarly, when values are expressed as approximations, by use of the antecedent “about,” it will be understood that the particular value forms another embodiment.
[0040] It is to be understood that the invention may assume alternative variations and step sequences, except where expressly specified to the contrary. It also is to be understood that the specific devices and processes illustrated in the attached drawings and described in this specification are simply exemplary embodiments of the invention. Hence, specific dimensions and other physical characteristics related to the embodiments disclosed herein are not to be considered as limiting.
[0041] While the disclosure has been described in detail and with reference to specific embodiments thereof, it will be apparent to one skilled in the art that various changes and modifications can be made thereto without departing from the spirit and scope of the embodiments. Thus, it is intended that this application covers modifications and variations of this disclosure provided they come within the scope of the appended claims and their equivalents.
[0042] The present invention can be used with any electronic smart devices 10 containing a gyroscope 40 (or gyroscope sensor 40) alone or as a part of an IMU 30, including but not limited to smart phones, tablets, mobile telephones, personal digital assistants, mobile computing devices, etc. (collectively referred to herein as a “smart device(s) 10”). As shown in
[0043] Receiver Hardware: For the smart device 10, one example embodiment of the present invention uses an Android smartphone running Android 5.0.1. It will be obvious to one skilled in the art that any smart device 10 that has a gyroscope 40 or an IMU 30 containing a gyroscope 40 can be used for the present invention as well and is hereby incorporated into the present invention. This particular smartphone (an LG Nexus 5) has an InvenSense MPU-6515 IMU 30 built in, which is capable of sampling the accelerometer 50 at 4 kHz and the gyroscope 40 at 32 kHz. The default Linux kernel drivers for Android support a maximum sample rate of 100 Hz, so this particular embodiment of the present invention utilizes a custom kernel driver that configures the IMU 30 into the 32 kHz high-speed sampling mode and disables the low-pass filter. Disabling this low-pass filter causes high-frequency vibrations (above the 16 kHz Nyquist frequency) to appear aliased in the sampled data, instead of being attenuated away. The driver commands the IMU 30 to store this data to an internal FIFO, which is emptied using a burst-read command every 20 ms by the kernel driver. The internal I.sup.2C bus is not fast enough to transmit all three axes at 32 kHz. Therefore, this one embodiment of the present invention uses only the z-axis of the gyroscope 40, which was determined to have the highest SNR in an informal test.
[0044] In one embodiment, the MPU-6515's gyroscope 40 is a “vibrating-comb” design—a common type of gyroscope 40 in which a micro-scale comb is vibrated at a fixed frequency fc near a second fixed comb and the capacitance between the two combs is continuously measured. External rotations of the vibrating comb cause it to move out-of-plane, changing the capacitance between the two combs. The comb's fundamental vibration frequency and two of its harmonics can be seen clearly in
[0045] When an external oscillation of frequency f is picked up by the gyroscope 40, the combination of this frequency and the comb frequency produces a pair of beat frequencies at f.sub.c+f and f.sub.c−f. For instance, as shown in
[0046] Finally, it is notable that the sampling rate of the device is not exactly 32 kHz. In fact, the sampling rate can vary by up to ±750 Hz depending on device imperfections and the device temperature. In order to correct for the sample rate variation, the driver reports the nanosecond timestamp and total accumulated byte count when reading from the internal FIFO, allowing the user code to compute the average sample rate over a period of time. The comb frequency also varies in approximately a 1:1 fashion with the sample rate, affecting the frequency of the received data.
[0047] These effects—sample rate instability, beat frequency, resonant band, and vibrating-comb noise—must be compensated for in order to provide the best possible data transmission rate. In particular, note that these effects are not specific or unique to this model of gyroscope 40—similar effects exist in most gyroscopes 40 of this type. Therefore, it will also be obvious to one skilled in the art that the present invention will work with any similar gyroscope 40 that is in a smart device 10 as long as these effects are compensated for to maximize the data transmission rate.
[0048] Alternative embodiments of the present invention utilize an accelerometer 50 instead of a gyroscope 40 as the receiver hardware. It will be obvious to one skilled in the art that an accelerometer 50 can be used to receive transmissions in a manner similar to the gyroscope 40.
[0049] Transmission: The present invention was tested with several objects 15 that were equipped with transducers 25 or one of a variety of transmission devices 20. The present invention works with a wide variety of transducers 25, including but not limited to speakers 20A, voice coils 20C, linear resonance actuators, and most piezoelectric elements, including but not limited to piezoelectric discs 20B. Testing of the present invention focused on one preferred embodiment that utilizes the piezo disc 20B (as a type of piezoelectric element)(shown in
[0050] One embodiment of the present invention uses a 27 mm PUI Audio piezo bender for the transmission device 20, hooked up via a 3.5 mm TRS connector to a MacBook Pro® computer which provides the modulated ultrasound signal. Piezoelectric elements also have the advantage of extreme low cost (pennies at scale), thin design and low weight, allowing them potentially to be integrated into everyday objects to make them capable of transmitting modulated vibrations. Again,
[0051] According to the present invention, the vibrations produced by the transducers 25 are “structured”, meaning the vibrations are intentionally designed to transmit data that can be detected by a gyroscope 40 and translated or decoded by an appropriately equipped and configured smart device 10. Data is transmitted by outputting one or more waves at known frequencies, called carrier frequencies. Carrier frequencies are used for some but not all embodiments of the present invention.
[0052] One embodiment of the present invention utilizes orthogonal frequency division multiplexing (“OFDM”) to improve resilience and channel utilization, but this approach was found to be less ideal than a rapidly-modulated single-carrier approach because the unpredictable shifts in the gyroscope's sample rate precluded clean recovery of the orthogonal subcarriers. Similarly, PSK and QAM approaches proved unreliable, as shifts in the sample rate during a message resulted in phase drifts that accumulated over time. However, DPSK, which naturally compensates for phase drift, works quite well, albeit at the cost of doubling the noise level.
[0053] For ASK, PSK, and DPSK, the present invention was tested utilizing a carrier frequency of 19200 Hz. For FSK, the present invention was tested at a frequency range of 17200-21200 Hz. In both cases, the goal was to use ultrasound frequencies for inaudibility (above ˜17 kHz), and to avoid the resonant band (which is excited by a 22-24 kHz external oscillation).
[0054] In one embodiment of the present invention, the data is packetized to be sent into 1-second long packets, and a header consisting of a fixed 50-symbol sequence is prepended (the left side of
[0055] Rx Software: In one embodiment, receive logic is implemented for ASK, DPSK, PSK, and FSK techniques. Each receiver consists of two parts: a header-search portion that looks specifically for the 50-symbol message preamble, and a decoder that decodes the subsequent message using the channel properties identified in the header. Data reception starts by converting the received data to a constant 32000 Hz sample rate using a phase-preserving sinc-based resample filter. Then, for each carrier frequency (1 carrier for ASK, DPSK, 1-8 carriers for FSK), the received frequency is estimated using a device-specific constant for the comb frequency and the received sample rate. Finally, a fourth-order Chebyshev band-pass filter is designed and applied to select the carrier. These steps correct for the received frequency offsets due to the comb frequency, as well as the sample rate drift.
[0056] ASK Decoding: Amplitude shift keying modulates the amplitude of the carrier frequency. The header is modulated using binary (two-level) ASK for ease of recognition. To find the header, the present invention, in one embodiment, scans across the filtered data, fitting sliding windows to this ideal ASK-modulated preamble. To fit a sliding window, the window is split into symbol-sized units of length N and the amplitude of the received carrier frequency fir is computed:
[0057] where x.sub.n is the n.sup.th sample in the symbol. One embodiment of the present invention uses the median of the amplitudes within the window as a threshold to produce a sequence of binary bits and compares this sequence to the preamble. If fewer than 8 errors (16%) are found—a 1 in 9.5 million chance of occurring at random—then an assumption is made that the present invention has located a header. The present invention, in one embodiment, then refines the header position by looking for the nearby window with the highest separation in amplitudes between ‘0’ bits and ‘1’ bits. Decoding then proceeds to walk over sliding windows of the data portion, computing amplitudes relative to the reference amplitudes from the header and outputting the relevant discrete symbol (after undoing the Gray code).
[0058] DPSK Decoding: Differential phase shift keying modulates the phase of the carrier frequency, where the data is encoded in the difference between phases in adjacent symbols. It is significantly more robust to phase drift issues than ordinary phase-shift keying, but it reduces the signal-to-noise ratio (“SNR”) by compounding errors from two symbols.
[0059] Because the sample rate drifts with time, and the signal is aliased across the (drifting) Nyquist boundary, the phase of the carrier frequency drifts with time. This is approximated by the relation θ′.sub.i=θ.sub.i+iAθ where θ.sub.i is the original phase at symbol i, θ′.sub.i is the received phase, and Δθ is the average phase drift.
[0060] The presence of this accumulating phase drift causes PSK data reception to fail. Even if it is carefully estimated, small variances will compound over the length of a message. By contrast, small variances in the phase drift have only small impacts on DPSK due to its differential nature.
[0061] To find the PSK-encoded header, the present invention, in one embodiment, scans across the filtered data and fits sliding windows to the ideal preamble. To fit a sliding window to the PSK preamble, this embodiment of the present invention computes the phase of each symbol:
The present invention then computes the estimated phase drifts φ.sub.i=φ′.sub.i−θ.sub.i (this assumes that the window matches the header). If these were not angles, one could use least-squares to estimate Δθ and θ.sub.0 in the linear relation φ.sub.i=iAθ+θ.sub.0. However, circular wraparound prevents this approach from working. Instead, the present invention makes an initial estimate using
where k=50, and then applies linear least-squares to
where [ ] is the rounding operator. This reconstructs the “unwrapped” phase values, allowing the invention to recover the phase drift accurately. It works as long as the initial estimate is off by no more than 2π/k radians. Once an estimated phase drift has been recovered, the present invention can binarize the adjusted phases in the window and match them to the PSK payload. As before, the present invention declares that it has found a header if fewer than 8 bits are in error. Decoding a DPSK transmission consists of walking over the sliding windows of the data portion, computing phases relative to the prior window, and outputting the corresponding symbol.
[0062] FSK Decoding: Frequency shift keying switches between different carrier frequencies depending on the symbol. The FSK header consists of a fixed random permutation h of [0, 1, 2, . . . , 49] modulo the number of levels, ensuring an even distribution of frequencies for sampling purposes. To find the header, one embodiment of the present invention fits sliding windows to the preamble by comparing the amplitudes of each carrier frequency c in each symbol i, producing a sequence of vectors v.sub.i. To account for variable frequency sensitivity and non-uniform background noise, a set of reference vectors is constructed
r.sub.c,c=
r.sub.c,d=
where
[0063] Evaluation: An experiment was run on certain embodiments of the invention to validate the robustness and performance of the invention. In order to ensure that the invention was not over-optimizing the system, the system was tested with three different Nexus 5s, one of which was never used during the development of the system. The three devices were calibrated by obtaining their baseline comb-sample rate offsets, but otherwise did not engage any device-specific configuration.
[0064] The experiment used the piezo throughout as the transmission device 20, pressing the phone 10 down lightly to ensure physical contact or acoustic touching 100. Data was collected for each combination of modulation (FSK, ASK, PSK, DPSK), bits per symbol (1, 2, 3, 4, with the exception of FSK which only went up to 3), and symbol rate (500, 1000, and 1250). In total, there were 45 conditions. For each condition, 25 different codes were generated, and each code was tested 6 times, for a total of 150 data points per condition per device. Thus, a total of 20,250 instances over the course of 30 hours of continuous data transfer were collected, representing a total of 44,550,000 of vibro-acoustically transmitted data.
[0065] An additional 2,334 instances of data were collected in an aircraft (Boeing B787) using one of the devices. An analysis of the bit errors in this set showed no significant differences, so it was merged with the main set. Two hundred seventy (270) instances of data also were collected with various music playing at maximum volume out of the phone's own speakers. An analysis of this data showed a significant increase in bit errors for ASK decoding, but no significant effect on FSK or DPSK techniques, suggesting that phase- and frequency-based modulations of the present invention are inherently robust to acoustic noise.
[0066] Results: Of the 22,584 packets transmitted, 78 (0.3%) were not detected at all (i.e. the header was not found). The vast majority (73) of these errors were from the 1250 Hz symbol rate condition; thus, for the 500 Hz and 1000 Hz conditions the table shown in
[0067] The 95th percentile bit error rates are useful for calculating the effective bit rate, after subtracting the necessary error correction overhead. A typical analysis uses a correction percentage of twice the error rate, and coding schemes such as Turbo Codes achieve this rate in practice.
[0068] The best overall condition was FSK modulation at 3 bits per symbol, 1000 symbols per second, which is a raw bit rate of 3000 bits/sec. With a 95th percentile BER of 16.20%, the effective bit rate is 2028 bits/sec. In other words, the present invention can transmit error-corrected data at 2028 bits/sec and expect that 95% of packets will be received successfully. For DPSK, the best condition was 4 bits per symbol and 1000 symbols per second. With a BER of 26.65%, this is an effective bit rate of 1868 bits/sec. With ASK, the best condition was 2 bits per symbol and 1250 symbols/sec (BER 17.80%), for an effective bit rate of 1627 bits/sec.
[0069] For reference, Ripple—the fastest IMU-based transmission technique that could be found in the literature—used 80th percentile BER (i.e. 20% of packets lost) to obtain an effective bit rate of 196.6 bits/sec. Viband was 155 bit/sec at 80th percentile BER. Neither paper reports 95.sup.th percentile BER, which would yield worse results. See Laput, G., Xiao, R. and Harrison, C. (2016), ViBand: High-Fidelity Bio Acoustic Sensing Using Commodity Smartwatch Accelerometers, In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16)(ACM, New York, N.Y., USA, 321-333. DOI: https://doi.org/10.1145/2984511.29845820; Roy, N., Gowda M. and Choudhury R. R. (2015), Ripple: communicating through physical vibration; In Proceedings of the 12th USENIX Conference on Networked Systems Design and Implementation (NSDI'15)(USENIX Association, Berkeley, Calif., USA, 265-278).
[0070] Non-limiting Example Uses: The overarching method embodied in the present invention is illustrated in
[0071] Payments are perhaps the most straightforward example use. 100 ms, or around 200 bits) would be sufficient to transfer a unique transaction identifier and ephemeral encryption key, allowing the transaction to be completed securely over a WiFi or cellular connection. Devices and systems according to the preset invention are significantly harder to spoof compared to wireless techniques like NFC (interception of signals is possible but would require specialized equipment for long range vibrometry). An attacker would also have to be in physical contact with the phone or reader (as opposed to merely being nearby).
[0072] Pairing devices according to the present invention has significant advantages over techniques that are currently in use. Today, pairing devices using technologies like Bluetooth is a cumbersome affair, often requiring expiring PINs to increase security. Part of the problem with wireless methods is the lack of guarantees that devices are actually next to one another (and not e.g., in a neighbor's apartment). Even “near field” technologies such as NFC have trouble disambiguating between true touch and being nearby, which is problematic in e.g., busy retail settings. Devices and systems designed according to the present invention offers stronger guarantees of physical proximity, as well as a pairing interaction that is much more explicit and intuitive: devices must be physically held together (e.g., a smartphone, and Bluetooth speaker).
[0073] Permanent device-device pairing is not required for many use cases, especially interactions that might only last for minutes or hours. One example of this type of short-lived pairing is guest WiFi access, where a user could touch the access point to confirm they are inside e.g., a cafe, which transmits a one-time token for temporary internet access. One such scenario is illustrated in
[0074] Another class of pairing occurs with semi-public, shared devices. A canonical example are office printers 20J. As shown in
[0075] There are also instances where a function or process on a smart device 10 might wish to be transferred to another. For example, transferring a call one took on a conference room speakerphone 20K to a smartphone 10 (
[0076] The present invention is especially useful for rapidly launching companion apps on smartphones. For example, touching one's smartphone 10 to a smart thermostat 20L launches the manufacturers app, offering touchscreen controls and advanced settings (see
[0077] Although the present invention may not be suitable for transmitting very large files, it could transmit server details (e.g., IP/port/password) or a URL that allows for download over faster means, such as WiFi of cellular. This could allow, for instance, a user to download a menu from a restaurant window 20M that has a transmitter 20D on it; see
[0078] In one embodiment, the current invention is limited to the receipt of data by the smart device 10, and not the transmission of data from the smart device 10. In other embodiments, the invention is combined with another transmission method (e.g., Wi-Fi, Bluetooth, cellular) for many applications. The preferred embodiment of the present invention involves pressing a smart device 10 to a transmission device 20 directly. In some embodiments of the present invention placing transducers/transmitters elsewhere (e.g., corner of a TV 20E), results in damping and multipath obfuscation of the signal significantly, and dramatically increased bit error rate. Nonetheless, even in those embodiments, the present invention is able to use the resonant frequency of the gyroscope 40 to transmit data through some physical objects, albeit at a much lower bandwidth.
[0079] Finally, the vibratory signals used in the present invention are very subtle, almost imperceptible to human touch. As such, poor acoustic coupling will render the signal unrecoverable. For example, some smartphone cases may reduce the signal by 10 dB or more and subsume the signal into the noise floor. Also, as there are innumerable IMU models 30 and vendors, different embodiments of the present invention may need to be adjusted to each model and vendor. That said, it will be obvious to one skilled in the art that the fundamental principles of operation should be applicable to all smart devices 10 that are equipped with gyroscopes 40.