Robust code / data hiding method against analog transmission (over the air) for digital audio
12166571 · 2024-12-10
Inventors
- Ismail Ertürk (Başiskele, TR)
- Yildiray YALMAN (Körfez, TR)
- Necla Bandirmali Ertürk (Başiskele, TR)
- Mehmet Yakut (Başiskele, TR)
- Cem Cihangir ÜstÜN (Ankara, TR)
- Mahmut Ogurlu (Bayrampaşa, TR)
Cpc classification
G06K19/0728
PHYSICS
H04K1/04
ELECTRICITY
International classification
Abstract
The present invention is relates a robust code/data hiding method against analog transmission (over the air) for digital audio, secret code/data hiding component to digital audio file and extracting the hidden code/data from the covered sound, which a digital audio where a secret code/data is embedded is played from the loudspeakers in any environment and media player, the digital audio is received by the user mobile phone or mobile device microphone without being affected by the distorting effects of the analog transmission medium and the noise in any way, enabling follow-up information based on user interaction and user product experience. The present invention can be used in many areas such as the participation of users in a campaign via mobile phone, increasing the TV/radio viewing rate (rating) & follow-ups, copyright tracking & management, participating in sweepstakes, distributing instant coupons, authentication and delivering advertisements to target customer groups.
Claims
1. A method for embedding a secret code/data in a covered audio file, enabling valuable information acquisition based on user interaction and user product experience, the method comprising: receiving the secret code/data within the covered audio file using a device with a microphone, after the covered audio file is played through a speaker in any music and sound system, without being affected by the distorting effects of the analog transmission environment; generating a marker signal having a pre-determined frequency distinct from the covered audio file; applying a windowing process to the covered audio file to minimize spectral leakage, resulting a window covered audio file; performing a sampling process to the windowed covered audio file to facilitate accurate processing of the as covered audio file in a digital format; conducting gain control to normalize the amplitude of the covered audio file; executing a loading process to prepare the covered audio file for data embedding in a processing unit; triggering a clock operation for synchronization of the embedding process; conducting a specific Fast Fourier Transform (FFT) transaction on the covered audio file; modifying the frequency domain of the covered audio file to embed at least one data bit based on the predetermined specific frequency of the marker signal; applying as specific Inverse Fast Fourier Transform (IFFT) transaction to convert the modified frequency domain back to the time domain; applying a specific filtering process to the modified covered audio file to minimize audible artifacts introduced by the data embedding process; and outputting the processed covered audio file containing the embedded secret code/data, wherein, the at least one data bit is represented by altering specific frequencies corresponding to logic-0 and logic-1 levels.
2. The method according to claim 1, wherein the device used to receive secrete code/data is selected form a mobile phone, computer, television, radio, etc., provided that the device is configured with the microphone.
3. The method according to claim 1, wherein the covered audio file received by the microphone comprises 256 samples.
4. The method according to claim 1, wherein the covered audio file is processed in real time.
5. The method according to claim 4, wherein real-time processing of the covered audio file is not affected from the distortion introduced by the analog air transmission environment from the speaker to the microphone.
6. The method according to claim 1, wherein the method further comprises detecting a specific pointer for synchronization and initiating data extraction based on the detected pointer.
7. The method according to claim 1, wherein the method further comprises dynamically detecting the starting point for data extraction.
8. The method according to claim 1, wherein the at least one data bits is decoded.
9. The method according to claim 1, wherein the covered audio file containing the embedded secret code/data is transmitted as RF, satellite, direct connection, or broadcast.
10. The method according to claim 1, wherein the marker signal comprises 256 samples representing logic-0 level and logic-1 levels at specifically defined frequencies.
11. The method according to claim 10, wherein the logic-0 level and logic-1 level are defined by 19 specific frequency values.
12. The method according to claim 10, wherein the 256 samples include certain sequences representing pointers, 1, 0, and space.
13. The method according to claim 1, wherein the real-time secret code/data decoding is performed by a processor within the device.
14. The method according to claim 1, wherein the device comprises an application icon, a voice receiving icon, and a voice processing icon.
15. The method according to claim 1, wherein the device further comprises a computerized web server, a database unit for a real-time secret code/data-specific functionality, and a data analysis unit, integrated with a cloud network.
Description
DESCRIPTION OF THE FIGURES
(1) The figures for the robust code/data hiding method against analog transmission (over the air) for digital audio, together with a web system, which is integrated with mobile phone (device) interaction and media players are shown below.
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
(13)
(14)
(15) The figures for the sample application areas of the developed robust code/data hiding method against analog transmission (over the air) for digital audio are shown below.
(16)
(17)
(18)
DESCRIPTION OF THE REFERENCE NUMBERS IN THE FIGURES
(19) The reference numbers shown on the figures of the robust code/data hiding method against analog transmission (over the air) for digital audio are described below: 100. Cover audio file 101. 44.100 Hz sampling 102. Windowing 103. FFT (FastFourierTransform) 104. Hidden code/data bits (D.sub.N-1-D.sub.0) 105. Process 106. FFT (Inverse Fast Fourier Transform) 107. Filtering 108. Covered digital audio output 200. Gain control 201. Windowed signal outputs 202. f.sub.M pointer signal input 203. f.sub.1 logic 1 level signal input 204. f.sub.0 logic 0 level signal input 205. Not logic gate 206. Control array, verification test and generator 207. f 44.100 Hz/512 clock (clk) signal 208. QSO shift register output (ShiftOut) 209. Q.sub.M pointer (Marker) output 210. Q.sub.DB code/data/space (Data Bits) output 211. Q.sub.W window output 212. LD load input 300. Valid data outputs 301. Empty frame 400. 256 sampled sound 500. Audio frequency zone in the audible band 501. 19 signal frequency values 600. Sound spectrum outside the working area of the method 700. Cover file (way array) 701. Obtaining 256-sample 702. Creating 256-sample blocks 703. 256-sample 19.464 Hz Marker series creation 704. 256-sample 18.604 Hz 0 series creation 705. 256-sample 20.324 Hz 1 series creation 706. 256-sample Space series creation 707. Marker frame, code/data bits, and space frame adding (ShiftRegister) selector 708. Selecting one of the 4 positions 709. Code/data hidden way array (covered file) 710. Original way FFT 711. Marker added FFT 712. 0 added FFT 713. 1 added FFT 714. Space added FFT 715. Process 800. Frame Markerpart (1 Bit) 801. Frame Country Code part (16 Bit) 802. Frame Firm Code part (16 Bit) 803. Frame Campaign Code part (16 Bit) 804. Frame Cyclic Redundancy Check (CRC) part (8 Bit) 805. Empty Frame part (1 Bit) 900. Microphone (mobile device/phone) 901. 512-sample buffer 902. Obtaining 256-sample blocks (frame0 and frame1) 903. Pointer detection 904. Pointer start detection 905. Code/data bits resolving 906. Code/data bits output 1000. Covered/coded sound V.sub.KS (t) 1001. Narrow band filter (f.sub.M Marker) V.sub.MDB (t) 1002. Narrow band filter (f.sub.1 1) V.sub.1DB (t) 1003. Narrow band filter (f.sub.0 0) V.sub.0DB (t) 1004. Bandpass filter V.sub.BG (t) 1005. Single shot repeat trigger multivibrator 1006. Absolute value and low pass filter V.sub.1DB (t) 1007. Absolute value and low pass filter V.sub.0DB (t) 1008. Absolute value and low pass filter V.sub.BG (t) 1008. Threshold (Th) 1009. Power amplifier (k) 1010. Comparator 1011. Process 1012. 0.5 constant input 1013. Data start (DS) 1014. Data output 1015. Valid Data 1016. Decoding logic component 1017. Reading data (RD) 1018. Data outputs (D.sub.N-1-D.sub.0) 1019. Data integrity information (DOK) 1100. Computerized code/data hiding component 1101. Audio encoder 1102. Cloud network (network) 1103. Data analytics 1104. Big data 1105. Database 1106. Web server 1107. Web browser (mobile device) 1108. Mobile phone (device) application 1109. Hidden code/data decoding component 1110. Mobile phone (device) 1111. Multimedia player 1112. Loudspeaker 1113. Covered sound output 1114. Covered media broadcast over RF, satellite, direct or internet 1200. Covered sound pickup & covered sound processing component 1201. Covered audio analysis and hidden code/data decoding components 1202. Mobile phone (device) screenshot 1203. Covered sound pickup & covered sound processing button 1204. Mobile application icon 1300. Computer 1301. Keyboard 1302. Mouse 1303. Monitor 1304. Web browser 1305. General system management, campaign management and data presentation web panel & webapp 1306. Adding a new campaign (code/dataurl matching definition) 1307. Campaign editing (code/dataurl matching change) 1308. Active Campaigns (data representation) 1400. Process timing (tn) 1500. A customized device that can receive covered sound (appliance) 1600. Covered sound broadcast medium
(20) The invention is generally about obtaining information based on user (audience/audience), user interaction and user product experience from private or public audio or multimedia broadcasts made via computer, television, radio, mobile phone, music and voice systems. In particular, it concerns the realization of code/confidential data transmission over the air (speaker to microphone) in noisy environments within audio or multimedia signals. As explained in detail in
(21) Invention: Within the scope of the general system architecture, which is shown in
(22) For example; the applications of the invention in
(23) The invention technically consists of two main subcomponents. In
(24) Code/Data Hiding Component: After the cover sound file (100), which is transmitted over the air, is obtained in digital environment, numerical data in the frequency band over the hearing frequency limits are superimposed as narrow band frequency regions. The method used is based on the sequential addition of code/data bits to the frequency values on the hearing region boundaries in the spectrum using Fast Fourier TransformFFT (103) and InverseFastFourier TransformIFFT (106) with the sampling frequency of 44100 Hz (101) and above. First, a pilot signal for synchronization is superimposed on the sound segment consisting of 128/256 samples, the secret code/data bits (104) to be added for subsequent sound blocks and then the signals of the control bit are added.
(25) Naturally, the intermittent overlay of frequency components not found in the sound on the spectrum leads to a disruptive effect that can be expressed as sizzling and audible by the human ear, even outside the hearing site boundaries. Therefore, filtering 107 is performed to soften these artificial noises, which become audible due to sudden transitions after the overlay. The general block diagram of the computerized code/data hiding process that can be realized with the use of signal processing tools is shown in
(26) Marker signal frequency is 19464 Hz, numerical buried data signal frequencies are determined as f.sub.1 logic 1 level signal input (203) to the right of this marker signal frequency and f.sub.0 logic 0 level signal input (204) to its left. Each coefficient obtained as a result of the FFT process;
(27) It has a frequency value of 44100/256=172,265 Hz.
(28) Marker signal frequency; (44100/2)(15172,265)=19464 Hz,
(29) The signal frequency values for binary (0 or 1) values that compose the code/hidden data are;
(44100/2)(10172,265)=18604 Hz and (44100/2)(20172,265)=20324 Hz.
(30) The marker frequency, the general audio spectrum, that is, the audio frequency region (500) in the audible band and the working frequency region of the invention, and 19 signal frequency values (501) used in the scope of the invention are presented in
(31) 19 different frequency values and working frequency region of the method selected for overlaying within the code/data hiding method are shown in
(32)
(33) Each frequency scheme (triple group) can be used for a different application. For each frequency scheme, a different named mobile application can be developed, or a mobile application can be used to decode hidden data by using specified signal frequency scheme.
(34) In the receiver application, a Marker signal other than the code/data bits signals to be hidden is used just before the binary code/data bits for easy and reliable code/data analysis; f.sub.M=19464 Hz.
(35) Sine signals are used to the right and left of the Marker signal frequency (5f) for binary code/data bits to be hidden. In other words;
19464 Hz(5171,265 Hz)=18604 Hz for 0 and
19464 Hz+(5171,265 Hz)=20324 Hz for 1.
(36) The 256-sample sine signal is shown in
(37) The pseudo code of the method of hiding the code/data bits into the way audio file data array presented in
(38) TABLE-US-00001 data = 45 file = read_wav(...) fs = 256 index = 0 frameM = 256 samples 19 Khz windowed sinus array (marker) frame0 = 256 examples 18 Khz windowed sinus array (0) frame1 = 256 samples 20 Khz windowed sinus array (1) loop: bit_no = 0 file (index) = file (index) + frameM (Add marker array from index) add: index = index + fs (move index to next frame) if (data (bit_no) == 1) (add frame 1 if the data bit is 1) file (index) = file (index) + frame1 else file (index) = file (index) + frame0 bit_no = bit_no + 1 (move to next code / data bit) if (bit_no<8) goto add (add until the last code / data bit is added) index = index + fs (skip one frame, leave empty) while (file (index) goto loop: (until the wav file reaches the end add as much code / data to the array - many times) write_wav (file) (save encoded file covered sound)
(39) The frame format used in the code/data hiding method is shown in
(40) With this method, 3 codes can be carried in 1 second voice (data can be hidden).
(41) Code/Data Decoding Component: The code/data extraction and decoding method components from over the air covered sound are shown in
(42) Microphone (900) of the mobile device/phone audio input is sampled at 44100 Hz. 512 sample buffers (901) are sampled to start. The first 256 samples of this block are taken and FFT is tested to see if the amplitude in the Marker frequency (19464 KHz) exceeds the predefined threshold value. If the threshold is exceeded, this may be a Marker signal. In this case, the starting point of the reference position of the Marker in the frame must be found. In order to synchronize with the code/data bits in time, FFTs are calculated on the existing 512 block of 64 samples right and left of 128 blocks of the first 256 blocks. Whether the marker frequency amplitude is higher on the right or left, 64 samples are headed in that direction, then FFTs are calculated for 32 right and left of this direction. In which direction the Marker frequency amplitude is higher 32 samples on the right or left, in that direction proceeded. Similarly, progress is continued up to 8, dividing by 2 at each step. As a result, a maximum of 8 samples will have the approximate slip and the location of the Marker (reference point). In this case, the data bits will start 256 samples ahead of the starting point. In addition to existing 512 samples, another 256 samples are collected for each bit and the data bits are detected in similar manner. In other words, some of the code/data bits are in this new blocks.
(43) For the next block of 256 samples from the marker starting sample, FFT is performed and the frequency component amplitudes are examined 5f left and 5f right (18604 Hz and 20324 Hz) of the marker frequency. If amplitude of the component on the left is larger than amplitude of the component on the right and above the default threshold level, the data bit for that block is 0. If amplitude of the component on the right (20324 Hz) is greater than amplitude of the component on the left (18604 Hz) and is above the default threshold level, the data bit of the block is 1. If amplitude of the larger component does not exceed the predefined threshold value, the Marker may be detected incorrectly, the data format is incompatible, or data is hidden in a different system (or the mobile device/phone microphone, the hidden sound source is too far from the speaker being played). In this case, Marker detection process is restarted. If the code/data bit is found and having sufficiently high amplitude, 256 new samples are sampled for the analysis of the next bits, and the process continues until 8 bits are completed. In the 256-element block at the end of 8 bits, both the Marker and the data bits should not be present (code silence interval).
(44) If this condition is also met, the data is considered valid as shown in
(45) An example of this is the Goertzel algorithm. In addition, frequency estimation techniques can be used with different approaches (instead of FFT) (e.g., periodogramspectral density estimation). The pseudo code for the secret code/data extraction component from the overlaid (over-the-air) covered voice is given below (on the next page):
(46) TABLE-US-00002 marker_start = 0 fs = 256 start: cnt = fs / 2 sht = cnt / 2 frame0 = new 256 examples frame1 = new 256 examples Repeat: Af ( ) = FFT (frame0 (0)) (Search marker) search: if (Af (m) <th) frame0 = frame1, (frame1 = new 256 examples) if (keep searching) goto again elsegoto end elseALf ( ) = FFT (frame0) cntsht) (If Marker signal is detected If ARf ( ) = FFT (frame0) cnt + sht) search starting block) if (ALf (m)>ARf (m)) cnt = cntsht elsecnt = cnt + sht sht = sht / 2 if (sht<8) goto search marker_start = cnt, bit_no = 0, data = 0 marker frame0 (0) = frame0 (marker start)... ... ... 256 elements copy 0. bit data frame1 (0) = frame1 (marker start) ... ... ... 256 elements copy bit_loop: ADf ( ) = FFT (frame1 (0)) (... binary 8 bit data is obtained) ifADf (m5)>ADf (m + 5) data (bit_no) = 0 else data (bit_no) = 1 frame0 = frame1 frame (marker_start) = new sample (256) if (bit_no<8) gotobit_loop data: There is embedded code in 8 bit audio process the data ... goto start
(47) As shown in
(48) Mobile Application: The general scheme of mobile device application, including the hidden code/data extraction component, is presented in
(49) Covered voice analysis and hidden code/data decoding (1201) are performed by taking the covered sound via the mobile telephone microphone (900) and processing it with the covered voice receiving and processing component block (1200). Invention's hidden voice analysis and hidden code/data decoding (1201) subcomponents can be realized by means of mobile phone hardware and operating system features, programming tools with libraries especially containing FFT operations. Using the single tap mobile application icon (1204) on the mobile phone (device) screen (1202), the code/data decoding component is activated instantly (in real time) by starting the covered voice receiving. After detection of valid hidden code in the sound mobile device application (1108) interacts with web server (1106). Mobile device application (1108) can be realized on different mobile application development platforms by considering mobile device operating systems (iOS, Android etc.).
(50) Web Management and Presentation Panel: The general web panel, software and sub-components of the data presentation are shown in
(51) By means of the general system management web panel software, campaign management and data presentation (1305), general system management, ad campaign participation and monitoring of the code/data hiding method, sample application model management and data presentation are realized. Campaign definition, which is hiding the code/data equivalent of the cover sound file, is made in the component (code/dataurl matching definition) (1306) and when necessary, campaign editing (code/dataurl matching change) can be made from the component (1307). All active campaigns can be tracked in the active campaigns (data representation) section (1308) to visualize the interactions of users and for data analysis.
(52) Three preferred application structures for the operation and systematic use of the invention in relation to the computerized code/data hiding, resistant to air transmission of the voice are shown in
(53) The general system, which is one of the application areas of the invention, and the preferred advertising campaign application processes are presented in
(54) With another preferred embodiment, the exemplary usage of the invention is shown in
(55) In
APPLICABILITY TO INDUSTRY
(56) The invention can be used in a wide range of industries ranging from security to entertainment, from authentication to copyright protection. Automatically generate TV/radio/channel view statistics, promote broadcast views, analyze radio/TV ad effectiveness and cost analysis, copyright and activity tracking, participation in campaigns, participation in surveys, participation in sweepstakes, participation in voting, coupon distributions, multimedia file stamping and general or customized identity can be used in verification fields and customer (user) interactivity.