METHOD FOR PROVIDING A SPATIALIZED SOUNDFIELD

Abstract

A signal processing system and method for delivering spatialized sound by optimizing sound waveforms from a sparse array of speakers to the cars of a user. The system can provide listening areas within a room or space, to provide spatialization sounds to create a 3D audio effect. In a binaural mode, a binary speaker array provides targeted beams aimed towards a user's cars.

Claims

1. A spatialized audio processor, comprising: an input port configured to receive an audio signal; at least one automated processor, configured to: determine a spatial relationship between a listener's ears and a real speaker array comprising a plurality of emissive elements; spatially filter the received audio signal to generate a virtual spatial array of virtual audio signals representing spatialized audio, having a larger number of virtual audio signals than a number of the emissive elements of the real speaker array; and map the virtual audio signals to respective emissive elements of the real speaker array, comprising control over an amplitude and delay of at least a subset of the plurality of the virtual audio signals to respective emissive elements of the real speaker array, to produce transducer signals; and an output port configured to convey the transducer signals for the plurality of emissive elements.

2. The spatial audio processor according to claim 1, wherein the received audio signal comprises a stereo audio signal.

3. The spatial audio processor according to claim 1, wherein mapped plurality of virtual audio signals are time-offset based on at least an estimated time difference of arrival at the listener's ears.

4. The spatial audio processor according to claim 1, wherein the received audio signal comprises sounds in a plurality of channels dependent on spatial relationships associated with distinct objects.

5. The spatial audio processor according to claim 1, wherein the at least one automated processor is further configured to predict an audible distortion of the audio signal represented in the transducer signals.

6. The spatial audio processor according to claim 5, wherein the at least one automated processor is further configured to selectively alter the delay of at least one virtual audio signal dependent on the predicted audible distortion.

7. The spatial audio processor according to claim 1, wherein the virtual spatial array of virtual audio signals comprises at least 12 virtual audio signals, and the real speaker array comprises between 2 and 6 emissive elements.

8. The spatial audio processor according to claim 1, wherein the virtual spatial array of virtual audio signals comprises two non-overlapping groups of 6 adjacent virtual audio signals, which are respectively mapped to define 2 transducer signals for two emissive elements.

9. The spatial audio processor according to claim 1, further comprising amplifiers configured to drive the emissive elements of the real speaker array from the transducer signals.

10. The spatial audio processor according to claim 1, wherein the audio signal is received in conjunction with a video signal associated with the audio signal.

11. The spatial audio processor according to claim 1, wherein each emissive element is mapped to a non-overlapping subset of the virtual spatial array of virtual audio signals with respect to the other emissive elements.

12. The spatial audio processor according to claim 1, wherein the at least one automated processor is further configured to cancel cross-talk.

13. The spatial audio processor according to claim 1, wherein the at least one automated processor is further configured to convolve ***** with a head related transfer function.

14. The spatial audio processor according to claim 1, wherein the at least one automated processor is configured to track a movement of the listener's ears over time.

15. A spatialized audio method, comprising: receiving an audio signal; determining a spatial relationship between a listener's ears and a real speaker array comprising a plurality of emissive elements; spatially filtering the received audio signal to generate a virtual spatial array of virtual audio signals representing spatialized audio, having a larger number of virtual audio signals than a number of the emissive elements of the real speaker array; mapping the virtual audio signals to respective emissive elements of the real speaker array, comprising control over an amplitude and delay of at least a subset of the plurality of the virtual audio signals to respective emissive elements of the real speaker array, to produce transducer signals; and outputting the transducer signals for the plurality of emissive elements.

16. The spatialized audio method according to claim 15, further comprising performing a head related transfer function convolution.

17. The spatialized audio method according to claim 15, further comprising combining a plurality of the virtual audio signals into a transducer signal for each of the plurality of emissive elements.

18. The spatialized audio method according to claim 15, further analyzing the transducer signals for peaks associated with a predicted amplitude-related distortion by a respective emissive element, and modifying the mapping of the virtual audio signals to reduce the predicted amplitude-related distortion by the respective emissive element.

19. A computer readable medium storing non-transitory instructions for controlling a programmable processor to spatialize audio, comprising: instructions for spatially filtering a received audio signal to generate a virtual spatial array of virtual audio signals representing spatialized audio, having a larger number of virtual audio signals than a number of the emissive elements of the real speaker array; instructions for mapping the virtual audio signals to respective emissive elements of the real speaker array, comprising control over an amplitude and delay of at least a subset of the plurality of the virtual audio signals to respective emissive elements of the real speaker array, to produce transducer signals; and instructions for modifying the mapping dependent on a predicted distortion by the real speaker array.

20. The computer readable medium according to claim 19, wherein the mapping is selectively dependent on an estimated time difference of arrival of sounds represented in the received audio signal at the ears of a listener.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0222] FIG. 1A is a diagram illustrating the wave field synthesis (WFS) mode operation used for private listening.

[0223] FIG. 1B is a diagram illustrating use of WFS mode for multi-user, multi-position audio applications.

[0224] FIG. 2 is a block diagram showing the WFS signal processing chain.

[0225] FIG. 3 is a diagrammatic view of an exemplary arrangement of control points for WFS mode operation.

[0226] FIG. 4 is a diagrammatic view of a first embodiment of a signal processing scheme for WFS mode operation.

[0227] FIG. 5 is a diagrammatic view of a second embodiment of a signal processing scheme for WFS mode operation.

[0228] FIGS. 6A-6E are a set of polar plots showing measured performance of a prototype speaker array with the beam steered to 0 degrees at frequencies of 10000, 5000, 2500, 1000 and 600 Hz, respectively.

[0229] FIG. 7A is a diagram illustrating the basic principle of binaural mode operation.

[0230] FIG. 7B is a diagram illustrating binaural mode operation as used for spatialized sound presentation.

[0231] FIG. 8 is a block diagram showing an exemplary binaural mode processing chain.

[0232] FIG. 9 is a diagrammatic view of a first embodiment of a signal processing scheme for the binaural modality.

[0233] FIG. 10 is a diagrammatic view of an exemplary arrangement of control points for binaural mode operation.

[0234] FIG. 11 is a block diagram of a second embodiment of a signal processing chain for the binaural mode.

[0235] FIGS. 12A and 12B illustrate simulated frequency domain and time domain representations, respectively, of predicted performance of an exemplary speaker array in binaural mode measured at the left ear and at the right ear.

[0236] FIG. 13 shows the relationship between the virtual speaker array and the physical speakers.

DETAILED DESCRIPTION

[0237] In binaural mode, the speaker array provides two sound outputs aimed towards the primary listener's cars. The inverse filter design method comes from a mathematical simulation in which a speaker array model approximating the real-world is created and virtual microphones are placed throughout the target sound field. A target function across these virtual microphones is created or requested. Solving the inverse problem using regularization, stable and realizable inverse filters are created for each speaker element in the array. The source signals are convolved with these inverse filters for each array element.

[0238] In a second beamforming, or wave field synthesis (WFS), mode, the transform processor array provides sound signals representing multiple discrete sources to separate physical locations in the same general area. Masking signals may also be dynamically adjusted in amplitude and time to provide optimized masking and lack of intelligibility of listener's signal of interest.

[0239] The WFS mode also uses inverse filters. Instead of aiming just two beams at the listener's cars, this mode uses multiple beams aimed or steered to different locations around the array.

[0240] The technology involves a digital signal processing (DSP) strategy that allows for the both binaural rendering and WFS/sound beamforming, either separately or simultaneously in combination. As noted above, the virtual spatialization is then combined for a small number of physical transducers, e.g., 2 or 4.

[0241] For both binaural and WFS mode, the signal to be reproduced is processed by filtering it through a set of digital filters. These filters may be generated by numerically solving an electro-acoustical inverse problem. The specific parameters of the specific inverse problem to be solved are described below. In general, however, the digital filter design is based on the principle of minimizing, in the least squares sense, a cost function of the type J=E+?V

[0242] The cost function is a sum of two terms: a performance error E, which measures how well the desired signals are reproduced at the target points, and an effort penalty BV, which is a quantity proportional to the total power that is input to all the loudspeakers. The positive real number ? is a regularization parameter that determines how much weight to assign to the effort term. Note that, according to the present implementation, the cost function may be applied after the summing, and optionally after the limiter/peak abatement function is performed.

[0243] By varying B from zero to infinity, the solution changes gradually from minimizing the performance error only to minimizing the effort cost only. In practice, this regularization works by limiting the power output from the loudspeakers at frequencies at which the inversion problem is ill-conditioned. This is achieved without affecting the performance of the system at frequencies at which the inversion problem is well-conditioned. In this way, it is possible to prevent sharp peaks in the spectrum of the reproduced sound. If necessary, a frequency dependent regularization parameter can be used to attenuate peaks selectively.

Wave Field Synthesis/Beamforming Mode

[0244] WFS sound signals are generated for a linear array of virtual speakers, which define several separated sound beams. In WFS mode operation, different source content from the loudspeaker array can be steered to different angles by using narrow beams to minimize leakage to adjacent areas during listening. As shown in FIG. 1A, private listening is made possible using adjacent beams of music and/or noise delivered by loudspeaker array 72. The direct sound beam 74 is heard by the target listener 76, while beams of masking noise 78, which can be music, white noise or some other signal that is different from the main beam 74, are directed around the target listener to prevent unintended eavesdropping by other persons within the surrounding area. Masking signals may also be dynamically adjusted in amplitude and time to provide optimized masking and lack of intelligibility of listener's signal of interest as shown in later figures which include the DRCE DSP block.

[0245] When the virtual speaker signals are combined, a significant portion of the spatial sound cancellation ability is lost; however, it is at least theoretically possible to optimize the sound at each of the listener's cars for the direct (i.e., non-reflected) sound path.

[0246] In the WFS mode, the array provides multiple discrete source signals. For example, three people could be positioned around the array listening to three distinct sources with little interference from each others' signals. FIG. 1B illustrates an exemplary configuration of the WFS mode for multi-user/multi-position application. With only two speaker transducers, full control for each listener is not possible, though through optimization, an acceptable (improved over stereo audio) is available. As shown, array 72 defines discrete sounds beams 73, 75 and 77, each with different sound content, to each of listeners 76a and 76b. While both listeners are shown receiving the same content (each of the three beams), different content can be delivered to one or the other of the listeners at different times. When the array signals are summed, some of the directionality is lost, and in some cases, inverted. For example, where a set of 12 speaker array signals are summed to 4 speaker signals, directional cancellation signals may fail to cancel at most locations. However, preferably adequate cancellation is preferably available for an optimally located listener.

[0247] The WFS mode signals are generated through the DSP chain as shown in FIG. 2. Discrete source signals 801, 802 and 803 are each convolved with inverse filters for each of the loudspeaker array signals. The inverse filters are the mechanism that allows that steering of localized beams of audio, optimized for a particular location according to the specification in the mathematical model used to generate the filters. The calculations may be done real-time to provide on-the-fly optimized beam steering capabilities which would allow the users of the array to be tracked with audio. In the illustrated example, the loudspeaker array 812 has twelve elements, so there are twelve filters 804 for each source. The resulting filtered signals corresponding to the same nth loudspeaker signal are added at combiner 806, whose resulting signal is fed into a multi-channel soundcard 808 with a DAC corresponding to each of the twelve speakers in the array. The twelve signals are then divided into channels, i.e., 2 or 4, and the members of each subset are then time adjusted for the difference in location between the physical location of the corresponding array signal, and the respective physical transducer, and summed, and subject to a limiting algorithm. The limited signal is then amplified using a class D amplifier 810 and delivered to the listener(s) through the two or four speaker array 812.

[0248] FIG. 3 illustrates how spatialization filters are generated. Firstly, it is assumed that the relative arrangement of the N array units is given. A set of M virtual control points 92 is defined where each control point corresponds to a virtual microphone. The control points are arranged on a semicircle surrounding the array 98 of N speakers and centered at the center of the loudspeaker array. The radius of the arc 96 may scale with the size of the array. The control points 92 (virtual microphones) are uniformly arranged on the arc with a constant angular distance between neighboring points.

[0249] An M?N matrix H(f) is computed, which represents the electro-acoustical transfer function between each loudspeaker of the array and each control point, as a function of the frequency f, where H.sub.p,l corresponds to the transfer function between the l.sup.th speaker (of N speakers) and the p.sup.th control point 92. These transfer functions can either be measured or defined analytically from an acoustic radiation model of the loudspeaker. One example of a model is given by an acoustical monopole, given by the following equation:

[00074]$H_{p,?\left(f\right)}=\frac{\exp [-j2?{\mathrm{fr}}_{p,?}/c}{4?r_{p,?}}$

[0250] where c is the speed of sound propagation, f is the frequency and r.sub.p,l is the distance between the l.sup.th loudspeaker and the p.sup.th control point.

[0251] Instead of correcting for time delays after the array signals are fully defined, it is also possible to use the correct speaker location while generating the signal, to avoid reworking the signal definition.

[0252] A more advanced analytical radiation model for each loudspeaker may be obtained by a multipole expansion, as is known in the art. (See, e.g., V. Rokhlin, Diagonal forms of translation operators for the Helmholtz equation in three dimensions, Applied and Computations Harmonic Analysis, 1:82-93, 1993.)

[0253] A vector p(f) is defined with M elements representing the target sound field at the locations identified by the control points 92 and as a function of the frequency f. There are several choices of the target field. One possibility is to assign the value of 1 to the control point(s) that identify the direction(s) of the desired sound beam(s) and zero to all other control points.

[0254] The digital filter coefficients are defined in the frequency (f) domain or digital-sampled (z)-domain and are the N elements of the vector a(f) or a(z), which is the output of the filter computation algorithm. The filer may have different topologies, such as FIR, IIR, or other types. The vector a is computed by solving, for each frequency f or sample parameter z, a linear optimization problem that minimizes e.g., the following cost function

[00075]$J\left(f\right)={.Math.H\left(f\right)a\left(f\right)-p\left(f\right).Math.}^2+?{.Math.a\left(f\right).Math.}^2$

[0255] The symbol ? . . . ? indicates the L.sup.2 norm of a vector, and ? is a regularization parameter, whose value can be defined by the designer. Standard optimization algorithms can be used to numerically solve the problem above.

[0256] Referring now to FIG. 4, the input to the system is an arbitrary set of audio signals (from A through Z), referred to as sound sources 102. The system output is a set of audio signals (from 1 through N) driving the N units of the loudspeaker array 108. These N signals are referred to as loudspeaker signals.

[0257] For each sound source 102, the input signal is filtered through a set of N digital filters 104, with one digital filter 104 for each loudspeaker of the array. These digital filters 104 are referred to as spatialization filters, which are generated by the algorithm disclosed above and vary as a function of the location of the listener(s) and/or of the intended direction of the sound beam to be generated.

[0258] The digital filters may be implemented as finite impulse response (FIR) filters; however, greater efficiency and better modelling of response may be achieved using other filter topologies, such as infinite impulse response (IIR) filters, which employ feedback or re-entrancy. The filters may be implemented in a traditional DSP architecture, or within a graphic processing unit (GPU, developer.nvidia.com/vrworks-audio-sdk-depth) or audio processing unit (APU, www.nvidia.com/en-us/drivers/apu/). Advantageously, the acoustic processing algorithm is presented as a ray tracing, transparency, and scattering model.

[0259] For each sound source 102, the audio signal filtered through the nth digital filter 104 (i.e., corresponding to the n.sup.th loudspeaker) is summed at combiner 106 with the audio signals corresponding to the different audio sources 102 but to the same n.sup.th loudspeaker. The summed signals are then output to loudspeaker array 108.

[0260] FIG. 5 illustrates an alternative embodiment of the binaural mode signal processing chain of FIG. 4 which includes the use of optional components including a psychoacoustic bandwidth extension processor (PBEP) and a dynamic range compressor and expander (DRCE), which provides more sophisticated dynamic range and masking control, customization of filtering algorithms to particular environments, room equalization, and distance-based attenuation control.

[0261] The PBEP 112 allows the listener to perceive sound information contained in the lower part of the audio spectrum by generating higher frequency sound material, providing the perception of lower frequencies using higher frequency sound). Since the PBE processing is non-linear, it is important that it comes before the spatialization filters 104. If the non-linear PBEP block 112 is inserted after the spatial filters, its effect could severely degrade the creation of the sound beam.

[0262] It is important to emphasize that the PBEP 112 is used in order to compensate (psycho-acoustically) for the poor directionality of the loudspeaker array at lower frequencies rather than compensating for the poor bass response of single loudspeakers themselves, as is normally done in prior art applications.

[0263] The DRCE 114 in the DSP chain provides loudness matching of the source signals so that adequate relative masking of the output signals of the array 108 is preserved. In the binaural rendering mode, the DRCE used is a 2-channel block which makes the same loudness corrections to both incoming channels.

[0264] As with the PBEP block 112, because the DRCE 114 processing is non-linear, it is important that it comes before the spatialization filters 104. If the non-linear DRCE block 114 were to be inserted after the spatial filters 104, its effect could severely degrade the creation of the sound beam. However, without this DSP block, psychoacoustic performance of the DSP chain and array may decrease as well.

[0265] Another optional component is a listener tracking device (LTD) 116, which allows the apparatus to receive information on the location of the listener(s) and to dynamically adapt the spatialization filters in real time. The LTD 116 may be a video tracking system which detects the listener's head movements or can be another type of motion sensing system as is known in the art. The LTD 116 generates a listener tracking signal which is input into a filter computation algorithm 118. The adaptation can be achieved either by re-calculating the digital filters in real time or by loading a different set of filters from a pre-computed database. Alternate user localization includes radar (e.g., heartbeat) or lidar tracking RFID/NFC tracking, breathsounds, etc.

[0266] FIGS. 6A-6E are polar energy radiation plots of the radiation pattern of a prototype array being driven by the DSP scheme operating in WFS mode at five different frequencies, 10,000 Hz, 5,000 Hz, 2,500 Hz, 1,000 Hz, and 600 Hz, and measured with a microphone array with the beams steered at 0 degrees.

Binaural Mode

[0267] The DSP for the binaural mode involves the convolution of the audio signal to be reproduced with a set of digital filters representing a Head-Related Transfer Function (HRTF).

[0268] FIG. 7A illustrates the underlying approach used in binaural mode operation, where an array of speaker locations 10 is defined to produce specially-formed audio beams 12 and 14 that can be delivered separately to the listener's cars 16L and 16R. Using this mode, cross-talk cancellation is inherently provided by the beams. However, this is not available after summing and presentation through a smaller number of speakers.

[0269] FIG. 7B illustrates a hypothetical video conference call with multiple parties at multiple locations. When the party located in New York is speaking, the sound is delivered as if coming from a direction that would be coordinated with the video image of the speaker in a tiled display 18. When the participant in Los Angeles speaks, the sound may be delivered in coordination with the location in the video display of that speaker's image. On-the-fly binaural encoding can also be used to deliver convincing spatial audio headphones, avoiding the apparent mis-location of the sound that is frequently experienced in prior art headphone set-ups.

[0270] The binaural mode signal processing chain, shown in FIG. 8, consists of multiple discrete sources, in the illustrated example, three sources: sources 201, 202 and 203, which are then convolved with binaural Head Related Transfer Function (HRTF) encoding filters 211, 212 and 213 corresponding to the desired virtual angle of transmission from the nominal speaker location to the listener. There are two HRTF filters for each source-one for the left ear and one for the right ear. The resulting HRTF-filtered signals for the left ear are all added together to generate an input signal corresponding to sound to be heard by the listener's left ear. Similarly, the HRTF-filtered signals for the listener's right ear are added together. The resulting left and right ear signals are then convolved with inverse filter groups 221 and 222, respectively, with one filter for each virtual speaker element in the virtual speaker array. The virtual speakers are then combined into a real speaker signal, by a further time-space transform, combination, and limiting/peak abatement, and the resulting combined signal is sent to the corresponding speaker element via a multichannel sound card 230 and class D amplifiers 240 (one for each physical speaker) for audio transmission to the listener through speaker array 250.

[0271] In the binaural mode, the invention generates sound signals feeding a virtual linear array. The virtual linear array signals are combined into speaker driver signals. The speakers provide two sound beams aimed towards the primary listener's earsone beam for the left ear and one beam for the right ear.

[0272] FIG. 9 illustrates the binaural mode signal processing scheme for the binaural modality with sound sources A through Z.

[0273] As described with reference to FIG. 8, the inputs to the system are a set of sound source signals 32 (A through Z) and the output of the system is a set of loudspeaker signals 38 (1 through N), respectively.

[0274] For each sound source 32, the input signal is filtered through two digital filters 34 (HRTF-L and HRTF-R) representing a left and right Head-Related Transfer Function, calculated for the angle at which the given sound source 32 is intended to be rendered to the listener. For example, the voice of a talker can be rendered as a plane wave arriving from 30 degrees to the right of the listener. The HRTF filters 34 can be either taken from a database or can be computed in real time using a binaural processor. After the HRTF filtering, the processed signals corresponding to different sound sources but to the same ear (left or right), are merged together at combiner 35 This generates two signals, hereafter referred to as total binaural signal-left, or TBS-L and total binaural signal-right or TBS-R respectively.

[0275] Each of the two total binaural signals, TBS-L and TBS-R, is filtered through a set of N digital filters 36, one for each loudspeaker, computed using the algorithm disclosed below. These filters are referred to as spatialization filters. It is emphasized for clarity that the set of spatialization filters for the right total binaural signal is different from the set for the left total binaural signal.

[0276] The filtered signals corresponding to the same nth virtual speaker but for two different cars (left and right) are summed together at combiners 37. These are the virtual speaker signals, which feed the combiner system, which in turn feed the physical speaker array 38.

[0277] The algorithm for the computation of the spatialization filters 36 for the binaural modality is analogous to that used for the WFS modality described above. The main difference from the WFS case is that only two control points are used in the binaural mode. These control points correspond to the location of the listener's cars and are arranged as shown in FIG. 10. The distance between the two points 42, which represent the listener's cars, is in the range of 0.1 m and 0.3 m, while the distance between each control point and the center 46 of the loudspeaker array 48 can scale with the size of the array used, but is usually in the range between 0.1 m and 3 m.

[0278] The 2?N matrix H(/) is computed using elements of the electro-acoustical transfer functions between each loudspeaker and each control point, as a function of the frequency f. These transfer functions can be either measured or computed analytically, as discussed above. A 2-element vector p is defined. This vector can be either [1,0] or [0,1], depending on whether the spatialization filters are computed for the left or right ear, respectively. The filter coefficients for the given frequency f are the N elements of the vector a(f) computed by minimizing the following cost function

[00076]$J\left(f\right)={.Math.H\left(f\right)a\left(f\right)-p\left(f\right).Math.}^2+?{.Math.a\left(f\right).Math.}^2$

[0279] If multiple solutions are possible, the solution is chosen that corresponds to the minimum value of the L.sup.2 norm of a(f).

[0280] FIG. 11 illustrates an alternative embodiment of the binaural mode signal processing chain of FIG. 9 which includes the use of optional components including a psychoacoustic bandwidth extension processor (PBEP) and a dynamic range compressor and expander (DRCE). The PBEP 52 allows the listener to perceive sound information contained in the lower part of the audio spectrum by generating higher frequency sound material, providing the perception of lower frequencies using higher frequency sound). Since the PBEP processing is non-linear, it is important that it comes before the spatialization filters 36. If the non-linear PBEP block 52 is inserted after the spatial filters, its effect could severely degrade the creation of the sound beam.

[0281] It is important to emphasize that the PBEP 52 is used in order to compensate (psycho-acoustically) for the poor directionality of the loudspeaker array at lower frequencies rather than compensating for the poor bass response of single loudspeakers themselves.

[0282] The DRCE 54 in the DSP chain provides loudness matching of the source signals so that adequate relative masking of the output signals of the array 38 is preserved. In the binaural rendering mode, the DRCE used is a 2-channel block which makes the same loudness corrections to both incoming channels.

[0283] As with the PBEP block 52, because the DRCE 54 processing is non-linear, it is important that it comes before the spatialization filters 36. If the non-linear DRCE block 54 were to be inserted after the spatial filters 36, its effect could severely degrade the creation of the sound beam. However, without this DSP block, psychoacoustic performance of the DSP chain and array may decrease as well.

[0284] Another optional component is a listener tracking device (LTD) 56, which allows the apparatus to receive information on the location of the listener(s) and to dynamically adapt the spatialization filters in real time. The LTD 56 may be a video tracking system which detects the listener's head movements or can be another type of motion sensing system as is known in the art. The LTD 56 generates a listener tracking signal which is input into a filter computation algorithm 58. The adaptation can be achieved either by re-calculating the digital filters in real time or by loading a different set of filters from a pre-computed database.

[0285] FIGS. 12A and 12B illustrate the simulated performance of the algorithm for the binaural modes. FIG. 12A illustrates the simulated frequency domain signals at the target locations for the left and right cars, while FIG. 12B shows the time domain signals. Both plots show the clear ability to target one ear, in this case, the left ear, with the desired signal while minimizing the signal detected at the listener's right ear.

[0286] WFS and binaural mode processing can be combined into a single device to produce total sound field control. Such an approach would combine the benefits of directing a selected sound beam to a targeted listener, e.g., for privacy or enhanced intelligibility, and separately controlling the mixture of sound that is delivered to the listener's ears to produce surround sound. The device could process audio using binaural mode or WFS mode in the alternative or in combination. Although not specifically illustrated herein, the use of both the WFS and binaural modes would be represented by the block diagrams of FIG. 5 and FIG. 11, with their respective outputs combined at the signal summation steps by the combiners 37 and 106. The use of both WFS and binaural modes could also be illustrated by the combination of the block diagrams in FIG. 2 and FIG. 8, with their respective outputs added together at the last summation block immediately prior to the multichannel soundcard 230.

Example

[0287] A 12-channel spatialized virtual audio array is implemented in accordance with U.S. Pat. No. 9,578,440. This virtual array provides signals for driving a linear or curvilinear equally-spaced array of e.g., 12 speakers situated in front of a listener. The virtual array is divided into two or four. In the case of two, the left e.g., 6 signals are directed to the left physical speaker, and the right e.g., 6 signals are directed to the right physical speaker. The virtual signals are to be summed, with at least two intermediate processing steps.

[0288] The first intermediate processing step compensates for the time difference between the nominal location of the virtual speaker and the physical location of the speaker transducer. For example, the virtual speaker closest to the listener is assigned a reference delay, and the further virtual speakers are assigned increasing delays. In a typical case, the virtual array is situated such that the time differences for adjacent virtual speakers are incrementally varying, though a more rigorous analysis may be implemented. At a 48 KHz sampling rate, the difference between the nearest and furthest virtual speaker may be, e.g., 4 cycles.

[0289] The second intermediate processing step limits the peaks of the signal, in order to avoid over-driving the physical speaker or causing significant distortion. This limiting may be frequency selective, so only a frequency band is affected by the process. This step should be performed after the delay compensation. For example, a compander may be employed. Alternately, presuming only rare peaking, a simple limited may be employed. In other cases, a more complex peak abatement technology may be employed, such as a phase shift of one or more of the channels, typically based on a predicted peaking of the signals which are delayed slightly from their real-time presentation. Note that this phase shift alters the first intermediate processing step time delay; however, when the physical limit of the system is reached, a compromise is necessary.

[0290] With a virtual line array of 12 speakers, and 2 physical speakers, the physical speaker locations are between elements 3-4 and 9-10. If (s) is the center-to-center distance between speakers, then the distance from the center of the array to the center of each real speaker is: A=3s. The left speaker is offset ?A from the center, and the right speaker is offset A.

[0291] The second intermediate processing step is principally a downmix of the six virtual channels, with a limiter and/or compressor or other process to provide peak abatement, applied to prevent saturation or clipping. For example, the left channel is:

[00077]$L_{\mathrm{out}}=\mathrm{Limit}\left(L_1+L_2+L_3+L_4+L_5+L_6\right)$

and the right channel is

[00078]$R_{\mathrm{out}}=\mathrm{Limit}\left(R_1+R_2+R_3+R_4+R_5+R_6\right)$

[0292] Before the downmix, the difference in delays between the virtual speakers and the listener's cars, compared to the physical speaker transducer and the listener's cars, need to be taken into account. This delay can be significant particularly at higher frequencies, since the ratio of the length of the virtual speaker array to the wavelength of the sound increases. To calculate the distance from the listener to each virtual speaker, assume that the speaker, n, is numbered 1 to 6, where 1 is the speaker closest to the center, and 6 is the farthest from center. The distance from the center of the array to the speaker is: d=((n?1)+0.5)*s. Using the Pythagorean theorem, the distance from the speaker to the listener can be calculated as follows:

[00079]$d_n=\sqrt{l^2+{\left(\left(\left(n-1\right)+0.5\right)*s\right)}^2}$

[0293] The distance from the real speaker to the listener is

[00080]$d_r=\sqrt{l^2+{\left(3*s\right)}^2}$

[0294] The sample delay for each speaker can be calculated by the different between the two listener distances. This can them be converted to samples (assuming the speed of sound is 343 m/s and the sample rate is 48 KHz.

[00081]$\mathrm{delay}=\frac{\left(d_n-d_r\right)}{343\frac{m}{s}}*48000\mathrm{Hz}$

[0295] This can lead to a significant delay between listener distances. For example, if the virtual array inter-speaker distance is 38 mm, and the listener is 500 mm from the array, the delay from the virtual far-left speaker (n=6) to the real speaker is:

[00082]$d_n=\sqrt{{\text{.5}}^2+{\left(5.5*\text{.038}\right)}^2}=\text{.541}m$$d_r=\sqrt{{\text{.5}}^2+{\left(3*\text{.038}\right)}^2}=\text{.513}m$$\mathrm{delay}=\frac{\text{.541}-.512}{343}*48000=4\mathrm{samples}$

[0296] At higher audio frequencies, i.e., 12 kHz an entire wave cycle is 4 samples, to the difference amounts to a 360? phase shift. See Table 1.

[0297] Thus, when combining the signals for the virtual speakers into the physical speaker signal, the time offset is preferably compensated based on the displacement of the virtual speaker from the physical one. The time offset may also be accomplished within the spatialization algorithm, rather than as a post-process.

[0298] The invention can be implemented in software, hardware or a combination of hardware and software. The invention can also be embodied as computer readable code on a computer readable medium. The computer readable medium can be any data storage device that can store data which can thereafter be read by a computing device. Examples of the computer readable medium include read-only memory, random-access memory, CD-ROMs, magnetic tape, optical data storage devices, and carrier waves. The computer readable medium can also be distributed over network-coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

[0299] The many features and advantages of the present invention are apparent from the written description and, thus, it is intended by the appended claims to cover all such features and advantages of the invention. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the invention to the exact construction and operation as illustrated and described. Hence, all suitable modifications and equivalents may be resorted to as falling within the scope of the invention.