MULTI-CHANNEL BINAURAL RECORDING AND DYNAMIC PLAYBACK

Abstract

Methods and systems are provided for enhanced audio experiences in VR/AR applications. The apparatuses of this disclosure are adapted to record multiple binaural stereo pairs and play back select binaural pairs corresponding to users' head positions. A substantially spherical microarray is utilized in various embodiments for recording multiple binaural stereo pairs. A VR/AR headset is further adapted to track a user's head positions and dynamically play back binaural sound pairs corresponding to the head positions.

Claims

1. An apparatus for binaural sound recording, comprising: a substantially spherical microphone array adapted to record multiple binaural sound pairs; and a multi-channel recording processor adapted to process the multiple binaural sound pairs thereby producing multiple pairs of binaural audio, wherein the substantially spherical microphone array comprises a substantially spherical frame and a plurality of microphones affixed to the surface of the frame, wherein each of the plurality of microphones is paired with a microphone on an orthodrome at its opposite side for recording binaural sound pairs.

2. The apparatus of claim 1, wherein the substantially spherical frame is a solid sphere.

3. The apparatus of claim 2, wherein the diameter of each orthodrome of the frame is substantially the distance in-between a pair of human ears.

4. The apparatus of claim 3, wherein the plurality of microphones comprises 18 microphones.

5. The apparatus of claim 3, wherein the plurality of microphones comprises 32 microphones.

6. The apparatus of claim 1, wherein the multi-channel recording processor further comprises a multi-channel amplifier adapted to filter sound signals.

7. A virtual reality and augmented reality (VR/AR) system for providing enhanced audio experiences to a user, comprising: a VR/AR headset adapted to track the user's head positions; the apparatus of claim 1; and a VR/AR ear set connected to the VR/AR headset and adapted to retrieve head positions of the user from the VR/AR headset, wherein the VR/AR ear set is further adapted to receive the multiple binaural sound pairs and dynamically reproduce a binaural sound pair corresponding to the head position.

8. The VR/AR system of claim 7, wherein the VR/AR headset and VR/AR ear set are physically joined.

9. The VR/AR system of claim 17, wherein the VR/AR headset and VR/AR ear set are wirelessly connected.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG. 1 depicts a substantially spherical microphone array according to one embodiment of this disclosure.

[0021] FIG. 2 depicts in a transparent view of a substantially spherical microphone array according to one embodiment.

[0022] FIG. 3 shows a headphone playback unit according to another embodiment.

[0023] FIG. 4 illustrates on the top certain head positions, and on the bottom the corresponding pairs of microphones viewed from different angles on a substantially spherical frame according to one embodiment.

[0024] FIG. 5 illustrates on the top certain other head positions, and on the bottom the corresponding pairs of microphones viewed from different angles on a substantially spherical frame according to another embodiment.

[0025] FIG. 6 illustrates on the top certain additional head positions, and on the bottom the corresponding pairs of microphones viewed from different angles on a substantially spherical frame according to another embodiment.

[0026] FIG. 7 illustrates on the top certain further head positions, and on the bottom the corresponding pairs of microphones viewed from different angles on a substantially spherical frame according to one embodiment.

[0027] FIG. 8 outlines a method for playing back binaural sound pairs according to one embodiment.

DETAILED DESCRIPTION OF THE VARIOUS EMBODIMENTS

[0028] The recording and playback systems of this disclosure are adapted to record multiple binaural stereo pairs and play back select binaural pairs corresponding to user's head positions. The binaural surround sound recording and dynamic playback methods in various embodiments enhance surround sound audio experience of a user in VR/AR applications.

Binaural Sound Recording

[0029] An exemplary system of this disclosure includes a spherical microphone array for collecting sound in the vicinity of each microphone in the array. The spherical microphone array in one embodiment comprises a substantially spherical frame and a plurality of microphones affixed to the frame. Referring to FIG. 1, the spherical frame is shown in a three-dimensional (3-D) perspective view. On the outer-surface of the spherical frame are affixed a plurality of microphones. The microphones are shown as protrusions dubbed over the entire surface of the sphere. Each microphone is positioned, and thus trackable, in a 3-D coordinate system. The 3-D coordinate system is a spherical coordinate system in one embodiment, and a 3-D cartesian coordinate system in an alternative embodiment. An illustration of the 3-D coordinate system is shown in the center of the spherical frame in FIG. 1.

[0030] Each of the plurality of microphones is paired with a microphone on an orthodrome at its opposite side according to one embodiment. Each microphone pairs are designed to record binaural sound pairs for the corresponding orientation or angle of a user's head, where the distance in-between the two paired microphones is substantially the same as the distance in-between the ears. That is, the diameter of each orthodrome of the spherical frame is substantially equal to the distance in-between a pair of human ears. Referring to FIG. 3, the distance in-between a pair of human ears in one embodiment is 12 cm. In alternative embodiments, variations are provided to accommodate different head sizes or ear distances in the population of users.

[0031] Referring to FIG. 2, an example of the placement of microphones on the spherical frame is shown in another illustration. A 3-D coordinate system is depicted as well at the center of the spherical frame. Each microphone in the spherical microphone array is addressable or trackable by its coordinates in the 3-D coordinate system; e.g., (a, b, c) in a 3-D cartesian coordinate system according to one embodiment, or (r, 0, (p) in a spherical coordinate system according to another embodiment.

[0032] The spherical frame is made of rubber, silicone, metal, plastics, or other composite materials in various embodiments. Microphones affixed on the spherical frame are of a variety of specifications suitable for various recording environment.

[0033] Referring to FIGS. 4-7, the positions of the microphones on the spherical frame are mapped to different orientations of a human head, including for example, from front facing to rear facing, from left side facing to right side facing, from upward facing to downward facing, as well as obliquely upward facing or obliquely downward facing orientations. These positions are tracked specifically using 3-D coordinates as discussed above.

[0034] In the exemplary arrangement shown in FIG. 2, 18 microphones are affixed on the spherical frame. The microphone array thus may record up to 9 channels of binaural audio. The number of microphones may be varied depending on the design of the microphone array in various embodiments. In an alternative embodiment, 32 microphones are affixed on the spherical frame, and up to 16 channels of binaural audio may be produced.

[0035] The system of this disclosure in addition includes a multi-channel recording processor, connected to the spherical microphone array. The multi-channel recording processor is adapted to produce multiple pairs of binaural audio from the collected sound signals of the microphone array. The multi-channel recording processor in one embodiment includes a multi-channel (e.g., in a number of N pairs) preamplifier for filtering the sound signals to generate n pairs of audio streams. The recording processor further includes a data acquisition card for sampling in another embodiment.

[0036] The resulting surround binaural sound from the multi-channel recording processor is capable of being played-back dynamically by the system of this disclosure, as discussed below.

Dynamic Playback

[0037] The recording and playback system of this disclosure is capable of binaural playback of the multi-channel surround sound recording in one embodiment. The multi-channel surround sound is dynamically rendered for the listener, delivering to the listener who may be rotating or moving his or her head in the surround sound field a pair of binaural sounds that matches his or her head positions or orientation. This enhances and improves audio experience for VR/AR applications.

[0038] For dynamic playback, the system of this disclosure includes a playback unit and a multi-channel playback processor. The playback unit is adapted to playback binaural sound pairs. In various embodiments, the playback unit is a headphone or earphone of various designs, whether wired or wireless. The playback unit is a VR/AR headset in certain embodiments. In an alternative embodiment, the playback unit includes multiple pairs of speakers configured for surround sound effect in a space.

[0039] The playback unit is connected to the multi-channel playback processor of this disclosure. In a certain embodiment, the playback unit and the playback processor are physically joined. The multi-channel playback processor is adapted to retrieve a head position of the user, and process the multiple pairs of binaural audio based on the head position, and thereby reproducing a binaural sound pair corresponding to the head position. This process of dynamic playback is outlined in FIG. 8.

[0040] Specifically, the playback processor of this disclosure is configured to perform a joint processing of the audio data and head position data (e.g., position and angle of the listener's head). In one embodiment, the playback processor further includes a tracking unit adapted to track head positions and orientation in a 3-D coordinate system. The 3-D coordinate system is a 3-D cartesian coordinate system in one embodiment, and a 3-D spherical system in another embodiment. The tracking unit may be hardware, software, firmware, or a combination thereof in various embodiments. In a certain embodiment, the tracking unit is a VR/AR headset with a tracking camera. In another embodiment, the tracking unit is a smartphone or smart camera device with a gyroscope.

[0041] The method of dynamic playback according to one embodiment is based on relating or comparing the coordinates or referencing points in the tracking unit's 3-D coordinate system to the coordinates or reference points in the spherical microphone array's 3-D coordinate system. Corresponding head positions as defined by one or a set of coordinates are retrieved, and then matched to the respective microphone pairs in the spherical microphone array based on their 3-D coordinates. In alternative embodiments, the microphone pairs and the head positions may be substantially or approximately matched based on the available microphone pairs.

[0042] The playback unit then receives the matched coordinates data, and playback binaural sound pairs based on the matched coordinates.

VR/AR System with Enhanced Audio

[0043] The systems and methods for surround sound recording and binaural playback in this disclosure may be advantageously employed in VR/AR applications.

[0044] In one embodiment, a VR/AR system with enhanced audio is provided, which includes a VR/AR headset adapted to track the user's head positions, a VR/AR ear set connected to the VR/AR headset and adapted to retrieve head positions of the user from the VR/AR headset, and a binaural recording apparatus with a spherical microphone array. The VR/AR ear set is adapted to receive multiple binaural sound pairs and dynamically reproduce a binaural sound pair corresponding to the head position. In an alternative embodiment, the VR/AR headset and the VR/AR ear set are configured jointly as one device.

[0045] In a VR setting, the dynamic rendering by the system of the approximately matched binaural audio pair gives the listener an audio experience that corresponds to and validates the visual experience, thereby delivering a total immersive VR experience as the user rotates his or her head and moves in the space where the multi-channel surround sound recording was made.

[0046] According to a certain embodiment, pre-calibration is performed on the VR/AR system. For example, the multi-channel playback processor first calibrates and build a map of the listener's head orientations that correspond to the microphone pairs' coordinates in the spherical microphone array. The playback processor then detects the head orientation or positions. It retrieves the two soundtracks recorded by the microphone pairs that match the current or then-current positions of the head orientation or position, and sends the corresponding pair of recordings to the playback unitthe VR headphone or ear set in one embodimentfor playback.

[0047] The binaural surround sound recording is achieved by the recording system of this disclosure in another embodiment with multiple spherical arrays strategically positioned in a VR surround sound space.

[0048] The descriptions of the various embodiments, including the drawings and examples, are to exemplify and not to limit the invention and the various embodiments thereof.