Wearable sound

Abstract

Vibratory motors are used to generate a haptic language for music or other sound that is integrated into wearable technology. The disclosed system enables the creation of a family of devices that allow people with hearing impairments to experience sounds such as music or other auditory input to the system. For example, a “sound vest” or one or more straps comprising a set of motors transforms musical input to haptic signals so that users can experience their favorite music in a unique way, and can also recognize auditory cues in the user's everyday environment and convey this information to the user using haptic signals.

Claims

1. A system for transforming audio information to a haptic language, comprising: a signal processor (20) configured to receive an audio input (10) and simultaneously generate a plurality of electrical driving signals (30) according to a predefined mapping from audio signals comprising portions of said audio input to each of said plurality of electrical driving signals; a wearable article (40, S1-S6); and a network of a plurality of vibratory motors (M1-M4) incorporated into the wearable article, wherein the plurality of electrical driving signals generated by the signal processor are used to drive the plurality of vibratory motors according to a predefined mapping of audio signals comprising portions of said audio input to a plurality of different regions of the article.

2. The system of claim 1, wherein the wearable article comprises a wearable vest configured to be worn on a user's abdomen.

3. The system of claim 1, wherein the wearable article comprises a strip configured to be worn as a strap around a selected location of a user's body.

4. The system of claim 1, wherein the wearable article comprises a plurality of separate strips each of which is configured to be worn as a strap around a selected location of a user's body.

5. A method for transforming audio information to a haptic language expressed through a wearable article, including a vest or one or more straps, comprising: using a signal processor to receive an audio input and simultaneously generate a plurality of electrical driving signals according to a predefined mapping from audio signals comprising portions of said audio input to each of said plurality of electrical driving signals; and using the plurality of electrical driving signals to drive a network of a plurality of vibratory motors incorporated into the wearable article, wherein the plurality of electrical driving signals generated by the signal processor are used to drive the plurality of electrical vibratory motors according to a predefined mapping of audio signals comprising portions of said audio input to a plurality of different regions of the wearable article.

6. The method of claim 5, further comprising transforming the audio data into the plurality of electrical driving signals by obtaining audio data in a prescribed format; organizing the audio data into tracks; mapping the tracks to one of the plurality of different regions of the wearable article; and using the respective tracks of data to drive the motors in the corresponding region.

7. The method of claim 6, wherein the prescribed format is based on a MIDI (Musical Instrument Digital Interface) format.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is a block diagram of an illustrative embodiment of the inventive system.

(2) FIG. 2 depicts the inventive system integrated into a wearable vest.

(3) FIG. 3 is a photograph of a prototype in which a network of motors is stitched into a piece of fabric.

(4) FIG. 4 is a flowchart of an inventive algorithm for converting audio data into signals for driving a network of vibrating motors incorporated into a wearable vest.

(5) FIGS. 5A, 5B, and 5C depict examples of “strip” layouts for motors, specifically in the form of a back strap, chest strap, and wrist and ankle strap, respectively.

(6) FIG. 6 depicts an example layout of strips (S1, S2, . . . , S6) in various locations on a person. The number of motors on a given strip and at any location on the body may be 3, 4, 5, or more.

(7) FIG. 7 illustrates a haptic driver data stream being input to signal processor the output of which is input to an end user's haptic drivers. This system may be employed to create desired vibratory sensations that are independent of music or that augment the music.

(8) FIG. 8 illustrates a tool that may be employed for recording a “vibration track” and adding this track to sound data, either through a new data stream or further encoding in the audio track.

(9) FIG. 9 illustrates a tool for crafting “augmented audio” experiences and for visualizing vibrations mapped to sites on the human body where the haptic drivers are worn.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

(10) As discussed above, the present disclosure relates to a system, or “sound vest”, that uses vibratory motors to generate a haptic language for music or other sound that is integrated into wearable technology. A technical challenge to creating such a system is to design a system that decomposes auditory input into control signals that can be streamed out to a network of motors. The present inventors have designed a preliminary system that performs entry-level signal processing techniques on the incoming sound in order to determine the spectral profile of the musical input. The motors are then powered based on the magnitude of the spectral power.

(11) A preliminary design of the system enables the use of up to 64 motors to represent the incoming audio. (A revised design utilizes 64 motors on each of the front and back sides of the vest, for a total of 128 motors. For example, each of M1, M2, M3, and M4 in FIG. 2 could represent up to 16 separate motors, for a total of 64 motors on the front side of the vest 40. A similar network of 64 motors could be deployed on the back side of the vest.) The user's entire torso is utilized to create a tono-topic map of the torso—that is, vibratory motors on the left (L) side of the vest are mapped to the left speaker, vibratory motors on the right (R) side of the vest are mapped to the right speaker, vibratory motors on the bottom of the vest are mapped to low frequencies, and vibratory motors on the top of the vest are mapped to high frequencies.

(12) FIG. 1 depicts the basic system including signal processor (20) that receives audio input (10) (e.g., from a microphone (50, see FIG. 2) or audio jack) and transforms the input audio signal into a haptic language for driving (30) a network of motors denoted M1, M2, M3, and M4. The signal processor may also include an analog-to-digital converter (ADC) (22) for digitizing real-time audio signals provided in analog form, memory or storage (24) for storing audio data, executable instructions, and the like; and a voice recognition module (26).

(13) As shown in FIG. 2, the motors M1 through M4 may be integrated into a wearable vest (40) such that M1 and M2 are on the right side of the user's torso and M3 and M4 are on the left side of the user's torso. Moreover, motors M1 and M3 vibrate to represent the higher frequency components of the audio input, whereas motors M2 and M4 vibrate to represent the lower frequency components. It should be understood that in a commercial implementation, there would likely be many more than four motors.

(14) As shown in FIG. 3, using conductive thread and relatively low-cost vibratory motors, an initial prototype was made by stitching thread into fabric, as illustrated.

(15) Applicants are aware of information in the public domain relating to wearable technology with haptic feedback. Documents relating to wearable technology with haptic feedback include the following (copies of these are being submitted herewith in an Information Disclosure Statement): Wojer, http://www.wearabletechworld.com/topics/wearable-tech/articles/361051-woojer-wearable-audio-accessory-that-lets-feel-sound.htm; https://www.kickstarter.com/projects/1382889335/woojer-feel-the-sound; https://www.youtube.com/watch?v=svEJ1CgkERM; Wearable speaker vest/jacket: https://www.google.com/patents/US20090262967?dq=wearable+technology+for+hearing+impaired+to+feel+music&h1=en&sa=X&ei=XIkwU9yXHoe9qAG_oYHgDQ&ved=0CGQQ6AEwBzgK Haptic chair sound enhancing system: https://www.google.com/patents/WO2010033086A1?c1=en&dq=wearable+technology+for+hearing+impaired+to+feel+music&h1=en&sa=X&ei=c4gwU5CaFonUqgGE1YGwDw&ved=0CDcQ6AEwAA; https://www.google.com/patents/US8582789?dq=wearable+technology+for+hearing+impaired+to+feel+music&h1=en&sa=X&ei=c4gwU5CaFonUqgGE1YGwDw&ved=0CEUQ6AEwAg Wearable tactile subwoofer and its use: https://www.google.com/patents/WO2006130559A2?c1=en&dq=wearable+technology+for+hearing+impaired+to+feel+music&h1=en&sa=X&ei=c4gwU5CaFonUqgGE1YGwDw&ved=0CEwQ6AEwAw Hey yaa: a Haptic Warning Wearable to Support Deaf People Communication: http://www.academia.edu/2997186/Hey_yaa_a_Haptic_Warning_Wearable_to_Support_Deaf_People_Communication Gaming: http://www.thebuttkicker.com/gaming/

(16) Some of these translate sound to vibration, but the present disclosure is different in that it goes beyond a simple sensory substitution. The brain is an amazingly “plastic” organ, and we will take advantage of its plasticity by giving the hearing impaired the opportunity to experience music through a haptic “language”. This difference lies in the real-time spectral analysis performed as the music streams into the micro-controller at the heart of the sound vest—the audio streams in and is broken down to a representation of its basic frequency components. Then, each frequency domain is sent to a different part of the body (i.e., if the user is listening to Alvin and the Chipmunks, he will feel a lot of vibration up by his collarbones, and not much down low; listen to Barry White, and it will be the other way around due to the dominance of Mr. White's low frequency components). The inventive system can also represent stereo by streaming to the left side of the body for the left speaker and right speaker to the right side.

(17) As shown in FIG. 4, an inventive algorithm for converting audio data into signals for driving a network of vibrating motors incorporated into a wearable vest comprises the following steps: First, in step S1, audio data in MIDI format is obtained. The data can either be downloaded to the system from a third party provider, or created using recorded audio and an audio production software tool. In step S2, the MIDI data is organized into 4 tracks representing vocals, drums, guitars, and bass. In step S3, the 4 tracks are mapped to different regions of the sound vest; and in step S4 the respective tracks of data are used to drive the motors in the different regions.

(18) Finally, the system may be enhanced by providing wireless links between the signal processor and the motors. In addition, a voice recognition module may be incorporated to enable the system to recognize specific spoken words for selective playback through the motors. For example, the user's name may be specifically recognized and used to signal the user through the motors.

(19) Further Developments

(20) During the course of further developing the system described above, we have discovered that the process of creating musical sensation though tactile stimuli can be improved in several ways:

(21) 1. The audio signals can be improved by converting them into the MIDI (i.e., Musical Instrument Digital Interface) data format, and then reducing the data to a small defined number of tracks, e.g., four (4) tracks representing drums, bass, guitars, and vocal. Other selections could be used as well, depending on the type of music. (Those skilled in the art understand that MIDI is a technical standard that enables a wide variety of electronic musical instruments, computers and other related devices to connect and communicate with one another. A single MIDI link can carry up to sixteen channels of information, each of which can be routed to a separate device.)

(22) 2. Instead of mapping the audio signals to the motors as described above (i.e., mapping higher frequencies to the top of the vest and mapping the lower frequencies to the bottom of the vest), it may be advantageous to map each of the 4 tracks to different parts of the vest. For example, the signals corresponding to vocals can be directed to the mid-section while the drums, bass, and guitar signals are directed to respective regions surrounding the mid-section. This mapping has been found to create less cross-over and less “muddiness” to the vibrations created by the motors.

(23) 3. If the system is unable to convert live audio to MIDI data in real time, it can be advantageous to provide a mode in which the music data is first downloaded and then played back through the vest. In this way, the user can experience the music albeit not in a real-time, “live” setting.

(24) 4. The mapping described above works well with some sorts of music, but not all types. Thus, each type of music needs its own mapping. For instance, a classical piece by Bach might have a region mapped to the organ, a region mapped to the English horn, a region mapped to the flute, and a region mapped to a violin.

(25) 5. Instead of mapping the signals to a vest, it is also possible and perhaps even superior to map the signals to haptic drivers distributed around the body. One way to do this might be to use “strips” containing motors, as depicted in FIGS. 5A-C and 6. For instance, as shown in FIG. 6, this mapping may be accomplished by mapping vocals to a strip on the chest (S1), harmonies like a guitar track to a strip on the back (S2), bass to strips attached to the wrists (S3, S4), and drums to strips attached to the ankles (S5, S6). This mapping has been found to improve a user's experience of the music both from an ability to recognize songs and the overall user experience. In certain situations, it may prove useful to only use a partial implementation, e.g., one strip on one ankle. For instance, many of the hearing impaired enjoy dancing, but have a hard time staying on the beat due to an inability to hear the music. One strip, attached to the ankle and driving the motors in sync with the beat of the music, either using real time audio processing described above or recorded tracks, could prove useful to the hearing impaired. As mentioned, FIGS. 5A, 5B, and 5C depict examples of “strip” layouts for motors in the form of a back strap, chest strap, and wrist and ankle strap, respectively.

(26) 6. Different parts of the body have different abilities to discern vibration. For instance, the wrists are more sensitive than the chest. Accordingly, varying the type of data mapped to each location by sensitivity, and varying the intensity of experience (both by overall amplitude of vibration and frequency) by the sensitivity of the region has also improved the user's ability to distinguish information in the vibrations.

(27) 7. It is possible to play the haptic drivers independently of music. For instance, a musician could create a “vibration track” to augment the auditory experience of the music, driving the haptic sensors directly. This is illustrated in FIG. 7, which shows a haptic driver data stream being input to a signal processor and the signal processor's out being input to an end user's haptic drivers for creating the desired vibratory sensations for the end user. This can happen both in live settings and in recorded settings, where the “vibration track” is recorded and added to the sound data, either through a new additional data stream, or utilizing further encoding in the audio track. Tools have been built for this purpose, and an example is illustrated in FIG. 8. In crafting these “augmented audio” experiences, seeing the visualization of the vibrations mapped to the sites on the human body where the haptic drivers are worn is useful. Tools have been created for this purpose, and are used to assist in the crafting of these experiences. An example of image produced by such a tool is illustrated in FIG. 9. The tool shown in FIG. 8 is like a finger powered drum machine that has sliders that may be moved to increase the amplitude of the pulse, and an open button for loading new recordings into the system. This allows the user to effectively play the haptic system like an instrument. The visualization, or image, depicted in FIG. 9 allows a producer or user to see what the levels are as they experience the haptic system, or as someone else experiences the system. The user could indicate desired changes in the experience, e.g., that the drums need to vibrate more strongly, and the producer could use the tool of FIG. 8 to do that.

(28) Conclusion

(29) The true scope the present invention is not limited to the illustrative embodiments disclosed herein. For example, the inventive “sound vest” described above is suitable for use as an assistive device for the hearing impaired, but it is by no means limited to such uses, since it has many applications for the hearing community. Similarly, the invention is not limited to a vest including vibratory motors as described. Instead, the invention could be implemented using wearable technology located on various, more appropriate under the circumstances, locations of the user's body. Accordingly, those skilled in the art will recognize alternative embodiments and uses within the scope of protection of the following claims.