METHOD FOR ANALYZING SOUNDS WITHIN VEHICLE INTERIOR AND TRANSMITTING CORRESPONDING DRUMKIT SOUNDS OVER SPEAKERS OF VEHICLE
20250266031 ยท 2025-08-21
Assignee
Inventors
Cpc classification
International classification
Abstract
A computer-implemented method and computer program product for capturing sounds caused by a user of a vehicle and transmitting corresponding sounds over speakers of the vehicle. The method includes the steps of: (a) connecting a software application to a vehicle network; (b) capturing a user-generated sound caused by an impact of the user on a component of a vehicle interior; (c) analyzing the captured user-generated sound and associating the user-generated sound with a recorded drum sound; and (d) playing the recorded drum sound over the speakers of the vehicle. The computer program is configured to perform the aforementioned steps.
Claims
1. A computer-implemented method for capturing user-generated sounds caused by a user of a vehicle and transmitting corresponding recorded drum sounds over speakers of the vehicle, said method comprising the steps of: (a) connecting a software application to a vehicle network; (b) capturing at least one user-generated sound caused by an impact of the user on a component of a vehicle interior; (c) analyzing the captured user-generated sound and associating the captured user-generated sound with at least one recorded drum sound; and (d) playing the at least one recorded drum sound over the speakers of the vehicle.
2. The computer-implemented method of claim 1, further comprising a memory in which is saved a plurality of data files representing pre-recorded sounds, wherein the step of analyzing the captured user-generated sound comprises comparing the captured user-generated sound with the pre-recorded sounds to determine a best fit between the captured user-generated sound and one of the pre-recorded sounds.
3. The computer-implemented method of claim 1, wherein each of the recorded drum sounds represents a specific user-generated sound corresponding to the impact of the user on the component of the vehicle interior.
4. The computer-implemented method of claim 1, further comprising transmitting the captured user-generated sound to a trained sound model of the software application that performs the analyzing step.
5. The computer-implemented method of claim 4, further comprising analyzing the captured user-generated sound by the trained sound model of the software application.
6. The computer-implemented method of claim 4, further comprising associating the captured user-generated sound by the trained sound model of the software application with at least one drum sound configuration of a plurality of user-configurable drum sound configurations programmed by the user and saved in a memory.
7. The computer-implemented method of claim 6, wherein at least one of the plurality of user-configurable drum sound configurations represents the specific sound corresponding to the impact of a hand of the user striking the component of the vehicle interior.
8. The computer-implemented method of claim 6, wherein at least one of the plurality of user-configurable drum sound configurations represents the specific sound corresponding to the impact of a finger of the user tapping the component of the vehicle interior.
9. The computer-implemented method of claim 6, wherein at least one of the plurality of user-configurable drum sound configurations comprises a snare drum sound.
10. The computer-implemented method of claim 1, wherein the component of the vehicle interior comprises at least one of a steering wheel, a center console, or a floor of the vehicle.
11. The computer-implemented method of claim 1, wherein the recorded drum sound is played over the speakers of the vehicle in real time.
12. The computer-implemented method of claim 1, wherein the recorded drum sound is played over the speakers of the vehicle as an independent drum sound.
13. The computer-implemented method of claim 1, wherein the recorded drum sound is played over the speakers of the vehicle as an overlay for music currently being played over the speakers of the vehicle.
14. The computer-implemented method of claim 1, wherein the recorded drum sound is played on an infotainment system of the vehicle.
15. The computer-implemented method of claim 1, wherein the software application is installed on a mobile device and the impact of the user on the component of the vehicle interior is captured by a microphone of the mobile device.
16. The computer-implemented method of claim 1, wherein the impact of the user on the component of the vehicle interior is captured by a microphone of an infotainment system of the vehicle.
17. A computer program product for capturing user-generated sounds caused by a user of a vehicle interior and transmitting corresponding recorded sounds over speakers of the vehicle, the computer program product being stored in a non-transitory computer-readable recording medium, wherein the computer program product is configured for: (a) connecting to a vehicle network; (b) capturing at least one user-generated sound caused by an impact of the user on a component of a vehicle interior; (c) analyzing the captured user-generated sound and associating the captured user-generated sound with at least one recorded drum sound; and (d) playing the at least one recorded drum sound over the speakers of the vehicle.
18. A vehicle comprising the computer program product of claim 17.
19. A smartphone comprising the computer program product of claim 17.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0006] The drawing figures depict one or more implementations, by way of example only, not by way of limitations. In the figures, like reference numerals refer to the same or similar elements.
[0007]
[0008]
[0009]
[0010]
DETAILED DESCRIPTION
[0011] In the following detailed description, numerous specific details are set forth by way of examples in order to provide a thorough understanding of the relevant teachings. However, it should be apparent to those skilled in the art that the present teachings may be practiced without such details. In other instances, well known methods, procedures, components, and/or circuitry have been described at a relatively high-level, without detail, in order to avoid unnecessarily obscuring aspects of the present teachings.
[0012]
[0013] Regardless of the type of vehicle 100, vehicle 100 has a computer 102 for controlling functions related to the operations of vehicle 100 and an infotainment system 107 including audio speakers 117, a smartphone mirroring system app (e.g., CarPlay), an optional microphone 118, navigation, telematics, internet access, etc. Infotainment system 107 may be considered as forming part of computer 102. An electronic mobile device 200 (otherwise referred to herein as a smartphone) is connected to vehicle computer 102 via a wired or wireless connection (e.g., Wi-Fi, cellular or Bluetooth connection). It is noted that components that are shown in broken lines in
[0014] Vehicle computer 102 includes a processor, a visual display 114 having means for both inputting data and displaying data (e.g., touchscreen), a transmitter/receiver for communicating with mobile device 200 in a conventional manner, and a GPS device 110. For example, vehicle computer 102 may send and receive data via Wi-Fi or 4G/LTE radios built into vehicle 100, by way of example. Mobile device 200 includes a processor, a visual display having means for both inputting data and displaying data (e.g., touchscreen), a transmitter/receiver for communicating with vehicle 100 in a conventional manner, and a microphone 118.
[0015] The interior 109 of vehicle 100 includes at least one component 111, such as a steering wheel, a center console, a dashboard, door panels, speaker surrounds, and/or the floor of the vehicle 100, for example. The microphone 118 of the mobile device 200 can be configured to capture sounds caused by an impact of the user (e.g., striking by a hand, tapping with fingers, tapping with feet, etc.) on a component 111 of the vehicle interior 109.
[0016] Alternatively, sounds caused by an impact of the user (e.g., striking by a hand, tapping with fingers, tapping with feet, etc.) on a component 111 of the vehicle interior 109 can be captured by the microphone 118 of the infotainment system 107 of the vehicle 100.
[0017] Turning back to
[0018] In certain embodiments, users of the vehicle 100 may be identified by program 104 based on paired device data maintained in the memory 112 of program 104. The paired device data can indicate, for example, unique device identifiers of mobile devices that were previously paired with program 104 of the vehicle 100, such that program 104 may automatically reconnect previously connected mobile devices without user intervention.
[0019] In certain embodiments, program 104 can be configured to interact with an AI chatbot model, such as ChatGPT, for example, which can run as a third-party software application (e.g., app) AI 106 on vehicle computer 102 or on the mobile device 200 of the user of the vehicle 100. In other embodiments, the AI model can be a different AI-enabled software application 106 running on vehicle computer 102 or on the mobile device 200 of the user of the vehicle 100. Program 104 can be configured to interact with the AI model or ChatGPT app 106 and use ChatGPT's API to perform an action (e.g., query) specified by the user. Further details of the AI model are described in U.S. patent application Ser. No. 18/201,799 filed on May 25, 2023, which is incorporated by reference herein in its entirety and for all purposes.
[0020] Program 104 can be designed and configured to connect directly to the network of the vehicle 100 and/or a third-party software applications (e.g., other apps) or devices via a direct connection application programming interface (API). For example, program 104 can be configured to interact with other apps installed on vehicle computer 102, such as different apps for social media, messaging, providing maps and navigation, playing music or podcasts, etc., that may be installed on, and be available via the display of, vehicle computer 102 (e.g., via Apple CarPlay or Android Auto), for example, or with other similar apps installed on mobile device 200. In these situations, because these apps are integrated with program 104, program 104 can use each specific app's API to perform an action specified by the user.
[0021] In certain embodiments, program 104 can appear as a selectable icon within Apple CarPlay. Program 104 can be configured to connect the network of the vehicle 100 with the Apple ecosystem (e.g., with the different Apple devices, such as iPhone, iPad, MacBook, Apple Watch, AirPods, Apple TV, Apple CarPlay, etc.) that may be connected and integrated with each other, and that may be accessible within the vehicle 100. Once connected to the Apple ecosystem, program 104 can run in the background of Apple CarPlay, for example.
[0022] The network of the vehicle 100 can include one or more of a vehicle controller area network (CAN) or an Ethernet network, for example. The network of the vehicle 100 can allow, after proper and successful authentication, program 104 to communicate with various systems or components of vehicle 100, such as a vehicle modem (if available), GPS device 110 and various electronic control units (ECUs) configured to interface with program 104. For example, the vehicle ECUs may include a vehicle interior control module configured to monitor and control power control functions, such as infotainment system 107, for example; a radio transceiver module configured to provide radio services and communicate with mobile devices inside vehicle 100; etc.
[0023] Program 104 can be configured to include different components to facilitate access to one or more features of vehicle 100 that may be provided with a default configuration by the manufactures of vehicle 100 and may be configured for changes or customization by the user of vehicle 100. For example, program 104 can include a Data Manager, a Connection Manager, an API Controller, and a Change Manager, as illustrated in
[0024] Program 104 can be configured to access a list of available features of the vehicle 100, including but not limited to initial or default values of the available features of the vehicle 100, before allowing the user of vehicle 100 access to modify or customize the available features of the vehicle 100. Once program 104 has gathered and parsed the initial vehicle data, program 104 can monitor for any changes to the initial vehicle data. For example, program 104 can open a WebSocket API as a two-way interactive communication session between the user's interface of program 104 and a server connected to a two-way interactive communication session between the user's browser and a server to observe, process, and store any and all changes by the user to the initial vehicle data. After the user's modifications to any of the available features of the vehicle 100 are processed and stored in the memory 112 of program 104, program 104 can be configured to synchronize the modified features of the vehicle 100 with the corresponding system or component of the vehicle 100.
[0025] In addition, program 104 can be configured to auto detect any smart devices (e.g., smart phones, smart watches, tablets, etc.) that are compatible with program 104, and that may be accessible within the vehicle 100, and auto connect with these devices as soon as they are setup, without the necessity for the user to set up these devices each time. As another example, program 104 can connect to a smart device via a direct connection application programming interface (API).
[0026] Program 104 can be configured to capture sound within the vehicle interior 109 using microphone 118 of the mobile device 200, for example. The sound within the vehicle interior can be caused by an impact of the user (e.g., striking by a hand, tapping with fingers, tapping with feet, etc.) on a component 111 of the vehicle interior 109. The sounds captured by microphone 118 are transmitted to program 104 installed on the mobile device 200, which analyzes the characteristics of the captured sounds. The characteristics of the captured sounds may include, but are not limited to, the direction of the sound or the source of the sound (e.g., which component 111 of the vehicle interior 109 the sound comes from), or parameters of the captured sound related to frequency, amplitude, volume, tempo, beats per minute (BPM), etc. . . . The program 104 can apply a trained sound model to the captured sounds. The trained sound model can be part of the AI model 106 or part of a different trained sound model programmed in the software application 104 running on the mobile device 200.
[0027] The trained sound model can be configured to apply a Fourier transform analysis to the captured sounds to determine the frequency/amplitude of the sound, the direction of the sound, or the source of the sound (e.g., whether the sound comes from the steering wheel of the vehicle, etc.) or distinguishing parameters of the sound (e.g., frequency, beats, tempo, etc.), for example. For example, the analysis could encompass the steps of comparing the captured user-generated sound with similar pre-recorded user-generated sounds to determine the origin of the captured sound. For example, the software application 104 can identify that the user is drumming on the steering wheel by comparing the captured sound of drumming on a steering wheel with a pre-recorded sample of sound of drumming on the steering wheel. The analysis may rely on the frequency, wavelength, pitch, etc. of the captured sound to perform the analysis. It should be understood that the analysis may involve comparing data files representing sound as opposed to the sound itself. Further details of a trained sound model that applies a Fourier transform analysis to the captured sounds are described in U.S. Patent Application Publication No. 2014/0328487 and U.S. Patent Application Publication No. 2009/0287323, which are incorporated by reference herein in their entireties and for all purposes.
[0028] Alternatively, sounds caused by an impact of the user (e.g., striking by a hand, tapping with fingers, tapping with feet, etc.) on a component 111 of the vehicle interior 109 can be captured by the microphone 118 of the infotainment system 107 of the vehicle 100. The sounds captured by microphone 118 can be transmitted to the processor of the vehicle computer 102, which can analyze the characteristics of the captured sounds in a similar way. After the program 104 analyses and processes the sounds caused by an impact of the user (e.g., striking by a hand, tapping with fingers, tapping with feet, etc.) on a component 111 of the vehicle interior 109, the program 104 assigns the identified user-generated sound to a particular recorded drum sound (e.g., bass drum sound). The recorded drum sound may be the sound of a tom drum, floor tom drum, high tom drum, mid tom drum, snare drum, bass drum, steel drum, hand drum, cymbal, etc. For example, the sound of striking by a hand could be assigned to a bass drum, the sound of tapping with fingers could be assigned to a high tom drum, the sound of tapping with feet could be assigned to a bass drum, etc.
[0029] The program 104 has saved in the memory 112 a plurality of different recorded drum sounds. As an example, saved in the memory 112 of the program 104 is a recorded bass drum sound that is assigned to the sound of a user striking the steering wheel by hand, a recorded snare drum sound that is assigned to the user tapping with fingers on the center console or tapping with feet on the floor, etc.
[0030] The user can also manually train the program 104 on the various tapping sounds generated by the user and their corresponding drum sounds.
[0031] The program 104 then plays the recorded drum sound, in real time, over the speakers 117 of the vehicle.
[0032] The program 104 can play the recorded drum sound over the speakers 117 of the vehicle 100 as an overlay for music currently being played over the speakers 117 of the vehicle 100. It is noted that the computer program 104 runs at a sufficient speed such that any time lag between a user's tap and a generated drum sound may be nearly imperceptible to the listener. For example, the music currently being played over the speakers 117 of the vehicle 100 can be music played by the vehicle infotainment system 107 or by different apps for streaming and playing music. The different music apps can run on the mobile device 200 or on the vehicle computer 112.
[0033] Alternatively, the program 104 can play the recorded drum sound over the speakers 117 of the vehicle 100 as an independent drum sound and without overlaying the drum sound over music.
[0034]
[0035] At the outset it is noted that method 400 uses mobile device 200 along with vehicle 100 to accomplish method 400, however, it should be understood that vehicle computer 102 of vehicle 100 may be omitted and the steps of method 400 may be completed using only mobile device 200. Alternatively, mobile device 200 may be omitted and the steps of method 400 may be completed using only vehicle computer 102 of vehicle 100.
[0036] At step 410 of method 400, mobile device 200 connects to vehicle 100 in either a wired or wireless manner, as described above. If the steps of method 400 are completed by vehicle 100 itself (and without the assistance of mobile device 200), then this step may be omitted.
[0037] Optionally, after the connection between the mobile device 200 and the vehicle 100 is established, program 104 can initiate an authentication of the mobile device 200 and/or the vehicle 100. The authentication can be performed using known methods, such as Transport Layer Security (TLS) 1.2 security protocol, public pinning hash, local certificate pinning, PIN numbers that can be entered by the user or QR codes that can be scanned by the mobile device 200, for example. Further functionality of program 104 can be disabled until after the mobile device 200 and/or the vehicle 100 successfully complete the authentication process (e.g., the mobile device 200 confirms that the saved user and vehicle information is stored in the vehicle 100).
[0038] At step 412 of method 400, the microphone 118 of mobile device 200 or microphone 118 of the infotainment system 107 of the vehicle 100 captures sound within the vehicle interior 109, as described above. The sound within the vehicle interior 109 can be caused by an impact of the user (e.g., striking by a hand, tapping with fingers, pressure applied by finger, fist, or palm, etc.) on a component 111 of the vehicle interior 109. The sounds captured by microphone 118 are transmitted to program 104 installed on the mobile device 200, which analyzes the characteristics of the captured sounds. The characteristics of the captured sounds may include, but are not limited to, the frequency of the sound, the direction of the sound or the source of the sound (e.g., which component 111 of the vehicle interior 109 the sound comes from), or parameters of the captured sound related to the tempo of the music, such as beats per minute (BPM), for example. The program 104 can apply the trained sound model described above to identify the user-generated sound and assign that user-generated sound to a recorded drum sound.
[0039] At step 420, without any user intervention, program 104 associates the captured sounds (i.e., sounds caused by an impact of the user on a component 111 of the vehicle interior 109, and which were analyzed by the Fourier transform analysis (for example)) with at least one recorded drum sound, and plays the recorded drum sound, in real time, over the speakers 117 of the vehicle 100. The program 104 can play the recorded drum sound over the speakers 117 of the vehicle 100 as an overlay for music currently being played over the speakers 117 of the vehicle 100. For example, the music currently being played over the speakers 117 of the vehicle 100 can be music played by the vehicle infotainment system or by different apps for streaming and playing music. The different music apps can run on the mobile device 200 or on the vehicle computer 112.
[0040] Alternatively, the program 104 can play the generated recorded sound over the speakers 117 of the vehicle 100 as an independent drum sound.
[0041] Steps 414, 416 and 418 describe an optional process where a user can customize and configure the program 104.
[0042] At optional step 414, if the program 104 is not yet launched, the user (e.g., driver or passenger of vehicle 100) launches program 104 on mobile device 200 or on vehicle computer 102. Step 414 may occur either prior to a vehicle trip or during the vehicle trip.
[0043] At optional step 416, program 104 displays a plurality of different user-configurable drum sound configurations. For example, each of the user-configurable drum sound configurations can include, but is not limited to, a plurality of different configurable drum sounds (e.g., bass drum sound, snare drum sound, etc.).
[0044] Before displaying to the user, in step 416, the internal listing of user-configurable drum sound configurations, the listing may be sorted, arranged and organized in a certain fashion. As one example for sorting the listing, program 104 checks for the most recently used or the most often used drum sound configurations and automatically organizes the appearances of the drum sound configurations such that the most recently used or the most often used drum sound configurations appear at the top of the list.
[0045] As another example, program 104 can recognize the user of the vehicle 100 (e.g., in case multiple users use the same vehicle 100) and can automatically display the drum sound configurations that were most recently used or are most often used by the recognized user.
[0046] At optional step 418, program 104 receives from the user (e.g., driver or passenger) of the vehicle 100 user input associating, in program 104, each of the user-configurable drum sound configurations with a different impact of the user (e.g., striking by a hand, tapping with fingers, etc.) and/or impact by the user on a different component 111 (e.g., steering wheel, center console, floor, etc.) of the vehicle interior 109. As an example, the user can configure in, and save in the memory 112 of, the program 104 a bass drum sound as sound corresponding to the user's striking the steering wheel by hand, snare drum sound corresponding to the user's tapping with fingers on the center console or the floor, etc. The user input can be in the form of the user selecting an icon or pressing a button on the touchscreen 114 of the user interface of program 104. Alternatively, the user input can be in the form of a voice command by the user over an audio system of the mobile device 200. In certain embodiments, for example, Apple Siri can be expanded to enable user input, selection, and modification of vehicle functions and external functions through the program 104.
[0047] The user-configurable drum sound configurations may be saved in the memory 112 of the mobile device 200, for example. The user's selection can be saved in the memory 112 of the mobile device 200 for either current use or later use.
[0048] The step 418 of configuring the plurality of user-configurable drum sound configurations based on the user's preferences can be executed by program 104 running on the smartphone 200 or on the vehicle computer 100, in real time between the smartphone 200 and the infotainment system of the vehicle 100. The method thereafter proceeds to step 420, which is described above.
[0049] It should be understood that method 400 is not limited to any particular step or sequence of steps.
[0050] The step of configuring the plurality of user-configurable drum sound configurations based on the user's preferences can be executed by program 104 running on the vehicle computer 100 or on the smartphone 200, in real time between the smartphone 200 and the infotainment system of the vehicle 100.
[0051] In the context of the present description, the functions of program 104 may be carried out by a processor. The processor can be understood to mean a machine or an electronic circuit, for example. In particular, a processor can be a central processing unit (CPU), a microprocessor or microcontroller, for example an application-specific integrated circuit or digital signal processor, possibly in combination with a data storage unit for storing program commands, etc. Additionally, a processor can be understood to be a virtual processor, a virtual machine or soft CPU. The program 104 may be stored in the memory of the machine.
[0052] It will be understood that the operational steps are performed by the computers or processors described herein upon loading and executing software code or instructions which are tangibly stored on a tangible, non-transitory computer readable storage medium, such as on a magnetic medium, e.g., a computer hard drive, an optical medium, e.g., an optical disc, solid-state memory, e.g., flash memory, or other storage media known in the art. Thus, any of the functionality performed by the computers or processors described herein described herein is implemented in software code or instructions which are tangibly stored on a tangible, non-transitory computer readable storage medium. Upon loading and executing such software code or instructions by the computers or processors, the computers or processors may perform any of the functionality of the computers or processors described herein, including any steps of the methods described herein.
[0053] The term software code or code used herein refers to any instructions or set of instructions that influence the operation of computers or processors. They may exist in a computer-executable form, such as machine code, which is the set of instructions and data directly executed by a computer's central processing unit or by a controller, a human-understandable form, such as source code, which may be compiled in order to be executed by a computer's central processing unit or by a controller, or an intermediate form, such as object code, which is produced by a compiler. As used herein, the term software code or code also includes any human-understandable computer instructions or set of instructions, e.g., a script, that may be executed on the fly with the aid of an interpreter executed by a computer's central processing unit or by a controller.
[0054] It will also be understood that the terms and expressions used herein have the ordinary meaning as is accorded to such terms and expressions with respect to their corresponding respective areas of inquiry and study except where specific meanings have otherwise been set forth herein. The terms comprises, comprising, includes, including, or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that has, comprises or includes a list of elements or steps does not include only those elements or steps but may include other elements or steps not expressly listed or inherent to such process, method, article, or apparatus. An element preceded by a or an does not, without further constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.
[0055] While the foregoing has described what are considered to be the best mode and/or other examples, it is understood that various modifications may be made therein and that the subject matter disclosed herein may be implemented in various forms and examples, and that they may be applied in numerous applications, only some of which have been described herein. It is intended by the following claims to claim any and all modifications and variations that fall within the true scope of the present concepts.