Enhanced graphical user interfaces for transcription of audio and video messages is disclosed. Audio data may be transcribed, and the transcription may include emphasized words and/or punctuation corresponding to emphasis of user speech. Additionally, the transcription may be translated into a second language. A message spoken by a user depicted in one or more images of video data may also be transcribed and provided to one or more devices.

A tool is needed for music instrument learners to get feedbacks on the correctness of their performances of a particular piece of music. The invention disclosed here is such a tool that can provide music instrument players instant feedback on note pitch and rhythms accuracy based on sheet music. This is accomplished through audio signal processing, sheet music image processing, and conversion of both analogue images and audio signals into standard digital music representation so a comparison can be done and hence a feedback can be presented to the player. An advanced feature will allow users to save the data to the cloud and retrieve later for comparison of progress. It also will allow user to participate an online competition with other players of the same piece of music.

A method for generating a musical score based on user performance during playing a keyboard instrument may include detecting a status change of a plurality of execution devices of the keyboard instrument. The method may include generating a first signal according to the detected status change. The method may include generating a second signal indicating a plurality of timestamps. The method may include determining a tune of the musical score based on the first signal. The method may include determining a rhythm of the musical score based on the second signal. The method may further include generating the musical score based on the tune and the rhythm of the musical score.

An audio processing method, for each time step of a plurality of time steps on a time axis: acquires encoded data that reflects current musical features of a tune for a current time step and musical features of the tune for succeeding time steps succeeding the current time step; acquires control data according to a real-time instruction provided by a user; and generates acoustic feature data representative of acoustic features of a synthesis sound in accordance with first input data including the acquired encoded data and the acquired control data.

Provided are systems and methods for transforming a digital score file into one or more of a plurality of levels of simplified visualization outputs. Methods of the present invention may be computer implemented. Systems of the present invention may include at least one display device, a non-transitory memory having instructions embedded thereon, and a processor in communication with the non-transitory memory and the at least one display device. Systems and methods of the present invention may be configured to receive at least one digital score file, upon which one or more simplification rules are executed, resulting in at least one simplified visualization output. Simplification rules may include, but are not limited to, song length, tempo adjustment, tie, rhythm, harmonic rhythm, and chord. One or more simplified visualization outputs are then provided.

A musical piece inference device includes an electronic controller configured to execute a data acquisition module, an inference module, and an output module. The data acquisition module is configured to acquire target data including an input token sequence that is arranged to indicate at least a part of a musical piece and includes a plurality of bar-line/beat tokens arranged to indicate bar-line/beat positions of at least the part of the musical piece. The bar-line/beat positions are positions of bar lines of at least the part of the musical piece, positions of beats of at least the part of the musical piece, or both. The inference module is configured to, by using a trained inference model, generate an output token sequence indicating a result of an inference with respect to the musical piece from the input token sequence. The output module is configured to output the result of the inference.

A system for generating audio content in dependence upon an input audio track comprising audio corresponding to one or more sound sources, the system comprising an audio input unit operable to input the input audio track to one or more models, each representing one or more of the sound sources, and an audio generation unit operable to generate, using the one or more models, one or more audio tracks each comprising a representation of the audio contribution of the corresponding sound sources of the input audio track, wherein the generated audio tracks comprise one or more variations relative to the corresponding portion of the input audio track.

Disclosed embodiments include systems and methods to teach and analyze a student's progress in learning to play a musical instrument, sing or perform other musical endeavors. Embodiments include the production of an AI score or music AI score which may be an extraction of performance parameters such as a student's tone, speed, rhythm, pitch loudness and other metrics. A musical AI score may also track changes in measured performance while playing a piece of music. Such changes within a piece of music or over time in performing various pieces of music can be valuable is a student's self-assessment or a music teacher's approach tailored to the particular student. A music or signing AI score can help a student to select and to prioritize their music repertoire, focus performance efforts, optimize time schedule, improve appreciation for music, improve the overall quality of music performance and instructor relationship.

In some implementations, a process for communicating data over audio is performed. In one aspect, one or more ordered sequences of audio attribute values that are selected based on a musical relationship between the audio attribute values and associated with data values may be played by a first device and received by a second device. This technique may allow for sound-based communications to take place between devices that listeners may find pleasant.

The present disclosure presents a novel approach to automatic transcription of piano music in a context-dependent setting. Embodiments described herein may employ an efficient algorithm for convolutional sparse coding to approximate a music waveform as a summation of piano note waveforms convolved with associated temporal activations. The piano note waveforms may be pre-recorded for a particular piano that is to be transcribed and may optionally be pre-recorded in the specific environment where the piano performance is to be performed. During transcription, the note waveforms may be fixed and associated temporal activations may be estimated and post-processed to obtain the pitch and onset transcription. Experiments have shown that embodiments of the disclosure significantly outperform state-of-the-art music transcription methods trained in the same context-dependent setting, in both transcription accuracy and time precision, in various scenarios including synthetic, anechoic, noisy, and reverberant environments.