Embodiments of the present technology are directed at features and functionalities of a VR/AR enabled digital audio workstation. The disclosed audio workstation can be configured to allow users to record, produce, mix, and edit audio in virtual 3D space based on detecting and manipulating human gestures in a virtual reality environment. The audio can relate to music, voice, background noise, speeches, background noise, one or more musical instruments, special effects music, electronic humming or noise from electrical/mechanical equipment, or any other type of audio.

The present invention is designed to allow easy synchronization of the movement of a computer graphics (CG) model with sound data. The data structure according to an embodiment of the present invention presents a data structure that relates to a computer graphics (CG) model, including first time-series information for designating the coordinates of the components of the CG model on a per beat basis, and the first time-series information is used on a computer to process the CG model.

A system for generating a signal includes a touchpad including touch cells and a touch detection device for detecting the location and intensity of a pressure exerted on the touchpad; a first computer generating a first instruction based on the location and intensity of the pressure; an optical detection device for detecting a movement and/or a position, including optics for capturing images; a second computer for determining a motion parameter based on the captured images and for generating a second instruction based on the parameter; and a signal generator for producing a second signal based on the first instruction or on a first signal extracted from the first instruction, to which there is applied a special effect extracted from the second instruction; or on the second instruction or on a first signal extracted from the second instruction, to which there is applied a special effect extracted from the first instruction.

Technologies are disclosed for improving the efficiency of real-time audio processing, and specifically for improving the efficiency of continuously modifying a real-time audio signal. Efficiency is improved by reducing memory bandwidth requirements and by reducing the amount of processing used to modify the real-time audio signal. In some configurations, memory bandwidth requirements are reduced by selectively transferring active samples in the frequency domain—e.g. avoiding the transfer samples with amplitudes of zero or near-zero. This has particular importance when the specialized hardware retrieves samples from main memory in real-time. In some configurations, the amount of processing needed to modify the audio signal is reduced by omitting operations that do not meaningfully affect the output audio signal. For example, a multiplication of samples may be avoided when at least one of the samples has an amplitude of zero or near-zero.

Systems for a touchless musical instrument, where the touchless musical instrument includes: multiple sensors, where each sensor detects a presence of an object within a predetermined distance of the sensor; an audio speaker; a processor electrically connected to the sensors and the audio speaker; and executes instructions on the process to: receive, from the plurality of sensors, an object detection signal indicating a location of the object with respect to the plurality of sensors; generate an audio output signal based on the location of the object indicated by the object detection signal; and generate an audible output from the audio speaker by transmitting the audio output signal to the audio speaker.

Provided is an information processing device including an acquisition unit that acquires subjective evaluation information from a second user about each performance performed by at least part of a body of a first user moving, a learning unit that performs machine learning on a relationship between the each performance and the corresponding subjective evaluation information and generates relationship information between the each performance and the corresponding subjective evaluation information, and a presentation unit that presents feedback information to the second user based on the relationship information.

Technologies are disclosed for improving the efficiency of real-time audio processing, and specifically for improving the efficiency of continuously modifying a real-time audio signal. Efficiency is improved by reducing memory bandwidth requirements and by reducing the amount of processing used to modify the real-time audio signal. In some configurations, memory bandwidth requirements are reduced by selectively transferring active samples in the frequency domain—e.g. avoiding the transfer samples with amplitudes of zero or near-zero. This has particular importance when the specialized hardware retrieves samples from main memory in real-time. In some configurations, the amount of processing needed to modify the audio signal is reduced by omitting operations that do not meaningfully affect the output audio signal. For example, a multiplication of samples may be avoided when at least one of the samples has an amplitude of zero or near-zero.

A virtual-musical-instrument-based audio processing method is provided. In the method, a video is played. A virtual musical instrument is displayed in the video when the virtual musical instrument is matched with at least one musical instrument graphic element in the video. Played audio of the virtual musical instrument is outputted according to interactions with the at least one musical instrument graphic element matched with the virtual musical instrument in the video. Apparatus and non-transitory computer-readable storage medium counterpart embodiments are also contemplated.

A method for controlling aspects of an artistic performance with a motion capture system includes modeling movements of a performer with a biomechanical skeleton, selecting parent and child segments on the biomechanical skeleton, positioning motion capture sensors on a motion capture subject at positions corresponding to the parent and child segments, selecting actions associated with movements of the child segment according to positions of the parent segment within at least two predefined spatial zones, executing actions in a first action group for the child segment when the parent segment is in a first spatial zone, and executing actions in a second action group for the child segment when the parent segment is in a second spatial zone.

Provided is a musical instrument controller capable of accurately controlling a musical sound parameter. This musical instrument controller includes: a reception means for receiving, from a musical performance device, a sound emission start signal transmitted on the basis of a musical performance operation; a sensor for detecting an amount of displacement from a reference position; and a control means for generating a control signal on the basis of the amount of displacement from the reference position and transmitting the control signal to a sound generation device. The control means sets the reference position on the basis of the sound emission start signal received from the musical performance device.