A method for generating virtual and real composite image data includes: acquiring captured image data capturing an image of a real space as seen from a user's point of view; inputting the captured image data into a trained model, the training model outputting segmentation data segmenting the captured image data into a first region in which a target object is displayed, a second region in which at least a part of the user's body is displayed, and a third region that is other than the first and second regions; and compositing data of the first region and data of the second region with a virtual space image data based on the segmentation data.

An imaging device includes an image sensor, a controller, and an annunciator. The image sensor is configured to capture an object image entering via an optical system. The controller is configured to control focusing operation to focus the object image by the optical system. The annunciator is configured to output focusing sound that has a predetermined frequency characteristic according to the focusing operation. The frequency characteristic of the focusing sound includes a first sound component based on first frequency, and a second sound component based on second frequency that is higher than the first frequency and lower than twice frequency of the first frequency.

Methods, systems, and computer program products for network-based processing and distribution of multimedia content of a live performance are disclosed. In some implementations, recording devices can be configured to record a multimedia event (e.g., a musical performance). The recording devices can provide the recordings to a server while the event is ongoing. The server automatically synchronizes, mixes and masters the recordings. The server performs the automatic mixing and mastering using reference audio data previously captured during a rehearsal. The server streams the mastered recording to multiple end users through the Internet or other public or private network. The streaming can be live streaming.

Disclosed are a sound source focus method and device in which the sound source focus device, in a 5G communication environment by amplifying and outputting a sound source signal of a user's object of interest extracted from an acoustic signal included in video content by executing a loaded artificial intelligence (AI) algorithm and/or machine learning algorithm. The sound source focus method includes playing video content including a video signal including at least one moving object and the acoustic signal in which sound sources output by the object are mixed, determining the user's object of interest from the video signal, acquiring unique sound source information about the user's object of interest, extracting an actual sound source for the user's object of interest corresponding to the unique sound source information from the acoustic signal, and outputting the actual sound source extracted for the user's object of interest.

The present teaching relates to method, system, medium, and implementations for facilitating skill learning. Multimedia data in different modalities are received, wherein such data are recorded based on a performance exhibiting a skill. The data in each of the modalities are analyzed to extract information exhibited in the performance that is relevant to the skill and is used to generate an animated tutoring script. Such generated animated tutoring script is then archived for future access to enable a skill learning session in an augmented reality.

The disclosure relates to a method and apparatus for creating and synthesizing music. The disclosed method comprises obtaining at least one image including at least one object related to at least one first musical instrument, identifying a user input associated with the at least one object, mapping the at least one object to at least one second musical instrument, and generating sound based on the user input and sound data of the at least one second musical instrument.

Disclosed are a sound source focus method and device in which the sound source focus device, in a 5G communication environment by amplifying and outputting a sound source signal of a user's object of interest extracted from an acoustic signal included in video content by executing a loaded artificial intelligence (AI) algorithm and/or machine learning algorithm. The sound source focus method includes playing video content including a video signal including at least one moving object and the acoustic signal in which sound sources output by the object are mixed, determining the user's object of interest from the video signal, acquiring unique sound source information about the user's object of interest, extracting an actual sound source for the user's object of interest corresponding to the unique sound source information from the acoustic signal, and outputting the actual sound source extracted for the user's object of interest.

Methods, systems, and computer program products for network-based processing and distribution of multimedia content of a live performance are disclosed. In some implementations, recording devices can be configured to record a multimedia event (e.g., a musical performance). The recording devices can provide the recordings to a server while the event is ongoing. The server automatically synchronizes, mixes and masters the recordings. The server performs the automatic mixing and mastering using reference audio data previously captured during a rehearsal. The server streams the mastered recording to multiple end users through the Internet or other public or private network. The streaming can be live streaming.

Gesture-controlled virtual reality systems and methods of controlling the same are disclosed herein. An example apparatus includes an on-body sensor to output first signals associated with at least one of movement of a body part of a user or a position of the body part relative to a virtual object and an off-body sensor to output second signals associated with at least one of the movement or the position relative to the virtual object. The apparatus also includes at least one processor to generate gesture data based on at least one of the first or second signals, generate position data based on at least one of the first or second signals, determine an intended action of the user relative to the virtual object based on the position data and the gesture data, and generate an output of the virtual object in response to the intended action.

The invention relates to a system for automatically creating a soundtrack, comprising a camera device (1, 1) for recording a user-generated video, at least one wearable sensor (3, 3), and a control unit (2, 2) in communication with the camera device (1, 1) and the at least one wearable sensor (3, 3). The control unit (2, 2) is adapted to generate the soundtrack based on data gathered from the at least one wearable sensor (3, 3) during the recording of the user-generated video. The invention further relates to a method for automatically creating a soundtrack, computer program product, a computer readable memory storage unit computing arrangement or mobile device (1, 11) for executing the method.