Device and method for controlling playback of digital multimedia data as well as a corresponding computer-readable storage medium and a corresponding computer program

Abstract

A device for controlling playback of digital multimedia displays in first and second portion a representation of a part of the multimedia data, where in the first portion a representation of a part of the multimedia data, which is displayed on the second portion, is displayed, and where the first portion operatable in at least a first and a second mode, where in the first mode independently of a position of a touch point a direction and a speed of the playback is controllable in dependence of a change of the touch point, and in the second mode a new playback position is definable in dependence of the position of the touch point,
for displaying in the second portion a marker for indicating a neighbourhood around the playback position, and
for switching the modus of the first portion, when the marker is moved by user input.

Claims

1. A device for controlling playback of digital multimedia data, the device comprising at least a first touch sensitive display portion and a second touch sensitive display portion, wherein the first and second touch sensitive display portions are configured for receiving data input and displaying data as output, wherein the device is configured for displaying, in the first and the second display portions, one or more representations of at least a part of the multimedia data, wherein the multimedia data is represented in a form of a waveform, wherein, in the first display portion, an excerpt representation of a part of the multimedia data is displayed and, in the second display, an entire representation of the multimedia data is displayed, wherein the excerpt representation is displayed in more detail than the entire representation, wherein the part of the multimedia data surrounds a current playback position, and wherein the first display portion is operable in at least a first mode and a second mode, where in the first mode independently of a position of a touch point, a direction and a speed of the playback is controllable in dependence of a change of the touch point, and in the second mode, a new playback position of the multimedia data is definable during playback of the multimedia data at a current playback position of the multimedia data in dependence of the position of the touch point, such that once and as soon as a new playback position is defined during playback of the multimedia data, the device takes over playback from the new playback position defined in dependence of the position of the touch point in sync with the beat-phase of the current playback position of the playback, wherein the device is configured for displaying, in the second display portion, a marker for indicating, on the second display portion within the entire representation of the multimedia data, a neighborhood around the playback position and the marker indicates a region of the entire representation of the multimedia data, wherein the region corresponds to part of the excerpt waveform displayed in the first display portion, and further wherein the device is configured for switching the mode of the first display portion, when the marker is moved by user input.

2. The device according to claim 1, wherein the device is configured for setting the at least one first display portion from the second mode into the first mode in reaction of a user input into the at least one first display portion.

3. The device according to claim 1, wherein the device is configured for displaying the representation in a predefined resolution within the first display portion.

4. The device according to claim 1, wherein the marker is realized as a frame and the device is configured in such a way that the representation, which is displayed on the first display portion, corresponds to the representation, which is displayed in the frame.

5. The device according to claim 1, wherein the device is configured for changing the size of the marker in reaction of a user input via the first or second touch sensitive display portion.

6. The device according to claim 1, wherein the device is configured for displaying, at least in one of the first and second display portion, an indicator for indicating the current playback position within the multimedia data.

7. The device according to claim 1, wherein the device comprises at least two first and at least two second touch sensitive portions, wherein a first pair of first and second portions representations of a first set of multimedia data, and in a second pair of first and second portions representations of a second set of multimedia data is displayed.

8. The device according to claim 1, wherein the device is realized as a portable data processing unit.

9. The device according to claim 1, wherein the marker, displayed in the second display portion, is a frame around the playback position and said frame is a scalable frame.

10. The device according to claim 1, wherein the marker, displayed in the second display portion, is a transparent or semi-transparent overlay superposing the entire representation of the multimedia data in the neighborhood of the playback position.

11. A method for controlling playback of digital multimedia data, the method comprising: providing at least a first touch sensitive display portion and a second touch sensitive display portion, wherein the first and second display portions receive data input and display data output; displaying, in the first and the second display portions, one or more representations of at least a part of the multimedia data, wherein the multimedia data is represented in a form of a waveform, wherein, in the first display portion, an excerpt representation of a part of the multimedia data around a playback position is displayed and, in the second display portion, an entire representation of the multimedia data is displayed, wherein the excerpt representation is displayed in more detail than the entire representation, and wherein the first display portion is operable in at least a first mode and a second mode, where in the first mode independently of a position of a touch point, a direction and a speed of the playback is controllable in dependence of a change of the touch point, and in the second mode, a new playback position of the multimedia data is definable during playback of the multimedia data at a current playback position of the multimedia data in dependence of the position of the touch point, such that, once and as soon as a new playback position is defined during playback of the multimedia data, playback is taken over from the new playback position defined in dependence of the position of the touch point in sync with the beat-phase of the current playback position of the playback; displaying, in the second display portions a marker for indicating, on the second display portion within the entire representation of the multimedia data, a neighborhood around the playback position; switching the mode of the first display portion, when the marker is moved by user input; and scrolling of an excerpt-waveform view starts at the time the playback is taken over at the new playback position, wherein the excerpt-waveform view is associated with the excerpt representation of the part of the multimedia data and displayable in the first display portion.

12. A non-transitory computer readable storage medium storing instructions, which when executed by one or more processors, cause the one or more processors to execute the method according to claim 11.

13. The method according to claim 11, further comprising: changing at least one of the speed and the direction of the playback of the multimedia data by touching or contacting the excerpt-waveform view displayed in the first display portion.

Description

(1) Exemplary embodiments of the invention will now be described in more detail with reference to the appended FIGURE, which show in

(2) FIG. 1 an exemplary user interface of a device for controlling digital music data, where two dominating excerpt-waveform views, and their corresponding entire-waveform views underneath are depicted.

(3) Particular embodiments of the invention will now be described in more detail. The following specification details five novel interaction methods of the touch-controlled inventive controller, each with an explanation in which the method differs from the state-of-the-art in DJ software. While the invention is described with the help of an example where digital music data is controlled, the invention is not limited to controlling digital music data, but the present invention comprises controlling of any multimedia data.

(4) Audio Waveform View and Playback Position Decoupling for In-beat On-the-fly Play-position Relocation

(5) While DJing, during playing back one or more tracks a DJ will intervene, modify or manipulate the play back. For example, a currently playing track is scratched, e.g. speed and/or direction of the play back of the track are changed, or the DJ will continue the currently playing track at a new play back position (on-the-fly play-position relocation).

(6) For DJ interaction for controlling play back of a track, electronic devices with a touch-sensitive user interface, e.g. a touch-sensitive display, are available. On the touch-sensitive user interface one or more waveform views for at least a part of the one or more tracks are displayed. In a preferred embodiment, at least one (excerpt-)waveform view moves through the user interface according to the progress of the play back. Preferably, also a (entire-)waveform view is displayed in addition to the (excerpt-)waveform view. The (entire-)waveform view may be a statically displayed on the user interface.

(7) On-the-fly play-position relocation in DJ software is usually performed by clicking onto the desired new position in the entire-waveform view. Clicking and dragging the excerpt-waveform view usually brings DJ software into scratch mode, in which the new playback speed and direction is determined by the user's subsequent mouse movement.

(8) On a touch-based input device however, touching the entire-waveform view at a desired position for on-the-fly play-position relocation would be too imprecise due to the accuracy shortcomings of the conventional input method. Hence for the touch-based inventive control device 100, a new method for on-the-fly play-position relocation is proposed:

(9) A semi-transparent overlay (excerpt-overlay 102, 104) on the entire-waveform view 106, 108 displays which part of the song is currently shown in the excerpt-waveform view 110, 112. Additionally, the entire-waveform view 106, 108 displays a play cursor 114, 116 indicating the current playback position of the song.

(10) The user can use two finger spreading and pinching gestures on the entire-waveform view 106, 108 to adjust the length of the section that is shown in the excerpt-waveform view 110, 112.

(11) In normal play mode, the section displayed in the excerpt-waveform view 110, 112 moves along with the play cursor 114, 116, making the displayed waveform 118, 120 scroll through the view's screen estate. If the user touches the excerpt-waveform view 110, 112 in normal play mode, the underlying waveform 118, 120 stops scrolling and its play position is now dependent on the user's finger movement (but not from the absolute position of the finger on the excerpt-waveform view 110, 112), enabling the user to scratch the audio. Scratching will continue the playback from the current playback position (at the time of touching the excerpt-waveform view 110, 112), but speed and direction depends from the speed and direction of the finger movement on the excerpt-waveform view 110, 112.

(12) To bring a portion of the waveform that is not centred around the current play cursor 114, 116 position into view, the user can reposition the excerpt-overlay 102, 104. In this seek mode, the excerpt-waveform view 110, 112 is static. By now tapping on the excerpt-waveform view 110, 112, the user can not scratch, but set a new playback position. Due to the higher time-resolution of the excerpt-waveform view 110, 112, the touch-based input method yields enough accuracy for precise play-position relocation.

(13) Once the user has set a new play-position, the control device takes over playback from this position in sync with the beat-phase. At the time the audio engine has taken over playback at the new position, the excerpt-waveform view 110, 112 starts scrolling again, letting the user scratch the audio the next time he touches the excerpt-waveform view 110, 112.

(14) The novelty of this concept is that the user can still scratch audio by touching on the excerpt-waveform view 110, 112, however, by repositioning the excerpt overlay 102, 104 still use the excerpt-views higher resolution over the audio material to precisely set a new playback position.

(15) Dynamic Beat Grid Segmentation Based on an Audio Waveform View's Zoom Factor

(16) Most conventional DJ software displays a visual beat-grid pattern on top of a waveform view. The beat-grid is usually based on an automated analysis of the audio data, and aids to indicate where the beats and the bars are located in the audio material. This can give visual cues to a DJ who manually beat-syncs two tracks. Apart from that, most conventional DJ software uses the beat-grid to auto-synchronize two songs that are playing simultaneously.

(17) A DJ technique that has gained momentum with digital DJing is the so-called cue-point juggling, or cue-point drumming. For this technique, the DJ pre-defines cue-points in a track, and, while performing, repeatedly jumps playback between the positions of the cue-points by pressing the cue-point's corresponding buttons on a hardware controller.

(18) When a number of cue-points are in line with the beats, and correspond to e.g. a kick drum, snare and hi-hat hit in the underlying audio material, the DJ can quite effectively remix a track using the cue-points while staying in synchronisation with the overall tempo and rhythm of his performance. The cue-points are then played very similar to drum samples in a digital drum machine, hence the term cue-point drumming.

(19) The touch-controlled inventive control device introduces a new way of remixing tracks on-the-fly called beat-grid drumming: A DJ can by touching a dedicated widget on the screen switch the excerpt-waveform view 110, 112 into freeze-mode, in which the currently displayed portion of the waveform 118, 120 stops scrolling. The DJ can then play the different parts of the waveform 118, 120, which is visually divided by the beat-grid, by touching the area between two successive beat-grid markers. In this mode, each of the beat-grid lines functions as a virtual cue-point, and each of the areas between the beat-grid lines as a button to trigger playback of the corresponding audio material. As the beat-grid is auto-detected, the DJ does not need to pre-define cue-points in order to use this routine.

(20) An additional novelty of this concept is that the resolution of the beat-grid is dependant on the duration and tempo of the audio material displayed in the excerpt-waveform view 110, 112. Usually a beat-grid in conventional DJ software displays only one marker per beat, and, hence due to the common 4/4 measure in modern dance music, 4 beats per bar. As a user can define the length of the section of the song that is shown in the excerpt-waveform view 110, 112 by a two-finger spreading and pinching gesture on the entire-waveform view 106, 108, it is possible that the displayed waveform 118, 120 represents audio of a duration between less than a bar up to multiple bars.

(21) If more than two bars of audio are visible, the beat-grid is subdivided into one-bar parts. With lesser bars of audio visible, the beat-grid is dynamically subdivided into smaller portions, i.e. depending on the zoom level, additional markers for half, quarter, and eight-beats become visible. Therefore a user can at a high zoom level use the beat-grid drumming technique to trigger parts of the audio material that begin offbeat, whereas on a low zoom level, only audio parts that begin with a beat can be triggered.

(22) Conventional DJ software does not allow for this functionality at all. The concept is novel in two ways: Employing the beat-grid for automated and dynamic cue-point setting, and dynamically refining the beat-grid based on the level of detail in a waveform view.

(23) Loop Selection on an Audio Waveform View Using Two-finger Touch Gestures

(24) Another routine commonly used by DJs in conjunction with digital DJ software is on-the-fly looping of a played song: Many conventional DJ routines include looping one portion of one track while playing another track over the loop, or shortening a currently playing loop over time to achieve tension in a mix.

(25) Conventionally, DJ software provides dedicated software controls to set in- and out-points for loops, and additional shortcuts to set loops of different duration, e.g. 4 bars, 2 beats, or different musically meaningful subdivisions of one beat. The touch-controlled inventive control device does not provide the user with dedicated software controls for setting loops of different lengths, but with only one dedicated loop button that will always engage a loop of a pre-defined length (commonly a four-bar loop, the length of a phrase in modern dance music) from the current playback position when tapped, or release that loop again.

(26) If the user wants to engage a loop of a different length than the four-bars as controlled with the loop button, he can use a two-finger tap gesture on the excerpt-waveform view 110, 112, whereby the distance between the user fingers in conjunction with the current zoom level is defining the length of the loop. The loop is quantized to the granularity of the currently displayed beat-grid.

(27) After the user has engaged a loop, the length of the loop is displayed by a semi-transparent visual overlay on top of the excerpt-waveform view. To disengage the loop, the user can again two-finger tap the overlay, or he can tap the dedicated loop button.

(28) Conventional DJ software does not allow for directly setting a loop visually on a waveform view.

(29) Loop Length Adjustment on an Audio Waveform View Using Touch Gestures

(30) Once a user has engaged a loop over the currently playing audio material, the length of the loop is adjustable. To modify the length of the loop, the user can with a two finger pinch or spread gesture modify the length of the loop overlay on the excerpt-waveform view. The newly set loop length is still quantized to the current beat-grid's level of detail.

(31) Conventional DJ software does not allow for a direct visual adjustment of the length of a loop, but dedicated buttons enable a user to adjust the length of a loop.

(32) Loop Slice Playback on an Audio Waveform View Using Touch Gestures

(33) Once a user has engaged a loop over the currently playing audio material, the user can additionally switch the excerpt-waveform view into freeze-mode. In this mode, the waveform representation of the looped region is spread out over the entire excerpt-waveform view, split into slices by the dynamic beat-grid markers. The duration of each slice is therefore depending on the initial length of the engaged loop. Without further interaction, the playback position keeps looping over the displayed waveform. If the user touches a slice, only this slice keeps looping. If a user uses two fingers, the slices between the two touch points are looping. The user can adjust the looped area by pinch or spread gestures.

(34) Differently from play mode in freeze-mode without an additionally engaged loop, the playback of a slice after a user touched it is not quantized to the beat of the audio material, but starting immediately. Further, if a user lifts the finger(s) from the touch screen, the playback is not resumed from the current playback position, but from the position in the loop where the playhead would currently be had the user not further interacted (flux-mode).

(35) Beat Grid Position Adjustment on an Audio Waveform View Using Touch Gestures

(36) Most conventional DJ software performs beat-grid analysis on the basis of lowlevel onset event and periodicity information from beat tracking on the audio data. These analysis methods generally yield results with a high-enough accuracy for beat-gridding of most standard dance music.

(37) If the beat-pattern of the analyzed music material is, however, non-conformist, the analysis results might be imprecise and even outright wrong. An example for a nonconformist beat-pattern would be that e.g. the kick drum of a drum beat is placed on the musical two, which might result that the beat-grid is off by one beat, i.e. that the downbeatthe musical one of the beatis misrecognized.

(38) The touch-controlled inventive control device provides the DJ with means to correct an inaccurate beat-grid. Similar to what most current DJ software provide, the DJ can tap the tempo of the song in order to correct for a wrongly identified tempo, or half/double the recognized beat-grid's resolution to correct analysis mistakes due to e.g. a half-time drum beat.

(39) On top of that, the touch-controlled inventive control device provides a novel way to correct for a misrecognized downbeat: On a dedicated beat-grid correction screen that can be brought into view, the user can displace the audio waveform of the song against the beat-grid using a swiping gesture. This interaction concept is novel to other DJ software in that the user does not have to delete a misrecognized beat-grid and manually set a new downbeat marker and the beats per minutes, but can reuse the mostly correctly recognized beat-pattern while just intuitively shifting the downbeat marker against the audio waveform.