Digital control of the sound effects of a musical instrument

Abstract

The object of the present invention concerns a control device (100) for a generation module (GM) of sound effects (EF.sub.A, EF.sub.B) of a musical instrument (MI), such device comprising computer software configured for: —the capture, using a digital camera (10), of at least one digital image (I) comprising at least one portion of the user's (U) face; —processing of such at least one image (I) to define expression data (D_EX.sub.i, i being a positive integer) containing information relating to facial expressions (EX.sub.a, EX.sub.b) of the user (U); —an analysis of such expression data (D_EX.sub.i) using a predefined first database (DB1) to determine a sound effect data (D_EF.sub.j, j being a positive integer) containing information relating to at least one sound effect (EF.sub.A, EF.sub.B) corresponding to the facial expression (EX.sub.a, EX.sub.b) of the user (U).

Claims

1. Control process for a generation module (GM) of sound effects to control one or several sound effects (EF.sub.A, EF.sub.B, EF.sub.C) of a musical instrument (MI), such process being implemented using computer software comprising the following stages: a capture stage (S1), consisting in capturing, using a digital camera (10), at least one digital image (I) comprising at least a portion of the user's (U) face, such capture module (M1) comprising emission of a source of infrared light (LUX_IR) in the direction of the user's (U) face, and such capture module (M1) comprising application of an infrared filtering (IRF) using an optical filter (30) to only allow the passage of an infrared wavelength band; a processing stage (S2) consisting of processing such at least one image (I) to define at least one expression data (D_EX.sub.i, i being a positive integer) containing information relating to at least one facial expression (EX.sub.a, EX.sub.b, EX.sub.c, EX.sub.d) of the user (U); and an analysis stage (S3) consisting of comparing such at least one expression data (D_EX.sub.i), determined during the processing stage (S2) to a predefined first database (DB1) to determine at least one sound effect data (D_EF.sub.j, j being a positive integer) containing information relating to at least one sound effect (EF.sub.A, EF.sub.B, EF.sub.C), corresponding to facial expression(s) (EX.sub.a, EX.sub.b, EX.sub.c, EX.sub.d) of the user (U), such first database (DB1) comprising a multitude of expression data (D_EX.sub.i) each associated with at least one sound effect data (D_EF.sub.j).

2. Control process according to claim 1, the capture stage (S1) consisting of capturing at least two digital images (I), wherein an interpolation stage (S4) occurs, consisting of interpolating sound effect data (D_EF.sub.j) and/or expression data (D_EX.sub.i) to obtain a multitude of intermediary sound effect data or expression data between two successive digital images (I).

3. Control process according to claim 1, wherein the processing stage (S2) comprises the comparison between facial descriptors (F_D.sub.k, k being a positive integer) with a second database (DB2) comprising a mapping between facial descriptors and one or several facial expressions predefined or recorded by the user.

4. Control process according to claim 3, wherein the second database (DB2), comprising a mapping between facial descriptors (F_D.sub.k, k being a positive integer) and one or several facial expressions, is learned partially or totally from digital images (I) of the face captured by the digital camera (10).

5. Computer readable medium containing program instructions for a computer to perform Computer program comprising instructions to implement the stages of the process according to claim 1, when such computer program is executed by at least one processor.

6. Computer readable medium containing program instructions for a computer to perform Computer program comprising instructions to implement t the stages of the process according to claim 2, when such computer program is executed by at least one processor.

7. Computer readable medium containing program instructions for a computer to perform Computer program comprising instructions to implement the stages of the process according to claim 3, when such computer program is executed by at least one processor.

8. Computer readable medium containing program instructions for a computer to perform Computer program comprising instructions to implement the stages of the process according to claim 4, when such computer program is executed by at least one processor.

9. A control device (100) for a generation module (GM) of sound effects to control one or several sound effects (EF.sub.A, EF.sub.B, EF.sub.C) of a musical instrument (MI), such device (100) comprising: a capture module (M1) comprising a digital camera (10) configured to capture at least one digital image (I) comprising at least a portion of the user's (U) face, such capture module (M1) comprising emission of a source of infrared light (LUX_IR) in the direction of the user's (U) face, and such capture module (M1) comprising application of an infrared filtering (IRF) using an optical filter (30) to only allow the passage of an infrared wavelength band; a computer processing module (M2) configured to process such at least one image (I) in order to define at least one expression data (D_EX.sub.i, i being a positive integer) containing information relating to at least on facial expression (EX.sub.a, EX.sub.b, EX.sub.c, EX.sub.d) of the user (U); and an analysis module (M3) configured to compare such at least one expression data (D_EX.sub.i) with a predefined first database (DB1) to determine at least one sound effect data (D_EF.sub.j, j being a positive integer) containing information relating to at least one sound effect (EF.sub.A, EF.sub.B, EF.sub.C), corresponding to facial expressions(s) (EX.sub.a, EX.sub.b, EX.sub.c, EX.sub.d) of the user (U), such first database (DB1) comprising a multitude of expression data (D_EX.sub.i) each associated with at least one sound effect data (D_Ef.sub.j).

10. Device (100) according to claim 9, capture module (M1) configured to capture at least two digital images (I), characterized in that it comprises an interpolation module (M4) temporally interpolating the effect data (D_EF.sub.j) and/or the expression data (D_EX.sub.i) to obtain a multitude of intermediary effect data or expression data between two successive digital images (I).

11. Device (100) according to claim 9, wherein the processing module (M2) comprises the comparison between facial descriptors (F_D.sub.k, k being a positive integer) with a second database (DB2) comprising a mapping between facial descriptors and one or several facial expressions predetermined or recorded by the user.

12. Device (100) according to claim 11, wherein the second database (DB2), comprising a mapping between facial descriptors (F_D.sub.k, k being a positive integer) and one or several facial expressions, is learned partially or totally from digital images (I) of the face captured by the camera.

Description

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

(1) Further characteristics and advantages of the present invention will be highlighted in the description herein, referring to FIGS. 1 to 3 in annex, which illustrate an example of a non-restrictive embodiment and for which:

(2) FIG. 1 schematically represents a control device for a generation module of sound effects to control one or several sound effects for a musical instrument according to an example of an embodiment;

(3) FIG. 2 schematically represents a non-restrictive example of the correspondence map between the user's facial expressions and sound effects; and

(4) FIG. 3 consists of a flow chart representing the various stages of implementation according to an example of an embodiment of the invention process.

DETAILED DESCRIPTION OF THE INVENTION

(5) A control process for a generation module GM of sound effects, along with the associated control device 100 will be described hereafter in the following description which refers collectively to FIGS. 1 to 3.

(6) Enables simultaneous control of one or several sound effects of a musical instrument MI, while enabling the musician U to have their hands and feet free, is one of the objectives of the present invention.

(7) The present invention therefore proposes a new way to interact with generation modules GM of sound effects. For this, the invention proposes a control device 100, based on an analysis in real-time of a video capture of the musician's U face; in other words, with the present invention, musicians U can now control sound effects EF.sub.A, EF.sub.B, and/or EF.sub.C with part of their face (for example, a mouth movement) or by simulating a general facial expression (for example, by making a “sad” or “happy” expression).

(8) To illustrate the numerous possible embodiments of the present invention, the various facial expressions considered here in the present example are as follows (see FIG. 2): a localized facial expression, with an open mouth illustrated in EX.sub.a, of a localized facial expression with a wink illustrated in EX.sub.b, a general “happy” facial expression illustrated in EX.sub.c, or a general “sad” facial expression illustrated in EX.sub.d.

(9) Evidently, it is understood that other facial expressions (localized or general) may be envisaged within the present invention (turning the head; cheek, nostril, or eyebrow movements; a “scared” or “tired” expression, etc.).

(10) In the example described here, the musician U first records their desired parameter settings: for each facial expression cited hereabove EX.sub.a, EX.sub.b, EX.sub.c, EX.sub.d, they associate at least one sound effect EF.sub.A, EF.sub.B, and/or EF.sub.C.

(11) The musician must therefore, during their first use, record on a first database DB1, during a parameter setting stage S0 a correspondence map between expression data D_EX.sub.1, D_EX.sub.2, D_EX.sub.3 and D_EX.sub.4 (D_EX.sub.i, where i is a positive integer ranging between 1 and N, N being here equal to 4) respectively related to expressions EX.sub.a, EX.sub.b, EX.sub.c, and EX.sub.d, and sound effect data D_EF.sub.1, D_EF.sub.2, D_EF.sub.3 and D_EF.sub.4 (D_EF.sub.j, where j is a positive integer ranging between 1 and M, M being here equal to 4) each respectively containing information relating to at least one sound effect EF.sub.A, EF.sub.B, and/or EF.sub.C.

(12) In one embodiment, the musician may create his own expressions from digital images I of their face captured by the camera, to record his own “happy” expression EX.sub.c, or charge a predefined bank of expressions. Then, he will record an association between each expression and at least one predefined sound effect.

(13) In the example described here, and as illustrated in FIG. 2, sound effect data D_EF.sub.1 comprises information relating to sound effects EF.sub.A; sound effect data D_EF.sub.2 comprises information relating to sound effects EF.sub.B; sound effect data D_EF.sub.3 comprises information relating to sound effects EF.sub.B and EF.sub.C; sound effect data D_EF.sub.4 comprises information relating to sound effects EF.sub.C.

(14) Evidently, this example is purely illustrative and is in no way restrictive; other combinations with other sound effects are possible and conceivable for someone skilled in the art.

(15) As stated hereabove, this parameter setting S0 may also be prerecorded on the device 100 which, in such case, comprises a first database DB1, predefined by default, comprising the correspondence map as detailed hereabove and illustrated in FIG. 2.

(16) The present invention exploits this database DB1 through automatic recognition of facial expressions EX.sub.a, EX.sub.b, EX.sub.c, EX.sub.d to control sound effects EF.sub.A, EF.sub.B, EF.sub.C and EF.sub.D.

(17) As such, the control device 100, according to the present invention, comprises a capture module M1 comprising a classic digital camera 10 configured to capture, during a capture stage S1, at least one digital image I comprising at least a portion of the musician's U face.

(18) Optionally, this digital camera 10 can be attached, remaining detachable, on a pivoting arm (not shown here) to easily turn the camera in all directions so that the musician U can adjust the pivoting arm so that the camera 10 points in the direction of their face.

(19) This pivoting arm may be attached directly on the musical instrument MI. Preferably, this camera 10 is linked to the module by a USB connector which is directly integrated within the pivoting arm.

(20) Alternatively, this camera 10 may also be positioned further away in order to capture the whole scene in it's field of vision.

(21) In any event, in the example described herein, the camera 10 captures at least one image I comprising a portion of the musician's U face.

(22) In the example described herein, the musician U performs on stage in an environment presenting numerous variations in lighting and luminosity. To resolve this issue and make recognition of facial expressions more precise and reliable, the present invention envisages, during the capture stage S1, the emission of a source of infrared light LUX_IR in the direction of the musician U, using an emission medium 20 such as infrared LED (here emitting a wavelength of 850 nanometers, but other wavelengths are possible).

(23) The capture module M1 furthermore comprises an infrared optical filter 30, to only allow, during IRF (“Infrared Filtering”), the passage of a narrow infrared frequency band set around the wavelength emitted (the filtered image here corresponds to a wavelength ranging between approximately 840 to 860 nanometers).

(24) The emission of a source of infrared light LUX_IR on the face of the musician U and the application of an infrared filtering IRF before the capture of the image I, make the process insensitive to variations of light and lighting on stage, thus making the analysis of facial expressions more reliable, whatever the surrounding lighting conditions.

(25) As stated hereabove, control of sound effects requires: on the one hand, the capture of one or several digital images I comprising at least a portion of the musician's U face, and, on the other hand, recognition of one or several facial expressions of the musician U.

(26) Such recognition of facial expressions for controlling the sound effects of a musical instrument is characteristic of the present invention.

(27) As such, in the example described herein, and as illustrated in FIGS. 1 and 3, the control device 100 comprises a computer processing module M2 which is configured to apply, during a processing stage S2, a mathematical image processing algorithm on each digital image I captured in order to recognize at least one of the user's U facial expressions.

(28) Furthermore, it is also possible to envisage that the processing stage S2 could consist of identifying facial descriptors F_D.sub.k (with k being an integer), comprising parameters of a 3D model of a face recognized in the image I and/or distinctive point coordinates in the image I, and compare such descriptors with a second database DB2 comprising a bank of facial expressions which are either predetermined or recorded by the musician. This comparison enables determination of the musician's facial expression.

(29) In the example described herein, and illustrated in FIG. 1, using this automatic recognition, the device 100 recognizes expressions EX.sub.A (open mouth) and EX.sub.B (wink) of the musician's U face in the image I.

(30) Expression data D_EX.sub.1 and expression data D_EX.sub.2 are obtained in the output of module M2, respectively comprising information relating to expressions EX.sub.A and EX.sub.B.

(31) These expression data D_EX.sub.1 and D_EX.sub.2 are then analyzed by a module M3 during an analysis phase S3. The analysis module M3 compares each expression data D_EX.sub.1 and D_EX.sub.2 to the first database DB1 described hereabove to determine the corresponding sound effect data, i.e. in the example described herein, a weighted combination of sound effect data D_EF.sub.1 for the “open mouth” expression and sound effect data D_EF.sub.2 for the “wink” expression.

(32) The combination of sound effect data D_EF.sub.1 and D_EF.sub.2 is then sent to the generation module GM which is linked to the amplification system HP, and which, during stage S5, enables the production and generation of the sounds of the musical instrument associated with sound effects EF.sub.A and EF.sub.B.

(33) As stated hereabove, the present invention also envisages a post-processing step at output of the analysis module M3 or pre-processing before the analysis module M3 to avoid staircase effects being perceived by the human ear.

(34) As such, and as explained hereabove, the device 100 may comprise an interpolation module M4 for a temporal interpolation of the values between two images I. This interpolation may also be used directly on expression data.

(35) As such, the present invention proposes a new innovative approach to effectively control the sound effects of a musical instrument. This approach provides musicians with a new tool enabling them not to be distracted from their musical performance, and enabling them to simultaneously control several sound effects.

(36) It should be observed that this detailed description focuses on a specific example of an embodiment of the present invention, but that in no case is this description restrictive of the possible uses of the invention; on the contrary: it aims rather to avoid any imprecision or false interpretation of the following claims.