Method for processing images having specularities and corresponding computer program product

Abstract

A method for processing images of a scene including a surface made of a material of unknown reflectance comprises the following steps: from at least 3 different positions of an image sensor, the positions and corresponding orientations of the sensor known, acquiring images of the scene illuminated by a light source, each image containing specularity generated by reflection of the light source from the surface and depending on the position, the shape and the intensity of the light source and the reflectance of the material; in each image, detecting each specularity and for each specularity, estimating a conic approximating the specularity. It comprises constructing a quadric representing the position, intensity and shape of the light source and the reflectance of the material, on the basis of the conics, and of the position and orientation of the image sensor during the acquisition of the images containing the specularities respectively approximated by these conics.

Claims

1. A method for processing images of a scene including a surface made of a material of unknown reflectance, the method comprising the following steps: from at least 3 different positions of an image sensor, the positions and corresponding orientations of the sensor being known, acquiring images of the scene illuminated by a light source, each image containing at least one specularity generated by reflection of the light source from said surface and depending on the position of the light source, on the shape of the light source and on the intensity of the light source and on the reflectance of the material; in each image, detecting each specularity; and for each specularity, estimating a conic approximating said specularity; constructing a quadric representing the position of the light source, the intensity of the light source, the shape of the light source, the reflectance of the material, on the basis of the conics, the position, and the orientation of the image sensor during the acquisition of the images containing the specularities respectively approximated by these conics.

2. The image-processing method as claimed in claim 1, the surface having an unknown roughness, the quadric also represents the roughness of the surface.

3. The image-processing method as claimed in claim 1, wherein the scene is illuminated by at least one other light source, each image containing at least one specularity generated by reflection of the at least one other light source from said surface, and further comprising for each specularity: temporal tracking of the specularity and matching said specularity, and therefore the conic approximating it, with a light source, and wherein the step of constructing a quadric is carried out for each light source, on the basis of the conics matched with said light source, and of the position and orientation of the image sensors during the acquisition of the images containing the specularities respectively approximated by these conics.

4. The image-processing method as claimed in claim 1, further comprising the following steps: for each specularity, minimizing the distance between the specularity and a conic resulting from projecting the quadric onto the image containing said specularity along an axis determined by the position and orientation of the sensor, in order to determine a new quadric; and iterating the preceding step until a preset stop criterion is reached, the new quadric resulting from the last iteration being a refined quadric.

5. The image-processing method as claimed in claim 1, comprising a step of choosing key images from the acquired images, depending on a predefined criterion of distribution of viewpoints of the sensor in the scene with respect to the light source.

6. The image-processing method as claimed in claim 1, wherein the light source is a point source and wherein the corresponding quadric is a sphere.

7. The image-processing method as claimed in claim 1, wherein the light source is an extended source and wherein the corresponding quadric is an ellipsoid.

8. The image-processing method as claimed in claim 1, comprising a step of calculating a prediction of the position and shape of a conic called the predicted conic approximating a specularity formed on a surface made of a material of known normal, depending on a new position and a new orientation of said image sensor during the acquisition of a new image and on the projection of the quadric onto said new image.

9. The image-processing method as claimed in claim 8, wherein quadric scales being preset, the conic prediction calculation is carried out for each scale so as to obtain as many conics as scales, each conic corresponding to an intensity level of the predicted specularity.

10. The image-processing method as claimed in claim 8, wherein the material of the predicted specularity is the same as the material associated with the quadric, and wherein the size of the predicted specularity is also predicted.

11. The image-processing method as claimed in claim 1, comprising a step of correcting error in the position and orientation of the image sensor on the basis of a calculation of a discrepancy between a specularity present in the image associated with said position and orientation and a conic resulting from the projection of the quadric onto said image.

12. The image-processing method as claimed in claim 1, wherein the different positions and orientations of the image sensor are obtained using the same moving image sensor or using a plurality of stationary image sensors.

13. A computer-program, comprising instructions stored on a tangible non-transitory storage medium for executing, on the basis of: at least 3 different positions of an image sensor, the positions and corresponding orientations of the sensor being known, and images of a scene including a surface, made of a material of unknown reflectance, illuminated by a light source, which images are acquired by said image sensor, each image containing at least one specularity generated by reflection of the light source from said surface and dependent on the position of the light source, on the shape of the light source, on the intensity of the light source and on the reflectance of the material, the following steps to be carried out when said program is executed on a computer: in each image, detecting each specularity; for each specularity, estimating a conic approximating said specularity; and constructing a quadric representing the position of the light source, the intensity of the light source, the shape of the light source, the reflectance of the material, on the basis of the conics, the position, and the orientation of the image sensor during the acquisition of the images containing the specularities respectively approximated by these conics.

14. A computer-program as claimed in claim 13, wherein the surface having an unknown roughness, the quadric also represents the roughness of the surface.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) Other features and advantages of the invention will become apparent on reading the following detailed description, which is given by way of nonlimiting example and with reference to the appended drawings, in which:

(2) FIG. 1 shows a flowchart of the main steps of the method according to the invention;

(3) FIG. 2 schematically shows an example of specularity on a planar surface for an incident ray using the Phong model;

(4) FIG. 3 illustrates an example of texture and specularity in two images taken from two different viewpoints;

(5) FIG. 4 illustrates two different approximations of a specularity by an ellipse;

(6) FIG. 5 schematically shows the links between a light source and a planar surface presenting specularities, and the corresponding virtual model according to the invention;

(7) FIGS. 6a and 6b schematically show a point light source (left-hand image) and an extended source represented by a set of point sources on a segment generating on a planar surface specular spots circumscribed by an ellipse (right-hand image);

(8) FIGS. 7a and 7b schematically show two examples of distances between a specularity and a corresponding ellipse, with the point/point distance (FIG. 7a) and the Jaccard distance (FIG. 7b);

(9) FIG. 8 illustrates a plurality of intensity levels of a specularity, respectively represented by conics; and

(10) FIG. 9 schematically shows a plurality of concentric quadrics respectively corresponding to a plurality of scales, each projected quadric being a conic associated with one intensity level of the predicted specularity.

(11) From one figure to the next, elements that are the same have been referenced with the same references.

(12) In the rest of the description, the expressions top and bottom are used with reference to the orientation of the described figures. Insofar as the image sensor may be positioned with other orientations, the directional terminology is given by way of illustration and is nonlimiting.

DETAILED DESCRIPTION

(13) Knowledge of primary sources is essential for a better comprehension of the scene. To find these primary sources, a plurality of optical artefacts, such as shadows and specularities, that are available in the images, may be used.

(14) 1. Shadows correspond to zones occulted by an opaque object in a scene. These elements depend on the size of the occulting object and on its distance with respect to the primary sources. In addition, they are greatly influenced by the shape of the latter.

(15) 2. Specularities correspond to a total reflection of the primary source from a reflecting surface. These artefacts are dependent on the viewpoint and the materials on which they are observed.

(16) A light sensor (eye, video camera) is sensitive to these optical artefacts. It is moreover the latter that generally assist a human being with interpretation of a scene (in particular information on depth, the coherence of the scene, etc.).

(17) For machine-vision applications, light is therefore an essential element. Knowledge thereof may be advantageous in various applications: For augmented-reality applications, realism is greatly dependent on the virtual insertion of these optical artefacts. The algorithms used in the field of video-camera localization may be disrupted by the presence of specularities. Knowledge of these elements could improve these algorithms. For the verification of properties of an object (geometry, materials) in the field of industrial inspection.

(18) The method according to the invention is based on a method for constructing a virtual model allowing light sources of various shapes (light bulbs, fluorescent strips, etc.), the reflectance of a material and optionally the roughness of the surface to be modelled at the same time on the basis of specular reflections from a planar surface of this material; this planar surface may be curved. These specular reflections are sensitive to the properties of the light (shape, intensity, position) and to the properties of the material on which they are represented. Constructing a model on the basis of specularities also implies inclusion of the reflectance properties of the material. The construction of this virtual model will allow the specularities to be estimated via projection of the model for unknown viewpoints.

(19) The light sources in question are stationary and are generally not very far from the surface presenting the specularities, as is the case for light sources located in the interior of a building or for an exterior streetlight. Nevertheless it will be noted that a source at infinity, such as the sun for example, may also be modelled by the method.

(20) The images with specularities that are processed by the method according to the invention are acquired by a sensor the intrinsic parameters of which (focal length and central point) are known.

(21) The situations of the image sensor are considered to be known for each image; a 3-D model of the scene is also known, in which only the normal to the plane (or the normals to the planes) on which the specularities are located is (are) used. These images may be obtained by a sensor moving along a known path, the images possibly themselves being obtained in video mode or in manual mode at times that are further apart, or by a plurality of stationary sensors located at various known locations.

(22) The primitives used for the construction of the source model including the reflectance of the material correspond to the specular reflections in an image. These specularities are caused by the reflection of the light sources from a reflective (non-Lambertian) surface. It will be noted that the properties of the light source, such as its size, its position and its shape, influence the shape of the specularity on a surface. In addition, depending on the reflectance of the material, the size of these specularity is also modified. Knowledge of these elements is essential if the shape of the specularity is to be predictable for unknown viewpoints.

(23) In the case of a planar surface, a specularity corresponds to the reflection of a light source with respect to a specular material (a perfectly specular material would correspond to a mirror). This property is important because it allows the number of light sources in a scene to be rapidly identified. In addition, tracking of these specularities allows the temporal state of the light sources associated with these artefacts to be observed.

(24) On the basis of images obtained by a sensor in at least 3 different positions, the positions and corresponding orientations of the sensor being known, the construction of the model includes the steps shown in the flowchart of FIG. 1:

(25) A. Detection of Specularities

(26) This step consists in detecting specularities in an image. At the end of this step, the resulting image is a gray-scale image. These specularities are detected using the method described in the publication by the inventors Generic and real-time detection of specular reflections in images VISAPP, 2014. This approach is based on the observation that these specularities stand out more in the Hue-Saturation-Value (HSV) color space than, for example, in the RGB space. Specifically, specularities manifest themselves as high values in the Value channel and low values in the Saturation channel.

(27) On the basis of this observation, this step is based on automatic thresholding of the Value and Saturation channels depending on the luminosity calculated from the image. In order to maximize the results, preprocessing (=before thresholding) and post-processing (=after thresholding) are implemented.

(28) Depending on the luminosity calculated from the image, the preprocessing allows difficult images (due to overexposure of the video camera, an underexposure, or abrupt changes in lighting, etc.) to be anticipated. A contrast equalization is applied in order to make the specularities stand out more in the case of underexposure and in order to limit the saturation of the image in the case of overexposure. The image resulting from this preprocessing allows detection to be performed under better conditions but also allows images that it will be hard for methods for localizing the sensor in a video sequence to exploit to be anticipated. Specifically, the sensor (eye, video camera) is very sensitive to variations in the light that it receives.

(29) The objective of the post-processing is to filter the various candidates resulting from the thresholding. After an equalization of the contrast, certain zones in the image remain falsely detected for an overexposed image in particular. Specifically, a white texture returns a large amount of light, this possibly leading to false detection, its light intensity being very high. To respond to this problem, a method for separating specular spots from ambiguous textures is implemented.

(30) If the image is observed in the Value channel of the HSV, it is easy to observe, for each specularity, a gradual decrease in the intensity of the latter from its center. In order to exploit this criterion, the image is divided into k-regions of specular candidates. This segmentation is carried out using a conventional binary-image-segmentation method; for example, the method as described in the publication by Suzuki and Satoshi Topological structural analysis of digitized binary images by border following Computer Vision, Graphics, and Image Processing, 30(1): 32-46, 1985 may be used. The maximum circumscribed box is employed for each outline. Specifically, a specularity is nonuniform and is generally very fragmented. By using a maximum circumscribed box, these fragments are included in the calculation. Next, the threshold in the Value channel is gradually modified by one increment of unitary size in each iteration, the resulting variation in the area of these maximum circumscribed boxes being observed. If this variation is constant (equivalent to a slight and regular decrease in intensity), this candidate is considered to be a specularity. If the area decreases suddenly, this candidate probably represents a texture and has therefore been wrongly detected as being specular.

(31) B. Tracking of the Specularities

(32) Each specularity is associated with a light source. In order to track the state of the light sources in real-time, temporal tracking of each specularity is carried out on the basis of the outline of each thereof in successive images, in order to track variation in these outlines. For two successive images, if the preceding and current outline intersect, then the outline is updated and the specularity is tracked. If the outline disappears with respect to the preceding image, the specularity is no longer tracked (poorly detected outline, light that has been switched off). This tracking allows each specularity to be matched in real-time with a light source reflecting from the specular surface and that is the origin of this specularity from one image to the next. It is a question so to speak of indexing each specularity with an index that is specific to the light source that is the cause of said specularity.

(33) In addition, this tracking allows specularities falsely detected in an image to be filtered. Specifically, white textures in the presence of strong light return light in large amounts, thereby possibly causing the image to saturate and leading to false detection of a specularity. However, for a plurality of images, the white texture remains stationary in the 3-D scene contrary to the specularity which, depending on the viewpoint of the sensor, will move in the scene. The principle of the filtering according to the invention is based on the dependency of the 3-D outline of the specularity, i.e. the outline obtained from the 3-D model of the scene, with respect to the position of the sensor. The warping method is used, which consists in calculating, for the specular candidates, the homography between two images from the situation (=position and orientation) of the sensor for these images. The candidate is considered to be a texture if its 3-D outline has not moved in the scene; it is considered to be a specularity if its 3-D outline has moved as illustrated in FIG. 3. The dotted outline of the candidate located in the center of the book has not moved in the scene between the right-hand image taken at a first viewpoint and the left-hand image taken at a second viewpoint and this candidate must be considered as to be a texture T; the solid-line outline of the candidate in the bottom right-hand corner of the book has moved in the scene between the right-hand image and the left-hand image and it is therefore a specularity S. To obtain this 3-D outline, each point of a 2-D outline is considered and ray tracing is carried out in the 3-D model of the scene on the basis of the situation of the sensor of the associated image. The intersection between the surface and these rays gives a 3-D outline that is analyzed for each image.

(34) C. Estimation of Conics Representing the Specularities

(35) According to the applicant, on a planar surface, an entire specularity (i.e. a specularity that is not cut-off) takes the form of an ellipse for various light sources (lightbulbs, fluorescent strips). A specularity more generally takes the form of a conic, but in the rest of the description an ellipse will be considered. The basis for this property is the physical model of reflection described by Phong et al. in the publication Illumination for computer generated pictures Communications of the ACM, 18(6):311-317, 1975, focusing only on the specular portion of the Phong equation, which gives the intensity of a 3-D point p as:
I.sub.s(p)=i.sub.sk.sub.s{circumflex over (R)}{circumflex over (V)} cos.sup.n ,
where {circumflex over (R)} is the normalized direction of a perfectly reflected ray, {circumflex over (V)} the normalized direction oriented towards the sensor, n the shininess (reflection coefficient of the light) of the material for which the ray cuts the plane, k.sub.s the ratio of reflection of the intensity of the light, the angle between R and V and i.sub.s the intensity incident on the surface.

(36) Let us consider a constant angle and a plane of specular material. Thus, for this angle, the value of I.sub.s(p) is constant. On the basis of the Phong equation, it is possible to note that there is a maximum intensity at the point p.sub.0 such that:
p.sub.0,I.sub.s(p.sub.0)=i.sub.sk.sub.s=0

(37) According to Fermat's principle, the path of a light ray between two points is always the shortest possible path. The Snell-Descartes law of reflection may be derived from this principle: the angle .sub.1 between the normal N and the vector L, and the angle .sub.2 between the normal N and the vector R are equal. This principle implies that the incident plane containing N, L and R is orthogonal to . This principle will be applied on the basis of two types of sources, point sources and extended sources.

(38) For a point source such as a light bulb: in the Phong model, for a given angle , considering R to be the stationary arm and V (defined by the optical center P and p a point of the specularity on ) the arm of variable length, it is possible to draw a conic on the plane using a perfect virtual pair of compasses such as described for example in the publication by J. Van Maanen Seventeenth century instruments for drawing conic sections The Mathematical Gazette, pages 222-230, 1992.

(39) It is interesting to note that the point p.sub.0 is not located at the center of the ellipse of the specularity. It is possible to consider that there exists a cut-off angle .sub.max such that:
.sub.max,>.sub.maxI.sub.s(p)=0

(40) In addition, this cut-off angle implies that there exists a threshold such that:
pS,I.sub.s(p)>, where S is the set of the specular points.

(41) Thus, in the Phong model, the applicant has proved that a point light source emits a specularity taking the form of a conic on the plane by using .sub.max, and the perfect pair of compasses as illustrated in FIG. 2. Specifically, according to Fermat's principal, the path of a light ray is always the shortest. Thus, the plane formed by the incident and reflected ray (plane of incidence) is perpendicular to the plane (reflecting surface) which may be associated with the drawing surface of the perfect pair of compasses. If a specularity is entirely visible in the image on the plane then the specularity is of elliptical shape.

(42) In the literature, for an extended source such as a fluorescent strip, the extended source is considered to be a discretized segment of point sources. It is known that these point sources produce specularities of elliptical shape on a planar surface of a reflective material. If this segment is considered to be infinitely discretized into point light sources, then it is possible to represent the specular spots by a maximum circumscribed ellipse. Thus, for an extended light source, the specularity obtained is also of elliptical shape.

(43) Thus, for each type of light source, for a specular planar surface, it is possible to represent these (whole) specularities as ellipses.

(44) Depending on the quality of the detection, two approaches for estimating ellipses on the basis of a gray-scale image delivered by the detector of specularities at the end of step A, are possible: Maximum circumscribed ellipse estimation, which allows precise ellipses to be obtained for detections with little error as indicated in the publication L. G. Khachiyan, M. J. Todd, On the complexity of approximating the maximal inscribed ellipsoid for a polytope, Math. Programming 61 (1993)137-159. Fitted ellipse estimation, which allows potential errors due to the detection to be limited as indicated in the publication by A. W. Fitzgibbon, R. B. Fisher, et al. A buyer's guide to conic fitting. DAI Research paper, 1996.
D. Choice of Key Images for the Light

(45) For the sake of execution rapidity and precision in the estimation, the steps of constructing and optimizing the model are not carried out for each image but only for certain images called key images. They are chosen by making a compromise between the quality of the specularities detected in the images and an angle criterion.

(46) It is important to have key images corresponding to viewpoints distributed over the scene. On the basis of the reflected angles obtained beforehand, a threshold is set (for example experimentally) between the reflected angle of the current image and each angle of the already existing key images in order to obtain viewpoints that are far apart such that there is no intersection between the specularity detected in one key image and the specularity detected in the following key image. These key images may be obtained directly (without subsequent sorting) when for example the sensors used are stationary and have already been localized in locations that are sufficiently far apart to meet this far-apart-viewpoint criterion.

(47) All these criteria allow optimal viewpoints for the construction of the model to be obtained. Moreover, the granularity of a specular surface may influence the shape of the specularity. Therefore, it may be difficult to obtain an ellipse corresponding exactly to the specularity as illustrated in FIG. 4 in which it may be observed that the calculated ellipse C corresponds better to the outline of the specularity S in the left-hand image than in the right-hand image. The quality of the specularities is estimated by comparing the outline of the specularity obtained from the detector (=from the detection of step A) with the ellipse calculated for this specularity. This quality is estimated via a point/point distance between points on the outline of the specularity and the estimated ellipse. The Jaccard distance may also be used in order to compare the similarity between the area of the specularity and the area of the ellipse. These distances are detailed in the optimizing step.

(48) E. Initialization

(49) In the Phong model, the direction of the point light, and its intensity, must be estimated directly on the basis of a point source. According to the Phong formula, it is known that a specularity is influenced in size depending on the intensity of the light source. In practice, the calculation of this intensity is not trivial because the sensor is very sensitive to variations in light and the perceived intensity may vary.

(50) It is proposed to reconstruct an empirical model of light sources as a virtual volume. This model includes the intensity and shape of the light source and the properties of the materials by reconstructing a quadric on the basis of conics estimated from the specularities associated with this light source.

(51) For a point source (lightbulb), the model according to the invention gives an equivalence to this point light source taking the form of a sphere as shown in FIG. 5. Specifically, for an omnidirectional point source at a fixed position G, there is a point p.sub.0 such that the angle between the normal and the incident ray is equal to the angle between the normal between the reflected ray according to the Snell-Descartes principle. In the Phong model, p.sub.0 is a point on the surface of maximum intensity. In FIG. 5, the intersection of the rays P.sub.1q.sub.1, P.sub.2 q.sub.2 and P.sub.3q.sub.3 corresponds to G.sub.s a point that is the reflection of G with respect to ; P.sub.1, P.sub.2, P.sub.3 are the various positions of the sensor, q.sub.1, q.sub.2 and q.sub.3 being the points of maximum intensity of the corresponding ellipses C.sub.1, C.sub.2, C.sub.3. The point of maximum intensity is not necessarily at the center of the conic as may be seen in FIG. 8. Returning to FIG. 5, the applicant has demonstrated that, for each viewpoint, on a planar surface for a given specular material, a point source G produces a specularity of elliptical shape. The intersection of the various cones of revolution of vertices P.sub.1, P.sub.2 and P.sub.3 (optical centers of the sensor) gives a sphere of radius r. This radius r is associated with an intensity and with the reflectance of the given material.

(52) An analogy may be made between the model and a result due to the wave nature of light. Specifically, according to Huygen's principle, light propagates in the form of wavelets. A light source is considered to be isotropic, i.e. it emits light waves that propagate in rays in all directions: at each of the points of space close to the source and reached by the light wave, the state of the electromagnetic field oscillates (at the frequency of the source) between a maximum value and a minimum value. At a given time, all the points located at an identical distance from the source are in the same state: they are in phase. The envelope of the points located at the same distance from the source makes a spherical shape in space. The surface of this sphere is called the wavefront, which may be considered to be a secondary source associated with the primary source.

(53) It may also be noted that for a given source, there are a plurality of wavefronts as shown in FIG. 6a in which each circle corresponds to a wavefront emitted by the central source in all directions. In the virtual model, it is possible to make an analogy with wavefronts by considering that, for each material, the specularity observed on its surface will embody a different wave-front level.

(54) For an extended source such as a fluorescent strip:

(55) In computer graphics, it is common to represent an extended source by a set of points (point sources). Thus a fluorescent strip is represented by a set of points distributed over a segment. For these points the waves, described by a sphere, are independent and originate from each point source. The interference of these spherical waves according to the Huygens-Fresnel principle will give a new wavefront the external envelope of which may be generalized by an ellipsoid as illustrated in FIG. 6b.

(56) Thus, for an extended light source, the model according to the invention represents this source with an ellipsoid by analogy with the shape of the wavefront. These observations allow the virtual light-source model to be generalized using a quadric that may represent a sphere, an ellipsoid, a cylinder or even a point.

(57) A few generalities on the construction of a quadric on the basis of conics will be recalled.

(58) The procedure for initializing a quadric on the basis of conics will be presented after the formalism of conics and quadrics has been presented.

(59) A quadric is a surface with 9 degrees of freedom which is represented by a symmetric matrix Q.sup.44 such that any point X on its surface satisfies the equation:
X.sup.TQX=0,
where X is a 3-D point in homogenous coordinates.

(60) The outline occulting a quadric in the image plane, for a given sensor situation, corresponds to a conic that is an outline with 5 degrees of freedom and that is represented by a symmetric matrix C.sup.33 such that any point x on its outline meets the constraint:
x.sup.TCx=0,
where x is a 2-D point in homogenous coordinates.

(61) In order to express C as a function of Q, the dual space is used instead of the primal space because the relationship between C and Q is not direct in the primal space. Using the dual space for the 3-D reconstruction makes it possible to dispense with depth information. In this space, the conics are represented using lines I such that:
I.sup.TC*I=0,
where C* is the dual conic.

(62) For a dual quadric Q*, a representation of the plane described by the following relationship is used:
.sup.TQ*=0.

(63) In the dual space, it is possible to use, to within a scale factor, the following relationship:
C*PQ*P.sup.T,
with Q*=adj (Q) and C*=adj (C), where adj is the adjucate (=the transposed cofactor matrix).

(64) The method described in the publication Cross et al. Quadric reconstruction from dual-space geometry in International Conference on Computer Vision, ICCV, 1998, is used, which allows Q* to be reconstructed by converting the relationship between Q* and C* into a linear system. By vectorising Q, P and C into the form Q.sub.v, B and C.sub.v, for n viewpoints with n3 we can reconstruct a system by using specular reflections taking the form of conics C and the model of the light source taking the form of Q:

(65) $\mathrm{Mw}=0\left(\begin{array}{ccccc}{B}_1& -C_{1,v}^{*}& 0& \mathrm{.Math.}& 0\\ B_2& 0& -C_{2,v}^{*}& & 0\\ \mathrm{.Math.}& 0& 0& & \mathrm{.Math.}\\ B_n& 0& 0& \mathrm{.Math.}& -C_{n,v}^{*}\end{array}\right)\left(\begin{array}{c}{Q}_v^{*}\\ {}_1\\ {}_2\\ \mathrm{.Math.}\\ {}_n\end{array}\right)=0,$
With the matrix B.sub.i .sup.610.

(66) The solution to this system is calculated on the basis of a singular-value decomposition (SVD) of M. It will be noted that .sub.1 corresponds to a scale factor such that:
.sub.iC*.sub.i,v=B.sub.iQ*.sub.i,v
for the viewpoint i.

(67) This procedure is illustrated in FIG. 5 for three viewpoints; the sphere of radius r is constructed on the basis of the three ellipses respectively associated with the three specularities. The sphere of radius r includes the intensity of the light source and properties of the material of the surface on which the specularities are present. The cones of revolution indicated by the dotted lines in the figure and respectively issued from the points P.sub.i, are not explicitly calculated to determine the quadric.

(68) This step is carried out for each indexed light source. Tracking specularities in the images allows a first model to be constructed for each light source, independently. At the end of this step, quadrics that each represent one light source will have been constructed.

(69) F. Optimization

(70) This phase allows the construction of the light-source model to be refined (for each of the light sources) via a minimization, for the various viewpoints, of the distance between the projections of the quadric and the outlines of the specularities of each image. Thus, the optimization is based on the minimization, for all the light sources, of a non-linear cost function that allows the ellipses generated from the projections to be made to correspond as best as possible to the corresponding specularities in the images. This cost function may be calculated differently depending on the chosen distance.

(71) F.1. Jaccard Distance

(72) The Jaccard distance quantifies the similarity between two sets (area of the specularity S bounded by a solid line and area of the projected conic (ellipse in our example) bounded by a dashed line) as shown in FIG. 7b in which each of the two areas and their intersection may be seen, and is given by the equation:

(73) $J\left(C,S\right)=\frac{.Math.C.Math.S.Math.}{.Math.C.Math.S.Math.}$
where C is a conic and S a specularity detected in an image.
Jaccard Cost Function:

(74) The objective of this cost function is to maximize, for all the viewpoints, the similarity between the area of the specularity and the conic generated by projecting the quadric.

(75) The cost function is described by the equation:

(76) $\underset{Q_k}{\mathrm{Max}}\underset{i=1}{\overset{n}{.Math.}}J\left(C_{i,k}^{},S_{i,k}\right)$
where S.sub.i are the specular pixels in the image and C.sub.i is the conic for the image of index i and the light source of index k.
F.2. Point/Point Distance for the Cost Function

(77) A point/conic distance between the outline of a specularity and that of its associated projected conic is not usable as such. Specifically, calculating the distance between a point and a conic involves calculating a root of order 4 of a polynomial, this being expensive in terms of computing time and not allowing analytical derivatives to be obtained.

(78) The desired solution is to obtain a parameterization of the conic into points respectively associated with each outline point, i.e. similar to the approach described in the publication P. Sturm and P. Gargallo Conic fitting using the geometric distance In Asian Conference on Computer Vision, ACCV.2007.

(79) Various cost-function calculations may be used to calculate this point/point distance, which is illustrated in FIG. 7a. Two thereof are presented.

(80) Cost Function with Point/Point Distance

(81) The objective of this cost function is to minimize all the Euclidean distances d between the projections C.sub.i of the quadric Q for a viewpoint of index i as a function of Q as shown in FIG. 7a. This distance is given by the equation:

(82) $\underset{Q_k}{\min }\underset{i=1}{\overset{n}{.Math.}}\underset{j=1}{\overset{m_i}{.Math.}}d\left(q_{i,j}^{},q_{i,j,k}\right)$
where q.sub.i,j,k is the point (represented by a solid circle) of index j of the outline of the specularity S of index i and q.sub.i,j,k is the point obtained from the parameterization of the conic C (represented by a cross) of index i. This operation is carried out for the light source k.

(83) In the case of the cost function for the point/point distance, the conic must be discretized in each iteration.

(84) Another solution is to use latent variables in order to preserve the same pairs of points (specularity/conic) instead of recalculating these pairs in each iteration. However, the constraint to be met is to have latent variables able to vary only on the outline of its associated conic. Now, a point of a conic is defined by the relationship: x.sup.T Cx=0,

(85) where C is a conic and x a 2-D point in homogenous coordinates.

(86) Thus the cost function may be re-written in the form:

(87) $\{\begin{array}{cc}\underset{Q_k,q_{1,1},\mathrm{.Math.},q_{n,m_i}}{\min }& \underset{i=1}{\overset{n}{.Math.}}\underset{j=1}{\overset{m_i}{.Math.}}d\left(q_{i,j,k}^{},q_{i,j,k}\right)\\ \mathrm{subject}\mathrm{to}& q_{i,j,k}^{}{C}_j{q}_{i,j,k}^T=0\end{array},$

(88) It will be noted that the quadric to be refined is in general represented by a matrix Q.sup.44 such that: X.sup.TQX=0,

(89) where X is a 3-D point belonging to the surface of the quadric.

(90) This quadric may also be represented parametrically (center of the quadric, scale of each axis, rotation about each axis).

(91) At the end of this step, refined quadrics that each represent one light-source model will have been constructed. When a plurality of quadrics coincide, they then represent a model of the same light source.

(92) G. Specularity Prediction:

(93) Once the model has been constructed and refined, it is possible to directly project this model onto the image plane .sub.i in order to predict the specularity Sp in a current new image acquired using the sensor with a new but known orientation and position P and with the same material. This function is particularly useful in the field of augmented reality. The projection of the (modelled) quadric onto this new image gives a conic bounding the zone predicted for the specularity.

(94) Although the model is applicable for a given material, it is possible to estimate a specularity, defined to within a factor, for another planar surface of a specular material different from that of the model but of known normal: the center of the projected conic indicates the center of the zone predicted for the specularity, the dimensions of the specularity being those of the conic to within a scale factor. It will also be noted that for complex objects, by discretization of the outline occulting the quadric for an unknown viewpoint, it is possible, by ray tracing, to predict the outline of the specularity on this object.

(95) Generally, a specularity possesses various intensity levels. Specifically, a specularity possesses a zone of maximum intensity and its intensity decreases with distance from this zone. In our model, the various intensity levels are all represented by conics as shown in FIG. 8 with 4 conics Cp1, Cp2, Cp3, Cp4 (and therefore 4 intensity levels). It is important to note that the latter are not concentric, as may be seen in this figure. The specularity-predicting method allows the maximum circumscribed outline of the specularity Sp to be estimated on the basis of the strict projection of the quadric, and the various intensity levels of the specularity to be estimated by projecting quadrics of various scales, as shown in FIG. 9 for 3 quadric scales. The 3 concentric quadrics Q.sub.1, Q.sub.2, Q.sub.3 (illustrating 3 quadric scales) projected onto the image plane .sub.i produce 3 (predicted) conics Cp1, Cp2, Cp3 corresponding to 3 intensity levels of the predicted specularity Sp. Therefore, our model also allows the zone of maximum intensity of the specularity to be predicted.

(96) H. Correction of the Position and Orientation of the Sensor

(97) When specularity is predicted on the basis of the constructed and optionally refined model, it is also possible to correct possible errors in sensor position and orientation on the basis of a calculation of a discrepancy between a specularity present in the image associated with said position and said orientation and a conic resulting from projecting the modelled quadric onto said image. In practice, this step is carried out using one of the aforementioned cost functions with the position and orientation of the video camera as minimization parameter; this cost function is calculated using the outline of the detected specularity and used to construct the model of the source and the outline of the predicted specularity.

(98) The method has been implemented (detection with temporal tracking of specularities, quadric reconstruction and nonlinear refinement) and validated quantitatively and quantitatively on synthetic sequences and on real sequences. These tests were carried out in complex environments with various light sources (lightbulbs and fluorescent lights) for objects (examples: table, kitchen, board, book, etc.) made of different materials (steel, plastic, varnished wood, china). The realized sequences highlight the precision of the method regarding the prediction of specular spots from viewpoints not used for the reconstruction. In addition, the video stream was treated in real-time (30 images per second on average) with the proposed approach.

(99) For all the applications of augmented reality, sensor localization, surface quality control, etc., machine vision using a method according to the invention therefore offers an inexpensive, practical and non-invasive solution.

(100) This method for processing images containing specularities may be implemented on the basis of images obtained by one movable sensor (or a plurality of stationary sensors) located in a scene, and on the basis of a 3-D geometric model of this scene.

(101) This method may in particular be implemented using a computer-program product, this computer program comprising code instructions allowing the steps of the method for processing images acquired by the sensor to be performed. It is then stored on a medium that is readable by computer, such as for example a computer that has access to a 3-D geometric model of the scene and the positions and orientations of the sensor. The medium may be electronic, magnetic, optical, electromagnetic or be a storage medium read using infrared light. Such media are for example semiconductor memories (random-access memory (RAM) or read-only memory (ROM)), tapes, floppy disks or magnetic or optical (Compact Disk-Read Only Memory (CD-ROM), Compact Disk-Read/Write (CD-R/W) or DVD) disks.

(102) Although the invention has been described with reference to particular embodiments, it will be obvious that it is in no way limited thereto and that it comprises any technical equivalent of the described means and combinations thereof if the latter fall within the scope of the invention.