APPARATUS FOR AND METHOD OF INSPECTING SURFACE TOPOGRAPHY OF A MOVING OBJECT

Abstract

A dynamic photometric stereo inspection technique usable to capture and analyse the topography of a moving surface. The technique includes an enhanced data capture method and apparatus comprising a spaced array of at least two coplanar illuminates to improve measurement range and accuracy. The apparatus can be used to inspect banknotes, e.g. to assist with fitness assessment and/or forgery detection. The method may comprise automatically assessing surface topography data to provide qualitative and quantitative information about 2D and 3D features, such as changes in reflectivity, colour, glossiness, 3D texture and the surface profile of the surface under inspection.

Claims

1. A method for inspecting a surface, the method comprising: illuminating a surface with three or more inspection beams, each inspection beam being output from a respective illuminate, the illuminates being spaced from each other over the surface; obtaining a plurality of digital images of the surface from a digital image capturing device; and calculating a magnitude and a direction for a surface normal component at each of a plurality of inspection points on the surface based on the plurality of digital images and a predetermined incident light vector from each of the illuminates at each inspection point, wherein the illuminates are arranged relative to the surface so that their predetermined incident light vectors are coplanar at each inspection point.

2. The method of claim 1 including moving the surface relative to the digital image capturing device while the plurality of digital images is obtained.

3. The method of claim 1, wherein calculating the magnitude and direction for the surface normal component at each inspection point comprises applying a surface reflectance lighting model to a detected light intensity at each inspection point obtained and the predetermined incident light vectors from the illuminates at each inspection point.

4. (canceled)

5. The method of claim 1 including: generating inspection data from the magnitude and direction of the surface normal components of the inspection points; and analyzing the inspection data to identify properties of the surface.

6. The method of claim 5, wherein analyzing the inspection data comprises any of: comparing the behavior of the surface normal components across the surface with characteristic surface normal behavior associated with one or more surface defects, and determining specular properties of the surface.

7. (canceled)

8. (canceled)

9. (canceled)

10. The method of claim 5, wherein the inspection data comprises one or more of: a bump map comprising a dense array of the surface normal component directions calculated for the plurality of inspection points; an albedo comprising a map of the surface normal component magnitudes calculated for the plurality of inspection points.

11. The method of claim 10, wherein the inspection data further includes any one or more of: a shadow pattern obtained from one or more of the plurality of digital images; a full colour image of the surface.

12. The method of claim 1, wherein illuminating the surface includes multiplexing the inspection beams in any of a temporal, spatial or spectral sense.

13. An apparatus for inspecting a surface, the apparatus comprising: three or more illuminates mounted in a spaced arrangement over an inspection plane, each of the three or more illuminates being arranged to output an inspection beam to illuminate a region of the inspection plane; a digital image capturing device having the inspection plane in its field of view; and a processor arranged to receive a plurality of digital images from the digital image capturing device, each of the plurality of digital images including an image of the inspection plane when illuminated by an inspection beam from a respective illuminate, wherein the processor is arranged to calculate a magnitude and a direction for a surface normal component at each of a plurality of inspection points on the inspection plane based on the plurality of digital images and a predetermined incident light vector from each of the illuminates at each inspection point, and wherein the three or more illuminates are arranged so that their predetermined incident light vectors are coplanar at each inspection point.

14. The apparatus of claim 13 including an object conveyor arranged to move a surface to be inspected across the inspection plane while the plurality of digital images are captured by the digital image capturing device.

15. (canceled)

16. (canceled)

17. The apparatus of claim 13, wherein each of the three or more illuminates comprises a line light for outputting a planar light beam that intersects with the inspection plane along an inspection line.

18. The apparatus of claim 17, wherein the three or more illuminates output illumination in different discrete wavelength bands, and wherein the planar light beams from the three or more illuminates intersect with the inspection plane along a common inspection line.

19. The apparatus of claim 17, wherein the inspection lines of the three or more illuminates lie adjacent one another on the inspection plane.

20. (canceled)

21. (canceled)

22. The apparatus of claim 13, wherein image capture by the digital image capturing device is synchronized with the movement of a surface across the inspection plane.

23. The apparatus of claim 13, wherein the three or more illuminates include a virtual illuminate created by simultaneous illumination of the surface with the inspection beams from two or more physical illuminates.

24. An apparatus for inspecting a surface, the apparatus comprising: an illuminate mounted adjacent to a predefined travel path for a inspection surface, the illuminate being arranged to output an inspection beam to illuminate a region of the inspection surface as it moves relative to the illuminate; a digital image capturing device having the illuminated region of the inspection surface in its field of view; and a processor arranged to receive a plurality of time-spaced digital images from the digital image capturing device, wherein the region of the inspection surface in the field of view of the digital image capturing device is curved; wherein the processor is arranged to calculate a magnitude and a direction for a surface normal component at each of a plurality of inspection points on the inspection surface based on the time-spaced plurality of digital images and a set of predetermined incident light vectors from the illuminate at each inspection point in each of the time-spaced plurality of digital images, and wherein the set of predetermined incident light vectors for each inspection point are coplanar.

25. An inspection apparatus for banknotes, comprising: a feed mechanism for conveying a banknote across an inspection plane; and a photometric stereo measurement system arranged to detect a surface topography of the banknote as it passes across the inspection plane; and an analysis processor arranged to identify defects in the banknote from the surface topography.

26. The inspection apparatus of claim 25, wherein the photometric stereo measurement system comprises: a plurality of illuminates mounted in a spaced arrangement over the inspection plane, each of the plurality of illuminates being arranged to output an inspection beam to illuminate a region of the inspection plane; and a digital image capturing device having the inspection plane in its field of view, wherein the digital image capturing device is arranged to capture a plurality of images, each image being of the surface in the inspection plane when illuminated by an inspection beam from a respective illuminate.

27. The inspection apparatus of claim 26, wherein the analysis processor is arranged to: calculate a magnitude and a direction for a surface normal component at each of a plurality of inspection points on the inspection plane based on the plurality of digital images and a predetermined incident light vector from each of the illuminates at each inspection point; generate inspection data from the magnitude and direction of the surface normal components of the inspection points, and to analyze the inspection data to identify defects in the banknote; and determine specular properties of the banknote from the surface topography.

28. (canceled)

29. (canceled)

30. (canceled)

31. The inspection apparatus of claim 27, wherein the inspection data comprises one or more of: a bump map comprising a dense array of the surface normal component directions calculated for the plurality of inspection points; an albedo comprising a map of the surface normal component magnitudes calculated for the plurality of inspection points; a shadow pattern obtained from one or more of the plurality of digital images; and a full color image of the surface.

32. (canceled)

33. A method of analyzing surface topography of a moving surface, the method comprising: obtaining a bump map comprising a dense array of surface normal component directions for a plurality of inspection points on a surface; modelling the behavior of the surface normal component directions in a region of the surface; identifying a property of the surface based on the modelled behavior.

34. The method of claim 33, wherein modelling the behavior of the surface normal component directions includes any one or more of: fitting the surface normal directions across the region to a polynomial expression; fitting the rate of angular change of the surface normal directions across the region to a polynomial expression; and running a sub-routine comprising the steps of: creating a computer-generated three-dimensional rendering of the region from the bump map; generating a first view of the computer-generated three-dimensional rendering using a first illumination location; and generating a second view of the computer-generated three-dimensional rendering using a second illumination location that is different to the first illumination location, wherein identifying a property of the surface includes comparing the first view with the second view.

35. (canceled)

36. The method of claim 33, wherein identifying a property of the surface includes any one or more of: comparing the modelled behavior with predetermined characteristic behavior of known surface properties; and quantifying a magnitude of a surface defect.

37. (canceled)

38. (canceled)

39. (canceled)

40. (canceled)

41. (canceled)

42. (canceled)

43. The method of claim 33, wherein the surface is the surface of a banknote, and wherein the method includes using the identified property of the surface to determine the fitness of the banknote.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Examples of the invention are discussed below in detail with reference to the accompanying drawings, in which:

[0068] FIG. 1 is a schematic view of a known photometric stereo measurement setup, and is discussed above;

[0069] FIG. 2 is a schematic view of a known dynamic photometric stereo measurement setup, and is discussed above;

[0070] FIG. 3 is a cross-sectional view of a photometric stereo measurement setup with closely spaced illuminates;

[0071] FIG. 4 is a cross-sectional view of a photometric stereo measurement setup with widely spaced illuminates;

[0072] FIG. 5 is a cross-sectional view of a photometric stereo measurement setup with a plurality of in plane illuminates;

[0073] FIG. 6 is a schematic view of a dynamic photometric stereo measurement setup that is an embodiment of the invention;

[0074] FIG. 7 is a cross-sectional view of a photometric stereo measurement setup with a plurality of in plane illuminates that illustrates the effect of a surface discontinuity;

[0075] FIG. 8 is a cross-sectional view of a dynamic photometric stereo measurement setup that is another embodiment of the invention;

[0076] FIG. 9 is a schematic view of a frequency multiplexing technique that can be used with the dynamic photometric stereo measurement setup of the invention;

[0077] FIG. 10 is a schematic view of a spatial multiplexing technique that can be used with the dynamic photometric stereo measurement setup of the invention;

[0078] FIG. 11 is a schematic view of a hybrid spectral-spatial multiplexing technique that can be used with the dynamic photometric stereo measurement setup of the invention;

[0079] FIGS. 12A, 12B and 12C are schematic cross-sectional views of different surface defect that may be characterised by the dynamic photometric stereo measurement setup of the invention.

DETAILED DESCRIPTION; FURTHER OPTIONS AND PREFERENCES

Enhanced Data Capture

[0080] The principle behind the enhanced data capture technique of the invention is now explained with reference to FIGS. 3 to 8.

[0081] In known photometric stereo arrangements, a compromise is struck between the sensitivity and accuracy of a measurement and the range over which measurements can be made. The closer together illuminates are placed, the greater is the range of surface normal recovery (i.e. the wider the range of surface normal angles that will be detected). However, the further apart the illuminates are placed, the greater is the sensitivity and accuracy in the surface normal recovery.

[0082] FIG. 3 shows a cross-sectional view of a photometric stereo measurement setup with closely spaced illuminates 21, 22 above an inspection surface 24. The illuminates 21, 22 may be point sources or line sources. The range of surface normal angles that can be recovered for a given inspection point 26 ranges between the horizon 28 that is orthogonal to the incident vector 30 from the first light source 21 to the horizon 32 that is orthogonal to the incident vector 34 from the second light source 22. This yields a large region of recovery 36, but with low sensitivity, i.e. images of 3D topography obtained using illuminates 21 and 22 tend to be similar.

[0083] In contrast, FIG. 4 shows a cross-sectional view of a photometric stereo measurement setup with widely spaced illuminates 21, 22 above an inspection surface 24. Again, the range of surface normal angles that can be recovered for a given inspection point 26 ranges between the horizon 28 that is orthogonal to the incident vector 30 from the first light source 21 to the horizon 32 that is orthogonal to the incident vector 34 from the second light source 22. In this case this yields a small region of recovery 36, but with high sensitivity, i.e. images of 3D topography obtained using illuminates 21 and 22 tend to very different.

[0084] The enhanced data capture technique of the present invention is based on the concept of multiple in-plane illumination, i.e. using more than two, e.g. three, four, five, six or more, illuminates arranged so that the incident vectors they provide for a given inspection point are coplanar. This arrangement provides redundancy that can be exploited to achieve higher accuracy as each illuminate contributes to defining the unknown normal.

[0085] FIG. 5 shows a cross-sectional view of a photometric stereo measurement apparatus 40 that is an embodiment of the invention. In this arrangement, six illuminates 41-46 are arranged over an inspection surface 24. The illuminates 41-16 are arranged such that the incident vectors used to calculate a surface normal at an inspection point 26 on the surface 24 are coplanar (in the plane of the page).

[0086] In known photometric stereo techniques, additional in-plane illuminates are avoided because they do not contribute further to the location of the normal in a three-dimensional coordinate system. However, the inventors have realised that the redundancy provided by additional in-plane illuminates can be exploited to particularly good effect where features of interest on the surface are very small.

[0087] In-plane illumination captures the component of the unknown surface normals in the plane formed by the illuminates only. This means that features that are entirely orthogonal to this plane will be missing. In practice, because real features are not perfect, this does not have a material effect. However, one or more additional illuminates may be provided out of the plane if it is desirable to resolve the remaining dimension of the surface normal. Multiple planes could be used to recover the full surface normal with increased sensitivity and accuracy. However, in certain applications, e.g. banknote inspection, it is possible to implement an inspection system with only the plurality of in plane illuminates, as these may give enough information about surface defects that are of interest.

[0088] In use, the multiple in-plane illuminates can provide both a wide range of recovery and high sensitivity. The lights which are far apart will give higher sensitivity, whereas by using multiple pairs we can obtain an overlapping region of recovery that is wider than can be achieved with only two illuminates. For example, a series of measurements for the inspection point 26 may be taken using different pairs of the illuminates, e.g. lights 41 and 43, then lights 42 and 44, then lights 43 and 45, then lights 44 and 46.

[0089] The illuminates 41-46 are arranged symmetrically over the inspection surface 24. This may facilitate more rapid processing, but is not essential.

[0090] The illuminates 41-46 may also be operated in combination to create virtual light sources at different locations. For example, in another embodiment lights 42 and 45 may not be physical light sources, but may instead be created by simultaneous operation of lights 41 and 43 and lights 44 and 46 respectively.

[0091] FIG. 6 is a schematic view of a banknote inspection apparatus 100 that incorporates the ideas above and which is an embodiment of the invention. The apparatus 100 comprises an object feed mechanism 102, which in this case is a conveyor for moving banknotes through an inspection plane 106. In FIG. 6 the banknote 104 is shown lying flat on the conveyor, but the present invention may also operate with non-flat surfaces as discussed below with reference to FIG. 8. The conveyor may be arranged to orientate the object to be inspected in a manner that accentuates the visibility of the features being detected.

[0092] A plurality of light sources 108, 110, 112, 114 are arranged over the inspection plane 106. Each light source 108, 110, 112, 114 generates a planar light beam which intersects the inspection plane 106 along a line. In this embodiment, the planar light beams overlap on the inspection plane along a common inspection line 116, which implies that a form of spectral multiplexing will be used (see below). However, in other embodiments the planar light beams may illuminate different areas on the inspection plane.

[0093] In this embodiment, each light source is a line light, e.g. comprising a plurality of high intensity LEDs arranged in a row. The light sources are parallel to one another. Each LED in the line produces a collimated beam to generate a composite planar beam. As discussed above, at each inspection point on the surface the planar beam may be treated as a single incident vector that extends the shortest distance between the inspection plane and the light source at that point. Each point on the common inspection line is therefore exposable to a plurality of light sources whose incident vectors lie in a common plane. An advantage of this arrangement is that it provides richer output data without a proportional change in image processing demand.

[0094] The illuminates 108, 110, 112, 114 may comprise broad band or narrow band sources of one or more wavelengths. As mentioned above, various combinations of wavelengths can be employed for implementing frequency multiplexing. For example, near infra-red illumination and or visible wavelengths may be used. Similarly, one or more of the illuminates may have a frequency (e.g. in the ultra violet or infra red range) selected to image a feature of interest on the surface that is sensitive to that frequency.

[0095] In another embodiment, the incident light may be polarised so that changes in polarisation upon reflection can be detected. Thus, a polarising filter may be placed in front of an illuminate and another filter, at 90 to the first, may be placed in front of the camera. This technique may be used for detecting the presence of different materials on the surface, e.g. transparent tape on a banknote.

[0096] In another embodiment, the incident light may be coherent, e.g. from a laser. The illuminate may comprise a laser source for outputting a beam of laser light, and an optical system, e.g. including a cylindrical lens, for manipulating the beam of laser light into an incident beam (e.g. an planar beam) on the inspection surface. In this arrangement, the light reflected from the inspection surface may be projected on a screen located in the field of view of the camera. The reflected light may form a pattern on the screen which may be indicative of the condition of the surface, i.e. may be influenced by the 3D texture and features on the note surface. This technique may be used in addition to photometric stereo (PS). The laser technique might be useful for detecting rather more specular features, such as tape or the varnish that is applied to the inspection surface.

[0097] Returning to FIG. 6, the inspection plane is located in the field of view of an image capture device, which in this embodiment comprises a pair of digital cameras 120. Using two (or more) cameras may increase the speed with which images are captured, but the invention can work with a single device.

[0098] A speed controller 118 may be provided on the conveyor to control the speed at which the object moves through the inspection plane. The speed of the object needs to be related to the speed that images are captured from the inspection line in order to allow full images for the object's surface to be generated for each desired illumination angle, e.g. one image for each of the illuminates. An encoder on the conveyor may measure the position of the surface to be inspected with reference to the inspection plane. The image can be built up by using the camera to capture one or more lines at a time. A trigger 124, e.g. an optical sensor or the like, can be used to initiate image capture by detecting a characteristic feature, e.g. an edge or other trigger image, on the object as it approaches the inspection plane.

[0099] In order to obtain multiple surface images of the surface of a moving object, some form of multiplexing is required. Any one of the following may be used: [0100] temporal multiplexing: each light source is projected on to the surface at different times. The light can be of any wavelength, as desired. Image capture is synchronised with the activation of each light in sequence. [0101] spatial multiplexing: each light source projects onto a different (i.e. non-overlapping) region of the surface. The lights can be of any wavelength and are projected to different (non-coincident) positions on the inspection plane. Camera scanning then occurs simultaneously at different locations, each image is built up as the surface moves through these locations. [0102] spectral multiplexing: each light source projects a beam at a different predominant wavelength (or band of wavelengths). Spectrally matched sensing is then used to simultaneously acquire the surface images. [0103] hybrid multiplexing: a combination of two or more of the multiplexing methods described above.

[0104] Examples of these techniques are discussed in further detail below.

[0105] The multiple surface images are sent to a computer processing device 122 for analysis. As mentioned above, the analysis in this case involves generating a bump map and albedo for the surface of the object by applying a lighting model (e.g. Lambert's law) to known information about the output intensity of each source, the incident light vector from each source at each inspection point and the measured intensity at each inspection point.

[0106] The use of in-plane illuminates may be particularly useful for detecting discontinuities caused by edge effects. This may be desirable in situations where sensitivity to broken surface (e.g. torn sheet material) is needed. Surface discontinuities can cause a rapid change in the sense of the surface normals in the region of the discontinuity and may also produce a shadow, that may provide useful information. A wide distribution of illuminates allows accurate detection and determination of discontinuities. By monitoring for sudden change between illuminates we can detect discontinuities, e.g. caused by small edges. Shadows manifest themselves as large changes between widely spaced illuminates.

[0107] FIG. 7 illustrates how multiple in-line illuminates provide enhanced sensitivity to edge effects. In FIG. 7 a surface 50 with an abrupt step therein is arranged under three in-plane illuminates 51, 52, 53. Incident vectors from illuminates 51, 52 are used to define normal A at a first inspection point 54 while incident vectors from illuminates 52, 53 are used to define normal B at a second inspection point 55. The presence of the step will cause a large difference in the intensities received by illuminate 51 and illuminate 53 (there will be a shadow in the image taken using illuminate 51). The information from all three images is thus richer than that obtained from only two. The shadow pattern obtained using illuminate 51 may itself be a source of useful information in conjunction with the albedo and bump map.

[0108] Another advantage of having multiple in-line illuminates is that it is possible to provide an illuminate position close to the camera, which enables detection of specular features, e.g. a significantly increased intensity with a rapid drop-off (i.e. a high rate of intensity change when moving between illuminates). This type of technique may permit reliable detection of foreign objects on a surface under inspection. For example, it may detect the presence of transparent tape on a matt surface (e.g. paper-based or polymer-based banknote) because the ratio of specular reflection to diffuse reflection is expected to be different between the tape and the matt surface.

[0109] It is also possible to implement a specular form of the photometric stereo method [7, 8] in order to recover dense surface normal data (perhaps representing a hidden signature) from shiny, glossy specular regions, e.g. a metallic surface or shiny plastic area. This type of photometric stereo analysis may use a specular reflection model rather than a Lambertian (diffuse) reflection model. The same illuminates and image capturing device may be used for the specular photometric stereo measurements as for the normal (diffuse) measurements.

[0110] FIG. 8 illustrates schematically an alternative arrangement for providing multiple in-plane illuminates that can be used when the object to be inspected is flexible, e.g. made from a sheet-like material such as paper or flexible plastic (e.g. polypropylene-based banknotes). Instead of providing multiple light source in separate physical locations or moving a single light source between separate physical locations, the arrangement in FIG. 8 uses a single light source 60 in conjunction with a curved inspection area 62, whereby from the point of view of the surface being inspected, the single light source 60 to appears to sweep through space.

[0111] Thus, as the flexible surface 64 passes around the curved inspection area 62, the incident vector from the light source has a different angle at different locations on the surface. Spatial multiplexing may be used to generate a series of images of the whole surface corresponding to different incident angles. FIG. 8 illustrates three inspection points 66, 68, 70. The three images captured at these locations using spatial multiplexing give effectively the same result as three separate illuminates and a flat inspection plane.

[0112] Sending the note through a curved path may also serve to accentuate features of interest. For example, tears in the surface may open which facilitates their detection.

[0113] Other types of surface manipulation may also be used for this purpose. For example, the surface may be twisted or bent at its edges by means of bevels or the like. In another example, a pressure difference may be applied across the surface to expose defects therein. For example, a partial vacuum may be applied to the surface note that may also be useful for opening tears. Alternatively, a blade of air could be used to induce a profile in the path of the note, or a vacuum could be used to pull the note into a recess to enable the edge of the surface to be examined for a discontinuity caused by a tear.

Banknote Inspection Apparatus

[0114] The technique of dynamic photometric stereo measurement may be particularly suitable for inspecting currency, i.e. sheet-like banknotes made of paper (i.e. cotton-based) or plastic (i.e. polypropylene-based), where 2D coloured patterns may be concomitant with important 3D topographic texture. Inspection may be performed either during manufacture, e.g. as part of a process control system, or on used notes to determine if they are fit for continued circulation.

[0115] The inspection of banknotes is characterised by two particular difficulties: the size and varied nature of potential defects, and the required processing speed, since banknotes may move at a rate of up to 40 notes per second (i.e. a sheet speed of up to 10 ms.sup.1) during processing and manufacture.

[0116] The latter problem has two aspects. As the speed of the notes increases, there is an expected increase in processing demand (i.e. an increase in the volume of collected data that needs to be assessed) and a decrease in the duration of the imaging window.

[0117] A solution to the increase in processing demand is to implement a more efficient processing system, i.e. high-power computation and data handling methods. For example, the processing device used to handle the output from the cameras may comprise a dedicated Graphics Processor Units (GPU) on a parallel computing platform (e.g. Nvidia CUDA). Such platforms are used in high-performance gaming workstations. Accelerated computing is possible by splitting processes between a GPU and Central Processing Unit (CPU). Computationally intensive algorithms, such as photometric stereo with surface normal manipulation, will be offloaded to the GPU. This is facilitated by the availability of more efficient and more abundant cores apparent in Nvidia GPUs and the parallel computing platform provided by Nvidia CUDA. This arrangement may enable inspection speeds of 2 ms.sup.1 or more, e.g. up to 10 ms.sup.1. This may correspond to single note processing times in a range from 25 ms (processing while acquiring) to 3 ms (processing following acquisition).

[0118] Some pre-processing may occur at the camera or at the interface between the camera and the processor. Preferably the image capture device communicates with the processor via an interface module (e.g. a frame grabber card which uses the Camera Link protocol standard) that incorporates an FPGA to enable some data pre-processing to be done on the cards, thereby reducing the amount of data to be transferred to the workstation and speeding up the processing.

[0119] The processing time may be further reduced through use of other known data management/reduction techniques (e.g. radial lens distortion reduction and image mosaicing).

[0120] Furthermore, as the majority of 3D features and/or defects on a banknote are generally vertical in orientation, the reduced dimensionality of the multiple in-plane illuminate detection system described above with reference to FIGS. 3 to 8 is applicable. This system may allow the defects to be detected with high sensitivity and accuracy without a proportional increase in the processing demand.

[0121] In order to address the problem of limited image capture duration, the image capture device may include high performance (200 kHz) line-scan cameras or contact image sensors (CIS) (1 or more) in combination with very high intensity light sources. For example, each illuminate may comprise a very high intensity LED line light which has the ability to deliver a suitable light level for each image. For example, one such light source is a CORONA II product line from Chromasens GmbH. High intensity light is desirable because it is necessary to use very short shutter times to capture the images. For surfaces moving at 10 m/s, for example, it may be desirable to capture a line with an exposure time of around 5 microseconds. The exact brightness will depend upon the response of the camera sensor. Each LED may output a collimated beam to enable the surface to be illuminated from a distance at a specified angle.

[0122] The overall apparatus for performing dynamic photometric stereo inspection of banknotes may use the apparatus shown in FIG. 6 (discussed above). The apparatus may be implemented with two or more illuminates (four are illustrated in FIG. 6). It may be preferable to use three or more in-plane illuminates to take advantage of the enhanced data capture concept discussed above.

[0123] In a preferred embodiment, the banknote inspection apparatus makes use of a methodology that combines two known dynamic photometric stereo techniques [5, 6]. These techniques are known as narrow band infrared (or colour) photometric stereo (NIRPS) and spatially multiplexed photometric stereo (SMPS), and are discussed below.

[0124] 1. Narrow Band Infrared (or Colour) Photometric Stereo

[0125] In a non-static scene the observation of the same point at the same location may not be possible if there is a temporal difference between observations. Images captured at separate times are likely to be subject to some degree of mis-registration and this in turn is likely to decouple the consistency of the Lambertian brightness-gradient based relationship [9] between the corresponding pixels of the photometrically disparate images. To overcome these issues, the banknote inspection apparatus of the present invention may capture multiple images of banknotes instantaneously, i.e. with spectral and spatial multiplexing rather than a temporal difference.

[0126] Spectral multiplexing employs illumination with different colours or frequencies of light with corresponding camera filters, and thereby offers the advantages of not needing high-speed light switching as in temporal multiplexing. However, a number of limitations usually apply when attempting to use a broadband colour photometric stereo approach.

[0127] Firstly, when deploying widely spaced channels of visible light, a coupling is found to exist between surface colour and surface gradient, in which it becomes difficult to determine whether an observed surface colour is due to an unknown arbitrary surface reflectance or whether it is due to unknown surface gradient. The problem arises due to the fact that similar components of coloured light may be reflected in similar proportions, both for a surface of particular colour or alternatively for a surface of a particular inclination. For example, a surface appearing blue to an observer may be either a white surface inclined towards a blue light or blue surface receiving equal illumination form three lights, one red, one green and one blue.

[0128] Secondly, in order to use a standard RGB colour camera, some 100 nm must separate each colour channel. This means that a surface of fixed arbitrary colour will appear at differing intensities under each coloured illuminate (e.g. a red surface will exhibit low radiance under blue illumination and high radiance under red illumination).

[0129] However, in the case of inspecting banknotes, it may be possible to overcome these problems since a full colour image of the banknote is known (or can be detected) in advance. This image can be later registered with the photometric images to resolve any potential ambiguity between colour and surface normal direction.

[0130] Alternatively the illuminates may operate in narrow frequency channels that are closely spaced at around only 20 nm intervals or less (hence the term narrow band photometric stereo).

[0131] Sensitivity to surface colour can be further reduced by locating the channels within the infrared (IR) region (i.e. 800-900 nm) of the spectrum. Approaching medium to long wave IR, i.e. in the wavelength range 1.4 mm to 10 mm, may further reduce sensitivity to changes in surface colour, i.e. different colours become metameric to one another. Also, it is known that both CCD and particularly CMOS cameras have excellent response in the IR region of the spectrum. Using this approach, surface colour data may be decoupled from gradient data. In addition, if required, an additional now decoupled superimposed white channel (i.e. visible light signal without any IR) may be simultaneously included to provide fully registered colour data. The approach is discussed in more detail in WO 03/014214, which is incorporated herein by reference.

[0132] FIG. 9 shows the detail of an optical assembly 150 used in a NIRPS apparatus. Reflected radiation from the inspection surface 152 is received in an object lens 154 of a image capture device. In this embodiment, the incident light is received from four illuminates: three narrow band IR illuminates and a white light illuminate. The incident light is delivered from all illuminates simultaneously. In order to separate the different signals, a plurality of infra red filters 158, 160, 162 are mounted in series along a common optical axis 156 beyond the object lens 154. Each infra red filter 158, 160, 162 acts to reflect a predetermined narrow band of infra red light on to a respective CCD sensor 164, 166, 168. The filters transmit radiation having wavelengths outside their respective band. A mirror 170 reflects the white light that is transmitted through the IR filters onto an RGB CCD to obtain the coloured image for registration.

[0133] 2. Spatially Multiplexed Photometric Stereo

[0134] Spatial multiplexing involves separate images of the same surface location being acquired at different points in space. Image acquisition at the separate locations occurs simultaneously, so in order to register images between viewing positions the scan lines of the digital camera must be carefully synchronised with the velocity of the moving surface of the banknotes. Therefore the relative movement between image system and object must be precisely controlled.

[0135] In practice the separate views may be closely spaced and imaged within a single camera array. However, for three lights or more, the lighting arrangement may be awkward to implement, since closely spaced and isolated bands of directional illumination must be produced. FIG. 10 shows a schematic setup using two lights producing strips of illumination that extend into the plane of the drawing. As depicted in FIG. 10, spatial multiplexing requires the illuminated areas to be kept completely separate, which may be impractical or have space implications. At the same time, a limitation of the NIRPS technique is the inherent complexity of camera optics, as shown in FIG. 9. These problems may be overcome by adopting a hybrid approach. Here images are isolated both in terms of spectral frequency and also by a close spatial displacement of one or two lines of pixels. A schematic configuration is shown in FIG. 11. This allows for considerable simplification of camera and optics. Instead of using multiple CODs arranged off a single optical axis, adjacent CCDs (as shown) or adjacent regions of a single CCD may be used (CCDs are shown but the sensors could employ CMOS or other sensor technology, as appropriate). As with spectral multiplexing, the various illuminates may be flooded simultaneously into the inspection area, simplifying illumination in comparison with spatial multiplexing techniques, while channel separation takes place at the camera.

[0136] The dynamic photometric stereo arrangement described above employs infrared light for illustrative purposes. The illumination in the present invention can be of any frequency, and a bump map and albedo will still be produced. However it is important to note that the albedo image generated will be for the surface as illuminated under the wavelength of illumination used in the dynamic photometric stereo. If this were infrared and a realistic white light albedo were required, then an additional colour camera would be employed (either line scan or area scan), with suitable illumination (such as broadband or white light) for producing the realistic colour albedo.

[0137] The rich 3D and 2D data set that will be made available from a technique such as NIRPS and/or SMPS, will enable the fold to be detected reliably, thereby providing considerable technological and cost benefits.

[0138] These techniques may also enable other banknote characteristics to be detected, e.g. unfit holograms (scratches, excessively crumpled) and heavy crumpled (limp) banknotes. For example, 3D data may be useful for detecting the presence of transparent tape and differentiating between a closed tear and a simple mark on the note.

[0139] Furthermore, the inspection technique may be applied to check security features of banknotes. Banknote security is a critical issue for the currency supply chain industry. Current security features include covert and overt elements, both visible and non-visible to the naked eye. The application of NIRPS/SMPS methodology in particular may enable quantifiable detection of a combination of overt and covert 2D or 3D characteristics. For example, raised features on the surface of the banknote may be measured with high resolution in 3D. The resulting bump map will provide auditable datasets. The photometric stereo technique also offers the potential to measure features in both 2D (albedo) and 3D (bump maps) with very high resolutions that are not available when using other techniques. The resolution of surface recovery with photometric stereo is limited only by the resolution of the camera and lens arrangement; therefore, if required, the morphology of 3D surface features could be recovered, and modelled, at very high resolutions.

[0140] Furthermore, 3D textures or patterns located under a transparent layer such as a transparent polymer, so that no 3D texture actually exists at the note surface, may also be recoverable in 3D by employing the photometric stereo approach with suitable wavelengths of light. Also, if required the bump map gradients can be integrated to reconstruct height maps of features.

Data AnalysisBump Map Modelling Considerations

[0141] Another advantage of the dynamic photometric stereo technique is the ability to generate both qualitative and quantitative information about the surface being inspected. In the context of inspecting banknotes, this may mean analysing any of an albedo, a bump map or a shadow pattern to identify and quantify defects that are detected. The method may be able to classify the defect type, e.g. fold, hole, tear, soiling or other foreign matter (e.g. tape). The quantification may comprise an analysis of the size or position of the defect, and may be based on threshold values which mark the boundary between what is an acceptable defect and what is not (e.g. a rip beyond a certain size may in fact make a banknote cease to be legal tender).

[0142] The data analysis may comprise a step of modelling detected features of interest on the inspected surface. This type of processing differs from conventional image processing, which involves analysis of changes in intensities of pixels in bitmap images. In contrast, the present invention bases its analysis on a bump map, which comprises a dense array of surface normals. The analysis may be performed on the behaviour of the angle of the surface normal, e.g. changes that occur in the gradient of the surface. This is implemented in terms of analysis/modelling of the components of the surface normal relative to the x and y axes by employing techniques that range from curve fitting through to AI techniques such as neural networks, i.e. to convert the surface normal angle data in a discrete or continuous state ready for modelling.

[0143] The idea above is illustrated in FIGS. 12A to 12C. FIGS. 12A and 12B shows schematically a cross-section through a folding portion of a sheet of material. The sense of folding is different in FIG. 12A from FIG. 12B. FIG. 12A is an underfold whereas FIG. 12B is an overfold. If one considers the surface normal vectors in a line on the banknote that passes through the fold, and is perpendicular to it, it can be appreciated that the components of these vectors, in a plane that passes through this line and is perpendicular to the surface of the banknote, change with the curve of the banknote.

[0144] In the case of FIGS. 12A and 12B, the curves may be modelled, e.g. using polynomial equations that are fitted to the curves. Statistical techniques such as regression analysis allow quantification of how well the polynomials fit the surfaces; and a good fit obtained in the form of a high correlation coefficient.

[0145] In contrast, FIG. 12C shows the case where a corner is torn rather than folded. Here there is an interruption in the continuous surface at the tear (i.e. a step change in gradient), so that a good fit of the polynomial cannot be obtained and there is a relatively low correlation coefficient. This enables reliable identification and quantification of both folded and torn corners.

REFERENCES

[0146] [1] M. L. Smith, T. Hill, G. Smith, Surface texture analysis based upon the visually acquired perturbation of surface normals, Image and Vision Computing 15 (1997) 949-955. [0147] [2] M. L. Smith, The analysis of surface texture using photometric stereo acquisition and gradient space domain mapping, Image and Vision Computing 17 (1999) 1009-1019. [0148] [3] M. L. Smith, G. Smith, T. Hill, Gradient space analysis of surface defects using a photometric stereo derived bump map, Image and Vision Computing 17 (1999) 321-332. [0149] [4] M. L. Smith, Surface inspection techniquesusing the integration of innovative machine vision and graphical modelling techniques, Professional Engineering Publishing, ISBN 1-86058-292-3, 2000. [0150] [5] M. L. Smith, L. N. Smith, A. R. Farooq, Dynamic Photometric StereoA New Technique for Moving Surface Analysis, Image and Vision Computing 23 (2005) 841-852. [0151] [6] A. R. Farooq, M. L. Smith, L. N. Smith and P. S. Midha, Dynamic Photometric Stereo for On Line Quality Control of Ceramic Tiles, Computers in Industry, Vol. 56, 8-9, 2005. [0152] [7] R. D. Wedowski, G. Atkinson, M. L. Smith and L. N. Smith, On-line deflectometry: A novel approach for the on-line reconstruction of specular freeform surfaces, Optics and Lasers in Engineering, Elsevier, 2011. [0153] [8] Wedowski R. D., Atkinson G. A, Smith M. L. and Smith L. N, A system for industrial on-line inspection of curved specular surfaces, Optics and Lasers in Engineering, 50 (2012) 632-644. [0154] [9] B. K. P. Horn, Understanding image intensities, Artificial Intelligence 8 (1977) 201-231.