IMAGE PROCESSING DEVICE, CALIBRATION BOARD, AND METHOD FOR GENERATING 3D MODEL DATA
20230162437 · 2023-05-25
Inventors
Cpc classification
H04N13/279
ELECTRICITY
G06T7/80
PHYSICS
International classification
G06T7/80
PHYSICS
Abstract
The present technique relates to an image processing device, a calibration board, and a method for generating 3D model data, which can easily achieve synchronization among devices. An image processing device including; an image synchronization unit that performs time synchronization on a plurality of images of a board on a basis of lighting conditions of a plurality of light-emitting parts included in the plurality of images captured by a plurality of imaging devices, the board including the plurality of light-emitting parts and a predetermined image pattern; and a calibration unit that calculates camera parameters of the plurality of imaging devices by using the plurality of images having been subjected to the time synchronization. The present technique can be applied to, for example, an image processing system that captures images to generate a 3D model.
Claims
1. An image processing device comprising: an image synchronization unit that performs time synchronization on a plurality of images of a board on a basis of lighting conditions of a plurality of light-emitting parts included in the plurality of images captured by a plurality of imaging devices, the board including the plurality of light-emitting parts and a predetermined image pattern; and a calibration unit that calculates camera parameters of the plurality of imaging devices by using the plurality of images having been subjected to the time synchronization.
2. The image processing device according to claim 1, wherein the plurality of light-emitting parts have a time display part that illuminates according to an imaging time, and the image synchronization unit performs the time synchronization on the plurality of images by selecting the images in which the time display part has the same lighting condition.
3. The image processing device according to claim 2, wherein the plurality of light-emitting parts further have a position display part that illuminates according to a position of the board, and the image synchronization unit performs the time synchronization on the plurality of images by selecting the images in which the time display part has the same lighting condition from the images selected such that the different lighting conditions of the position display part are provided in a predetermined ratio.
4. The image processing device according to claim 3, wherein an imaging range of the plurality of imaging devices is divided into a plurality of sections, and the position display part of the plurality of light-emitting parts illuminates according to the sections.
5. The image processing device according to claim 3, wherein the image synchronization unit selects the images such that the different lighting conditions of the position display part are equally allocated.
6. The image processing device according to claim 1, wherein the plurality of light-emitting parts have a board display part that illuminates to identify the board, and the image synchronization unit performs the time synchronization on the plurality of images by selecting the images in which the board display part has the same lighting condition.
7. The image processing device according to claim 1, wherein the board is configured with the light-emitting parts disposed in a pattern of the predetermined image pattern.
8. The image processing device according to claim 1, wherein the board is configured with the light-emitting parts disposed in a region different from a region of the predetermined image pattern.
9. The image processing device according to claim 2, wherein the time display part of the board changes the lighting condition at each lapse of a unit time.
10. The image processing device according to claim 3, wherein the board further includes a position information detection unit that detects position information, and the position display part changes the lighting condition according to a detection result of the position information detection unit.
11. The image processing device according to claim 3, wherein the board further includes an operation unit that receives a user operation, and the position display part changes the lighting condition in response to an operation on the operation unit.
12. The imaging device according to claim 1, wherein the light-emitting parts of the board illuminate in different colors according to 0 or 1.
13. The imaging processing device according to claim 1, wherein the light-emitting parts of the board are illuminated or unilluminated according to 0 or 1.
14. The imaging processing device according to claim 1, further comprising an extraction unit that determines whether the lighting conditions of the plurality of light-emitting parts included in the images have changed, and extracts the changed images, wherein the image synchronization unit performs time synchronization on the plurality of images on the basis of the lighting conditions of the plurality of light-emitting parts included in the plurality of extracted images.
15. A calibration board comprising a plurality of light-emitting parts that change lighting conditions at each lapse of a unit time, and a predetermined image pattern, wherein the plurality of light-emitting parts are caused to illuminate to perform time synchronization on a plurality of images captured by a plurality of imaging devices.
16. A method for generating 3D model data, the method comprising: performing time synchronization on a plurality of images of a board on a basis of lighting conditions of a plurality of light-emitting parts included in the plurality of images captured by a plurality of imaging devices, the board including the plurality of light-emitting parts and a predetermined image pattern; calculating camera parameters of the plurality of imaging devices by using the plurality of images having been subjected to the time synchronization; generating a 3D model of a predetermined subject from a plurality of subject images of the predetermined subject, the subject images being captured by the plurality of imaging device by using the calculated camera parameters; and generating a virtual viewpoint image by viewing the generated 3D model of the predetermined subject from a predetermined viewpoint.
Description
BRIEF DESCRIPTION OF DRAWINGS
[0019]
[0020]
[0021]
[0022]
[0023]
[0024]
[0025]
[0026]
[0027]
[0028]
[0029]
[0030]
[0031]
[0032]
[0033]
[0034]
[0035]
[0036]
[0037]
[0038]
[0039]
DESCRIPTION OF EMBODIMENTS
[0040] A mode for embodying the present disclosure (hereinafter referred to as an embodiment) will be described below with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration will be denoted by the same reference numerals, and thus repeated descriptions thereof will be omitted. The description will be given in the following order.
1. Outline of volumetric capture
2. Configuration example of image processing system
3. Calibration using calibration board
4. Block diagram
5. Position pattern assignment
6. Time information lighting
7. Position information lighting
8. Image extraction
9. Calibration
[0041] 10. Modification example of calibration board
11. Configuration example of 3D model generation
12. Flowchart of 3D model generation
13. Computer configuration example
[0042] <1. Outline of Volumetric Capture>
[0043] An image processing system according to the present disclosure relates to volumetric capture for providing an image with a free viewpoint (free viewpoint image) by generating a 3D model of a subject from images captured from multiple viewpoints and generating a virtual viewpoint image of the 3D model according to any viewing position.
[0044] First, referring to
[0045] For example, a plurality of captured images can be obtained by imaging a predetermined imaging space, in which subjects such as a person are disposed, from the outside by using a plurality of imaging devices. The captured image includes, for example, a moving image. In the example of
[0046] A 3D object MO1, which is a 3D model of the subject Ob1 to be displayed in the imaging space, is generated by using the captured images obtained from the plurality of imaging devices CAM in different directions (3D modeling). The 3D object MO1 is generated by using, for example, a scheme such as Visual Hull in which a three-dimensional shape of a subject is cut out using images captured in different directions.
[0047] From one or more 3D objects in the imaging space, data on the one or more 3D objects (hereinafter also referred to as 3D model data) is transmitted to a device on the reproduction side and is reproduced therein. In other words, by rendering the acquired 3D object based on the data on the 3D object in the device on the reproduction side, a two-dimensional image of the 3D object is displayed on the viewing device of a viewer.
[0048] On the reproduction side, only a 3D object to be viewed can be requested from one or more 3D objects in the imaging space and can be displayed on the viewing device. For example, on the reproduction side, a virtual camera is assumed to have an imaging range equivalent to the viewing range of a viewer, and only a 3D object to be captured by the virtual camera is requested from multiple 3D objects in the imaging space and is displayed on the viewing device. The viewpoint (virtual viewpoint) of the virtual camera may be set at any position such that a viewer can view a subject from any viewpoint in the real world. A background image representing a predetermined space can be optionally combined with the 3D object.
[0049] <2. Configuration Example of Image Processing System>
[0050]
[0051] An image processing system 1 of
[0052] The N cameras 12-1 to 12-N are imaging devices that capture subject images. As described with reference to
[0053] The image processing device 11 includes, for example, a personal computer and a server. The image processing device 11 controls the timing of imaging by the cameras 12-1 to 12-N, acquires moving images captured by the cameras 12, and performs predetermined image processing such as the generation of a 3D model based on the acquired moving images.
[0054] In order to allow the image processing device 11 to generate a 3D model of a subject by using moving images captured by the cameras 12, the positional relationship among the cameras 12 needs to be a known relationship. Moreover, in order to calculate the positional relationship among the cameras 12, the moving images of the cameras 12 need to be synchronized with one another. Thus, before imaging for generating a 3D model, the image processing device 11 performs calibration for calculating the positional relationship among the cameras 12, specifically, the positions and orientations of the cameras 12 on world coordinates by using the synchronized moving images of the cameras 12. The positions and orientations of the cameras 12 are the external parameters of the cameras 12. The internal parameters of the cameras 12 are assumed to be known parameters.
[0055] When the image processing device 11 causes the cameras 12 to capture images, the image processing device 11 generates a control signal for providing an instruction to start or terminate the capturing of images and a synchronizing signal, and supplies the signals to each of the cameras 12-1 to 12-N.
[0056] Referring to
[0057]
[0058] The timings to start capturing the moving image 14-1 and the moving image 14-2 are not synchronized with each other, and the phases of exposure timing (the timings to start and terminate exposure) are not synchronized with each other.
[0059] The timings to start capturing the moving image 14-2 and the moving image 14-3 are not synchronized with each other, but the phases of exposure timing are synchronized with each other.
[0060] The timings to start capturing the moving image 14-2 and the moving image 14-4 are synchronized with each other, and the phases of exposure timing are also synchronized with each other.
[0061] Moving images generated by the cameras 12-1 to 12-N on the basis of the control signal for starting and terminating the capturing of images and the synchronizing signal have the synchronization relationship like the moving image 14-2 and the moving image 14-3, the control signal and the synchronizing signal being supplied from the image processing device 11. In other words, the moving images are generated by the cameras 12-1 to 12-N such that the phases of exposure timing are synchronized with each other, but the timings to start capturing the moving images are not synchronized with each other.
[0062] Returning to
[0063] In a state where the positional relationship among the cameras 12 is known due to the calibration, the image processing device 11 causes the cameras 12 to capture images of a predetermined subject serving as a target of 3D model generation. For example, the cameras 12 capture images of a person making a predetermined motion, as a predetermined subject serving as a target of 3D model generation. The image processing device 11 generates a 3D model of an object, which is a person imaged as a subject, from multiple moving images supplied from the cameras 12-1 to 12-N.
[0064] Furthermore, the image processing device 11 can generate a virtual viewpoint image by viewing the generated 3D model of the object from any virtual viewpoint and display the image on the display device 13. The display device 13 includes, for example, the display D1 or the head-mounted display (HMD) D2 that are illustrated in
[0065] Communications between the image processing device 11 and the cameras 12-1 to 12-N and communications between the image processing device 11 and the display device 13 may be direct communications via a cable or the like or communications via a predetermined network, e.g., a LAN (Local Area Network) or the Internet. The communications may be wire communications or radio communications. The image processing device 11 and the display device 13 may be integrated into a single unit.
[0066] In the image processing system 1 configured thus, calibration for calculating the positional relationship among the cameras 12, that is, the externa parameters of the cameras 12 is first performed by using moving images of a predetermined calibration board imaged by the cameras 12.
[0067] When the positional relationship among the cameras 12 is known, a predetermined subject as a target of 3D model generation is imaged by the cameras 12, and a 3D model of an object, which is the predetermined subject, is generated on the basis of multiple moving images captured by the cameras 12.
[0068] <3. Calibration Using Calibration Board>
[0069] First, calibration using the calibration board will be described in detail.
[0070]
[0071] A calibration board 21 in
[0072] Moreover, at least one operation button 24 is disposed at a predetermined point of the calibration board 21. The operation button 24 is operated by a user when an operation is performed to start or terminate a light-emitting operation in the 44 light-emitting parts 23.
[0073] The light-emitting part 23 disposed in each of the black patterns of the image pattern 22 is composed of an LED (Light Emitting Diode) or the like. The light-emitting part 23 can provide two lighting conditions, for example, an illuminated state and an unilluminated state of white light. Alternatively, the light-emitting parts 23 may illuminate in multiple colors, for example, red and green.
[0074] The 44 light-emitting parts 23 are grouped into a time display part 31 that illuminates according to a time, and a position display part 32 that illuminates according to a position. In the example of
[0075] The time display part 31 illuminates to perform time synchronization on moving images captured by the plurality of cameras 12. The time display part 31 associates “illuminated” or “unilluminated” of the light-emitting part 23 with a bit of “0” or “1.” The 39 light-emitting parts 23 display 39-bit time information.
[0076] The position display part 32 associates “illuminated” or “unilluminated” of the light-emitting part 23 with a bit of “0” or “1.” The five light-emitting parts 23 display 5-bit position information.
[0077] For example, as illustrated in
[0078] The position display part 32 constitutes 5-bit bit strings in which the light-emitting part 23 on the left end serves as a least significant bit (LSB) and the light-emitting part 23 on the right end serves as a most significant bit (MSB). When the light-emitting parts 23 of the position display part 32 are illuminated or unilluminated as illustrated in the example of
[0079] If the light-emitting parts 23 can provide illumination in multiple colors, “0” or “1” may be represented by a color difference instead of illuminated/unilluminated. For example, “red illumination” may be represented as “1” and “green illumination” may be represented as “0.”
[0080] Referring to
[0081] The time display part 31 increments (updates) a 39-bit bit value on the basis of the internal timer of the calibration board 21 at each lapse of a predetermined unit time. In the calibration, the cameras 12 capture the moving images of the calibration board 21, and the lighting pattern of the time display part 31 of the calibration board 21 is identified in the moving images, thereby determining an imaging time.
[0082]
[0083] The lighting pattern of the time display part 31 of the calibration board 21 in the moving images of
[0084] The p-th frame (p is a natural number) of the moving image captured by the camera 12-1 in the calibration includes the calibration board 21 in which the time display part 31 has a lighting pattern “11010011.” The (p+1)-th frame includes the calibration board 21 in which the time display part 31 has a lighting pattern “11010100.” The (p+2)-th frame includes the calibration board 21 in which the time display part 31 has a lighting pattern “11010101.”
[0085] The p-th frame of the moving image captured by the camera 12-2 in the calibration includes the calibration board 21 in which the time display part 31 has a lighting pattern “11010101.” The (p+1)-th frame includes the calibration board 21 in which the time display part 31 has a lighting pattern “11010110.” The (p+2)-th frame includes the calibration board 21 in which the time display part 31 has a lighting pattern “11010111.”
[0086] Thus, in the (p+2)-th frame of the camera 12-1 and the p-th frame of the camera 12-2, the frames being surrounded by frames in
[0087] As described with reference to
[0088] As illustrated in
[0089] Referring to
[0090]
[0091] An imaging space 41 is determined on the basis of the imaging range of the eight cameras 12-1 to 12-8. In the example of
[0092] In the present embodiment, on the assumption that a user or a self-propelled robot moves with the calibration board 21 on the floor of the imaging space 41 when an image is captured, only a two-dimensional region corresponding to the floor of the imaging space 41 is examined. The imaging space 41 is also referred to as an imaging region 41.
[0093] The N cameras 12-1 to 12-N are annularly disposed at predetermined intervals (for example, regular intervals) outside the imaging region 41 so as to face the center of the imaging region 41.
[0094] The square imaging region 41 is divided into a plurality of sections 42. A predetermined bit value that can be represented by the position display part 32 is allocated to each of the sections 42.
[0095] For example, as illustrated in
[0096] The calibration board 21 includes a position information detection unit, for example, a GPS module capable of acquiring position information. The 5-bit position information of the position display part 32 is controlled according to the position of the calibration board 21 in one of the four sections 42A to 42D of the imaging region 41. In the calibration, the cameras 12 capture the moving images of the calibration board 21, and the lighting pattern of the position display part 32 of the calibration board 21 is identified in the moving images, thereby determining the position of the calibration board 21, specifically, which one of the sections 42A to 42D has the calibration board 21 when an image is captured.
[0097] It is known that the accuracy of calibration for calculating the positional relationship among the cameras 12 is increased by detecting the feature points of the image pattern 22 of the calibration board 21 at various positions evenly in the imaging region (imaging space) 41.
[0098] The section of the frame image captured as the calibration board 21 can be determined by identifying the lighting pattern of the position display part 32 of the calibration board 21 included in the frame images of in the moving images captured by the cameras 12, so that the frame image used for calibration can be evenly selected from the four sections 42A to 42D in the imaging region 41.
[0099] In the example of
[0100] Referring to
[0101] As illustrated in
[0102] In this way, if the two cameras 12 have the common imaging range 46, the positional relationship between the two cameras 12 can be directly calculated on the basis of the feature points of the image pattern 22 of the calibration board 21 in frame images captured in synchronization with each other by the two cameras 12.
[0103] For example, even if the two cameras 12-1 and 12-N do not have the common imaging range 46 as illustrated in
[0104] <4. Block Diagram>
[0105]
[0106] The calibration board 21 includes a position information detection unit 51, an operation unit 52, a control unit 53, and an information display unit 54.
[0107] The position information detection unit 51 includes, for example, a GPS (Global Positioning System) module. The position information detection unit 51 detects current position information on the calibration board 21 and supplies the information to the control unit 53.
[0108] The operation unit 52 corresponds to the operation button 24 of
[0109] The control unit 53 controls the display of the information display unit 54, specifically, the lighting of the 44 light-emitting parts 23 on the basis of the position information supplied from the position information detection unit 51 and the operation signal supplied from the operation unit 52.
[0110] The information display unit 54 corresponds to the 44 light-emitting parts 23 of
[0111]
[0112] The image processing device 11 includes a moving image acquisition unit 71, an image extraction unit 72, an extracted image storage unit 73, an image synchronization unit 74, a calibration unit 75, and a camera parameter storage unit 76.
[0113] The moving image acquisition unit 71 acquires the moving images of the calibration board 21 from the plurality of cameras 12 and supplies the moving images to the image extraction unit 72.
[0114] The image extraction unit 72 performs image extraction for extracting a time lighting-pattern changed frame image from the moving image supplied from the plurality of cameras 12. More specifically, the image extraction unit 72 extracts, as a time lighting-pattern changed frame image, a frame image after the lighting pattern of the time display part 31 of the calibration board 21 in the moving image changes from that of the preceding frame image, and the image extraction unit 72 supplies the extracted time lighting-pattern changed frame image to the extracted image storage unit 73.
[0115] The extracted image storage unit 73 stores a plurality of time lighting-pattern changed frame images that are extracted from the moving images of the cameras 12 in the image extraction unit 72.
[0116] The image synchronization unit 74 selects the time lighting-pattern changed frame images such that the four sections 42A to 42D of the imaging region 41 are allocated in a predetermined ratio on the basis of the lighting conditions of the position display part 32 of the calibration board 21 in the time lighting-pattern changed frame images stored in the extracted image storage unit 73. For example, the image synchronization unit 74 selects the time lighting-pattern changed frame image such that the four sections 42A to 42D are equally allocated.
[0117] Furthermore, the image synchronization unit 74 performs time synchronization on the plurality of time lighting-pattern changed frame images, which are selected such that the sections 42 are allocated in a predetermined ratio, on the basis of the lighting conditions of the time display part 31 in the frame images. In other words, the image synchronization unit 74 collects frame images in which the lighting conditions of the time display part 31 indicate the same time. The plurality of time lighting-pattern changed frame images captured at the same time are supplied to the calibration unit 75.
[0118] The calibration unit 75 performs calibration for calculating the external parameters of the N cameras 12 by using the plurality of lighting-pattern changed frame images that are time-synchronized images. More specifically, by using a plurality of time lighting-pattern changed frame images captured by two cameras 12-A and 12-B (A and B are natural numbers of 1 to N and are different from each other) at the same time, the calibration unit 75 sequentially performs, on the N cameras 12-1 to 12-N, processing for calculating the positional relationship between the cameras 12-A and the camera 12-B. The external parameters of the N cameras 12 are stored in the camera parameter storage unit 76, the external parameters being obtained by the calibration.
[0119] The camera parameter storage unit 76 accommodates the external parameters of the N cameras 12, the external parameters being supplied from the calibration unit 75.
[0120] The image processing device 11 is configured as above:
[0121] <5. Position Pattern Assignment>
[0122] Referring to the flowchart of
[0123] First, in step S1, the control unit 53 of the calibration board 21 acquires the position information of the imaging region 41. For example, when a user carrying the calibration board 21 moves in the outer edge of the imaging region 41, position information corresponding to the outer edge of the imaging region 41 is supplied from the position information detection unit 51 to the control unit 53 and is stored in internal memory, so that the position information of the imaging region 41 is acquired. The method of acquiring the position information of the imaging region 41 is not particularly limited. For example, position information on the four corners of a rectangle corresponding to the imaging region 41 may be inputted.
[0124] In step S2, the control unit 53 divides the imaging region 41 into the plurality of sections 42 after the position information is acquired. For example, as illustrated in
[0125] In step S3, the control unit 53 sets a correlation between the plurality of sections 42 split in the imaging region 41 and the lighting pattern of the position display part 32 and stores the correlation. Specifically, as illustrated in
[0126] In step S3, when the correlation between the plurality of sections 42 split in the imaging region 41 and the lighting pattern of the position display part 32 is stored in the control unit 53, the position pattern assignment is completed.
[0127] At the completion of the position pattern assignment of
[0128] In the processing for capturing the images of the calibration board 21, the control signal for providing an instruction to start capturing images and the synchronizing signal are supplied from the image processing device 11 to the cameras 12. The cameras 12 start capturing images in response to the control signal for providing an instruction to start capturing images, and capture moving images (capture a moving image for each frame) in response to the synchronizing signal.
[0129] While the cameras 12 capture moving images, for example, the user carrying the calibration board 21 moves in the imaging region 41. The cameras 12 capture at least the moving images of the calibration board 21 in the imaging region 41.
[0130] <6. Time Information Lighting>
[0131]
[0132] First, in step S21, the control unit 53 sets “0” for a variable tb corresponding to the time information of the time display part 31. The variable tb corresponds to a value obtained by expressing, as a decimal number, a 39-bit bit value in binary form.
[0133] In step S22, the control unit 53 illuminates the time display part 31 (39 light-emitting parts 23) in a lighting pattern corresponding to a time tb. The lighting pattern corresponding to the time tb is a pattern in which the variable tb in decimal form is represented as a 39-bit bit string (binary), “0” represents an unilluminated state, and “1” represents an illuminated state.
[0134] In step S23, the control unit 53 determines whether a predetermined unit time has elapsed. The processing of step S23 is repeated until it is determined that the predetermined unit time has elapses. The predetermined unit time corresponds to the time of one bit of the time display part 31.
[0135] If it is determined that the predetermined unit time has elapsed in step S23, the processing advances to step S24, and the control unit 53 increments the variable tb, which corresponds to time information, by “1.”
[0136] In step S25, it is determined whether an operation to terminate the lighting of the information display unit 54 has been performed.
[0137] If it is determined in step S25 that the operation to terminate the lighting of the information display unit 54 has not been performed, the processing returns to step S22, and the processing of steps S22 to S25 is performed again. In other words, the time display part 31 (39 light-emitting parts 23) is illuminated for the predetermined unit time in the lighting pattern corresponding to the variable tb incremented by “1.”
[0138] In step S25, if it is determined that the operation to terminate the lighting of the information display unit 54 has been performed, the time information lighting is terminated.
[0139] As described above, the information display unit 54 of the calibration board 21 changes the lighting condition at each lapse of the unit time.
[0140] <7. Position Information Lighting>
[0141]
[0142] First, in step S41, the control unit 53 acquires current position information from the position information detection unit 51 and illuminates the position display part 32 (5 light-emitting parts 23) in a lighting pattern corresponding to the current position. The lighting pattern corresponding to the current position is a pattern in which “0” represents an unilluminated state, and “1” represents an illuminated state in a 5-bit bit string (binary) assigned to the section 42 including the current position.
[0143] In step S42, the control unit 53 determines whether position information supplied from the position information detection unit 51 has changed. The processing of step S42 is repeated until it is determined that the position information has changed.
[0144] In step S42, if it is determined that the position information has changed, the processing advances to step S43. The control unit 53 determines whether a movement across the section 42 of the imaging region 41 has been made before and after a change of the position information.
[0145] In step S43, if it is determined that a movement across the section 42 has been made before and after a change of the position information, the processing advances to step S44, and the control unit 53 illuminates the position display part 32 (five light-emitting parts 23) in the lighting pattern corresponding to the current position.
[0146] If it is determined in step S43 that a movement across the section 42 has not been made, step S44 is skipped.
[0147] In step S45, the control unit 53 determines whether an operation to terminate the lighting of the information display unit 54 has been performed.
[0148] If it is determined in step S45 that the operation to terminate the lighting of the information display unit 54 has not been performed, the processing returns to step S42, and the processing of steps S42 to S45 is performed again. In other words, the processing for illuminating the position display part 32 (five light-emitting parts 23) is continued in the lighting pattern corresponding to the current position.
[0149] In step S45, if it is determined that the operation to terminate the lighting of the information display unit 54 has been performed, the position information lighting is terminated.
[0150] As described above, the position display part 32 of the calibration board 21 changes the lighting condition according to the sections 42.
[0151] The time information lighting in
[0152] <8. Image Extraction>
[0153] Referring to the flowchart of
[0154] The moving images of the calibration board 21 are inputted from the plurality of cameras 12 to the image processing device 11. The image extraction of
[0155] In the present embodiment, it is assumed that the unit time of one frame based on the frame rate of a moving image is shorter than a unit time during which the lighting pattern of the time display part 31 of the calibration board 21 changes and timings to start capturing images are synchronized with each other in a frame immediately after the lighting pattern of the time display part 31 changes.
[0156] First, in step S61, the moving image acquisition unit 71 acquires (a frame image of) one frame of a moving image inputted from the camera 12 and supplies the frame to the image extraction unit 72.
[0157] In step S62, the image extraction unit 72 determines whether the frame image supplied from the moving image acquisition unit 71 includes the calibration board
[0158] If it is determined in step S62 that the frame image supplied from the moving image acquisition unit 71 does not include the calibration board 21, the processing returns to step S61 and the processing of steps S61 and S62 is performed again. Thus, the frame images of moving images are searched until it is determined that the frame image includes the calibration board 21.
[0159] In step S62, if it is determined that the frame image includes the calibration board 21, the processing advances to step S63. The image extraction unit 72 identifies the lighting pattern of the time display part 31 of the calibration board 21 in the frame image and stores the lighting pattern therein.
[0160] Subsequently, in step S64, the moving image acquisition unit 71 determines whether (a frame image of) the subsequent frame of a moving image is present, in other words, whether a subsequent frame has been inputted from the camera 12.
[0161] In step S64, if it is determined that the subsequent frame of a moving image is absent, the image extraction is terminated.
[0162] If it is determined in step S64 that the subsequent frame of a moving image is present, the processing advances to step S65. The moving image acquisition unit 71 acquires (a frame image of) the subsequent frame inputted from the camera 12 and supplies the frame to the image extraction unit 72. The frame acquired in step S61 will be referred to as the preceding frame, and the frame acquired in step S65 will be referred to as the current frame.
[0163] In step S66, the image extraction unit 72 determines whether the frame image of the current frame supplied from the moving image acquisition unit 71 includes the calibration board 21.
[0164] If it is determined in step S66 that the frame image of the current frame does not include the calibration board 21, the processing returns to step S61 and the processing of step S61 and later is performed again. Specifically, if the calibration board 21 is not included in any one of the frame images of the preceding frame and the current frame, the image processing device 11 acquires a preceding frame again.
[0165] If it is determined in step S66 that the frame image of the current frame includes the calibration board 21, the processing advances to step S67. The image extraction unit 72 determines whether the lighting pattern of the time display part 31 in the frame image of the current frame has changed from the frame image of the preceding frame.
[0166] If it is determined in step S67 that the lighting pattern of the time display part 31 of the current frame has not changed from the frame image of the preceding frame, the processing returns to step S64 and the foregoing steps S64 to S67 are repeated. In steps S64 to S67, the subsequent frame of the moving image is acquired as the current frame, and it is determined whether the lighting pattern of the time display part 31 has changed.
[0167] If it is determined in step S67 that the lighting pattern of the time display part 31 of the current frame has changed from the frame image of the preceding frame, the processing advances to step S68. The image extraction unit 72 associates time information identified from the lighting pattern of the time display part 31 and position information (section 42) identified from the lighting pattern of the position display part 32 with the frame image of the current frame, and stores the frame image in the extracted image storage unit 73. The stored frame image of the current frame in the extracted image storage unit 73 is the time lighting-pattern changed frame image.
[0168] After step S68, the processing returns to step S63, and the foregoing processing is repeated. Specifically, the lighting pattern of the time display part 31 of the calibration board 21 in the frame image of the current frame is internally stored as the information of the preceding frame, the subsequent frame is acquired as the current frame, it is determined whether the lighting pattern of the time display part 31 has changed in the current frame from the preceding frame, and if the lighting pattern has been changed, time information and position information are stored while being associated with the frame image of the current frame. If it is determined that the subsequent frame of the moving image is absent, the image extraction is terminated.
[0169] Through the image extraction, at least one time lighting-pattern changed frame image is extracted from a moving image and is stored in the extracted image storage unit 73 along with the time information and position information of the calibration board 21 in the frame image.
[0170] The image extraction of
[0171] The image processing device 11 may temporarily store moving images, which are outputted from the cameras 12, in the device and then perform the image extraction of
[0172] In the image extraction, as described above, a frame image is extracted when the lighting pattern of the time display part 31 changes. Thus, the unit time of one frame of a moving image is preferably shorter than the unit time during which the lighting pattern of the time display part 31 of the calibration board 21 changes. The unit time of one frame of a moving image may be set longer than or as long as the unit time during which the lighting pattern of the time display part 31 changes.
[0173] <9. Calibration>
[0174] Referring to the flowchart of
[0175] First, in step S81, the image synchronization unit 74 selects the time lighting-pattern changed frame images such that the four sections 42A to 42D of the imaging region 41 are allocated in a predetermined ratio on the basis of position information associated with the time lighting-pattern changed frame images of the extracted image storage unit 73. For example, the image synchronization unit 74 selects the time lighting-pattern changed frame image such that the four sections 42A to 42D are equally allocated.
[0176] In step S82, the image synchronization unit 74 performs time synchronizations on the time lighting-pattern changed frame images on the basis of time information associated with the time lighting-pattern changed frame images of the extracted image storage unit 73. In other words, the image synchronization unit 74 selects (collects) the time lighting-pattern changed frame images captured at the same time, on the basis of the time information associated with the time lighting-pattern changed frame images. The selected time lighting-pattern changed frame images are supplied to the calibration unit 75.
[0177] In step S83, the calibration unit 75 performs calibration for calculating the external parameters of the N cameras 12 by using the time lighting-pattern changed frame images that are time-synchronized and supplied from the image synchronization unit 74. More specifically, by using time lighting-pattern changed frame images captured by the two cameras 12-A and 12-B at the same time, the calibration unit 75 sequentially performs, on the N cameras 12-1 to 12-N, processing for calculating the positional relationship between the cameras 12-A and the camera 12-B. The external parameters of the N cameras 12 are supplied to the camera parameter storage unit 76 and are stored therein, the external parameters being obtained by the calibration.
[0178] Hence, the calibration using the time lighting-pattern changed frame images, which are time-synchronized, is completed.
[0179] By the calibration of
[0180] Moreover, on the basis of time information associated with the time lighting-pattern changed frame images, the plurality of time lighting-pattern changed frame images captured at the same time can be easily selected. In other words, synchronization can be easily achieved among the devices. Hence, calibration for calculating the external parameters of the cameras 12 can be performed by using the synchronized time lighting-pattern changed frame images.
[0181] The foregoing example described equal allocation in which the four sections 42A to 42D are allocated in a predetermined ratio. The sections 42A to 42D do not always need to be equally allocated. For example, if the locations of subjects serving as targets of 3D model generation are biased in the imaging region 41, the sections may be distributed according to the ratio of the locations.
[0182] <10. Modification Example of Calibration Board>
[0183] In the example of
[0184] The image pattern 22 is formed over a wide range in the calibration board 21, so that the multiple light-emitting parts 23 (44 in the example of
[0185] Since the multiple light-emitting parts 23 are provided, a sufficient amount of information can be obtained even if the information display unit 54 displays two kinds of information in the time display part 31 that illuminates according to a time and the position display part 32 that illuminates according to a position. Specifically, a large number of light-emitting parts 23 is allocated to the time display part 31 and thus the same lighting pattern does not periodically appear from the start to the end of image capturing, so that an elapsed time can be uniquely indicated. This can easily achieve synchronization among the plurality of cameras 12 that start capturing images at different times or start capturing the calibration board 21 at different times.
[0186] <Another Layout Example of Light-Emitting Parts 23>
[0187] However, the light-emitting parts 23 of the calibration board 21 do not always need to be disposed in the pattern of the image pattern 22 and may be disposed in a region other than the region of the image pattern 22.
[0188] For example, as illustrated in
[0189] In addition to the time display part 31 and the position display part 32, the information display unit 54 may further include a board display part that illuminates to identify the calibration board 21 when the plurality of calibration boards 21 are used at the same time. In this case, images can be captured by using the plurality of calibration boards 21, and frame images are selected with the board display part in the same lighting condition, so that frame images including the same calibration board 21 can be selected.
[0190] <Manual Display Example of Position Display Part 32>
[0191] In the foregoing example, the calibration board 21 is provided with the position information detection unit 51 including a GPS module, and the lighting condition of the position display part 32 changes according to the detection result of the position information detection unit 51.
[0192] However, the lighting condition of the position display part 32 may be changed by a user operation on the operation button 24 serving as the operation unit 52. For example, the floor is marked with the plurality of sections 42 determined by dividing the imaging region 41, and the operation button 24 is operated according to the section 42 where a user carrying the calibration board 21 is located, so that the lighting condition can be changed according to the section 42. In this case, the position information detection unit 51 can be omitted. Moreover, the lighting condition of the position display part 32 may include an unavailable state indicating unavailability to calibration. The user operates the operation button 24 to set the lighting condition of the position display part 32 to an unavailable state during a movement between the sections 42 or at a location unavailable to calibration, so that the frame image can be excluded in the image extraction of the image extraction unit 72.
[0193] <Addition of Communication Function>
[0194] The calibration board 21 can be configured with the communication function of radio communications of Wi-Fi (registered trademark) and bluetooth (registered trademark) or cable communications. This achieves a configuration that transmits, for example, the detection result of the position information detection unit 51 to a self-propelled robot via radio communications and causes the self-propelled robot to move in the imaging region 41 according to received position information. Alternatively, the calibration board 21 transmits the detection result of the position information detection unit 51 to the smartphone (portable terminal) of the user carrying the calibration board 21, allowing the user to move in the imaging region 41 while confirming position information displayed on a map application of the smartphone. Alternatively, the detection result of the position information detection unit 51 may be transmitted to the image processing device 11 and the position of the calibration board 21 may be displayed on the display device 13, allowing the user carrying the calibration board 21 to move while confirming position information displayed on the display device 13.
[0195] <11. Configuration Example of 3D Model Generation>
[0196] When calibration for calculating the positional relationship among the cameras 12 is performed by using the moving images of the calibration board 21 imaged by the cameras 12 and the external parameters of the cameras 12 are stored in the camera parameter storage unit 76, a predetermined subject as a target of 3D model generation is prepared for imaging with the cameras 12.
[0197] 3D model generation will be described below. The 3D model generation includes processing for capturing the moving images of the predetermined subject as a target of 3D model generation with the cameras 12 in the image processing system 1 and generating a 3D model of an object, which is the predetermined subject, on the basis of the moving images captured by the cameras 12, and rendering for displaying a two-dimensional image of a 3D object on the viewing device of a viewer on the basis of the generated 3D model.
[0198]
[0199] The image processing device 11 includes the camera parameter storage unit 76 and a 3D-model operation unit 81. The 3D-model operation unit 81 includes a moving image acquisition unit 91, a 3D-model generation unit 92, a 3D model DB 93, and a rendering unit 94.
[0200] The moving image acquisition unit 91 acquires the images (moving images) of a subject from the N cameras 12-1 to 12-N and supplies the images to the 3D-model generation unit 92.
[0201] The 3D-model generation unit 92 acquires the camera parameters of the N cameras 12-1 to 12-N from the camera parameter storage unit 76. The camera parameters include at least an external parameter and an internal parameter.
[0202] The 3D-model generation unit 92 generates a 3D model of the subject on the basis of the images captured by the N cameras 12-1 to 12-N and the camera parameters, and stores moving image data on the generated 3D model (3D-model data) in the 3D model DB 93.
[0203] The 3D model DB 93 stores the 3D-model data generated in the 3D-model generation unit 92 and supplies the data to the rendering unit 94 in response to a request from the rendering unit 94. The 3D model DB 93 and the camera parameter storage unit 76 may be the same storage medium or different storage media.
[0204] The rendering unit 94 acquires, from the 3D model DB 93, moving image data (3D model data) on a 3D model specified by a viewer of the reproduced image of the 3D model. The rendering unit 94 then generates (reproduces) a two-dimensional image by viewing the 3D model from the viewing position of the viewer and supplies the two-dimensional image to the display device 13, the viewing position being supplied from the operation unit, which is not illustrated. On the assumption that a virtual camera is used with an imaging range equivalent to the viewing range of the viewer, the rendering unit 94 generates the two-dimensional image of the 3D object captured by the virtual camera and displays the two-dimensional image on the display device 13. The display device 13 includes the display D1 or the head-mounted display (HMD) D2 as illustrated in
[0205] <12. Flowchart of 3D Model Generation>
[0206] Referring to the flowchart of
[0207] First, in step S81, the moving image acquisition unit 91 acquires the images (moving images) of a subject from the N cameras 12-1 to 12-N and supplies the images to the 3D-model generation unit 92.
[0208] In step S82, the 3D-model generation unit 92 acquires the camera parameters of the N cameras 12-1 to 12-N from the camera parameter storage unit 76.
[0209] In step S83, the 3D-model generation unit 92 generates a 3D model of the subject on the basis of the images captured by the N cameras 12-1 to 12-N and the camera parameters, and stores moving image data on the generated 3D model (3D-model data) in the 3D model DB 93.
[0210] In step S84, the rendering unit 94 acquires, from the 3D model DB 93, moving image data (3D model data) on a 3D model specified by a viewer. The rendering unit 94 then generates (reproduces) a two-dimensional image by viewing the 3D model from the viewing position of the viewer and causes the display device 13 to display the two-dimensional image, the viewing position being supplied from the operation part, which is not illustrated.
[0211] The process of step S84 is continuously performed until the end of viewing of the reproduced image of the 3D model. When an exit operation is detected, the 3D model generation is terminated.
[0212] The 3D model generation in steps S81 to S83 and rendering for displaying a two-dimensional image of a 3D object on the viewing device of a viewer in step S84 do not have to be consecutively performed and may be performed at different timings.
[0213] As described above, the image processing device 11 can perform processing for calculating the camera parameters of the cameras 12 on the basis of the moving images of the calibration board 21, processing for generating a 3D model of an object, which is the predetermined subject, on the basis of the moving images of the predetermined subject imaged by the cameras 12 by using the calculated camera parameters, and processing for generating a two-dimensional image as a virtual viewpoint image obtained by viewing the generated 3D model of the object from a predetermined viewpoint.
[0214] <13. Computer Configuration Example>
[0215] The series of processing can be performed by hardware or software. When the series of processing is performed by software, a program constituting the software is installed in a computer. In this configuration, the computer includes a microcomputer embedded in dedicated hardware or includes, for example, a general-purpose personal computer in which various functions can be performed by installing various programs.
[0216]
[0217] In the computer, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103 are connected to one another via a bus 104.
[0218] An input/output interface 105 is further connected to the bus 104. An input unit 106, an output unit 107, a storage unit 108, a communication unit 109, and a drive 110 are connected to the input/output interface 105.
[0219] The input unit 106 includes a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 107 includes a display, a speaker, and an output terminal. The storage unit 108 includes a hard disk, a RAM disc, and a nonvolatile memory. The communication unit 109 includes a network interface. The drive 110 drives a removable recording medium 111, e.g., a magnetic disk, an optical disc, a magneto-optical disc, or a semiconductor memory.
[0220] In the computer configured thus, for example, the CPU 101 performs the series of processing by loading a program, which is stored in the storage unit 108, into the RAM 103 via the input/output interface 105 and the bus 104 and executing the program. In the RAM 103, data necessary for performing a variety of processing by the CPU 101 is also optionally stored.
[0221] The program to be executed by the computer (the CPU 101) can be recorded on, for example, the removable recording medium 111 serving as a package medium and provided in this form. The program can also be provided via wire or wireless transmission media such as a local area network, the Internet, or digital satellite broadcasting.
[0222] In the present description, the steps having been described in the flowcharts may be carried out in parallel or with necessary timing, for example, when evoked, even if the steps are not executed in time series along the order having been described therein, as well as when the steps are executed in time series.
[0223] In the present specification, a system means a collection of a plurality of constituent elements (devices, modules (components), or the like) regardless of the presence or absence of all the constituent elements in the same casing. Accordingly, a plurality of devices stored in separate casings and connected via a network and a single device including a plurality of modules stored in a casing are all systems.
[0224] Note that embodiments of the present disclosure are not limited to the foregoing embodiment and can be modified in various manners without departing from the gist of the present disclosure.
[0225] Furthermore, for example, a plurality of techniques relating to the present technique can be independently implemented as separate techniques if no contradiction arises. Of course, any present techniques may be implemented in combination. For example, at least a part of the present technique described in any one of the embodiments can be implemented in combination with at least a part of the present technique described in other embodiments. Alternatively, at least a part of the present technique can be implemented in combination with other techniques that are not described above.
[0226] For example, the present technique may be configured for cloud computing in which one function is shared and cooperatively processed by a plurality of devices via a network.
[0227] In addition, the steps described in the flowchart can be executed by a single device or can be shared among a plurality of devices.
[0228] Furthermore, if one step includes a plurality of processes, the plurality of processes included in the step can be executed by a single device or can be shared among a plurality of devices.
[0229] The advantageous effects described in the present specification are merely exemplary and are not limited, and other advantageous effects may be achieved in addition to the advantageous effects described in the present specification.
[0230] The present technique can be configured as follows:
[0231] (1)
[0232] An image processing device including: an image synchronization unit that performs time synchronization on a plurality of images of a board on a basis of lighting conditions of a plurality of light-emitting parts included in the plurality of images captured by a plurality of imaging devices, the board including the plurality of light-emitting parts and a predetermined image pattern; and a calibration unit that calculates camera parameters of the plurality of imaging devices by using the plurality of images having been subjected to the time synchronization.
[0233] (2)
[0234] The image processing device according to (1), wherein the plurality of light-emitting parts have a time display part that illuminates according to an imaging time, and the image synchronization unit performs the time synchronization on the plurality of images by selecting the images in which the time display part has the same lighting condition.
[0235] (3)
[0236] The image processing device according to (2), wherein the plurality of light-emitting parts further have a position display part that illuminates according to a position of the board, and
[0237] the image synchronization unit performs the time synchronization on the plurality of images by selecting the images in which the time display part has the same lighting condition from the images selected such that the different lighting conditions of the position display part are provided in a predetermined ratio.
[0238] (4)
[0239] The image processing device according to (3), wherein an imaging range of the plurality of imaging devices is divided into a plurality of sections, and the position display part of the plurality of light-emitting parts illuminates according to the sections.
[0240] (5)
[0241] The image processing device according to (3) or (4), wherein the image synchronization unit selects the images such that the different lighting conditions of the position display part are equally allocated.
[0242] (6)
[0243] The image processing device according to any one of (1) to (5), wherein the plurality of light-emitting parts have a board display part that illuminates to identify the board, and
[0244] the image synchronization unit performs the time synchronization on the plurality of images by selecting the images in which the board display part has the same lighting condition.
[0245] (7)
[0246] The image processing device according to any one of (1) to (6), wherein the board is configured with the light-emitting parts disposed in a pattern of the predetermined image pattern.
[0247] (8)
[0248] The image processing device according to any one of (1) to (6), wherein the board is configured with the light-emitting parts disposed in a region different from a region of the predetermined image pattern.
[0249] (9)
[0250] The image processing device according to any one of (2) to (8), wherein the time display part of the board changes the lighting condition at each lapse of a unit time.
[0251] (10)
[0252] The image processing device according to any one of (3) to (9), wherein the board further includes a position information detection unit that detects position information, and
[0253] the position display part changes the lighting condition according to a detection result of the position information detection unit.
[0254] (11)
[0255] The image processing device according to any one of (3) to (10), wherein the board further includes an operation unit that receives a user operation, and the position display part changes the lighting condition in response to an operation on the operation unit.
[0256] (12)
[0257] The imaging device according to any one of (1) to (11), wherein the light-emitting parts of the board illuminate in different colors according to 0 or 1.
[0258] (13)
[0259] The imaging processing device according to any one of (1) to (11), wherein the light-emitting parts of the board are illuminated or unilluminated according to 0 or 1.
[0260] (14)
[0261] The imaging processing device according to any one of (1) to (13), further including an extraction unit that determines whether the lighting conditions of the plurality of light-emitting parts included in the images have changed, and extracts the changed images,
wherein the image synchronization unit performs time synchronization on the plurality of images on the basis of the lighting conditions of the plurality of light-emitting parts included in the plurality of extracted images.
[0262] (15)
[0263] A calibration board including a plurality of light-emitting parts that change lighting conditions at each lapse of a unit time, and a predetermined image pattern, wherein the plurality of light-emitting parts are caused to illuminate to perform time synchronization on a plurality of images captured by a plurality of imaging devices.
[0264] (16)
[0265] A method for generating 3D model data, the method comprising: performing time synchronization on a plurality of images of a board on a basis of lighting conditions of a plurality of light-emitting parts included in the plurality of images captured by a plurality of imaging devices, the board including the plurality of light-emitting parts and a predetermined image pattern; calculating camera parameters of the plurality of imaging devices by using the plurality of images having been subjected to the time synchronization;
generating a 3D model of a predetermined subject from a plurality of subject images of the predetermined subject, the subject images being captured by the plurality of imaging device by using the calculated camera parameters; and generating a virtual viewpoint image by viewing the generated 3D model of the predetermined subject from a predetermined viewpoint.
REFERENCE SIGNS LIST
[0266] 1 Image processing system [0267] 11 Image processing device [0268] 12-1 to 12-N Camera (imaging device) [0269] 13 Display device [0270] 21 Calibration board [0271] 22 Image pattern [0272] 23 Light-emitting part [0273] 24 Operation button [0274] 31 Time display part [0275] 32 Position display part [0276] 41 Imaging region (imaging space) [0277] 42 (42A to 42D) Section [0278] 46 Common imaging range [0279] 51 Position information detection unit [0280] 52 Operation unit [0281] 53 Control unit [0282] 54 Information display unit [0283] 71 Moving image acquisition unit [0284] 72 Image extraction unit [0285] 73 Extracted image storage unit [0286] 74 Image synchronization unit [0287] 75 Calibration unit [0288] 76 Camera parameter storage unit [0289] 81 3D-model operation unit [0290] 91 Moving image acquisition unit [0291] 92 3D-model generation unit [0292] 93 3D model DB [0293] 94 Rendering unit [0294] 101 CPU [0295] 102 ROM [0296] 103 RAM [0297] 106 Input unit [0298] 107 Output unit [0299] 108 Storage unit [0300] 109 Communication unit [0301] 110 Drive