Image processing system, server apparatus, image processing method, and image processing program

Abstract

An image processing system which can decide an image processing target region considering a decrease in image quality is provided. The image processing system includes an image acquisition unit that acquires an image obtained by capturing a subject, an image quality determination information acquisition unit that acquires image quality determination information representing a determination result of an image quality in the image, an image processing target region decision unit that decides an image processing target region in the image using the image quality determination information, and an image processing unit that executes at least one of a detection process or a composition process for the image processing target region. A region represented as a logical product of two or more temporary image processing target regions respectively corresponding to two or more pieces of the image quality determination information is decided as the image processing target region.

Claims

1. An image processing system that executes at least one of a detection process of detecting a detection target which is damage in a structure and included in an image and a composition process of calculating a correspondence relationship between a plurality of images and compositing the plurality of images based on the correspondence relationship, the system comprising a processor configured to: acquire an image obtained by capturing a subject; acquire two or more pieces of image quality determination information, each representing a determination result of an image quality in the acquired image; derive two or more temporary image processing target regions respectively corresponding to the two or more pieces of the acquired image quality determination information; decide a logical product of the two or more temporary image processing target regions as an image processing target region in the acquired image; and execute the at least one of the detection process and the composition process for the decided image processing target region.

2. The image processing system according to claim 1, wherein the processor is configured to: perform at least one of: acquiring an imaging condition in the capturing of the acquired image, and analyzing the acquired image to obtain an analysis result of the acquired image; and derive at least one of the two or more temporary image processing target regions using at least one of the acquired imaging condition and the obtained analysis result.

3. The image processing system according to claim 2, wherein the processor is configured to: acquire an imaging angle and a focus position as the imaging condition; and decide a non-end portion of the acquired image that is determined based on the imaging angle and the focus position as the at least one of the two or more temporary image processing target regions.

4. The image processing system according to claim 2, wherein the processor is configured to: acquire an imaging angle, a focus position, a subject distance, a focal length, an F number, and a diameter of a permissible circle of confusion as the imaging condition; and decide a non-end portion of the acquired image that is determined based on the imaging angle, the focus position, the subject distance, the focal length, the F number, and the diameter of the permissible circle of confusion as the at least one of the two or more temporary image processing target regions.

5. The image processing system according to claim 4, wherein the processor is configured to: calculate a depth of field in an imaging range using the subject distance, the focal length, the F number, and the diameter of the permissible circle of confusion; calculate a focal point shift amount representing a distance of a focal point shift in the imaging range using the imaging angle and the focal length; and decide a region in which the calculated focal point shift amount falls in a range of the calculated depth of field as the at least one of the two or more temporary image processing target regions.

6. The image processing system according to claim 2, wherein the processor is configured to: acquire a type of lens and a focus position as the imaging condition; and based on the type of lens and the focus position, in a case where the focus position is in a center portion of an imaging range, decide a region that includes the center portion and is determined from characteristics of the lens as the at least one of the two or more temporary image processing target regions, and in a case where the focus position is in a peripheral portion of the imaging range, decide a region that includes the peripheral portion and is determined from the characteristics of the lens as the at least one of the two or more temporary image processing target regions.

7. The image processing system according to claim 2, wherein the processor is configured to: acquire presence or absence of light emission of a strobe as the imaging condition; and decide a strobe light reaching region in which strobe light radiated to the subject from the strobe reaches and thereby makes the strobe light reaching region brighter than a strobe light non-reaching region as the at least one of the two or more temporary image processing target regions.

8. The image processing system according to claim 7, wherein the processor is configured to: acquire a subject distance as the imaging conditions; and decide the strobe light reaching region that is determined depending on the subject distance as the at least one of the two or more temporary image processing target regions.

9. The image processing system according to claim 2, wherein the processor is configured to: divide an image of an analysis target into a plurality of regions and generate a spatial frequency spectrum distribution of each region; and decide a high image quality region that is determined based on the generated spatial frequency spectrum distribution of each region as the at least one of the two or more temporary image processing target regions.

10. The image processing system according to claim 2, wherein the processor is configured to: divide an image of an analysis target into a plurality of regions and generate a histogram of a gradation value of each region; and decide a high image quality region that is determined based on the generated histogram of the gradation value of each region as the at least one of the two or more temporary image processing target regions.

11. The image processing system according to claim 1, wherein the processor is configured to: store, in a storage, a relationship between the acquired image quality determination information and each of the temporary image processing target regions in association; and acquire each of the temporary image processing target regions corresponding to the image quality determination information from the storage using the acquired image quality determination information.

12. The image processing system according to claim 1, wherein the processor is configured to display, on an image display, an image on which the image processing is performed, and an image processing target exclusion region that is excluded from a target of the image processing.

13. The image processing system according to claim 1, wherein the processor is configured to change the decided image processing target region.

14. The image processing system according to claim 1, wherein the processor is configured to execute a process of detecting at least one of a crevice of a concrete member, a chalk line, free lime, water leakage, stripping, rebar exposure, a float, a crack of a steel member, or corrosion as the detection target.

15. The image processing system according to claim 1, further comprising: a server apparatus; and a client apparatus that is communicably connected to the server apparatus through a network, wherein the server apparatus includes the processor configured to acquire the image obtained by capturing the subject, acquire the two or more pieces of image quality determination information, decide, the image processing target region, and execute the at least one of the detection process and the composition process.

16. The image processing system according to claim 15, wherein the client apparatus includes a transmitter configured to transmit image data representing the captured image to the server apparatus.

17. A server apparatus included in an image processing system that executes at least one of a detection process of detecting a detection target which is damage in a structure and included in an image and a composition process of calculating a correspondence relationship between a plurality of images and compositing the plurality of images based on the correspondence relationship, the apparatus comprising a processor configured to: acquire an image obtained by capturing a subject; acquire two or more pieces of image quality determination information, each representing a determination result of an image quality in the acquired image; derive two or more temporary image processing target regions respectively corresponding to the two or more pieces of the acquired image quality determination information; decide a logical product of the two or more temporary image processing target regions as an image processing target region in the acquired image; and execute the at least one of the detection process and the composition process for the decided image processing target region.

18. An image processing method of executing at least one of a detection process of detecting a detection target which is damage in a structure and included in an image and a composition process of calculating a correspondence relationship between a plurality of images and compositing the plurality of images based on the correspondence relationship, the method comprising: acquiring an image obtained by capturing a subject; acquiring two or more pieces of image quality determination information, each representing a determination result of an image quality in the acquired image; deriving two or more temporary image processing target regions respectively corresponding to the two or more pieces of the acquired image quality determination information; deciding a logical product of the two or more temporary image processing target regions as an image processing target region in the acquired image; and executing the at least one of the detection process and the composition process for the decided image processing target region.

19. The image processing method according to claim 18, wherein: the acquiring the image quality determination information includes performing at least one of: acquiring an imaging condition in the capturing of the acquired image, and analyzing the acquired image to obtain an analysis result of the acquired image; and the deriving the two or more temporary image processing target regions includes deriving at least one of the two or more temporary image processing target regions using at least one of the acquired imaging condition and the obtained analysis result.

20. A non-transitory, computer-readable recording medium which records an image processing program for executing at least one of a detection process of detecting a detection target which is damage in a structure and included in an image and a composition process of calculating a correspondence relationship between a plurality of images and compositing the plurality of images based on the correspondence relationship, the program causing a computer to implement: acquiring an image obtained by capturing a subject; acquiring two or more pieces of image quality determination information, each representing a determination result of an image quality in the acquired image; deriving two or more temporary image processing target regions respectively corresponding to the two or more pieces of the acquired image quality determination information; deciding a logical product of the two or more temporary image processing target regions as an image processing target region in the acquired image; and executing the at least one of the detection process and the composition process for the decided image processing target region.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is a block diagram of an image processing system according to an embodiment.

(2) FIG. 2 is a descriptive diagram of a specific example of image processing.

(3) FIG. 3 is a block diagram of a case where an image processing target region is decided using an imaging condition in the image processing system illustrated in FIG. 1.

(4) FIG. 4 is a block diagram of a case where the image processing target region is decided using an image quality analysis result in the image processing system illustrated in FIG. 1.

(5) FIG. 5 is a diagram illustrating one example of an analysis region.

(6) FIG. 6 is a schematic diagram of a spatial frequency spectrum distribution image in an in-focus region.

(7) FIG. 7 is a diagram illustrating one example of the spatial frequency spectrum distribution image in a case where the analysis region is a normal image.

(8) FIG. 8 is a diagram illustrating one example of the spatial frequency spectrum distribution image in a case where the analysis region is a blurred image or a shaky image.

(9) FIG. 9 is a diagram illustrating one example of a high frequency component determination region.

(10) FIG. 10 is a diagram illustrating one example of a shade histogram.

(11) FIG. 11 is a schematic diagram of imaging of a subject.

(12) FIG. 12 is a diagram illustrating one example of a temporary image processing target region in a case where a focus area is the center of an imaging range.

(13) FIG. 13 is a diagram illustrating one example of the temporary image processing target region in a case where the focus area is positioned below the center of the imaging range.

(14) FIG. 14 is a schematic diagram illustrating a relationship between a focal point shift amount and a depth of field.

(15) FIG. 15 is a diagram illustrating one example of a region in the depth of field in a case where the focus area is the center of the imaging range.

(16) FIG. 16 is a diagram illustrating one example of the region in the depth of field in a case where the focus area is positioned below the center of the imaging range.

(17) FIG. 17 is a diagram illustrating one example of a field curvature in a case where the focus area is the center of the imaging range.

(18) FIG. 18 is a diagram illustrating one example of the field curvature in a case where the focus area is a peripheral portion of the imaging range.

(19) FIG. 19 is a diagram illustrating one example of a non-out-of-focus region in which the effect of the field curvature is considered in a case where the focus area is the center of the imaging range.

(20) FIG. 20 is a diagram illustrating one example of the non-out-of-focus region in which the effect of the field curvature is considered in a case where the focus area is the peripheral portion of the imaging range.

(21) FIG. 21 is a diagram illustrating another example of the non-out-of-focus region in which the effect of the field curvature is considered in a case where the focus area is the center of the imaging range.

(22) FIG. 22 is a diagram illustrating one example of a bright region and a dark region in strobe light emission.

(23) FIG. 23 is a diagram illustrating one example of the image processing target region based on a logical product of two temporary image processing target regions.

(24) FIG. 24 is a description of one example of an image composition process.

(25) FIG. 25 is a description of a display example of an image processing result.

(26) FIG. 26 is a block diagram illustrating an application example for a client server type network system.

(27) FIG. 27 is a flowchart illustrating a flow of procedure of an image processing method according to the embodiment.

(28) FIG. 28 is a flowchart illustrating a flow of procedure of an image processing target region decision step illustrated in FIG. 27.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

(29) Hereinafter, a preferred embodiment of the present invention will be described in accordance with the appended drawings. In the present specification, the same constituents will be designated by the same reference signs, and descriptions of such constituents will not be repeated.

(30) [Summary of Image Processing System]

(31) FIG. 1 is a block diagram of an image processing system according to the embodiment. An image processing system 10 illustrated in FIG. 1 comprises an image acquisition unit 12, an image quality determination information acquisition unit 14, an image processing target region decision unit 16, and an image processing unit 18. The image processing system 10 may comprise an image display unit, not illustrated.

(32) <Image Acquisition Unit>

(33) The image acquisition unit 12 acquires image data as a target of image processing that uses the image processing unit 18. An example of the image data is exemplified by image data that is obtained by imaging a subject using an imaging apparatus. An example of the imaging apparatus is exemplified by an electronic camera that can capture a color image. The concept of imaging in the present specification includes filming.

(34) An example of an image pick-up device comprised in the electronic camera is exemplified by a color CCD linear image sensor. CCD is the abbreviation for Charge-Coupled Device and refers to a charge-coupled element.

(35) The color CCD linear image sensor is an image sensor in which a light-receiving element comprising a color filter of each color of R, G, and B is linearly arranged. R denotes red. G denotes green. B denotes blue.

(36) Instead of the color CCD linear image sensor, a color CMOS linear image sensor can be used. CMOS is the abbreviation for Complementary Metal Oxide Semiconductor and refers to a complementary metal oxide film semiconductor.

(37) <Image Quality Determination Information Acquisition Unit>

(38) The image quality determination information acquisition unit 14 acquires image data 20 that is acquired using the image acquisition unit 12. The image quality determination information acquisition unit 14 may acquire EXIF information 22 of the image data 20 illustrated in FIG. 1 as information of an imaging condition. EXIF is the abbreviation for Exchangeable Image File Format.

(39) As the imaging condition, the image quality determination information acquisition unit 14 may acquire at least one of user designation information 24 designated by a user, user selection information 26 selected by the user, and sensor information 28 acquired from a sensor.

(40) That is, the image processing system 10 may comprise an EXIF information acquisition unit that acquires the EXIF information 22. The image processing system 10 may comprise a user designation information acquisition unit that acquires the user designation information 24. The image processing system 10 may comprise a user selection information acquisition unit that acquires the user selection information 26. The image processing system 10 may comprise a sensor information acquisition unit that acquires the sensor information 28.

(41) The image processing system 10 may comprise an imaging condition acquisition unit that functions as at least one of the EXIF information acquisition unit, the user designation information acquisition unit, the user selection information acquisition unit, and the sensor information acquisition unit. Illustrations of the EXIF information acquisition unit, the user designation information acquisition unit, the user selection information acquisition unit, and the sensor information acquisition unit are not provided. The imaging condition acquisition unit is designated by reference sign 40 and illustrated in FIG. 3.

(42) The image quality determination information acquisition unit 14 performs an image quality analysis process on the acquired image data 20 and acquires an image quality analysis result 30. Alternatively, the image quality determination information acquisition unit 14 acquires the imaging condition. The image quality determination information acquisition unit 14 may acquire both of the image quality analysis result 30 and the imaging condition.

(43) The image quality determination information acquisition unit 14 determines the image quality of an image represented by the image data 20 using at least one of the image quality analysis result 30 and the imaging condition. The image quality analysis result 30 is one example of an analysis result of analysis of an image using an analysis unit.

(44) The image quality determination information acquisition unit 14 generates two different types of image quality determination information including first image quality determination information 32A and second image quality determination information 32B as a determination result of the image quality of the image represented by the image data 20. The two different types of image quality determination information may be read as two different pieces of image quality determination information.

(45) The first image quality determination information 32A and the second image quality determination information 32B may be generated using two different types of image quality analysis results 30. The first image quality determination information 32A and the second image quality determination information 32B may be generated using two different types of imaging conditions. The two different types of imaging conditions may be read as two different imaging conditions.

(46) The first image quality determination information 32A may be generated using the image quality analysis result 30, and the second image quality determination information 32B may be generated using the imaging condition. The first image quality determination information 32A may be generated using the imaging condition, and the second image quality determination information 32B may be generated using the image quality analysis result 30.

(47) In the present embodiment, an aspect in which two types of image quality determination information are generated is illustrated. Alternatively, two or more types of image quality determination information may be generated, or three or more types of image quality determination information may be generated. Details of the image quality determination using the image quality determination information acquisition unit 14 will be described below.

(48) <Image Processing Target Region Decision Unit>

(49) The image processing target region decision unit 16 decides an image processing target region in the image data 20 based on the first image quality determination information 32A and the second image quality determination information 32B. The image processing target region decision unit 16 decides a first temporary image processing target region based on the first image quality determination information 32A. In addition, the image processing target region decision unit 16 decides a second temporary image processing target region based on the second image quality determination information 32B. That is, the image processing target region decision unit 16 derives two or more temporary image processing target regions corresponding to the two types of image quality determination information.

(50) The two or more different temporary image processing target regions are exemplified by a region excluding an out-of-focus region caused by an imaging angle, a region excluding an out-of-focus region caused by the characteristics of the lens, a region excluding a region that becomes dark due to strobe light emission, a region excluding an excessively bright region, a region excluding an excessively dark region, and a region excluding a region having broken gradation.

(51) The image processing target region decision unit 16 combines the two or more temporary image processing target regions. An example of the combining is exemplified by an example in which a region represented as the logical product of the two or more temporary image processing target regions is decided as the image processing target region. The image processing target region decision unit 16 generates image processing target region information 34 that represents the image processing target region.

(52) <Image Processing Unit>

(53) The image processing unit 18 acquires the image data 20 and the image processing target region information 34. The image processing unit 18 performs image processing on the image processing target region in the image data 20 and generates an image processing result 36.

(54) An example of the image processing using the image processing unit 18 is exemplified by a detection process of detecting a detection target included in the image represented by the image data 20. Another example of the image processing using the image processing unit 18 is exemplified by a composition process of compositing divided images obtained by dividing and imaging a subject.

(55) <Specific Example of Image Processing>

(56) FIG. 2 is a descriptive diagram of a specific example of the image processing. The image processing illustrated in FIG. 2 is the composition process of compositing three divided images including a divided image 100, a divided image 102, and a divided image 104 that are obtained by dividing a concrete wall of a building into three parts and imaging the divided parts.

(57) The divided image 100, the divided image 102, and the divided image 104 illustrated in FIG. 2 are images obtained by arranging the imaging apparatus on a line that is normal to the wall as a subject and passes through the center of the wall, performing a pan operation of the imaging apparatus, and imaging the wall by changing an imaging region on the wall. The center of the wall may not be an exact center position and may be an approximate center position.

(58) Regions that are designated by reference sign 108 and reference sign 110 and illustrated in the divided image 100 are out-of-focus regions. Regions that are designated by reference sign 112 and reference sign 114 and illustrated in the divided image 104 are out-of-focus regions. The out-of-focus region is a region that is out of focus. A region excluding the out-of-focus region is an in-focus region that is in focus.

(59) In the divided image 100, the out-of-focus region 108 and the out-of-focus region 110 are present in both end portions of the divided image 100 in the lateral direction. In the divided image 104, the out-of-focus region 112 and the out-of-focus region 114 are present in both end portions of the divided image 104 in the lateral direction.

(60) In other words, the in-focus region is present in the non-end portion of the divided image 100 in the lateral direction and the non-end portion of the divided image 104 in the lateral direction. The non-end portion is a region excluding both end portions. For the up-down direction, a region excluding both end portions in the up-down direction is a non-end portion in the up-down direction.

(61) Meanwhile, the out-of-focus region is not present in the divided image 102. In other words, the in-focus region is present across the whole range of the divided image 102 in the lateral direction.

(62) The lateral direction of the divided image is a direction in which the optical axis of the imaging apparatus in the subject moves in the pan operation of the imaging apparatus. The longitudinal direction of the divided image is a direction in which the optical axis of the imaging apparatus in the subject moves in a tilt operation of the imaging apparatus. The optical axis of the imaging apparatus represents the optical axis of an optical image formation system comprised in the imaging apparatus.

(63) An overlapping region is present in the divided image 100 and the divided image 102. An overlapping region is present in the divided image 102 and the divided image 104. A composite image 106 illustrated in FIG. 2 is a composite image that is generated by performing the composition process of compositing the divided image 100, the divided image 102, and the divided image 104.

(64) In the composite image 106, a region designated by reference sign 116 is a superimposed region between the divided image 100 and the divided image 102. In the composite image 106, a region designated by reference sign 118 is a superimposed region between the divided image 102 and the divided image 104.

(65) In a case where the pixels of the out-of-focus region 110 of the divided image 100 are included in the superimposed region 116 of the composite image 106, the superimposed region 116 in the composite image 106 is blurred due to the pixels of the out-of-focus region 110 of the divided image 100.

(66) Similarly, in a case where the pixels of the out-of-focus region 112 of the divided image 104 are included in the superimposed region 118 of the composite image 106, the superimposed region 118 in the composite image 106 is blurred due to the pixels of the out-of-focus region 112 of the divided image 104.

(67) The image processing system according to the present embodiment uses the pixels of the divided image 102 which is not out of focus for the superimposed region 116 of the composite image 106. In addition, the image processing system uses the pixels of the divided image 102 which is not out of focus for the superimposed region 118 of the composite image 106. Accordingly, the occurrence of blurriness in the superimposed region 116 and the superimposed region 118 of the composite image 106 may be avoided.

(68) [Description of Decision of Image Processing Target Region Using Imaging Condition]

(69) FIG. 3 is a block diagram of a case where the image processing target region is decided using the imaging condition in the image processing system illustrated in FIG. 1. The image processing system 10 illustrated in FIG. 3 comprises an imaging condition acquisition unit 40 as the image quality determination information acquisition unit 14 illustrated in FIG. 1. In addition, the image processing system 10 illustrated in FIG. 3 comprises an image processing target region database 42.

(70) The imaging condition acquisition unit 40 may acquire at least one condition of the imaging angle, a subject distance, a focal length, an F number, a permissible circle of confusion diameter, the type of lens, the type of camera, a focus position, the presence or absence of strobe light emission, and the like as the imaging condition.

(71) The imaging condition acquisition unit 40 may acquire the imaging condition included in the EXIF information 22 transmitted from the image acquisition unit 12 as the imaging condition. The imaging condition acquisition unit 40 may acquire the imaging condition designated by the user, the imaging condition selected by the user, and the imaging condition obtained using the sensor as the imaging condition. The imaging condition acquisition unit 40 may acquire the imaging condition linked to the model of the camera.

(72) The imaging condition acquisition unit 40 transmits imaging condition information 44 representing the imaging condition to the image processing target region decision unit 16. The imaging condition information 44 is one example of the first image quality determination information 32A or the second image quality determination information 32B illustrated in FIG. 1.

(73) In the image processing target region database 42, the relationship between the imaging condition and the temporary image processing target region is stored as a database. Based on the imaging condition as an index, the image processing target region decision unit 16 can acquire the temporary image processing target region corresponding to the imaging condition using the image processing target region database 42.

(74) By using the temporary image processing target region which is acquired from the image processing target region database 42 and corresponds to the imaging condition, the image processing target region decision unit 16 decides the image processing target region as a target of the image processing using the image processing unit 18. The image processing target region database 42 is one example of a storage unit in which the relationship between the image quality determination information and the temporary image processing target region is stored in association with each other.

(75) That is, the image processing target region decision unit 16 decides one or more temporary image processing target regions using the imaging condition.

(76) The image processing target region decision unit 16 may decide the image processing target region as a region represented by the logical product of the temporary image processing target region based on the imaging condition and the temporary image processing target region based on the image quality analysis result 30 illustrated in FIG. 1.

(77) The image processing target region decision unit 16 may decide two or more different temporary image processing target regions using two or more different types of imaging conditions. The image processing target region decision unit 16 may decide the image processing target region as a region represented by the logical product of two or more different temporary image processing target regions.

(78) The image processing target region decision unit 16 may decide an image processing target exclusion region. The image processing target exclusion region is a region that is excluded from the target of the image processing using the image processing unit 18. The same applies to the image processing unit 18 in the image processing system 10 illustrated in FIG. 4.

(79) The image processing target region decision unit 16 transmits the image processing target region information 34 to the image processing unit 18. The image processing unit 18 executes the image processing based on the image data 20 and the image processing target region information 34 and outputs the image processing result 36.

(80) [Specific Example of Imaging Condition]

(81) A specific example of imaging information acquired using the imaging condition acquisition unit 40 illustrated in FIG. 3 will be described. A specific example of the imaging information designated by the user or the imaging information selected by the user is exemplified by the model of the camera, the type of lens, an imaging position condition, and the setting of the camera. The setting of the camera may be the setting of the lens.

(82) The size of an imaging element for each camera may be perceived from the model of the camera. The size of the imaging element can be used for calculating the depth of field as the permissible circle of confusion diameter. In the case of a camera in which the lens is not replaceable, the characteristics of the lens for each camera may be perceived from the model of the camera. The characteristics of the lens can be used for determining the presence or absence of the field curvature.

(83) In the image processing target region database 42 illustrated in FIG. 3, the relationship between the model of the camera and the size of the imaging element may be stored as a database. In the image processing target region database 42 illustrated in FIG. 3, the relationship between the model of the camera and the characteristics of the lens may be stored as a database for the camera in which the lens is not replaceable.

(84) The user may designate or select the size of the imaging element. The user may designate or select the characteristics of the lens.

(85) In the case of the camera in which the lens is not replaceable, the type of lens may be acquired as the imaging condition. The characteristics of the lens may be perceived from the type of lens. The characteristics of the lens can be used for determining the presence or absence of the field curvature.

(86) In the image processing target region database 42 illustrated in FIG. 3, the relationship between the type of lens and the characteristics of the lens may be stored as a database. The characteristics of the lens may be designated or selected.

(87) At least one of the subject distance and the imaging angle may be perceived from the imaging position condition. Details of the subject distance and the imaging angle will be described below.

(88) At least one of the focus position, the focal length, the F number, and the presence or absence of strobe light emission may be perceived from the setting of the camera or the setting of the lens.

(89) A specific example of the imaging condition that may be acquired from metadata of the image is exemplified by the model of the camera, the type of lens, and the setting of the camera. The setting of the camera may be the setting of the lens. The metadata of the image is information that is not the image data and is related to the image data. An example of the metadata is exemplified by the EXIF information 22 illustrated in FIG. 1 and FIG. 3.

(90) The model of the camera, the type of lens, and the setting of the camera or the setting of the lens are described above. Descriptions of details of the model of the camera, the type of lens, and the setting of the camera or the setting of the lens will not be repeated.

(91) A specific example of the imaging condition that may be acquired using the sensor is exemplified by the imaging position and the imaging angle. The subject distance may be perceived from the imaging position condition. In the case of imaging using a robot, the imaging condition acquisition unit 40 illustrated in FIG. 3 may acquire information that is obtained from various sensors comprised in the robot. Examples of the various sensors are exemplified by a sensor measuring the subject distance, a sensor measuring the imaging angle, and the like. The subject distance is measured as a distance from the lens of the imaging apparatus to the subject. The subject distance may be measured by a filmer using a measurement assistance tool such as a measure. The subject distance may be measured using a measurer such as a laser measurer. The imaging angle may be measured using a measurer such as an angle meter. The imaging angle may be derived from the distance between the imaging apparatus and the subject in the horizontal direction and the distance between the imaging apparatus and the subject in the optical axis direction of the imaging apparatus using a trigonometric function.

(92) [Description of Decision of Image Processing Target Region Using Image Quality Analysis Result]

(93) FIG. 4 is a block diagram of a case where the image processing target region is decided using the image quality analysis result in the image processing system illustrated in FIG. 1. The image processing system 10 illustrated in FIG. 4 comprises an image quality analysis result acquisition unit 50 as the image quality determination information acquisition unit 14 illustrated in FIG. 1. In addition, the image processing system 10 illustrated in FIG. 4 comprises a threshold value input unit that inputs a threshold value into the image processing target region decision unit 16. An illustration of the threshold value input unit is not provided.

(94) The image quality analysis result acquisition unit 50 illustrated in FIG. 4 sets a plurality of analysis regions in the image data 20, executes the image quality analysis on each analysis region, and acquires the image quality analysis result 30. An example of the image quality analysis is exemplified by generation of a spectrum distribution of a spatial frequency and generation of a shade histogram that is a histogram of a shade value in a gray scale image. The image quality analysis result acquisition unit 50 is one example of an analysis unit.

(95) Determination information for out-of-focus and shake may be acquired from the spectrum distribution of the spatial frequency in each analysis region. Determination information for an excessively dark region, an excessively bright region, and a region having broken gradation may be acquired from the shade histogram of each analysis region in the gray scale image. The image quality determination information acquisition unit 14 transmits the acquired image quality analysis result 30 to the image processing target region decision unit 16.

(96) The image processing target region decision unit 16 decides one or more temporary image processing target regions by setting a threshold value 54 for the image quality analysis result 30. The threshold value 54 may be a predetermined value that is set in advance, or a designated value that is designated by the user.

(97) The image processing target region decision unit 16 may decide the image processing target region as a region represented by the logical product of the temporary image processing target region decided using the image quality analysis result 30 and the temporary image processing target region decided using the imaging condition.

(98) The image processing target region decision unit 16 may decide two or more different temporary image processing target regions using two or more different types of image quality analysis results. The image processing target region decision unit 16 may decide the image processing target region as a region represented by the logical product of two or more different temporary image processing target regions decided using two different types of image quality analysis results.

(99) The image processing target region decision unit 16 transmits the image processing target region information 34 to the image processing unit 18. The image processing unit 18 executes the image processing based on the image data 20 and the image processing target region information 34 and outputs the image processing result 36.

(100) Various processing units illustrated in FIG. 1 to FIG. 4 may be represented as processing units using English representation. A processor may be represented as a processor using English representation. The processing units include a substantial processing unit that is a constituent not having the name “processing unit” but executes any process.

(101) Various processors include a CPU that is a general-purpose processor functioning as various processing units by executing a program, a PLD that is a processor such as an FPGA of which the circuit configuration can be changed after manufacturing, a dedicated electric circuit that is a processor such as an ASIC having a circuit configuration dedicatedly designed to execute a specific process, and the like. The program is the same definition as software.

(102) FPGA is the abbreviation for Field Programmable Gate Array. PLD is the abbreviation for Programmable Logic Device. ASIC is the abbreviation for Application Specific Integrated Circuit.

(103) One processing unit may be configured with one of the various processors or may be configured with two or more processors of the same type or different types. For example, one processing unit may be configured with a plurality of FPGAs or a combination of a CPU and an FPGA. In addition, a plurality of processing units may be configured with one processor.

(104) As an example of configuring a plurality of processing units with one processor, a first form is configuring one processor with a combination of one or more CPUs and software and implementing a plurality of processing units by the processor as represented by a computer such as a client and a server.

(105) A second form is using a processor that implements the function of the whole system including the plurality of processing units using one IC chip as represented by an SoC and the like. Various processing units are configured using one or more of the various processors as a hardware structure. Furthermore, the hardware structure of the various processors is more specifically an electric circuit in which circuit elements such as a semiconductor element are combined.

(106) SoC is the abbreviation for System On Chip. IC is the abbreviation for Integrated Circuit. The electric circuit may be represented as circuitry using English representation.

(107) [Detailed Description of Image Quality Analysis]

(108) Next, the image quality analysis using the image quality analysis result acquisition unit 50 illustrated in FIG. 4 will be described in detail. Hereinafter, the image quality analysis in the divided image 100 illustrated in FIG. 2 will be described.

(109) FIG. 5 is a diagram illustrating one example of the analysis region. An analysis region 150 illustrated in FIG. 5 is a region as the unit of the image quality analysis. The analysis region 150 represents any one of a plurality of analysis regions set in the divided image 100. Hereinafter, in a case where the term analysis region is used without a reference sign, the analysis region represents the analysis region illustrated in FIG. 5.

(110) In the present embodiment, a plurality of analysis regions are set in the divided image 100 in each of the longitudinal direction and the transverse direction. The divided image may be divided in each of any two orthogonal directions.

(111) While the analysis region 150 having a square shape is illustrated in the present embodiment, any shape such as a polygonal shape and a circle other than the square shape may be applied as the shape of the analysis region 150. Furthermore, the number of pixels of the analysis region 150 can be appropriately determined.

(112) The image quality analysis result acquisition unit 50 illustrated in FIG. 4 acquires the result of the image quality analysis process in each analysis region of the plurality of analysis regions. The acquisition of the result of the image quality analysis process includes an aspect in which the result of the image quality analysis process performed using the image quality analysis result acquisition unit 50 is acquired.

(113) <One Example of Image Quality Analysis Process>

(114) Next, acquisition of a spatial frequency spectrum distribution in each analysis region will be described as one example of the image quality analysis process result. FFT is applied in the acquisition of the spatial frequency spectrum distribution. FFT is the abbreviation for fast Fourier transform that is an English representation representing a fast Fourier transformation process.

(115) FIG. 6 is a schematic diagram of a spatial frequency spectrum distribution image in a case where the analysis region is the in-focus region. A center 162 of a spatial frequency spectrum distribution image 160 represents a spatial frequency of 0. A corner 164 of the spatial frequency spectrum distribution image 160 represents the maximum spatial frequency.

(116) A center portion in the vicinity of the center 162 of the spatial frequency spectrum distribution image 160 represents a region in which a low frequency component spectrum 166 is present. A peripheral portion in the vicinity of the corner 164 of the spatial frequency spectrum distribution image 160 represents a region in which a high frequency component spectrum 168 is present.

(117) The center portion in the vicinity of the center 162 of the spatial frequency spectrum distribution image 160 is a region that includes the center 162 of the spatial frequency spectrum distribution image 160. The center portion is determined depending on the low frequency component.

(118) The peripheral portion in the vicinity of the corner 164 of the spatial frequency spectrum distribution image 160 is a region that includes the corner 164 of the spatial frequency spectrum distribution image 160. The peripheral portion is determined depending on the high frequency component.

(119) The high frequency component and the low frequency component are appropriately determined depending on the resolution of the image. Any high frequency component and low frequency component that represent a relative relationship between frequency components may be used. The unit of the spatial frequency is represented using the multiplicative inverse of the unit of a length. The unit of the length may be millimeter or inch.

(120) FIG. 7 is a diagram illustrating one example of the spatial frequency spectrum distribution image in a case where the analysis region is a normal image. The normal image means a non-out-of-focus image. In a case where the analysis region is the normal image, the low frequency component spectrum 166 and the high frequency component spectrum 168 are present in the spatial frequency spectrum distribution image 160.

(121) FIG. 8 is a diagram illustrating one example of the spatial frequency spectrum distribution image in a case where the analysis region is a blurred image or a shaky image. As illustrated in FIG. 8, in a case where the analysis region is a blurred image that is out of focus or a shaky image that is shaky, the low frequency component spectrum 166 is present in the spatial frequency spectrum distribution image 160, but the high frequency component spectrum 168 is not present.

(122) Accordingly, a determination as to whether to set the analysis region as the temporary image processing target region or the image processing target exclusion region may be performed depending on the presence or absence of the high frequency component spectrum 166 of the spatial frequency spectrum distribution image 160 derived from each analysis region.

(123) The analysis region corresponding to the spatial frequency spectrum distribution image 160 illustrated in FIG. 7 is one example of a high image quality region that is determined based on the spatial frequency spectrum distribution. The analysis region corresponding to the spatial frequency spectrum distribution image 160 illustrated in FIG. 8 is one example of a low image quality region that is determined based on the spatial frequency spectrum distribution.

(124) FIG. 9 is a diagram illustrating one example of a high frequency component determination region. As illustrated in FIG. 9, the range of r pixels from each corner 164 of the spatial frequency spectrum distribution image 160 is set as a high frequency component determination region 170.

(125) In a case where a spectrum is present in the high frequency component determination region 170, it may be determined that the high frequency component spectrum is present in the spatial frequency spectrum distribution image 160. In a case where the high frequency component spectrum is present in the spatial frequency spectrum distribution image 160, it is determined that the analysis region is a non-out-of-focus region or a non-shaky region. The non-out-of-focus analysis region or the non-shaky analysis region is decided as the temporary image processing target region.

(126) Meanwhile, in a case where a spectrum is not present in the high frequency component determination region 170, it may be determination the high frequency component spectrum is not present in the spatial frequency spectrum distribution image 160. In a case where the high frequency component spectrum is not present in the spatial frequency spectrum distribution image 160, it is determined that the analysis region is an out-of-focus region or a shaky region. The out-of-focus analysis region or the shaky analysis region is decided as the image processing target exclusion region. The spatial frequency spectrum distribution image 160 is one example of the spatial frequency spectrum distribution.

(127) A determination threshold value is denoted by r that decides the high frequency component determination region 170. The determination threshold value may be a predetermined value that is set in advance, or a designated value that may be designated by the user. An example of the determination threshold value is exemplified by ¼ of the number of pixels of the analysis region in the longitudinal direction or ¼ of the number of pixels of the analysis region in the lateral direction. For example, in a case where the number of pixels of the analysis region in the longitudinal direction is 100, and the number of pixels of the analysis region in the lateral direction is 100, the determination threshold value r may be set to 25 pixels. Here, ¼ is for illustrative purposes. An arbitrary ratio may be applied.

(128) While an example in which the determination threshold value is represented using the number of pixels is illustrated in FIG. 9, a length in millimeters or the like may be used for the determination threshold value. The number of pixels may be represented as pixels.

(129) <Another Example of Image Quality Analysis Process>

(130) Next, the acquisition of the shade histogram in each analysis region will be described as another example of the image quality analysis process result. FIG. 10 is a diagram illustrating one example of the shade histogram. A shade histogram 180 illustrated in FIG. 10 is obtained by converting the analysis region from a color image represented using R, G, and B into a gray scale image and calculating the shade histogram of the gray scale image.

(131) The shade value of each pixel of the gray scale image is calculated using R×0.30+G×0.59+B×0.1. In the expression, R denotes the shade value of red. G denotes the shade value of green. B denotes the shade value of blue. A color conversion look-up table from RGB into a gray scale can be applied in the conversion from the color image into the gray scale image. The conversion from the color image into the gray scale image is for illustrative purposes. Other conversion expressions for conversion into an image from which the shade histogram can be calculated may be used.

(132) The horizontal axis of the shade histogram 180 of the analysis region illustrated in FIG. 10 denotes the shade value. The vertical axis of the shade histogram 180 denotes a frequency. The shade value is represented using the absolute value of 0 to 255. The shade value 0 represents the highest darkness. The shade value 255 represents the highest brightness.

(133) In a case where the analysis region is excessively dark, the frequency of the shade histogram 180 is biased to a dark side. In a case where the analysis region is excessively bright, the frequency of the shade histogram 180 is biased to a bright side.

(134) A determination as to whether each analysis region is an excessively bright region or an excessively dark region can be performed using the shade histogram of each analysis region. The histogram of each shade value is denoted by G(i). The shade value is denoted by i. In the present embodiment, an integer greater than or equal to 0 and smaller than or equal to 255 is used for i.

(135) A shade value threshold value of the excessively bright region is denoted by k.sub.b, and a pixel threshold value of the excessively bright region is denoted by h.sub.b. The analysis region that satisfies Expression 1 below may be determined as the excessively bright region. The excessively bright region may be set as the image processing target exclusion region.

(136) $\frac{\underset{j=k_b}{\overset{255}{.Math.}}{G\left(j\right)}_{}^{}}{\underset{i=0}{\overset{255}{.Math.}}G\left(i\right)}\geqq h_b$

(137) That is, in a case where the ratio of the number of pixels having the shade value k.sub.b or higher to the total number of pixels of the analysis region is greater than or equal to h.sub.b, the analysis region is determined as the excessively bright region. The shade value threshold value k.sub.b of the excessively bright region and the pixel threshold value h.sub.b of the excessively bright region may be predetermined values that are set in advance, or may be designated values that may be designated by the user.

(138) An example of the shade value threshold value k.sub.b of the excessively bright region is exemplified by 205. An example of the pixel threshold value h.sub.b of the excessively bright region is exemplified by 0.5. In a case where 50 percent of all pixels of the analysis region have the shade value of 205 or higher, the analysis region is determined as the excessively bright region. The excessively bright region is set as the image processing target exclusion region.

(139) The shade threshold value of the excessively dark region is denoted by k.sub.d, and the pixel threshold value of the excessively dark region is denoted by h.sub.d. The analysis region that satisfies Expression 2 below may be determined as the excessively dark region. The excessively dark region may be set as the image processing target exclusion region.

(140) $\frac{\underset{j=0}{\overset{k_d}{.Math.}}G\left(j\right)}{\underset{i=0}{\overset{255}{.Math.}}G\left(i\right)}\geqq h_d$

(141) An example of the shade threshold value k.sub.d of the excessively dark region is exemplified by 50. An example of the pixel threshold value h.sub.d of the excessively dark region is exemplified by 0.5. In a case where 50 percent of all pixels of the analysis region has the shade value of 50 or lower, the analysis region is determined as the excessively dark region. The excessively dark region is set as the image processing target exclusion region.

(142) Meanwhile, a region that is not determined as the excessively bright region, and a region that is not determined as the excessively dark region may be set as the temporary image processing target region. That is, the analysis region that does not satisfy the condition illustrated in Expression 1 and the condition illustrated in Expression 2 may be set as the temporary image processing target region.

(143) The region that is not determined as the excessively bright region, and the region that is not determined as the excessively dark region are one example of a high image quality region that is determined based on a histogram of a gradation value. The shade histogram 180 is one example of the histogram of the gradation value.

(144) The frequency of the analysis region having broken gradation on a shade side is present at the shade value 0 of the shade histogram 180. The frequency of the analysis region having broken gradation on a highlight side is present at the shade value 255 of the shade histogram 180.

(145) That is, in a case where a determination threshold value for broken gradation on the shade side is denoted by T.sub.d, the analysis region that satisfies G(0)>T.sub.d may be determined as the region having broken gradation on the shade side. The region having broken gradation on the shade side may be set as the image processing target exclusion region.

(146) In addition, in a case where a determination threshold value for broken gradation on the highlight side is denoted by T.sub.b, the analysis region that satisfies G(255)>T.sub.b may be determined as the region having broken gradation on the highlight side. The region having broken gradation on the highlight side may be set as the image processing target exclusion region.

(147) Meanwhile, a region that is not determined as the region having broken gradation on the shade side, and a region that is not determined as the region having broken gradation on the highlight side may be set as the temporary image processing target region. That is, the analysis region that does not satisfy G(0)>T.sub.d and G(255)>T.sub.b may be set as the temporary image processing target region.

(148) The analysis region that does not satisfy G(0)>T.sub.d and G(255)>T.sub.b is one example of the high image quality region that is determined based on the histogram of the gradation value.

(149) [Description of Decision of Temporary Image Processing Target Region Using Imaging Condition]

(150) <Example in which Imaging Angle and Focus Position are Used>

(151) Next, decision of the temporary image processing target region using the imaging condition will be described. First, an example in which the imaging angle and the focus position are used as the imaging condition will be described.

(152) FIG. 11 is a schematic diagram of imaging of the subject. In FIG. 11, imaging of a subject 202 using an imaging apparatus 200 is schematically illustrated. A divided image 204 is an image that is obtained by imaging the subject 202 by setting the direction of the optical axis of the imaging apparatus 200 to a direction parallel to the horizontal direction.

(153) A divided image 206 is an image that is obtained by performing the tilt operation of the imaging apparatus 200, directing the optical axis of the imaging apparatus 200 by α degrees upward with respect to the horizontal direction, and imaging the subject 202. A divided image 208 is an image that is obtained by performing the tilt operation of the imaging apparatus 200, directing the optical axis of the imaging apparatus 200 by β degrees upward with respect to the horizontal direction, and imaging the subject 202. The angle α and the angle β have a relationship of α<β.

(154) FIG. 12 is a diagram illustrating one example of the temporary image processing target region in a case where a focus area is the center of an imaging range. In FIG. 12, three divided images that are obtained by imaging a bridge 210 as a main subject are illustrated. A circle of a broken line that is designated by reference sign 212 and illustrated in FIG. 12 represents the focus area in the divided image 204, the divided image 206, and the divided image 208. The center of the focus area is the focus position.

(155) A frame designated by reference sign 214 in the divided image 206 and the divided image 208 represents a boundary between a non-out-of-focus region that is in focus with respect to the bridge 210, and an out-of-focus region that is not in focus with respect to the bridge 210. The inside of the frame 214 is the non-out-of-focus region, and the outside of the frame 214 is the out-of-focus region. The frame 214 is included in the non-out-of-focus region.

(156) In the divided image 206 and the divided image 208 illustrated in FIG. 12, a region that is separated upward from a focus area 212 deviates from the depth of field and is out of focus. The divided image 208 that is above the divided image 206 and has a larger imaging angle than the divided image 206 has a larger area of the out-of-focus region than the divided image 206.

(157) FIG. 13 is a diagram illustrating one example of the temporary image processing target region in a case where the focus area is positioned below the center of the imaging range. The focus area 212 of a divided image 204A illustrated in FIG. 13 is positioned below the focus area 212 of the divided image 204 illustrated in FIG. 12. The same applies to a divided image 206A and a divided image 208A illustrated in FIG. 13.

(158) The divided image 206A illustrated in FIG. 13 has a larger area of the out-of-focus region than the divided image 206 illustrated in FIG. 12. The same applies to the divided image 208A illustrated in FIG. 13.

(159) That is, a region that is separated from the focus position in a direction in which the angle with respect to the subject changes deviates from the depth of field and is out of focus. In a case where the imaging angle with respect to the imaging apparatus as a reference extends in the up-down direction of the imaging apparatus by performing the tilt operation of the imaging apparatus, at least one of the upper end portion and the lower end portion of the divided image is out of focus.

(160) In a case where the imaging angle with respect to the imaging apparatus as a reference extends in the left-right direction of the imaging apparatus by performing the pan operation of the imaging apparatus, at least one of the right end portion and the left end portion of the divided image is out of focus. The up-down direction may be read as the longitudinal direction. The left-right direction may be read as the lateral direction.

(161) In FIG. 12, a non-out-of-focus region 204B of the divided image 204, a non-out-of-focus region of the divided image 206 designated by reference sign 206B, and a non-out-of-focus region of the divided image 208 designated by reference sign 208B may be decided as the temporary image processing target region.

(162) The relationship among the focus position, the imaging angle, and the non-out-of-focus region is stored as a database in the image processing target region database 42 illustrated in FIG. 3. Information of the non-out-of-focus region in the divided image may be acquired from the image processing target region database 42 illustrated in FIG. 3 using the focus position and the imaging angle.

(163) A non-main subject such as a background in the divided image 204, the divided image 206, and the divided image 208 illustrated in FIG. 12 is the actual out-of-focus region. However, since the final result is generated by performing the process with focus on the main subject, the temporary image processing target region may be decided by focusing on blurriness or shake of the main subject. The non-main subject may be included in the temporary image processing target region.

(164) In other words, all divided images can be handled as the same plane, and the temporary image processing target region in the divided image can be decided based on blurriness or shake of the main subject.

(165) <Example in which Depth of Field and Focal Point Shift Amount are Calculated>

(166) Next, another example of the decision of the temporary image processing target region using the imaging condition will be described. Hereinafter, an example in which the imaging angle, the focus position, the depth of field, the focal length, the F number, and the permissible circle of confusion are used as the imaging condition will be described.

(167) Specifically, the depth of field and a focal point shift amount that represents the distance of a focal point shift in the imaging range are calculated, and a region 229 of which the focal point shift amount falls in the depth of field is decided as the temporary image processing target region. A region in which the focal point shift amount does not fall in the depth of field is decided as the image processing target exclusion region. An illustration of the region in which the focal point shift amount does not fall in the depth of field is not provided.

(168) FIG. 14 is a schematic diagram illustrating a relationship between the focal point shift amount and the depth of field. Reference sign D.sub.N illustrated in FIG. 14 denotes a front depth of field. Reference sign D.sub.f denotes a rear depth of field. The front depth of field D.sub.N is represented using Expression 3 below. The rear depth of field D.sub.f is represented using Expression 4 below. The depth of field is represented using Expression 5 below.

(169) $\mathrm{Front}\mathrm{depth}\mathrm{of}\mathrm{field}=\frac{\begin{array}{c}\mathrm{Diameter}\mathrm{of}\mathrm{permissible}\\ \mathrm{circle}\mathrm{of}\mathrm{confusion}\end{array}\times F\mathrm{number}\times \mathrm{Subject}{\mathrm{distance}}^2}{\begin{array}{c}\begin{array}{c}\mathrm{Focal}\mathrm{length}+\\ \begin{array}{c}\mathrm{Diameter}\mathrm{of}\mathrm{permissible}\\ \mathrm{circle}\mathrm{of}\mathrm{confusion}\end{array}\times \end{array}\\ F\mathrm{number}\times \mathrm{Subject}\mathrm{distance}\end{array}}$$\mathrm{Rear}\mathrm{depth}\mathrm{of}\mathrm{field}=\frac{\begin{array}{c}\mathrm{Diameter}\mathrm{of}\mathrm{permissible}\\ \mathrm{circle}\mathrm{of}\mathrm{confusion}\end{array}\times F\mathrm{number}\times \mathrm{Subject}{\mathrm{distance}}^2}{\begin{array}{c}\begin{array}{c}\mathrm{Focal}{\mathrm{length}}^2-\\ \begin{array}{c}\mathrm{Diameter}\mathrm{of}\mathrm{permissible}\\ \mathrm{circle}\mathrm{of}\mathrm{confusion}\end{array}\times \end{array}\\ F\mathrm{number}\times \mathrm{Subject}\mathrm{distance}\end{array}}$$\mathrm{Depth}\mathrm{of}\mathrm{field}=\mathrm{Front}\mathrm{depth}\mathrm{of}\mathrm{field}+\mathrm{Rear}\mathrm{depth}\mathrm{of}\mathrm{field}$

(170) In a case where the front depth of field D.sub.N in Expression 3, the rear depth of field D.sub.f in Expression 4, and the depth of field in Expression 5 are in units of millimeters, the permissible circle of confusion diameter, the subject distance, and the focal length in Expression 3 and Expression 4 are in units of millimeters.

(171) The permissible circle of confusion diameter means the diameter of the permissible circle of confusion. The size of the imaging element comprised in the imaging apparatus 200 is applied as the permissible circle of confusion diameter.

(172) Reference sign 220 in FIG. 14 denotes an in-focus plane of the imaging apparatus 200. A solid line designated by reference sign 220A represents the imaging range of the imaging apparatus 200. The imaging range in the longitudinal direction is calculated by dividing a value obtained by multiplying the subject distance with the size of the sensor in the longitudinal direction by the focal length. A unit that represents a length is used for the imaging range in the longitudinal direction.

(173) The longitudinal direction represents a direction in which a broken line representing the in-focus plane 220 illustrated in FIG. 14 is directed. The size of the sensor in the longitudinal direction is the size of the imaging element comprised in the imaging apparatus 200 in the longitudinal direction.

(174) The imaging range in the lateral direction is calculated by dividing a value obtained by multiplying the subject distance with the size of the sensor in the lateral direction by the focal length. A unit that represents a length is used for the imaging range in the lateral direction.

(175) The lateral direction represents a direction that is orthogonal to the longitudinal direction and passes through the page of FIG. 14. The size of the sensor in the lateral direction is the size of the imaging element comprised in the imaging apparatus 200 illustrated in FIG. 14 in the lateral direction. An imaging angle θ is the angle between a subject plane perpendicular direction 222 and a direction 224 of the optical axis of the imaging apparatus 200. The subject plane perpendicular direction 222 is the direction of a line normal to a subject plane 202A.

(176) Reference sign 226 illustrated in FIG. 14 denotes the front end of the depth of field. Reference sign 228 denotes the rear end of the depth of field. Reference sign 230 denotes the focal point shift amount in front of the in-focus plane 220. Reference sign 232 denotes the focal point shift amount behind the in-focus plane 220. The focal point shift amount can be calculated a distance from the subject plane 202A to the in-focus plane 220 using geometric calculation.

(177) The imaging condition acquisition unit 40 illustrated in FIG. 3 includes a depth of field calculation unit and a focal point shift amount calculation unit. Illustrations of the depth of field calculation unit and the focal point shift amount calculation unit are not provided.

(178) FIG. 15 is a diagram illustrating one example of a region in the depth of field in a case where the focus area is the center of the imaging range. In FIG. 15, a temporary image processing target region 240 that is decided based on the depth of field and the focal point shift amount in the divided image 206 illustrated in FIG. 12 is illustrated.

(179) FIG. 16 is a diagram illustrating one example of the region in the depth of field in a case where the focus area is positioned below the center of the imaging range. In FIG. 16, a temporary image processing target region 240A that is decided based on the depth of field and the focal point shift amount in the divided image 206A illustrated in FIG. 13 is illustrated.

(180) <Example in which Type of Lens and Focus Position are Used>

(181) Next, another example of the decision of the temporary image processing target region using the imaging condition will be described. Hereinafter, an example in which the type of lens and the focus position are used as the imaging condition will be described. In a case where the imaging apparatus is a fixed lens type, the type of lens may be specified using the type of camera.

(182) FIG. 17 is a diagram illustrating one example of a field curvature in a case where the focus area is the center of the imaging range. In a case where the field curvature caused by the characteristics of the lens occurs, a center 252 of an imaging range 250 or a position in the vicinity of the center 252 is in focus, and a peripheral portion 254 of the imaging range 250 is out of focus.

(183) FIG. 18 is a diagram illustrating one example of the field curvature in a case where the focus area is the peripheral portion of the imaging range. In a case where the field curvature caused by the characteristics of the lens occurs, the center 252 of the imaging range 250 or the vicinity of the center 252 of the imaging range 250 is out of focus in a case where the peripheral portion 254 of the imaging range 250 is in focus. The vicinity of the center 252 of the imaging range 250 is a range including the center 252 of the imaging range 250. The vicinity of the center 252 of the imaging range 250 is decided depending on the type of lens.

(184) FIG. 19 is a diagram illustrating one example of the non-out-of-focus region in which the effect of the field curvature is considered in a case where the focus area is the center of the imaging range. FIG. 20 is a diagram illustrating one example of the non-out-of-focus region in which the effect of the field curvature is considered in a case where the focus area is the peripheral portion of the imaging range.

(185) FIG. 19 and FIG. 20 illustrate one example of a non-out-of-focus region 260 in which the effect of the field curvature is considered in the divided image 102 illustrated in FIG. 2. Reference sign 262 illustrated in FIG. 19 and FIG. 20 denotes the focus area.

(186) FIG. 21 is a diagram illustrating another example of the non-out-of-focus region in which the effect of the field curvature is considered in a case where the focus area is the center of the imaging range. The non-out-of-focus region 260 illustrated in FIG. 21 may be decided depending on the characteristics of the lens.

(187) The relationship among the type of lens, the focus position, and the non-out-of-focus region is stored as a database in the image processing target region database 42 illustrated in FIG. 3. Information of the non-out-of-focus region in the divided image may be acquired from the image processing target region database 42 illustrated in FIG. 3 using the type of lens and the focus position.

(188) The non-out-of-focus region 260 illustrated in FIG. 19 and FIG. 21 is one example of a region that includes the center portion and is determined from the characteristics of the lens. The non-out-of-focus region 260 illustrated in FIG. 20 is one example of a region that includes the peripheral portion and is determined from the characteristics of the lens.

(189) <Example in which Presence or Absence of Strobe Light Emission is Used>

(190) Next, another example of the decision of the temporary image processing target region using the imaging condition will be described. Hereinafter, an example in which the presence or absence of strobe light emission is used as the imaging condition will be described. Specifically, the peripheral portion is dark in the strobe light emission. Thus, a bright region other than the peripheral portion is set as the temporary image processing target area. The dark area which is the peripheral portion is set as the image processing target exclusion region.

(191) FIG. 22 is a diagram illustrating one example of the bright region and the dark region in the strobe light emission. A strobe light reaching region 270 that is a region in which strobe light radiated to the subject from a strobe arrives in the strobe light emission is the bright region. A strobe light non-reaching region 272 that is a region in which the strobe light does not arrive in the strobe light emission is the dark region.

(192) A region that is likely to be dark differs depending on the position of the strobe in the imaging apparatus. For example, in a case where the strobe is arranged in the upper portion of the imaging apparatus, the lower side of the captured image is likely to be dark. In a case where the subject distance is relatively decreased, the bright region is relatively decreased. In a case where the subject distance is relatively increased, the bright region is relatively increased.

(193) Reference sign 274 illustrated in FIG. 22 denotes a boundary between the strobe light reaching region 270 and the strobe light non-reaching region 272. In a case where the subject distance is relatively short, the boundary 274 moves in a direction indicated by an arrow line designated by reference sign 276.

(194) That is, the boundary 274 in a case where the subject distance is relatively decreased moves in a direction in which the area of the strobe light reaching region 270 is relatively decreased and the area of the strobe light non-reaching region 272 is relatively increased.

(195) In a case where the subject distance is relatively increased, the boundary 274 moves in a direction indicated by an arrow line designated by reference sign 278. That is, the boundary 274 in a case where the subject distance is relatively increased moves in a direction in which the area of the strobe light reaching region 270 is relatively increased and the area of the strobe light non-reaching region 272 is relatively decreased.

(196) The relationship between the subject distance and the strobe light reaching region 270 or the strobe light non-reaching region 272 is stored in advance as a database in the image processing target region database 42 illustrated in FIG. 3. Information of the strobe light reaching region 270 or the strobe light non-reaching region 272 in the divided image may be acquired from the image processing target region database 42 illustrated in FIG. 3 using the subject distance.

(197) [Description of Decision of Image Processing Target Region]

(198) Next, decision of the image processing target region will be described. FIG. 23 is a diagram illustrating one example of the image processing target region based on the logical product of two temporary image processing target regions. In FIG. 23, an image processing target region 304 that is decided by combining a first temporary image processing target region 300 which is decided using the imaging angle with a second temporary image processing target region 302 which is decided using the lens characteristics is illustrated for the divided image 206 illustrated in FIG. 12. Reference sign 310 illustrated in FIG. 23 denotes the focus area in the divided image 206.

(199) In the example illustrated in FIG. 23, in the divided image 206, it is assumed that the first temporary image processing target region 300 based on the imaging angle corresponds to 1, and an image processing target exclusion region 306A corresponds to 0. In addition, in the divided image 206, it is assumed that the second temporary image processing region 302 based on the lens characteristics corresponds to 1, and an image processing target exclusion region 306B corresponds to 0.

(200) A region that is represented using the logical product of the first temporary image processing target region 300 and the second temporary image processing target region 302 based on the effect of each factor is set as the image processing target region 304 of the divided image 206.

(201) In other words, in the divided image 206, it is assumed that the first temporary image processing target region 300 based on the imaging angle corresponds to 0, and the image processing target exclusion region 306A corresponds to 1. In addition, in the divided image 206, it is assumed that the second temporary image processing region 302 based on the lens characteristics corresponds to 0, and an image processing target exclusion region 306B corresponds to 1.

(202) A region that is represented using the logical product of the image processing target exclusion region 306A and the image processing target exclusion region 306B based on the effect of each factor is set as an image processing target exclusion region 306 of the divided image 206.

(203) The image processing target region 304 of the divided image 206 illustrated in the lower part of FIG. 23 is decided by superimposing the effect of the imaging angle and the effect of the lens characteristics. In FIG. 23, an example in which the image processing target region is decided as the logical product of two different types of temporary image processing target regions which are decided using two different types of imaging conditions is illustrated.

(204) The image processing target region may be decided using the temporary image processing target region which is decided using the image quality analysis result, and the temporary image processing target region which is decided using the imaging condition. The image processing target region may be decided as the logical product of two different types of temporary image processing regions which are decided using two different types of image quality analysis results.

(205) The logical product may include an aspect in which the first temporary image processing target region is decided, and then, the second temporary image processing target region is decided by limiting the processing region to the first temporary image processing target region. For example, the first temporary image processing target region may be decided for the divided image based on the imaging condition, and the image processing target region may be decided by acquiring the image quality analysis result using the decided first temporary image processing target region as a target.

(206) [Image Processing]

(207) An image processing method according to the present embodiment is applied to only damage detection, only image composition, and both of the damage detection and the image composition. Both of the damage detection and the image composition includes composition of the damage detection result. Hereinafter, the damage detection and the image composition will be described in detail.

(208) <Damage Detection Process>

(209) In a damage detection process, damage is detected for the image processing target region in the image obtained by capturing using the imaging apparatus. A damage detection apparatus that is prepared in advance is applied in the damage detection. In the damage detection process, the damage detection is executed for the image processing target region using the damage detection apparatus.

(210) An example of the subject as a target of the damage detection is exemplified by a structure such as a wall, a pillar, a ceiling, and a window of a building, a road, a tree, a rock face, a sea surface, a ship, a vehicle, and the like.

(211) An example of damage in the wall and the pillar of the building is exemplified by a crevice of a concrete member, free lime, water leakage, stripping, rebar exposure, a float, a crack of a steel member, corrosion, a chalk line drawn on the wall or the like, and the like. Another example of the damage is exemplified by breakage, loss, a hole, discoloration, peeling of a coating, corrosion, and the like in the structure.

(212) <Image Composition Process>

(213) An image composition process includes an image association step and an image combining step. The image association process step includes a feature point extraction step, a correspondence point extraction step, and a projective transformation matrix calculation step.

(214) An image association process is executed for all divided images. An image combining process may be executed for only the image processing target region. The image combining process may be executed by prioritizing the image processing target region.

(215) In the image combining process, a blending process or an overwriting process is executed for a superimposed region that extends in two divided images. The blending process or the overwriting process is executed for only pixels included in the image processing target region.

(216) The blending process is a process of blending the pixels of the two divided images included in the superimposed region in accordance with a predetermined blending process rule. The overwriting process is a process of overwriting the superimposed region using the pixels of any one of the two divided images constituting the superimposed region. In other words, the overwriting process is a process of replacing the superimposed region with any one of the two divided images constituting the superimposed region.

(217) In a case where the superimposed region does not include the image processing target region, the pixels of the image processing target exclusion region may be used in the processing of the superimposed region. In a case where the superimposed region does not include the image processing target region, the pixel values of the superimposed region may be set to a predetermined value.

(218) An example of the predetermined value is exemplified by a pixel value representing black and a pixel value representing white. An example of the pixel value representing black is exemplified by the minimum pixel value. An example of the pixel value representing white is exemplified by the maximum pixel value.

(219) The pixels of each divided image are used in a single region that is a region other than the superimposed region and is configured using a single divided image.

(220) FIG. 24 is a description of one example of the image composition process. Hereinafter, an example in which the divided image 204, the divided image 206, and the divided image 208 illustrated in FIG. 12 are composited will be described. Reference sign 209 in FIG. 24 denotes a composite image that represents a composition result obtained by compositing the divided image 204, the divided image 206, and the divided image 208.

(221) In the image association process step, an association among the divided image 204, the divided image 206, the divided image 208, and the composite image 209 is decided. First, in the feature point extraction step, the feature point of each of the divided image 204, the divided image 206, and the divided image 208 is extracted. Specifically, the coordinates of the feature point are extracted. The coordinates are coordinates set in the divided image 204, the divided image 206, and the divided image 208. For example, the coordinates are exemplified by coordinates in a two-dimensional orthogonal coordinate system.

(222) Next, in the correspondence point extraction step, a correspondence point that corresponds to the feature point of each of the divided image 204, the divided image 206, and the divided image 208 in the composite image 209 is extracted. In the projective transformation matrix calculation step, the coordinates of the correspondence point corresponding to the coordinates of the feature point of each of the divided image 204, the divided image 206, and the divided image 208 in the composite image 209 are calculated. Details of the projective transformation matrix calculation are disclosed in Paragraph 0053 of JP2004-072533A.

(223) A region 320 in the composite image 209 illustrated in FIG. 24 is a region that corresponds to the divided image 204. A region 322 in the composite image 209 is a region that corresponds to an image processing target exclusion region 206D on the lower side of the divided image 206.

(224) A region 324 in the composite image 209 is a region that corresponds to the image processing target region 206B of the divided image 206. A region 326 in the composite image 209 is a region that corresponds to an image processing target exclusion region 206C on the upper side of the divided image 206.

(225) A region 328 in the composite image 209 is a region that corresponds to an image processing target exclusion region 208C of the divided image 208. A region 330 in the composite image 209 is a region that corresponds to the image processing target region 208B of the divided image 208.

(226) By doing so, the correspondence relationship among the divided image 204, the divided image 206, the divided image 208, and the composite image 209 is derived.

(227) In the image combining step, a combining process of combining the divided image 204, the divided image 206, and the divided image 208 is executed. The blending process or the overwriting process is executed for a superimposed region 352 in the composite image 209.

(228) The superimposed region 352 is configured with a region 352A and a region 352B. The region 352A is a region in which the divided image 204 and the image processing target exclusion region of the divided image 206 extend. In the region 352A, the pixels of the divided image 204 are employed. The region 352B is a region in which the divided image 204 and the image processing target region 206B of the divided image 206 extend. In the region 352B, the pixels of the divided image 204 may be used, or the pixels of the image processing target region 206B of the divided image 206 may be used. In the region 352B, the pixels of the divided image 204 and the pixels of the image processing target region 206B of the divided image 206 may be blended.

(229) A superimposed region 354 is configured with a region 354A and a region 354B. The region 354A is a region in which the image processing target region 206B of the divided image 206 and the image processing target exclusion region 208C of the divided image 208 extend. In the region 354A, the pixels of the image processing target region 206B of the divided image 206 are used.

(230) The region 354B is a region in which the image processing target exclusion region 206C of the divided image 206 and the image processing target exclusion region 208C of the divided image 208 extend. In the region 354B, the pixels of the image processing target exclusion region 206C of the divided image 206 may be used, or the pixels of the image processing target exclusion region 208C of the divided image 208 may be used.

(231) In the region 354B, the pixels of the image processing target exclusion region 206C of the divided image 206 and the pixels of the image processing target exclusion region 208C of the divided image 208 may be blended. A predetermined pixel value such as the pixel value representing white or the pixel value representing black may be applied in the region 354B.

(232) A region 356A is a region that corresponds to the image processing target exclusion region 208C of the divided image 208. In the region 356A, the pixels of the image processing target exclusion region 208C of the divided image 208 may be used, or a predetermined pixel value such as the pixel value representing white or the pixel value representing black may be applied.

(233) The image processing target exclusion region 356 in the composite image 209 is configured with the region 354B and the region 356A. A process rule for the image processing target exclusion region is used for the image processing target exclusion region 356. An example of the process rule for the image processing target exclusion region is exemplified by the process for the region 354B and the process for the region 356A.

(234) The following process rule is applied in the image combining process. In a case where a plurality of image processing target regions are present, the blending process or the pixels of any one image processing target region are employed. This rule applies to the region 324 in the composite image 209 illustrated in FIG. 24.

(235) In a case where one image processing target region is present, the pixels of the image processing target region are employed. This rule applies to a region 340, a region 344, and a region 350 in the composite image 209 illustrated in FIG. 24.

(236) In a case where the image processing target region is not present, the blending process, the pixels of any one image processing target exclusion region, or the predetermined pixel value is employed. This rule applies to a region 346 and a region 348 in the composite image 209 illustrated in FIG. 24.

(237) <Damage Detection Process and Image Composition Process>

(238) In a case where the damage detection process and the image composition process are executed, a process of compositing the damage detection result may be executed. An example of the composition of the damage detection result is exemplified by an example in which a crevice is detected in each of the divided image 204, the divided image 206, and the divided image 208 illustrated in FIG. 24, and the divided image 204, the divided image 206, and the divided image 208 in which the crevice is detected are composited.

(239) The composite image obtained by compositing the divided image 204, the divided image 206, and the divided image 208 in which the crevice is detected is illustrated as an image processing result image 209B in FIG. 25.

(240) In the composition of the damage detection result, the damage detection result is composited based on the correspondence relationship between the divided images. An example of the correspondence relationship between the divided images is exemplified by the projective transformation matrix. In the superimposed region, the detection result of any divided image included in the image processing target region is employed and composited.

(241) The image processing target region in the damage detection may be set as the image processing target region in the image composition process. The image processing target region in the image composition process may be derived separately from the image processing target region in the damage detection.

(242) [Display of Image Processing Result]

(243) FIG. 25 is a description of a display example of the image processing result. In FIG. 25, an image processing result image 209A that is the composite image obtained by compositing the divided image 204, the divided image 206, and the divided image 208 and in which the image processing target exclusion region is displayed distinctively from other regions is illustrated.

(244) The image processing result image 209A or the image processing result image 209B illustrated in FIG. 25 is displayed on an image display unit, not illustrated.

(245) In the image processing result image 209A, a frame that indicates an image processing target exclusion region 209C is displayed. In the example illustrated in FIG. 25, the pixels of the divided image 206 or the pixels of the divided image 208 are employed in the image processing target exclusion region 209C.

(246) The image processing result image 209B illustrated in FIG. 25 is the composite image obtained by compositing the divided image 204, the divided image 206, and the divided image 208 in which the crevice is detected. In the image processing result image 209B, a crevice 209E of the detection target is displayed in a highlighted manner. The crevice 209E is not displayed in an image processing target exclusion region 209D in the image processing result image 209B.

(247) In the display of the image processing result, in a case where the image processing target region is present in the image processing result image, the image processing target region in the divided image may be changed, and the image processing may be executed again. For example, the image processing target region 206B of the divided image 206 illustrated in FIG. 25 may be manually changed. The change of the image processing target region in the divided image is executed using an image processing target region changing unit, not illustrated.

(248) In the change of the image processing target region in the divided image, an aspect in which imaging is performed by changing the imaging condition such as the imaging angle using the imaging apparatus and the image processing target region is set for the newly obtained divided image may be applied.

(249) [Effect]

(250) According to the image processing system configured as above, a region that is represented as the logical product of the first temporary image processing target region 300 and the second temporary image processing target region 302 which are two or more temporary image processing target regions respectively corresponding to two or more pieces of image quality determination information is decided as the image processing target region 304. Accordingly, the image processing target region 304 in which a plurality of causes of decrease in image quality are considered can be decided.

(251) The temporary image processing target region is decided using at least one of the imaging condition or the image quality analysis result.

(252) As the image quality analysis, the spatial frequency spectrum distribution may be acquired for each analysis target region, and a determination of the presence or absence of the high frequency component may be applied. A region in which the high frequency component is present may be decided as the temporary image processing target region.

(253) As the image quality analysis, the histogram of the shade value may be acquired for each analysis target region, and the analysis of the histogram may be applied. The temporary image processing target region may be decided based on a bias in frequency of the histogram, whether or not the frequency is present at the minimum shade value, and whether or not the frequency is present at the maximum shade value.

(254) The temporary image processing target region may be decided by applying the imaging angle and the focus position as the imaging condition. In addition, a region in which the focal point shift amount falls in the range of the depth of field may be decided as the temporary image processing target region by applying the imaging angle, the focus position, the subject distance, the focal length, the F number, and the diameter of the permissible circle of confusion as the imaging condition.

(255) Furthermore, the in-focus region in a case where a field curvature is present in the lens may be decided as the image processing target region by applying the type of lens and the focus position as the imaging condition. In the case of the imaging apparatus of a fixed lens type, the type of lens may be specified using the model of the camera.

(256) Furthermore, a range in which the strobe light arrives may be decided as the image processing target region by applying the presence or absence of the strobe light emission as the imaging condition. Furthermore, by applying an imaging distance as the imaging condition, the image processing target region may be decided by adjusting the range in which the strobe light arrives depending on the imaging distance.

(257) [Application Example for Client Server Type Network System]

(258) FIG. 26 is a block diagram illustrating an application example for a client server type network system. An image processing system 500 illustrated in FIG. 26 comprises a server apparatus 502 and one or more client apparatuses. In FIG. 26, a first client apparatus 504, a second client apparatus 506, and a third client apparatus 508 are illustrated as the client apparatus.

(259) The server apparatus 502 is communicably connected to the first client apparatus 504, the second client apparatus 506, and the third client apparatus 508 through a network 510.

(260) Any communication network such as a public communication network and a local area network can be applied as the network 510. The local area network may be represented as LAN that is the abbreviation for English representation Local Area Network.

(261) As data communication between the server apparatus 502 and the network 510, wired type data communication may be applied, or wireless type data communication may be applied. Similarly, as data communication between the client apparatus and the network 510, wired type data communication may be applied, or wireless type data communication may be applied.

(262) As data communication between the client apparatus and the network 510, the same type as the data communication between the server apparatus 502 and the network 510 may be applied, or a different type may be applied.

(263) The client apparatus is a collective term for the first client apparatus 504, the second client apparatus 506, and the third client apparatus 508 illustrated in FIG. 26. The term client apparatus may represent any of the first client apparatus 504, the second client apparatus 506, and the third client apparatus 508.

(264) The server apparatus 502 illustrated in FIG. 26 is configured using one or more computers. Hardware that implements the function of the image processing system 10 illustrated in FIG. 1, FIG. 3, and FIG. 4 may be applied in the server apparatus 502. A configuration example of the server apparatus 502 is exemplified by an example in which a function block for acquiring machine learning data and a function block for executing the image processing illustrated in FIG. 1, FIG. 3, and FIG. 4 are comprised.

(265) The server apparatus 502 may comprise a storage apparatus in which a database implementing the function of the image processing target region database 42 illustrated in FIG. 3 is stored. The storage apparatus may be incorporated in the server apparatus 502 illustrated in FIG. 26. Alternatively, the storage apparatus may be externally attached to the server apparatus 502.

(266) The server apparatus 502 comprises a communication interface. The server apparatus 502 is connected to the network 510 using the communication interface. The server apparatus 502 comprises a program storage unit.

(267) The client apparatus comprises an image data transmission unit that transmits image data to the server apparatus 502. The client apparatus comprises a communication interface. The client apparatus is connected to the network 510 using the communication interface. In FIG. 26, a computer is illustrated as an example of the client apparatus. Alternatively, a portable type terminal apparatus may be applied as the client apparatus.

(268) [Image Processing Method]

(269) FIG. 27 is a flowchart illustrating a flow of procedure of the image processing method according to the embodiment. The image processing method according to the present embodiment is configured to include an image acquisition step S10, an image quality determination information acquisition step S12, an image processing target region decision step S14, an image processing step S16, and an image output step S18.

(270) In the image acquisition step S10, the image data 20 is acquired using the image acquisition unit 12 illustrated in FIG. 1. In the image quality determination information acquisition step S12 illustrated in FIG. 27, two different types of image quality determination information are generated and acquired using the image quality determination information acquisition unit 14 illustrated in FIG. 1. The image quality determination information acquisition step S12 illustrated in FIG. 27 may include at least one of an imaging condition acquisition step of acquiring the imaging condition or an analysis step of analyzing the image.

(271) In the image processing target region decision step S14, the image processing target region is decided based on the two different types of image quality determination information using the image processing target region decision unit 16 illustrated in FIG. 1.

(272) In the image processing step S16 illustrated in FIG. 27, the image processing is executed using the image processing unit 18 illustrated in FIG. 1. In the image output step S18 illustrated in FIG. 27, the image processing result is output. The image processing method is finished after the image processing result is output in the image output step S18.

(273) FIG. 28 is a flowchart illustrating a flow of procedure of the image processing target region decision step illustrated in FIG. 27. The image processing target region decision step S14 illustrated in FIG. 27 is configured to include a first temporary image processing target region decision step S100, a second temporary image processing target region decision step S102, and an image processing target region decision step S104 illustrated in FIG. 28.

(274) In the first temporary image processing target region decision step S100, the first temporary image processing target region in the image of the image processing target is decided based on the image quality analysis result 30 or the imaging condition using the image processing target region decision unit 16 illustrated in FIG. 1.

(275) In the second temporary image processing target region decision step S102 illustrated in FIG. 28, the second temporary image processing target region in the image of the image processing target is decided based on the image quality analysis result 30 or the imaging condition using the image processing target region decision unit 16 illustrated in FIG. 1.

(276) In the image processing target region decision step S104 illustrated in FIG. 28, the image processing target region that is represented as the logical product of the first temporary image processing target region and the second temporary image processing target region is decided using the image processing target region decision unit 16 illustrated in FIG. 1.

(277) In the image processing target region decision step S104 illustrated in FIG. 28, a transition is made to the image processing step S16 illustrated in FIG. 27 after the image processing target region is decided. In the image processing step S16, at least one process of a detection target detection process or the image composition process is executed for the image of the image processing target.

(278) [Application Example for Image Processing Program]

(279) An image processing program that corresponds to the image processing system and the image processing method illustrated in the present embodiment may be configured. That is, an image processing program that causes a computer to implement the function of each unit of the image processing system 10 illustrated in FIG. 1, FIG. 3, and FIG. 4 may be configured.

(280) For example, an image processing program that causes one or a plurality of computers (processors) to implement an image acquisition function corresponding to the image acquisition unit, an image quality determination information acquisition function corresponding to the image quality determination information acquisition unit, an image processing target region decision function corresponding to the image processing target region decision unit, and an image processing function corresponding to the image processing unit may be configured. A computer-readable recording medium on which the program is recorded is also included in the present embodiment.

(281) [Terminology]

(282) The term same in the present specification includes aspects in which the same or approximately the same effect is obtained and that may be regarded as being substantially the same among aspects that are strictly different.

(283) The term up in the present specification refers to a direction opposite to the direction of gravity. The term down refers to the direction of gravity. Up and down may represent a relative positional relationship.

(284) The term orthogonal in the present specification includes substantial orthogonality in which the same effect as a case where the angle formed between two intersecting directions is 90 degrees can be obtained while the angle formed between two intersecting directions is strictly smaller than 90 degrees or exceeds 90 degrees.

(285) The term parallel in the present specification includes substantial parallelism in which the same effect as a case where two directions are strictly in parallel can be obtained while the two directions are strictly not in parallel.

Combination of Embodiment, Modification Example, and Like

(286) Configurations described in the embodiment and matters described in the modification example can be appropriately used in combination. In addition, a part of constituents may be replaced.

(287) In the embodiment of the present invention described thus far, constituents can be appropriately changed, added, or removed without departing from the gist of the present invention. The present invention is not limited to the embodiment described thus far. Those skilled in the art can carry out various modifications within the technical idea of the present invention.

EXPLANATION OF REFERENCES

(288) 10, 500: image processing system 12: image acquisition unit 14: image quality determination information acquisition unit 16: image processing target region decision unit 18: image processing unit 20: image data 22: EXIF information 24: user designation information 26: user selection information 28: sensor information 30: image quality analysis result 32A: first image quality determination information 32B: second image quality determination information 34: image processing target region information 40: imaging condition acquisition unit 42: image processing target region database 44: imaging condition information 50: image quality analysis result acquisition unit 54: threshold value 100, 102, 104: divided image 106: composite image 108: out-of-focus region 110, 112, 114: out-of-focus region 116, 118: superimposed region 150: analysis region 160: spatial frequency spectrum distribution image 162: center of spatial frequency spectrum distribution image 164: corner of spatial frequency spectrum distribution image 166: low frequency component spectrum 168: high frequency component spectrum 170: high frequency component determination region 180: shade histogram 200: imaging apparatus 202: subject 202A: subject plane 204, 204A: divided image 204B: non-out-of-focus region 206, 206A: divided image 206B, 208B, 304: image processing target region 206C, 206D, 208C, 209C, 209D, 306, 306A, 306B, 356: image processing target exclusion region 208, 208A: divided image 209: composite image 209A, 209B: image processing result image 210: bridge 212, 262, 310: focus area 214: frame 220: in-focus plane 220A: imaging range 222: subject plane perpendicular direction 224: direction of optical axis of imaging apparatus 226: front end of depth of field 228: rear end of depth of field 230: focal point shift amount in front of in-focus plane 232: focal point shift amount behind in-focus plane 240, 240A: temporary image processing target region 250: imaging range 252: center of imaging range 254: peripheral portion of imaging range 260: non-out-of-focus region 270: strobe light reaching region 272: strobe light non-reaching region 274: boundary between strobe light reaching region and strobe light non-reaching region 276, 278: movement direction of boundary 300: first temporary image processing target region 302: second temporary image processing target region 320, 322, 324, 326, 328, 330, 340, 342, 344, 346, 348, 350, 352A, 352B, 354A, 354B, 356A: region 352, 354: superimposed region 500: image processing system 502: server apparatus 504: first client apparatus 506: second client apparatus 508: third client apparatus 510: network S1 to S18: each step of image processing method S100 to S114: each step of image processing target region decision step