METHODS AND SYSTEMS FOR GRAY COMPONENT REPLACEMENT IN COLOR ELECTROPHORETIC DISPLAY DEVICES

Abstract

A driving method for a color electrophoretic display includes receiving an RGB input color image; mapping the RGB color image to an electrophoretic destination space defined by an electrophoretic color gamut to generate a color destination space image; dithering the color destination space image to generate a dithered color image; identifying neutral portions of the dithered color image; converting the RGB color image to a black and white image; mapping the black and white image to a black and white electrophoretic destination space to generate a black and white destination space image; dithering the black and white destination space image to generate a dithered black and white image; merging the dithered color image and the dithered black and white image by replacing neutral portions of the dithered color image with corresponding portions of the dithered black and white image to generate a merged color image; and displaying the merged image.

Claims

1. A method of driving a color electrophoretic display, comprising: (a) receiving a red-green-blue (RGB) input color image; (b) mapping the RGB input color image to an electrophoretic destination space defined by an electrophoretic color gamut to generate a color destination space image; (c) dithering the color destination space image to generate a dithered color image; (d) identifying neutral portions of the dithered color image; (e) converting the RGB input color image to a black and white image; (f) mapping the black and white image to a black and white electrophoretic destination space to generate a black and white destination space image; (g) dithering the black and white destination space image to generate a dithered black and white image; (h) merging the dithered color image and the dithered black and white image by replacing the neutral portions of the dithered color image with corresponding portions of the dithered black and white image to generate a merged color image; and (i) displaying the merged color image on the electrophoretic display.

2. The method of claim 1, wherein the black and white electrophoretic destination space is defined by black, white, and a plurality of gray colors.

3. The method of claim 1, further comprising performing sigmoidal contrast enhancement of the black and white destination space image in step (f) to compensate for reduced dynamic range.

4. The method of claim 1, wherein the color destination space image and the black and white destination space image are dithered using a threshold mask or a dither mask.

5. The method of claim 1, wherein the RGB input color image is converted to the black and white image in step (e) using weighted additions of red, green, and blue channels of the RGB input color image.

6. The method of claim 1, wherein identifying the neutral portions of the dithered color image comprises calculating a degree of neutrality for each pixel in the dithered color image and determining whether the degree of neutrality for each pixel meets a given threshold.

7. The method of claim 6, wherein the degree of neutrality for each pixel in the dithered color image is calculated using an RGB ratio method, a neighborhood RGB ratio method, or a uniform color space chroma method.

8. The method of claim 7, wherein the RGB ratio method comprises calculating an RGB ratio metric value for each pixel of the color image, and comparing the RGB ratio metric value to a threshold to identify neutral portions of the color image.

9. The method of claim 1, wherein the color electrophoretic display comprises a light-transmissive electrode, an active matrix of pixel electrodes, and an electrophoretic medium comprising multiple types of electrophoretic particles having different optical properties, the electrophoretic medium being disposed between the light-transmissive electrode and the active matrix of pixel electrodes, the electrophoretic display being capable of producing a plurality of primary colors at each pixel electrode.

10. The method of claim 1, wherein the color electrophoretic display includes: a processor; a controller coupled to the processor, and configured to provide pixel color instructions to the electrophoretic display; a non-transitory memory coupled to the processor having a program stored therein executable by the processor for controlling operation of the electrophoretic display including performing steps (a) to (i).

11. A color display, comprising: an electrophoretic display comprising a light-transmissive electrode, an active matrix of pixel electrodes, and an electrophoretic medium comprising multiple types of electrophoretic particles having different optical properties, the electrophoretic medium being disposed between the light-transmissive electrode and the active matrix of pixel electrodes, the electrophoretic display being capable of producing a plurality of primary colors at each pixel electrode; a processor; a controller coupled to the processor, and configured to provide electrophoretic display pixel color instructions to the active matrix of pixel electrodes; a non-transitory memory coupled to the processor having a program stored therein containing a plurality of instructions which, when executed by the processor, cause the processor to: (a) receive a red-green-blue (RGB) input color image; (b) map the RGB input color image to an electrophoretic destination space defined by an electrophoretic color gamut to generate a color destination space image; (c) dither the color destination space image to generate a dithered color image; (d) identify neutral portions of the dithered color image; (e) convert the RGB input color image to a black and white image; (f) map the black and white image to a black and white electrophoretic destination space to generate a black and white destination space image; (g) dither the black and white destination space image to generate a dithered black and white image; (h) merge the dithered color image and the dithered black and white image by replacing the neutral portions of the dithered color image with corresponding portions of the dithered black and white image to generate a merged color image; and (i) instruct the controller to cause the electrophoretic medium to display the merged color image.

12. The color display of claim 11, wherein the black and white electrophoretic destination space is defined by black, white, and a plurality of gray colors.

13. The color display of claim 11, wherein the program further comprises instructions for performing sigmoidal contrast enhancement of the black and white destination space image in step (f) to compensate for reduced dynamic range.

14. The color display of claim 11, wherein the color destination space image and the black and white destination space image are dithered using a threshold mask or a dither mask.

15. The color display of claim 11, wherein the RGB input color image is converted to the black and white image in step (e) using weighted additions of red, green, and blue channels of the RGB input color image.

16. The color display of claim 11, wherein the neutral portions of the dithered color image are identified by calculating a degree of neutrality for each pixel in the dithered color image and determining whether the degree of neutrality for each pixel meets a given threshold.

17. The color display of claim 16, wherein the degree of neutrality for each pixel in the dithered color image is calculated using an RGB ratio method, a neighborhood RGB ratio method, or a uniform color space chroma method.

18. The color display of claim 17, wherein the RGB ratio method comprises calculating an RGB ratio metric value for each pixel of the color image, and comparing the RGB ratio metric values to a threshold to identify neutral portions of the color image.

19. The color display of claim 11, wherein the electrophoretic medium includes at least four types of electrophoretic particles and is capable of producing eight primary colors at each pixel electrode.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0041] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0042] FIG. 1 is a schematic cross-section showing an embodiment of an encapsulated electrophoretic display suitable for use with the methods of the invention.

[0043] FIG. 2 is a schematic cross-section showing an embodiment of an encapsulated electrophoretic display suitable for use with the methods of the invention.

[0044] FIG. 3A illustrates an exemplary equivalent circuit of a single pixel of an electrophoretic display wherein the voltage on the single pixel is controlled with a transistor. The circuit of FIG. 3A is commonly used in active matrix backplanes.

[0045] FIG. 3B illustrates an exemplary color display including a display module that can be any electro-optic display module, but is preferably a color electrophoretic display module. The color display also includes a processor, memory, one or more power supplies, and a controller.

[0046] FIG. 4 is a schematic cross-section showing the positions of the various colored particles in a colored electrophoretic medium when displaying black, white, three subtractive primary colors and three additive primary colors.

[0047] FIG. 5 shows exemplary push-pull drive schemes for addressing an electrophoretic medium including three subtractive particles and a scattering (white) particle.

[0048] FIGS. 6A-6D are simplified diagrams illustrating basic principle of GCR for a blue color.

[0049] FIGS. 7A and 7B are simplified diagrams illustrating basic principle of GCR for a neutral color.

[0050] FIGS. 8A and 8B show a uniform gray source image and a dithered uniform gray image, respectively, demonstrating how dithering can be used to successfully render uniform areas.

[0051] FIGS. 9A and 9B show a single-pixel gray line source image and a dithered single-pixel gray line image, respectively, demonstrating the difficulty in rendering fine details using a common electrophoretic dithering method.

[0052] FIG. 10 is a flowchart illustrating an exemplary process for to gray component replacement in images to be displayed on color electrophoretic display devices according to one aspect of the invention.

[0053] FIGS. 11A and 11B show an exemplary test source image in its original RGB format and calculated black and white format, respectively.

[0054] FIG. 12 is a graph showing addition of sigmoidal contrast enhancement during the compression from Input RGB values to output E Ink destination RGB values.

[0055] FIGS. 13A and 13B show dithered renderings of the test image in the color destination-space and the black and white destination-space, respectively.

[0056] FIGS. 14A and 14B show image maps used for determining the degree of neutrality using the RGB ratio method.

[0057] FIG. 15 shows a function to indicate the likelihood a pixel will be classified as black and white.

[0058] FIG. 16 is a graph showing Sigmoidal color likelihood and black and white likelihood functions.

[0059] FIGS. 17A and 17B show threshold mask images with =1 and =1.5, respectively.

[0060] FIG. 18 illustrates an exemplary process of merging dithered black and white and dithered color test images.

[0061] FIGS. 19A and 19B show the test image rendered using only color palette primaries and using color and black and white primaries, respectively.

[0062] FIGS. 20A and 20B show close-up views of text rendered using only color palette primaries and using color and black and white primaries, respectively.

[0063] FIGS. 21A and 21B show close-up views of an area of the test image that includes both a black and white image and color image rendered using only color palette primaries and using color and black and white primaries, respectively.

[0064] FIGS. 22A and 22B show close-up views of a shadow gradient area of the test image rendered using only color palette primaries and using color and black and white primaries, respectively.

DETAILED DESCRIPTION

[0065] The present invention generally relates to transforming input image files (typically RGB files) to files that can be displayed on color electrophoretic display devices. More particularly, the invention relates to gray component replacement in images to render fine neutral details, such as black and gray lines and text, using only neutral primaries with lower contrast between groups of dithered primaries. This makes dithering patterns less visible and makes black and white regions appear smoother, increasing the resolution of fine details.

[0066] Electrophoretic display devices may be constructed using an electrophoretic fluid in several ways that are known in the prior art. The electrophoretic fluid may be encapsulated in microcapsules or incorporated into microcell structures that are thereafter sealed with a polymeric layer. The microcapsule or microcell layers may be coated or embossed onto a plastic substrate or film bearing a transparent coating of an electrically conductive material. This assembly may be laminated to a backplane bearing pixel electrodes using an electrically conductive adhesive. Alternatively, the electrophoretic fluid may be dispensed directly on a thin open-cell grid that has been arranged on a backplane including an active matrix of pixel electrodes. The filled grid can then be top-sealed with an integrated protective sheet/light-transmissive electrode.

[0067] FIGS. 1 and 2 show electrophoretic displays 101, 102 including a top light-transmissive electrode 110, an electrophoretic medium 120, and bottom drive electrodes 130, 135, which are often pixel electrodes of an active matrix of pixels controlled with thin film transistors (TFT). Alternatively, bottom drive electrodes 130, 135 may be directly wired to a controller or some other switch that provides voltage to the bottom drive electrodes 130, 135 to effect a change in the optical state of the electrophoretic medium 120, i.e., segmented electrodes. Importantly, it is not necessary that a junction between drive electrodes 130, 135 corresponds with an intersection of microcapsules or with a wall 127 of a microcell. Because the electrophoretic medium 120 is sufficiently thin, and the capsules or microcells sufficiently wide, the pattern of the drive electrodes (square, circles, hexagons, wavy, text, or otherwise) will show when the display is viewed from the viewing surface; not the pattern of the containers. The electrophoretic medium 120 contains at least one type of electrophoretic particle 121, however a second type of electrophoretic particle 122, or a third type of electrophoretic particle 123, a fourth type of electrophoretic particle 124, or more types of particles is feasible. (It should be noted that third electrophoretic particles 123 and fourth electrophoretic particles 124 can be included within the microcapsules 126 of FIG. 1, but have been omitted for ease of illustration.) The electrophoretic medium 120 typically includes a solvent, such as isoparaffins, and may also include dispersed polymers and charge control agents to facilitate state stability, e.g. bistability, i.e., the ability to maintain an electro-optic state without inputting any additional energy.

[0068] The electrophoretic medium 120 is typically compartmentalized such as by a microcapsule 126 (FIG. 1) or the walls of a microcell 127 (FIG. 2). The entire display stack is typically disposed on a substrate 150, which may be rigid or flexible. The displays 101, 102 typically also include a protective layer 160, which may simply protect the top electrode 110 from damage, or it may envelop the entire displays 101, 102 to prevent ingress of water, etc. Electrophoretic displays 101, 102 may also include one or more adhesive layers 140, 170, and/or sealing layers 180 as needed. In some embodiments, an adhesive layer may include a primer component to improve adhesion to the electrode layer 110, or a separate primer layer (not shown in FIG. 1 or 2) may be used. (The structures of electrophoretic displays and the component parts, pigments, adhesives, electrode materials, etc., are described in many patents and patent applications published by E Ink Corporation, such as U.S. Pat. Nos. 6,922,276; 7,002,728; 7,072,095; 7,116,318; 7,715,088; and 7,839,564, all of which are incorporated by reference herein in their entireties.

[0069] Thin-film-transistor (TFT) backplanes usually have only one transistor per pixel electrode or propulsion electrode. Conventionally, each pixel electrode has associated therewith a capacitor electrode such that the pixel electrode and the capacitor electrode form a capacitor; see, e.g., International Patent Application WO2001007961. In some embodiments, N-type semiconductor (e.g., amorphous silicon) may be used to form the transistors and the select and non-select voltages applied to the gate electrodes can be positive and negative, respectively.

[0070] FIG. 3A depicts an exemplary equivalent circuit of a single pixel of an electrophoretic display. As illustrated, the circuit includes a capacitor 10 formed between a pixel electrode and a capacitor electrode. The electrophoretic medium 20 is represented as a capacitor and a resistor in parallel. In some instances, direct or indirect coupling capacitance 30 between the gate electrode of the transistor associated with the pixel and the pixel electrode (usually referred to a as a parasitic capacitance) may create unwanted noise to the display. Usually, the parasitic capacitance 30 is much smaller than that of the storage capacitor 10, and when the pixel rows of a display is being selected or deselected, the parasitic capacitance 30 may result in a small negative offset voltage to the pixel electrode, also known as a kickback voltage, which is usually less than 2 volts. In some embodiments, to compensate for the unwanted kickback voltage, a common potential V.sub.com, may be supplied to the top plane electrode and the capacitor electrode associated with each pixel, such that, when V.sub.com is set to a value equal to the kickback voltage (V.sub.KB), every voltage supplied to the display may be offset by the same amount, and no net DC-imbalance experienced.

[0071] Typically, the transistors are arranged in a matrix having gate and signal lines to each transistor, as well as a drain electrode typically coupled to a pixel electrode. This active matrix backplane is coupled to the electrophoretic medium, e.g., as illustrated in FIGS. 1 and 2, and typically sealed to create a display module 55, as depicted in FIG. 3B. Such a display module 55 becomes the focus of a color display 100. The color display 100 will typically include a processor 50, which is configured to coordinate the many functions relating to displaying content on the display module 55, and to transform standard images, such as RGB images to a color regime that best duplicates the image on the display module 55. The processor 50 is typically a mobile processor chip, such as made by Freescale or Qualcomm, although other manufacturers are known. The processor 50 is in frequent communication with a non-transitory memory 70, from which it pulls image files and/or look up tables to perform the color image transformations described below. The color display 100 may have more than one non-transitory memory chip. The memory 70 may be flash memory. Once the desired image has been converted for display on the display module 55, the specific image instructions are sent to a controller 60, which facilitates voltage sequences being sent to the respective thin film transistors (described above). Such voltages typically originate from one or more power supplies 80, which may include, e.g., a power management integrated chip (PMIC). The color display 100 may additionally include a communication module 85, which may implement one or more communications protocols, e.g., WIFI protocols or BLUETOOTH, and allows the color display 100 to receive images and instructions, which also may be stored in the memory 70. The color display 100 may additionally include one or more sensors 90, which may include a temperature sensor and/or a photo sensor, and such information can be fed to the processor 50 to allow the processor to select an optimum look-up-table when such look-up-tables are indexed for ambient temperature or incident illumination intensity or spectrum. In some instances, multiple components of the color display 100 can be implemented in a singular integrated circuit. For example, a specialized integrated circuit may fulfill the functions of the processor 50 and the controller 60.

[0072] In the instance of ACeP, each of the eight principal colors (red, green, blue, cyan magenta, yellow, black and white) corresponds to a different arrangement of the four pigments, such that the viewer only sees those colored pigments that are on the viewing side of the white pigment (i.e., the only pigment that scatters light). More specifically, when the cyan, magenta and yellow particles lie below the white particles (Situation [A] in FIG. 4), there are no particles above the white particles and the pixel simply displays a white color. When a single particle is above the white particles, the color of that single particle is displayed, yellow, magenta and cyan in Situations [B], [D], and [F] respectively in FIG. 4. When two particles lie above the white particles, the color displayed is a combination of those of these two particles; in FIG. 4, in Situation [C], magenta and yellow particles display a red color, in Situation [E], cyan and magenta particles display a blue color, and in Situation [G], yellow and cyan particles display a green color. Finally, when all three colored particles lie above the white particles (Situation [H] in FIG. 4), all the incoming light is absorbed by the three subtractive primary colored particles and the pixel displays a black color.

[0073] It is possible that one subtractive primary color could be rendered by a particle that scatters light, so that the display would comprise two types of light-scattering particle, one of which would be white and another colored. In this case, however, the position of the light-scattering colored particle with respect to the other colored particles overlying the white particle would be important. For example, in rendering the color black (when all three colored particles lie over the white particles) the scattering colored particle cannot lie over the non-scattering colored particles (otherwise they will be partially or completely hidden behind the scattering particle and the color rendered will be that of the scattering colored particle, not black). It would not be easy to render the color black if more than one type of colored particle scattered light.

[0074] It has been found that waveforms to sort the four pigments into appropriate configurations to make these colors are best achieved with at least seven voltage levels (high positive, medium positive, low positive, zero, low negative, medium negative, high negative). FIG. 5 shows typical waveforms (in simplified form) used to drive a four-particle color electrophoretic display system described above. Such waveforms have a push-pull structure: i.e., they consist of a dipole comprising two pulses of opposite polarity. The magnitudes and lengths of these pulses determine the color obtained. In general, the higher the magnitude of the high voltages, the better the color gamut achieved by the display. The high voltage is typically between 20V and 30V, more typically around 25V, e.g., 24V. The medium (M) level is typically between 10V and 20V, more typically around 15V, e.g., 15V or 12V. The low (L) level is typically between 3V and 10V, more typically around 7V, e.g., 9V or 5V. Of course, the values for H, M, L will depend somewhat on the composition of the particles, as well as the environment of the electrophoretic medium. In some applications, H, M, L may be set by the cost of the components for producing and controlling these voltage levels.

[0075] As shown in FIG. 5, if the top electrode is held at a constant voltage (i.e., not top plane switched), even simple waveforms for the ACeP system require that the driving electronics provide seven different voltages to the data lines during the update of a selected pixel of the display (+H, +M, +L, 0, L, M, H). While multi-level source drivers capable of delivering seven different voltages are available, most commercially-available source drivers for electrophoretic displays permit only three different voltages to be delivered during a single frame (typically a positive voltage, zero, and a negative voltage).

[0076] Of course, achieving the desired color with the driving pulses of FIG. 5 is contingent on the particles starting the process from a known state, which is unlikely to be the last color displayed on the pixel. Accordingly, a series of reset pulses precede the driving pulses, which increases the amount of time required to update a pixel from a first color to a second color. The reset pulses are described in greater detail in U.S. Pat. No. 10,593,272, incorporated by reference. The lengths of these pulses (refresh and address) and of any rests (i.e., periods of zero voltage between them may be chosen so that the entire waveform (i.e., the integral of voltage with respect to time over the whole waveform) is DC balanced (i.e., the integral of voltage over time is substantially zero). DC balance can be achieved by adjusting the lengths of the pulses and rests in the reset phase so that the net impulse supplied in the reset phase is equal in magnitude and opposite in sign to the net impulse supplied in the address phase, during which phase the display is switched to a particular desired color.

[0077] While modifying the rail voltages provides some flexibility in achieving differing electro-optical performance from a four-particle electrophoretic system, there are many limitations introduced by top-plane switching. For example, it is typically preferred, in order to make a white state with displays of the present invention, that the lower negative voltage V.sub.M is less than half the maximum negative voltage V.sub.H.

[0078] An alternative solution to the complications of top-plane switching can be provided by fabricating the control transistors from less-common materials that have a higher electron mobility, thereby allowing the transistors to switch larger control voltages, e.g., +/30V, directly. Newly-developed active matrix backplanes may include thin film transistors incorporating metal oxide materials, such as tungsten oxide, tin oxide, indium oxide, and zinc oxide. In these applications, a channel formation region is formed for each transistor using such metal oxide materials, allowing faster switching of higher voltages. Such transistors typically include a gate electrode, a gate-insulating film (typically SiO.sub.2), a metal source electrode, a metal drain electrode, and a metal oxide semiconductor film over the gate-insulating film, at least partially overlapping the gate electrode, source electrode, and drain electrode. Such backplanes are available from manufacturers such as Sharp/Foxconn, LG, and BOE.

[0079] One preferred metal oxide material for such applications is indium gallium zinc oxide (IGZO). IGZO-TFT has 20-50 times the electron mobility of amorphous silicon. By using IGZO TFTs in an active matrix backplane, it is possible to provide voltages of greater than 30V via a suitable display driver. Furthermore, a source driver capable of supplying at least five, and preferably seven levels provides a different driving paradigm for a four-particle electrophoretic display system. In an embodiment, there will be two positive voltages, two negative voltages, and zero volts. In another embodiment, there will be three positive voltages, three negative voltages, and zero volts. In an embodiment, there will be four positive voltages, four negative voltages, and zero volts. These levels may be chosen within the range of about 27V to +27V, without the limitations imposed by top plane switching as described above.

[0080] Using advanced backplanes, such as metal oxide backplanes, it is possible to directly address each pixel with a suitable push-pull waveform, i.e., as described in FIG. 5. This greatly reduces the time required to update each pixel, in some instances transforming a six-second update to less than one second. While, in some cases, it may be necessary to use reset pulses to establish a starting point for addressing, the reset can be done quicker at higher voltages. Additionally, in four-color electrophoretic displays having reduced color sets, it is possible to directly drive from a first color to a second color with a specific waveform that is only slightly longer than the push-pull waveforms shown in FIG. 5.

[0081] The bulk of electronic color images in the world are formatted in a red-green-blue (RGB) color space, corresponding to the red, green, and blue subpixels that are commonly used in liquid crystal displays (LCD), light emitting diode (LED) displays, or cathode ray tube (CRT) displays. A common format is an 8-bit RGB that assigns red, green, and blue subpixel values to each pixel in the image as a set of three numbers, each number spanning from 0-255. Accordingly, a standard RGB image file consists of a set of numbers corresponding to pixels in the image. When those color levels are provided to the assigned pixels, the image appears on the display. The next image file, corresponding to a new photograph or a next frame of a video, has a new set of numbers at each pixel.

[0082] The RGB values do not map directly into the color space used with electrophoretic displays. The shape of the RGB gamut is different from the shape of an electrophoretic device gamut, e.g., an ACeP gamut. Thus, it is necessary to transform the RGB image files to an electrophoretic file format. Methods of mapping RGB image files to electrophoretic image files like ACeP files are known in the art including, e.g., as described in U.S. Pat. No. 11,984,088, which is incorporated by reference herein in its entirety. U.S. Pat. No. 11,984,088 describes transforming RGB image data to image data for the ACeP gamut. The process includes mapping the RGB source space into the ACeP device space using a tetrahedral decomposition of the RGB source space.

[0083] After the mapping process, the image set can be dithered in the electrophoretic device color space in order to produce a greater number of perceived colors using a limited set of color primaries (typically red, green, blue, cyan, yellow, magenta, white, and black). Dithering creates the illusion of color depth in images with the limited color palette. Colors not available in the palette are approximated by a diffusion of colored pixels from within the available palette. When a dithered image is viewed at a sufficient distance, the individual colored pixels are merged by the human visual system into perceived uniform colors. Because of the trade-off between color depth and spatial resolution, dithered images when viewed closely have a characteristic graininess as compared to images in which the color palette available at each pixel location has the same depth as that required to render images on the display as a whole. Standard dithering algorithms such as threshold mask dithering and error diffusion algorithms (in which the error introduced by printing one pixel in a particular color that differs from the color theoretically required at that pixel is distributed among neighboring pixels so that overall the correct color sensation is produced) can be employed with limited palette displays. There is an enormous literature on error diffusion; for a review see Pappas, Thrasyvoulos N. Model-based halftoning of color images, IEEE Transactions on Image Processing 6.7 (1997): 1014-1024. Use of dithering in color electrophoretic displays is known in the art; see, e.g., U.S. Pat. No. 11,869,451, which is incorporated by reference herein in its entirety.

Grayscale Replacement in Color Electrophoretic Displays

[0084] The printing industry has long used the GCR methods to replace some or all of the cyan, magenta, and yellow (CMY) inks in built colors with black. The amount of amount of cyan, magenta, and yellow that equals black at different tone levels is referred to as the gray component. The primary motivation is that black ink is less expensive than the respective combined amounts of cyan, magenta, and yellow inks. This concept is illustrated in the simplified drawings of FIG. 6A-6D. FIG. 6A shows some amount of cyan, magenta, and yellow ink needed to make a blue color. Assume that equal parts of cyan, magenta, and yellow can be replaced by a respective amount of black as depicted in FIG. 6B. The amount of cyan, magenta, and yellow equal to the minimum amount of the three colors is replaced with black as shown in FIG. 6C. FIG. 6D shows how the same blue color is now made from a lesser amount of cyan and magenta, with black.

[0085] In other cases, pure neutral colors built with cyan, magenta, and yellow, may be replaced entirely by black ink, as shown in FIGS. 7A and 7B. FIG. 7A shows how an equal amount of cyan, magenta, and yellow ink can make a gray. FIG. 7B depicts the same gray printed with only black ink.

[0086] Applying GCR concepts to electrophoretic devices requires a somewhat different process than the GCR method described above. It may be advantageous to include additional gray tones in the color dithering process to decrease the size of the dithering tetrahedra and make the tetrahedra vertices closer together in the color space. However, that process would not replace cyan, magenta, and yellow equivalents with gray, only dither gray alongside cyan, magenta, and yellow, as well as red, green, and blue. In applying GCR processes to electrophoretic devices, methods in accordance with various embodiments replace neutral areas dithered using some combination of the color palette primaries (typically, but not limited to, cyan, magenta, yellow red, green, blue, white, and black) with dither patterns containing only black, white, and a plurality of grays (typically, but not limited to, two grays).

[0087] In many current electrophoretic display device renderings, any RGB value/region not exactly [0 0 0] or [1 1 1] is dithered using black or white plus colored primaries. This is satisfactory for large areas where the human visual system is able to average over that area to create the sensation of a patch of uniform color, as illustrated in FIGS. 8A and 8B. FIG. 8A shows a uniform gray source image at 1200% zoom, and FIG. 8B shows a dithered uniform gray image also at 1200% zoom. FIG. 8B demonstrates how dithering can be used in successfully rendering uniform areas. However, displayed text and thin lines have very noticeable defects since there is not enough area to visually average groups of colored pixels as illustrated in FIGS. 9A and 9B. FIG. 9A shows a black and white source image with single-pixel lines of values (from top to bottom) of 0, 50, 100, 150, and 200 (at 1200% zoom). FIG. 9B shows a dithered rendering of the black and white source image using the mask-method also at 1200% zoom. FIG. 9B demonstrates the difficulty in rendering fine details using dithering in electrophoretic displays. It shows that the small details cannot be resolved amid the dithering noise. A need thus exists for reducing defects in high-frequency areas in electrophoretic displays resulting from color dithering.

[0088] One aspect of the invention relates to a process for gray component replacement in color electrophoretic display renderings. The process merges a dithered color image with a dithered black and white version of the same image by replacing the neutral portions of the dithered color image with corresponding portions of the dithered black and white image. The end result is an image in which fine neutral details, such as black and gray lines and text, are rendered using only neutral primaries with lower contrast between groups of dithered primaries. This makes dithering patterns less visible and makes black and white regions appear smoother, increasing the resolution of fine details. In one or more embodiments, black is dithered with gray 1, and white is dithered with gray 2. As with standard color dithering, a lookup table (LUT) can be used to transform RGB to the black and white dithering matrix.

[0089] FIG. 10 is a flowchart generally illustrating an exemplary gray component replacement process for color electrophoretic displays in accordance with one aspect of the invention. The steps of the process can be performed by a device processor such as, e.g., the processor 50 previously discussed above in connection with FIG. 3B.

[0090] At step 102 of the process, an input color image to be displayed on an electrophoretic device is received by the device processor. The image is in an RGB format.

[0091] The RGB input color image is mapped to an electrophoretic destination space defined by an electrophoretic color gamut to generate a color destination space image at step 104.

[0092] The color destination space image is dithered at step 106 to generate a dithered color image.

[0093] At step 108, neutral portions of the dithered color image are identified.

[0094] The RGB input color image is also converted to a black and white image at step 110.

[0095] The black and white image is mapped to a black and white electrophoretic destination space defined by electrophoretic black, white, and a plurality (e.g., two) of gray primaries to generate a black and white destination space image at step 112.

[0096] The black and white destination space image is dithered using the four neutral primaries at step 114 to generate a dithered black and white image.

[0097] The dithered color image and the dithered black and white image are merged at step 116 by replacing the neutral portions of the dithered color image identified in step 108 with corresponding portions of the dithered black and white image to generate a merged color image.

[0098] The merged color image is then displayed on the electrophoretic display at step 118. For example, the device processor 50 instructs the device controller 60 to cause the electrophoretic display module 55 to display the merged color image.

[0099] The source RGB image can be converted to black and white (step 110) using one of a number of known methods for converting RGB images to black and white images. One such method utilizes the weighted addition of red, green, and blue channels. The green channel is typically weighted higher than the red and blue channels, as indicated in Eq. 1 below (which provides an approximation of luminance), because the green channel most closely represents the visual luminous efficiency function. However, the specific weights can be varied.

[00001]$\begin{array}{cc}{I}_{BW}=0.2126*R_{x,y}+0.7152*G_{x,y}+0.0722*B_{x,y}& \left(1\right)\end{array}$

[0100] FIGS. 11A and 11B show an example source test image its original RGB format and the calculated black and white format determined using Eq. 1.

[0101] The black and white image is mapped from the source to destination space (step 112) only along the luminance dimension since there is no color information. The source RGB values, in the range [0, 1], are scaled down to the maximum digital values of the destination space, about [0.2, 0.6], depending on the specific display unit. A sigmoidal contrast enhancement can be added to compensate for the reduced dynamic range. This tonal compression with sigmoidal enhancement is illustrated in the graph shown in FIG. 12.

[0102] The black and white pixel value range is divided by the pixel values of the plurality of N gray primaries, where Gray 1 and Gray N are the black and white primaries, respectively, from the set of color primaries. There are N1 sequential pairs of gray primaries ([1, 2], [2, 3], . . . [N1, N]). Image pixel values are sorted into the N1 groups based on the bounding pixel values in each group and are then dithered using a combination of the two gray pixel values in that group. For example, if Gray 2 has a pixel value of 0.2 and Gray 3 has a pixel value of 0.4, then all image pixel values in the range [0.2, 0.4] will be dithered using Gray 2 and Gray 3. The amount of each Gray primary used in the dithering pattern can be determined using a barycentric mapping with the following equation.

[00002]$\begin{array}{cc}\left[\begin{array}{ccc}{}_{i,1}& .Math.& {}_{i,M}\\ {}_{j,1}& .Math.& {}_{j,M}\end{array}\right]={\left[\begin{array}{cc}1& 1\\ p_i& p_j\end{array}\right]}^{-1}\left[\begin{array}{ccc}1& .Math.& 1\\ I_{\left[i,j\right],1}& .Math.& I_{\left[i,j\right],M}\end{array}\right]& \left(2\right)\end{array}$

where 1i<jN for N Gray primaries, .sub.i and .sub.j are the barycentric coordinates for Gray primaries i and j, and I.sub.[i,j],M are the image pixel values for the for M pixels in group [i, j].

[0103] After calculating the barycentric coordinates, a black and white dither pattern for the image is computed using a threshold mask or a dither mask, which may include blue-noise masks, Bayer masks, or similar masks used for dithering. A blue-noise mask (BNM) is disclosed, e.g., in U.S. Pat. No. 11,527,216, which is incorporated by reference herein in its entirety. FIGS. 13A and 13B show dithered renderings of the color destination-space and the black and white destination-space images, respectively.

[0104] The dithered color and black and white destination-space images are merged at step 116. The merge process begins with the dithered color image, where each pixel is represented by a single color primary. Those pixels in areas of the color image where there is little color information, a.k.a. neutral areas, are replaced by respective pixels from the dithered black and white image. Identifying the neutral portions of the dithered color image (step 108) includes determining the degree of neutrality (DoN) for each pixel in the dithered color image. Then a neutrality threshold is set. Pixels with a degree of neutrality greater than the threshold are selected for replacement with black and white pixels. The DoN may be calculated using one of a plurality of known methods, including but not limited to (1) RGB ratio method, (2) neighborhood RGB ratio method, and (3) uniform color space chroma method discussed in further detail below.

RGB Ratio Method

[0105] The RGB ratio is calculated using the source RGB image to identify black and white and color areas. Each color pixel is defined by a vector, [R, G, B], of red, green, and blue pixel values. The difference between the minimum and maximum values in [R, G, B] approaches zero with maximum neutrality, and approaches 1 with minimum neutrality. The RGB Ratio metric, .sub.i,j, in Eq. 3, is computed for each color pixel value, (i, j).

[00003]$\begin{array}{cc}{}_{i,j}={\left(1-\frac{\max \left(\left[R_{i,j},G_{i,j},B_{i,j}\right]\right)-\min \left(\left[R_{i,j},G_{i,j},B_{i,j}\right]\right)}{255}\right)}^{}& \left(3\right)\end{array}$

[0106] The exponent, , with values ideally in the range, but not constrained to, [1.5, 1.8] gives greater discrimination between neutral and non-neutral areas. Image maps of with =1 and =1.5 are shown in FIGS. 14A and 14B, respectively, where regions approaching yellow (having a higher RGB ratio) are more neutral, and regions approaching blue are less neutral. The map with =1.5 (FIG. 14B) has more blue regions, thus determining a smaller number of pixels as neutral.

[0107] A BNM (or other threshold or dither mask) is used to dither regions in which there is a transition between color and black and white pixels. This avoids edges, which may show as artifacts if there is inconsistent color rendering in the displayed image. First, a threshold, .sub.T, is defined. The default value for .sub.T=0.90. The input to the BNM is a function that defines the likelihood a pixel will be classified as black and white. This function has a sigmoidal shape, with a domain between 0 and 1, where 1 is definitely black and white, and 0 is definitely color, and a median value equal to .sub.T. The likelihood function is calculated using a MATLAB function, LikelihoodColorCurve, shown as Selection 1 (FIG. 15). (MATLAB 2022, version R2022a available from The Math Works Inc.). The inputs are .sub.T (RhoT) and .sub.M, the RGB Ratio values for the image (ImageRGBRatio). The function outputs the likelihoods pixels in the image are color (ColorOut) and the likelihoods pixels in the image are black and white (black and whiteOut), the latter of which is the input to the BNM. The sigmoidal color likelihood functions and black and white likelihood functions are shown in FIG. 16.

[0108] A binary image, in which values of 1 are color pixels and value of 0 are black and white pixels, called the Threshold Mask is created by dithering the black and whiteOut values using the BNM, similar to the method disclosed in U.S. Pat. No. 11,527,216, which is incorporated by reference herein in its entirety. The BNM is a 6464 matrix with 4096 values between 0 and 1 strategically placed such that sequential values are not in close proximity. The BNM is tessellated across the area of the image such that a new BNM has the same dimensions as the image. For a given pixel in the, BNM, BNM.sub.(i,j), and in black and whiteOut, BNM.sub.(i,j), if black and whiteOut.sub.(i,j)>BNM.sub.(i,j), then the merged image pixel I.sub.(i,j) will use the respective pixel from the dithered black and white image (FIG. 13B). Otherwise, I.sub.(i,j) will use the respective pixel from the dithered color image (FIG. 13A).

[0109] The Threshold Mask images, with =1, and =1.5 are shown in FIG. 17. White areas represent color pixels and black areas represent black and white pixels.

[0110] FIG. 18 depicts the workflow for merging the dithered black and white and dithered color images.

Neighborhood RGB Ratio Method

[0111] The neighborhood RGB ratio method is an alternative to the RGB ratio method discussed above for merging color and black and white dithered images. Rather than using a to reduce the use of black and white pixels in color areas, a smoothing kernel, K, can be used to smooth .sub.i,j by convolving the image with K. The value may be, but is not limited to, 1 in this application.

[00004]$\begin{array}{cc}{}_{N,\left(i,j\right)}=\mathrm{conv}\left(K,{}_{i,j}\right)& \left(4\right)\end{array}$

[0112] One potential drawback to this method is that it may create artifacts in areas where there are gradual changes from color to black and white, such as transitioning from a color gradient to a neutral gradient. One option, rather than using a convolution kernel, is to use an edge-preserving filter, such as a bilateral filter. However, edge-preserving filters were designed for noise-reduction, not blurring, and so may have a minimal effect and not provide needed reduction in black and white dithering in colored areas.

Uniform Color Space Chroma Method

[0113] The RGB ratio method and the neighborhood RGB ratio method make neutrality decisions based on device-dependent RGB values. Alternatively, neutrality decisions can be made by converting the image RGB values to one of a plurality of device-independent uniform color spaces including, but not limited to: CIE L*a*b*, IPT, CIECAM02 UCS, CAM16-UCS. (CIECAM02 UCS is described in Moroney, Nathan; Fairchild, Mark; Hunt, Robert; and Li, Changjun, The CIECAM02 color appearance model (2002). CAM16-UCS is described in Li C, Li Z, Wang Z, et al. Comprehensive color solutions: CAM16, CAT16, and CAM16-UCS. Color Res Appl. 017; 42:703-718. Https://doi.org/10.1002/col.2213.)

[0114] The chroma values in each of these spaces indicate the level of neutrality, where 0 is perfectly neutral and increasing values are more chromatic. An example flow from RGB to CIELAB C* is RGB.fwdarw.CIE XYZ.fwdarw.CIE L*a*b*.fwdarw.C*. The C* values can then be used to generate a Threshold Mask and dither the images. The advantages of using RGB over a UCS is that the values are linear and bounded between 0 and 1. However, the UCS method provides a more perceptually uniform mapping, so there is less likelihood that color pixels are rendered using black and white, if the threshold is set properly.

EXAMPLES

[0115] Various examples discussed below compare image renderings using only color palette primaries and using merged color and black and white primaries generated using the processes disclosed above. In general, it has been found that the original dithered color image and merged color and black and white image look similar if the color of the middle gray primaries falls inline between the white and black primaries. Tinting of the gray tones, however, may lead to less pleasing color, depending on how far the middle grays deviate from the trace between black and white.

[0116] FIGS. 19A and 19B show the test image rendered using only color palette primaries and using merged color and black and white primaries, respectively.

[0117] FIGS. 20A and 20B are close-up views of text in the images of FIGS. 19A and 19B, respectively.

[0118] FIGS. 21A and 21B are close-up views of an area of the images of FIGS. 19A and 19B, respectively, that includes both black and white and color elements.

[0119] FIGS. 22A and 22B are close-up views of a shadow gradient area of the images of FIGS. 19A and 19B, respectively.

[0120] Having thus described several aspects and embodiments of the technology of this application, it is to be appreciated that various alterations, modifications, and improvements will readily occur to those of ordinary skill in the art. Such alterations, modifications, and improvements are intended to be within the spirit and scope of the technology described in the application. For example, those of ordinary skill in the art will readily envision a variety of other means and/or structures for performing the function and/or obtaining the results and/or one or more of the advantages described herein, and each of such variations and/or modifications is deemed to be within the scope of the embodiments described herein. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments described herein. It is, therefore, to be understood that the foregoing embodiments are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, inventive embodiments may be practiced otherwise than as specifically described. In addition, any combination of two or more features, systems, articles, materials, kits, and/or methods described herein, if such features, systems, articles, materials, kits, and/or methods are not mutually inconsistent, is included within the scope of the present disclosure.