METHOD AND APPARATUS FOR ASSOCIATING THE COLOR OF AN OBJECT WITH AN EVENT

Abstract

Method and apparatus for associating the color of an object with an event preferably provides linked information for the object in any one of a plurality of succeeding video frames based only on the color value of the object. Preferably, at least one computer is coupled to a GUI and to a pointing structure, and is configured to: (i) identify coordinates of the object in a first video frame where the pointing structure has pointed; (ii) determine the color values of the identified object; (iii) store the color values of the identified object; (iv) detect an action of the object being selected in any one of the plurality of succeeding video frames; (v) determine the color values of the selected object; (vi) determine whether the selected object color values correspond to the stored color values; and (vii) automatically provide the linked information to the GUI only when the color values correspond.

Claims

1. A processor-implemented method for automatically identifying an object in a plurality of Internet video frames and associating the object with a link action, comprising: providing at least one processor which executes the steps of: determining a location in one of said Internet video frames where an action by a pointing device on the object has occurred on a display device; determining a color value of the determined location on the object, including the step of determining a color boundary of the object by providing color pixels on the boundary with an edge which is not visible to a user, the edge comprising a portion of an invisible wire frame in which color values of the determined object pixels define the invisible wire frame edges, the color values of the invisible wire frame varying from frame to frame, the color value comprising a combination of color values for first, second, and third different colors; determining a color value range for each of the color values for the first, second, and third different colors of the determined object, each color value range including a continuous series of integer color values, the step of determining each color value range including the steps of (i) determining a histogram for each of the color values for the first, second, and third different colors of the determined object, (ii) determining the peak color value of the histogram for each of the first, second, and third different colors, (iii) determining a variance of the peak color value of the histogram for each of the first, second, and third different colors, and (iv) for each of the first, second, and third different colors, assigning color range values corresponding to at least one standard deviation of the determined variance, the peak color value of each histogram consisting essentially of the highest number of data values for each color value for each of the first, second, and third different colors; and automatically associating a link action with the combined color value of said determined location on the object in said one video frame, and automatically associating said link action with said combined color value on the object in succeeding video frames, without pre-processing the invisible wire frame of said succeeding video frames.

2. The method according to claim 1, wherein the at least one processor executes the step of determining a color value pattern for the selected object, including the step of locating plural alternating peaks and valleys in the color pattern.

3. The method according to claim 2, wherein the at least one processor executes the steps of automatically associating a link decision with said color value pattern of the selected object in said one video frame, and automatically associating said color value pattern peaks and valleys in succeeding video frames.

4. Computer apparatus for automatically identifying an object in a plurality of Internet video frames, and associating the object with a link action, comprising: a graphical user interface (GUI); user pointing structure; and at least one computer coupled to the graphical user interface and to said user pointing structure, said at least one computer executing the steps of: determine a location in one of said Internet video frames displayed on said GUI where said user pointing structure has identified the object; determine a color value of the object in said one video frame displayed on said GUI, including determining a color boundary of the object by providing color pixels on the boundary with an edge which is not visible to a user, the edge comprising a portion of an invisible wire frame in which color values of the determined object pixels define the invisible wire frame edges, the color values of the invisible wire frame varying from frame to frame, the color value comprising a combination of color values for first, second, and third different colors; determine a color value range for each of the color values for the first, second, and third different colors of the determined object, each color value range including a continuous series of integer color values, the step of determining each color value range including the steps of (i) determining a histogram for each of the color values for the first, second, and third different colors of the determined object, (ii) determining the peak color value of the histogram for each of the first, second, and third different colors, (iii) determining a variance of the peak color value of the histogram for each of the first, second, and third different colors, and (iv) for each of the first, second, and third different colors, assigning color range values corresponding to at least one standard deviation of the determined variance, the peak color value of each histogram consisting essentially of the highest number of data values for each color value for each of the first, second, and third different colors; automatically associate a link action with the combined color value of said determined location on the object in said one video frame; and automatically associate said link action with the combined color value on the object in succeeding video frames, without pre-processing the invisible wire frame of said succeeding video frames.

5. The apparatus according to claim 4, wherein the at least one computer executes the step of determining a color value pattern for the object, including the step of locating plural alternating peaks and valleys in the color pattern.

6. The apparatus according to claim 5, wherein the at least one computer processor executes the steps of automatically associating a link decision with decision with said color value pattern of the object in said one video frame, and automatically associating said color value pattern peaks and valleys in succeeding video frames.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0029] These and other features, aspects and advantages of the present invention will become better understood with regard to the following description, appended claims and accompanying drawings where:

[0030] FIG. 1 is a flowchart showing how to associate the color of an object with an event according to an embodiment of the present invention.

[0031] FIG. 2 is a flowchart showing how to define the color characteristics of an object according to an embodiment of the present invention.

[0032] FIG. 3 is a flowchart showing how to define the color characteristics of an object according to another embodiment of the present invention.

[0033] FIG. 4 is a flowchart showing how to define the color characteristics of an object according to another embodiment of the present invention.

[0034] FIG. 5 is a flowchart showing how to associate the color of an object with an event using a table according to an embodiment of the present invention.

[0035] FIG. 6 is a computer execution environment where one or more embodiments of the present invention may be implemented.

[0036] FIG. 7 shows an embodiment of the present invention where the frame buffer is utilized.

[0037] FIG. 8 shows an embodiment of the present invention where color ranges are utilized.

DETAILED DESCRIPTION OF THE INVENTION

[0038] The invention is a method and apparatus for associating the color of an object with an event. In the following description, numerous specific details are set forth to provide a more thorough description of embodiments of the invention. It is apparent, however, to one skilled in the art, that the invention may be practiced without these specific details. In other instances, well known features have not been described in detail so as not to obscure the invention.

[0039] One or more embodiments of the present invention are directed to enhancing a user's ability to interact with a computer, set-top box, interactive television set, or other processing device. According to embodiments of the present invention, the user provides input on some area of a display device, for instance by depressing a mouse button at a specified location. The invention then determines where the user has depressed the mouse button and what color value appears on the display device at the time the user has depressed the mouse button. Then, the invention initiates an event that corresponds to that color value.

[0040] Event Processing. In one or more embodiments, the invention takes a streaming or static media and associates an active, dynamic, or static object with an event by its given color. The event may be, for instance, the retrieval of a document at a specified URL in a web browser or the retrieval of other information in non-web browser implementations. Other implementations include, for instance, the viewing of media, via a television, DVD or other media player, set-top devices, web-TV, interactive TV, and video-games.

[0041] Take, for example, a web page implementing streaming video. In the example, a game show host is wearing a green shirt. If a user provides input to the screen by depressing a mouse with the cursor in a location where the green shirt is located, the invention links to a URL associated with the green shirt. Thus, the moving color may substitute for a conventional, static, text link. In another embodiment, the game show host with the green shirt may be in a television show. In this embodiment, providing input on the moving green shirt may initiate a different event, such as providing information about the shirt or the game show host.

[0042] One embodiment of the present invention is shown in FIG. 1. At step 100, an action occurs. At step 110, the location of the action (a region of interest) is determined. Locating a region of interest is well known in the art.

[0043] One method to locate a region of interest is to obtain an ordered pair corresponding to the location of the cursor at the moment of the action. For instance, a computer screen may be represented as a two-dimensional graph wherein the axis of the graph resides in the lower left hand corner of the computer screen. Thus all ordered pairs will correspond to positive values of x,y pairs on the graph.

[0044] After defining the region of interest, the value associated with a color in the region of interest is determined at step 120. Thereafter, at step 130, an event associated with that color is initiated.

[0045] Color Characteristics. In one embodiment, the invention defines the color by its mathematical properties. For instance, one embodiment uses the RGB standard which separates the red, green, and blue properties of the color as numerical values. Another embodiment also uses the HSV characteristics of the color, which define hue, saturation, and value as components of color. HSV characteristics further define the properties of a color, for instance V represents an analog of the intensity of a color or how bright the color appears. Another embodiment of the present invention uses both RGB and HSV characteristics.

[0046] One embodiment defines color characteristics as shown in FIG. 2. At step 200, it is determined whether an action has occurred. If no action occurred, step 200 repeats until an action occurs. If the action has occurred, it is determined where the action has occurred at step 210.

[0047] The location is shown herein with respect to a single pixel, pixel N. At step 220, a red component of a color for pixel A is determined. Then, at step 230, a green component of the color for pixel A is determined. Next, at step 240, a blue component of the color for pixel A is determined. Thereafter, at step 250, the color characteristics are combined to form a color value for pixel A

[0048] After step 250, it is determined whether any event is associated with the color value at step 255. If no event is associated with that color value, no event is initiated and the process repeats at step 200. If an event is associated with the color value, then at step 260 an event associated with the color value is initiated.

[0049] Another embodiment defines color characteristics as shown in FIG. 3. At step 300, it is determined whether an action has occurred. If no action occurred, step 300 repeats until an action occurs. If the action has occurred, it is determined where the action has occurred at step 310. The location is shown herein with respect to a single pixel, pixel B. At step 320, a hue component of a color for pixel B is determined. Then, at step 330, a saturation component of the color for pixel B is determined. Next, at step 340, a value component of the color for pixel B is determined.

[0050] Thereafter, at step 350, the color characteristics are combined to form a color value for pixel B. After step 350, it is determined whether any event is associated with the color value at step 355. If no event is associated with that color value, no event is initiated and the process repeats at step 300. If an event is associated with the color value, then at step 360 an event associated with the color value is initiated.

[0051] Another embodiment defines color characteristics as shown in FIG. 4. At step 400, it is determined whether an action has occurred. If no action occurred, step 400 repeats until an action occurs. If the action has occurred, it is determined where the action has occurred at step 410. The location is shown herein with respect to a single pixel, pixel C. At step 420, a hue component of a color for pixel C is determined. Then, at step 430, a saturation component of the color for pixel C is determined. Next, at step 440, a value component of the color for pixel C is determined. At step 450, a red component of a color for pixel C is determined. Then, at step 460, a green component of the color for pixel C is determined. Next, at step 470, a blue component of the color for pixel C is determined.

[0052] Thereafter, at step 480, the color characteristics are combined to form a color value for pixel B. After step 480, it is determined whether any event is associated with the color value at step 485. If no event is associated with that color value, no event is initiated and the process repeats at step 400. If an event is associated with the color value, then at step 490 an event associated with the color value is initiated.

[0053] Frame Buffer. In a computer or other comparable device, images are often used as a form of output. A common way that such devices handle images or streams of images (e.g., video) is by using a frame buffer. For offline pre-processing or for real-time processing using high speed computers, the RGB frame buffers reside in memory. Alternatively, a special high speed video board can be used to process an RGB frame buffer for real-time processing.

[0054] A video signal, for instance, comprises a series of still images shown sequentially where the series of images appears so quickly that a user perceives the series of pictures to be a continuous stream of motion. As the computer processes each image, data relating to the image is stored in the frame buffer. Such data includes, for instance, the color values associated with each pixel at each location on the screen.

[0055] As a new image is displayed, new data about the image is stored in the frame buffer. Depending on the size of the frame buffer, many sequential images may be stored there before it fills up and old image data must be overwritten.

[0056] One embodiment of the present invention takes advantage of the data already existing in the frame buffer, for instance to perform step 120 of FIG. 1. FIG. 7 shows an embodiment of the present invention where the frame buffer is utilized.

[0057] At step 700, an action occurs. At step 710, the location of the action (a region of interest) is determined. Next, at step 720, the invention access the frame buffer to obtain color values at the location of the action when it occurred. Then, it is determined whether any event is associated with the color value at step 730. If no event is associated with that color value, no event is initiated and the process repeats at step 700. If an event is associated with the color value, then at step 740 an event associated with the color value is initiated.

[0058] Events. As described above, one event is a link to another URL (i.e., to facilitate a transaction over the Internet). Thus, one event may comprise the retrieval of a web page from a server computer connected to the Internet.

[0059] In the context of streaming media, an event may be the retrieval of a text page. For instance, assume the user is watching a DVD program showing several of the great pyramids of Egypt. If the user causes an event to occur in a region of interest defined by a specific pyramid, then the event may comprise the retrieval of a text page telling the user exactly which pyramid this is, when it was built, the history associated with this particular pyramid, and any other pertinent information. Thus, the event would cause the DVD to move to a different data track (i.e., to pull up different information).

[0060] In the same great pyramid example, another event may comprise the retrieval of audio information, for instance a narrator providing information similar to that given in the text track in the above example. Furthermore, the event may comprise the retrieval of a new media stream, such as the close up of the pyramid, alternate angles showing different views of the same pyramid, or views from inside the pyramid, for example.

[0061] In one embodiment an event comprises the retrieval of a data track. A data track comprises information associated with the color acted upon. Assume for instance, that this embodiment of the present invention takes place using an interactive television set. While the interactive television set is operating, a game show host appears wearing a green shirt. Then assume, for instance, that a user caused an action to occur at the location of the green shirt. In this instance, the data track may be information about the shirt, the game show host, or any other relevant information.

[0062] Whether an event is associated with a color value may be determined in a plurality of ways. One such way is to use a table. An example of one embodiment of a table is shown in Table 1.

TABLE-US-00001 TABLE 1 Color Value Event 100 retrieve document at URL www.info.comltest 300 obtain data track at a specified location 200 initiate media stream at specified location 288 initiate audio track at specified location

[0063] The table may be arranged in a plurality of ways and may be accessed in methods well known to those skilled in the art, including hash tables, hash functions, pointers, indices, or any other suitable form of table lookup. In addition, different color values may initiate the same event. Alternatively, every color value on an output device may be associated with the same event.

[0064] FIG. 5 shows an embodiment of the present invention where a table is implemented. At step 500, an action occurs. At step 510, the location of the action (a region of interest) is determined. Then at step 520, the value associated with a color in the region of interest is determined. Next, a table is searched to locate the color value to obtain an event that is paired with the color value at step 530. Then, at step 535, it is determined whether the color value is in the table. If it is not, then that color value is not linked to an event and no event is initiated. In that case the process repeats at step 500. Otherwise, at step 540, the event is initiated.

[0065] Color Ranges. In another embodiment of the present invention regions of interest (i.e., those regions where a user has performed an action) are associated with events based on color ranges. A color range, for instance, includes those shades of blue that correspond to the body of a bluebird in a video stream. Realistic image and video streams will not have a uniform shade of blue at all areas of the bluebird's body since light and other factors will cause natural variations in the perceived color reflected from an object. In this embodiment, this reality is accounted for, which gives this embodiment specific application in dynamic and high-quality video streams.

[0066] Color ranges are defined by color characteristics. For instance, a color range may translate into values on an RGB or HSV scale, or could be combinations of both. Typically, in a computer system such values range from 0-255. Thus, the bluebird's body may be defined as a region of interest where the color range includes a B value on the RGB scale of 50-75, for instance.

[0067] Thus, whether the action occurred at an upper left or lower right portion of the bluebird's body, the same event is initiated, despite the potential that at these corners of the regions of interest, the color values differed. All that is required in this embodiment is that both differing color values fall within the defined color range.

[0068] In many images, the colors may be very similar, yet they may be aspects of different components of the image and associated with different events. In one embodiment, the ranges are defined with respect to a plurality of color characteristics including RGB and HSV.

[0069] For instance, a range may be defined where the R is selected to have a range of 50-80, the G is selected to have a range of 150-160, and the B is selected to have a range of 200-220. An example color is selected by the user where the example color has RGB characteristics of 60, 155, and 205 respectively. Another color on the screen that is associated with a different event may appear to be substantially identical to the example color but has RGB characteristics of 190, 100, 33. Since the ranges have been defined by a plurality of characteristics, however, such minute differences in color shading are accounted for by this embodiment of the present invention.

[0070] In one embodiment, the characteristics of the color are defined by R, G, B, H, S, V, and R-G, and G-B, where R-G and G-B have red and green and green and blue components respectively. In this embodiment, any three of these variables is sufficient to uniquely define all colors displayed on an output device or stored in a frame buffer. To determine whether the variables are within the color range for the purpose of determining whether an event will be associated with the area of interest, the following algorithm is used (shown in FIG. 8).

[0071] At step 800, it is determined whether an action has occurred. If no action occurred, step 800 repeats until an action occurs. If the action has occurred, it is determined where the action has occurred at step 810. Then at step 820, three variables are chosen from a list of possible color components, for instance, R, G, B, H, S, V, R-G, or G-B. For the chosen variables, histograms are produced at step 830

[0072] A histogram is a graphical method for describing data. It is a presentation of quantitative data (i.e., color components) that have been summarized in a frequency distribution. The values of the quantitative variable are shown on the horizontal axis. A rectangle is drawn above each class such that the base of the rectangle is equal to the width of the class interval and its height is proportional to the number of data values in the class.

[0073] Using the histogram, the color ranges are determined at step 840, for instance by determining the variable peak on the histogram+ or three times its variance. After step 840, it is determined whether any event is associated with the color range at step 850. If no event is associated with that color range, no event is initiated and the process repeats at step 800. If an event, is associated with that color range, then at step 860 an event associated with the color range is initiated.

[0074] Color Patterns. It is often beneficial to define a regions of interest that is not uniform in color. A region of interest, for instance, may be a plaid tablecloth. In this case, it is not adequate to use a single color value, nor a color range to define the region of interest. To account for this, one embodiment of the present invention uses color patterns to define the region of interest.

[0075] A color pattern comprises a representation of the colors in the region of interest, for instance a graphical representation, defined by a plot on an x,y axis. The varying colors in the region of interest create peaks and valleys in the color pattern, graph representation. By locating patterns of peaks and valleys in such a color pattern, multi-color patterns, such as the plaid tablecloth, are used to associate actions with events.

[0076] Edge Detection. One embodiment of the present invention uses an edge detection scheme to define the boundaries for the colors. The color component of an object is described by its pixels. An algorithm gives those pixels an edge which is not visible to the user, nor drawn on the screen, nor pre-processed prior to the user viewing the images formed by the pixels. For instance, an invisible wire frame is created to describe the shape of the given colored object to associate with the event. But the value of the pixels defines the wire frame rather than actually drawing a wire frame on the screen.

[0077] One embodiment of the present invention uses Quicktime, a software package available from Apple Computers. In this embodiment, Quicktime is used to support the use of sprites. A sprite is a clickable element within a video stream Using Quicktime, a sprite is modified to be defined by its color component. Thus, one or more embodiments of the present invention create the associations between objects and events dynamically, in that the data which creates the association is a property of the image itself (i.e., its color values). The present invention requires no alteration or pre-processing of the image to create the associations.

[0078] Embodiment of Computer Execution Environment (Hardware). An embodiment of the invention may be implemented as computer software in the form of computer readable program code executed in a general purpose computing environment such as environment 600 illustrated in FIG. 6, or in the form of byte code class files executable within a Java Runtime Environment running in such an environment, or in the form of byte codes running on a processor (or devices enabled to process byte codes) existing in a distributed environment (e.g., one or more processors on a network). A keyboard 610 and mouse 611 are coupled to a system bus 618. The keyboard and mouse are for introducing user input to the computer system and communicating that user input to central processing unit (CPU) 613. Other suitable input devices may be used in addition to, or in place of, the mouse 611 and keyboard 610. I/O (input/output) unit 619 coupled to bi-directional system bus 618 represents such I/O elements as a printer, A/V (audio/video) I/O, etc.

[0079] Computer 601 may include a communication interface 620 coupled to bus 618. Communication interface 620 provides a two-way data communication coupling via a network link 621 to a local network 622. For example, if communication interface 620 is an integrated services digital network (ISDN) card or a modem, communication interface 620 provides a data communication connection to the corresponding type of telephone line, which comprises part of network link 621. If communication interface 620 is a local area network (LAN) card, communication interface 620 provides a data communication connection via network link 621 to a compatible LAN. Wireless links are also possible. In any such implementation, communication interface 620 sends and receives electrical, electromagnetic or optical signals which carry digital data streams representing various types of information.

[0080] Network link 621 typically provides data communication through one or more networks to other data devices. For example, network link 621 may provide a connection through local network to local server computer 623 or to data equipment operated by ISP 624. ISP 624 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the Internet 625. Local network 622 and Internet 625 both use electrical, electromagnetic or optical signals which carry digital data streams. The signals through the various networks and the signals on network link 621 and through communication interface 620, which carry the digital data to and from computer 600, are exemplary forms of carrier waves transporting the information.

[0081] Processor 613 may reside wholly on client computer 601 or wholly on server 626 or processor 613 may have its computational power distributed between computer 601 and server 626. Server 626 symbolically is represented in FIG. 6 as one unit, but server 626 may also be distributed between multiple tiers. In one embodiment, server 626 comprises a middle and back tier where application logic executes in the middle tier and persistent data is obtained in the back tier. In the case where processor 613 resides wholly on server 626, the results of the computations performed by processor 613 are transmitted to computer 601 via Internet 625, Internet Service Provider (ISP) 624, local network 622 and communication interface 620. In this way, computer 601 is able to display the results of the computation to a user in the form of output.

[0082] Computer 601 includes a video memory 614, main memory 615 and mass storage 612, all coupled to bi-directional system bus 618 along with keyboard 610, mouse 611 and processor 613. As with processor 613, in various computing environments, main memory 615 and mass storage 612, may reside wholly on server 626 or computer 601, or they may be distributed between the two. Examples of systems where processor 613, main memory 615, and mass storage 612 are distributed between computer 601 and server 626 include the thin-client computing architecture developed by Sun Microsystems, Inc., the palm pilot computing device and other personal digital assistants, Internet ready cellular phones and other Internet computing devices, and in platform independent computing environments, such as those which utilize the Java technologies also developed by Sun Microsystems, Inc.

[0083] The mass storage 612 may include both fixed and removable media, such as magnetic, optical or magnetic optical storage systems or any other available mass storage technology. Bus 618 may contain, for example, thirty-two address lines for addressing video memory 614 or main memory 615. The system bus 618 also includes, for example, a 32-bit data bus for transferring data between and among the components; such as processor 613, main memory 615, video memory 614 and mass storage 612. Alternatively, multiplex data/address lines may be used instead of separate data and address lines.

[0084] In one. embodiment of the invention, the processor 613 is a microprocessor manufactured by Motorola, such as the 680XO processor or a microprocessor manufactured by Intel, such as the 80X86, or Pentium processor, or a SPARC microprocessor from Sun Microsystems, Inc. However, any other suitable microprocessor or microcomputer may be utilized. Main memory 615 is comprised of dynamic random access memory (DRAM). Video memory 614 is a dual-ported video random access memory. One port of the video memory 614 is coupled to video amplifier 616. The video amplifier 616 is used to drive the cathode ray tube (CR 1) raster monitor 617. Video amplifier 616 is well known in the art and may be implemented by any suitable apparatus. This circuitry converts pixel data stored in video memory 614 to a raster signal suitable for use by monitor 617. Monitor 617 is a type of monitor suitable for displaying graphic images.

[0085] Computer 601 may send messages and receive data, including program code, through the network(s), network link 621, and communication interface 620. In the Internet example, remote server computer 626 might transmit a requested code for an application program through Internet 625, ISP 624, local network 622 and communication interface 620. The received code may be executed by processor 613 as it is received, and/or stored in mass storage 612, or other non-volatile storage for later execution. In this manner, computer 600 may obtain application code in the form of a carrier wave. Alternatively, remote server computer 626 may execute applications using processor 613, and utilize mass storage 612, and/or video main memory 615. The results of the execution at server 626 are then transmitted through Internet 625, ISP 624, local network 622 and communication interface 620. In this example, computer 601 performs only input and output functions.

[0086] Application code may be embodied in any form of computer program product. A computer program product comprises a medium configured to, store or transport computer readable code, or in which computer readable code may be embedded. Some examples of computer program products are CD-ROM disks, ROM cards, floppy disks, magnetic tapes, computer hard drives, servers on a network, and carrier waves.

[0087] The computer systems described above are for purposes of example only. An embodiment of the invention may be implemented in any type of computer system or programming or processing environment.

[0088] Thus, a method and apparatus for associating the color of an object with an event is described in conjunction with one or more specific embodiments. The invention is defined by the claims and their full scope of equivalents.