Enhanced error recovery for data storage drives

Abstract

Methods and systems for enhanced error recovery are described. A first one or more data blocks to write to a first drive are received by a first drive controller module. A first parity block is calculated by the first drive controller module based on a first data block parity group. The first one or more data blocks are written by the first drive controller module to the first drive. The first parity block is written by the first drive controller module to the first drive.

Claims

1. A method performed by a storage system comprising a plurality of drive controller circuitries associated with a plurality of drives, respectively, the method comprising: receiving at least a plurality of first data blocks and a plurality of second data blocks; in a first writing step, writing the plurality of first data blocks to a first subset of the plurality of drives, respectively, leaving a first one of the plurality of drives unwritten to in the first writing step; in a second writing step, writing the plurality of second data blocks to a second subset of the plurality of drives, respectively, leaving a second one of the plurality of drives unwritten to in the second writing step, a first data block of the plurality of first data blocks being written to a third one of the plurality of drives in the first writing step, and a second data block of the plurality of second data blocks being written to the third one of the plurality of drives in the second writing step; calculating a first parity block based on the plurality of first data blocks written to the first subset of the plurality of drives; calculating a second parity block based on the plurality of second data blocks written to the second subset of the plurality of drives; writing the first parity block to the first one of the plurality of drives unwritten to in the first writing step; writing the second parity block to the second one of the plurality of drives unwritten to in the second writing step; calculating, by drive controller circuitry associated with at least the third one of the plurality of drives, at least a third parity block based at least on the first data block and the second data block written to the third one of the plurality of drives; and writing the third parity block to the third one of the plurality of drives.

2. The method of claim 1, wherein at least the first data block and the second data block written to the third one of the plurality of drives include a pre-determined number of data blocks.

3. The method of claim 1, further comprising: determining, by the drive controller circuitry associated with at least the third one of the plurality of drives, a number of data blocks in at least the first data block and the second data block written to the third one of the plurality of drives based on a number of data blocks in the plurality of first data blocks and the plurality of second data blocks; and writing, by the drive controller circuitry associated with at least the third one of the plurality of drives, the determined number of data blocks to the third one of the plurality of drives.

4. The method of claim 1, wherein the calculating of the third parity block by the drive controller circuitry associated with at least the third one of the plurality of drives includes performing an exclusive OR operation on at least the first data block and the second data block written to the third one of the plurality of drives.

5. A storage system comprising: a plurality of drives; and a plurality of drive controller circuitries associated with the plurality of drives, respectively, wherein one or more of the plurality of drive controller circuitries are configured: to receive at least a plurality of first data blocks and a plurality of second data blocks; in a first writing step, to write the plurality of first data blocks to a first subset of the plurality of drives, respectively, leaving a first one of the plurality of drives unwritten to in the first writing step; in a second writing step, to write the plurality of second data blocks to a second subset of the plurality of drives, respectively, leaving a second one of the plurality of drives unwritten to in the second writing step, a first data block of the plurality of first data blocks being written to a third one of the plurality of drives in the first writing step, and a second data block of the plurality of second data blocks being written to the third one of the plurality of drives in the second writing step; to calculate a first parity block based on the plurality of first data blocks written to the first subset of the plurality of drives; to calculate a second parity block based on the plurality of second data blocks written to the second subset of the plurality of drives; to write the first parity block to the first one of the plurality of drives unwritten to in the first writing step; and to write the second parity block to the second one of the plurality of drives unwritten to in the second writing step, and wherein drive controller circuitry associated with at least the third one of the plurality of drives is configured: to calculate at least a third parity block based at least on the first data block and the second data block written to the third one of the plurality of drives; and to write the third parity block to the third one of the plurality of drives.

6. The storage system of claim 5, wherein at least the first data block and the second data block written to the third one of the plurality of drives include a pre-determined number of data blocks.

7. The storage system of claim 5, wherein the drive controller circuitry associated with at least the third one of the plurality of drives is further configured: to determine a number of data blocks in at least the first data block and the second data block written to the third one of the plurality of drives based on a number of data blocks in the plurality of first data blocks and the plurality of second data blocks; and to write the determined number of data blocks to the third one of the plurality of drives.

8. The storage system of claim 5, wherein the drive controller circuitry associated with at least the third one of the plurality of drives is further configured: to calculate third parity block by performing an exclusive OR operation on at least the first data block and the second data block written to the third one of the plurality of drives.

9. A storage system comprising: a plurality of drives in a RAID configuration; and a plurality of drive controller circuitries associated with the plurality of drives, respectively, wherein one or more of the plurality of drive controller circuitries are configured: to receive at least a plurality of first data blocks and a plurality of second data blocks; in a first writing step, to write the plurality of first data blocks to a first subset of the plurality of drives, respectively, leaving a first one of the plurality of drives unwritten to in the first writing step; in a second writing step, to write the plurality of second data blocks to a second subset of the plurality of drives, respectively, leaving a second one of the plurality of drives unwritten to in the second writing step, a first data block of the plurality of first data blocks being written to a third one of the plurality of drives in the first writing step, and a second data block of the plurality of second data blocks being written to the third one of the plurality of drives in the second writing step; to calculate a first parity block based on the plurality of first data blocks written to the first subset of the plurality of drives; to calculate a second parity block based on the plurality of second data blocks written to the second subset of the plurality of drives; to write the first parity block to the first one of the plurality of drives unwritten to in the first writing step; and to write the second parity block to the second one of the plurality of drives unwritten to in the second writing step, and wherein drive controller circuitry associated with at least the third one of the plurality of drives is configured: to calculate at least a third parity block based at least on the first data block and the second data block written to the third one of the plurality of drives; and to write the third parity block to the third one of the plurality of drives.

10. The storage system of claim 9, wherein the RAID configuration is selected from the group consisting of RAID 0, RAID 1, RAID 2, RAID 3, RAID 4, RAID 5, and RAID 6 configurations.

11. The storage system of claim 9, wherein the drive controller circuitry associated with at least the third one of the plurality of drives is further configured to: calculate the first third parity block parity group by performing an exclusive OR operation on at least the first data block and the second data block written to the third one of the plurality of drives.

12. A method performed by a storage system comprising a plurality of drive controller circuitries associated with a plurality of drives, respectively, the method comprising: determining that a data block stored on a respective drive is unreadable, wherein (1) in a first writing step, a plurality of first data blocks is written to a first subset of the plurality of drives, respectively, leaving a first one of the plurality of drives unwritten to in the first writing step, (2) in a second writing step, a plurality of second data blocks is written to a second subset of the plurality of drives, respectively, leaving a second one of the plurality of drives unwritten to in the second writing step, a first data block of the plurality of first data blocks being written to a third one of the plurality of drives in the first writing step, and a second data block of the plurality of second data blocks being written to the third one of the plurality of drives in the second writing step, (3) a first parity block is calculated based on the plurality of first data blocks written to the first subset of the plurality of drives, (4) a second parity block is calculated based on the plurality of second data blocks written to the second subset of the plurality of drives, (5) the first parity block is written to the first one of the plurality of drives unwritten to in the first writing step, (6) the second parity block is written to the second one of the plurality of drives unwritten to in the second writing step, (7) at least a third parity block is calculated by drive controller circuitry associated with at least the third one of the plurality of drives based at least on the first data block and the second data block written to the third one of the plurality of drives, and (8) the third parity block is written to the third one of the plurality of drives, and wherein the respective drive storing the unreadable data block corresponds to the third one of the plurality of drives; reading, by the drive controller circuitry associated with at least the third one of the plurality of drives, at least the first data block and the second data block from the third one of the plurality of drives; reading, by the drive controller circuitry associated with at least the third one of the plurality of drives, the third parity block from the third one of the plurality of drives; and calculating, by the drive controller circuitry associated with at least the third one of the plurality of drives, a value of the unreadable data block based on at least the first data block, the second data block, and the third parity block, each of the first data block, the second data block, and the third parity block being read from the third one of the plurality of drives.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The foregoing and other objects, features, and advantages of the present technology, as well as the technology itself, will be more fully understood from the following description of various embodiments, when read together with the accompanying drawings, in which:

(2) FIG. 1 depicts a storage system, in accordance with the technology.

(3) FIGS. 2A-2C illustrate data written to a drive in accordance with the technology.

(4) FIG. 3 depicts drives, in accordance with the technology.

(5) FIG. 4 depicts drives in accordance with the technology.

(6) FIG. 5 depicts a flowchart illustrating a method of enhancing error recovery for a data storage drive.

(7) FIG. 6 depicts a flowchart illustrating a method of recovering a data block on a drive.

DETAILED DESCRIPTION

(8) The technology described herein relates to enhanced error recovery capabilities of data storage drives. The technology can, for example, facilitate recovering data after single or multiple sector failures and/or a drive failure. In an exemplary application, the technology can be used to improve error recovery for drives in a mirrored pair configuration. In such a configuration, data written to the first drive of the mirrored pair is also written to the second drive of the mirrored pair. In accordance with aspects of the technology, when a series of data blocks are written to a drive in the mirrored pair, a parity block is also written to the drive. A data block can be any size. For example, a data block can correspond to a magnetic disk drive's sector (e.g., 512 bytes or 4 kilobytes). As another example, a data block can be another pre-determined size (e.g., multiple sectors of data). The parity block can be calculated based on the series of data blocks being written to the drive. The parity block can also be calculated based on data blocks already stored on the drive. If one of the data blocks on a drive in the mirrored pair becomes unreadable, the data block can be recovered using the parity block associated with the unreadable data block and the other data blocks that were used to calculate the parity block. Beneficially, the technology can improve error recovery in the mirrored pair of drives. For example, if one of the mirrored drives fails entirely, the remaining functional drive can still recover from losses of individual data blocks (e.g., due to one or more sector failures) by using the parity blocks stored on the remaining functioning drive.

(9) Exemplary Storage System Environment

(10) FIG. 1 depicts storage system 100, in accordance with the technology. Storage system 100 includes storage processor enclosure 105. Storage processor enclosure 105 can include one or more storage processors (not shown). The storage processors can be any computing devices, such as servers, blade servers, etc. Storage system 100 includes disk array enclosure 110 and disk array enclosure 115. Disk array enclosures 110 and 115 include drives 120a-120h. Drives 120a-120h can be any type of storage devices, such as magnetic disk drives, flash memory drives, solid state drives, etc. Drives 120a-120h include drive controller modules 125a-125h. In the illustrated embodiment, storage processor enclosure 105 is connected to disk array enclosure 110, which is connected to disk array enclosure 115. In illustrated storage system 100, the storage processors can read data from and write data to drives 120a-120h. Drive controller modules 125a-125h can control how data is written to and read from drives 120a-120h. Storage system 100 can be connected to data networks 130. The illustrated connections can each be any type of connection for communicating data.

(11) In general, the storage processors in storage processor enclosure 105 service IO requests (e.g., commands to read data from or write data to storage system 100) received from data networks 130. It should be appreciated that storage system 100 in FIG. 1 is an exemplary storage system in accordance with the technology. Other configurations are contemplated. For example, in some embodiments, the storage processors and disk enclosures can be in the same enclosure. In some embodiments, more or fewer disk array enclosures can be connected to the storage processors in the storage processor enclosure. In some embodiments, the drive controller modules can be hardware. In some embodiments, the drive controller modules can be software (e.g., software executing on the service processors) or firmware. Each drive controller module can control a single drive or multiple drives.

(12) Data Block Storage and Parity Block Calculation

(13) FIGS. 2A-2C illustrate data written to drive 205 in accordance with the technology. For example, drive 205 can be any of drives 120a-120h shown in FIG. 1. Drive 205 can store data blocks at addresses 210a-210r. As noted above, a data block can be any size.

(14) With reference to FIG. 2A, drive 205 is illustrated with data blocks A.sub.1-A.sub.8 stored at addresses 210a-210h. In accordance with the technology, parity block P.sub.1 can be stored on drive 205 at address 210i. Parity block P.sub.1 can be calculated based on data blocks A.sub.1-A.sub.8 by, for example, a drive controller module associated with drive 205. In some embodiments, the data blocks (e.g., data blocks A.sub.1-A.sub.8) used when calculating the parity block can be referred to as the data block parity group. Parity block P.sub.1 can be calculated by performing an exclusive OR operation on data blocks A.sub.1-A.sub.8. For example, the nth bit of parity block P.sub.1 can be calculated by performing an exclusive OR operation on the nth bits from each of data blocks A.sub.1-A.sub.8. Other methods of error correction are contemplated. For example, error correcting codes can be calculated across data blocks A.sub.1-A.sub.8 and stored in place of parity block P.sub.1.

(15) The drive controller module can receive additional data blocks for writing to drive 205. Referring to FIG. 2B, drive 205 is illustrated with data blocks A.sub.1-A.sub.8 stored at addresses 210a-210h and data blocks B.sub.1-B.sub.8 stored at addresses 210j-210q. In accordance with the technology, parity block P.sub.2 can be written to drive 205 at address 210r. Parity block P.sub.2 can be calculated, for example, by the drive controller module associated with drive 205 based on the data block parity group containing data blocks B.sub.1-B.sub.8. With reference to FIG. 2C, drive 205 is illustrated with data blocks A.sub.1-A.sub.8 stored at addresses 210a-210h, data blocks C.sub.1-C.sub.4 stored at addressed 210j-210m, and data blocks B.sub.5-B.sub.8 stored at addresses 210n-210q. As illustrated, data blocks C.sub.1-C.sub.4 are written over data blocks B.sub.1-B.sub.4 of FIG. 2B. In accordance with the technology, parity block P.sub.3 can be written to drive 205 at address 210r. Parity block P.sub.3 can be calculated, for example, by the drive controller module associated with drive 205 based on the data block parity group containing data blocks C.sub.1-C.sub.4 and B.sub.5-B.sub.8. In some embodiments, conventional techniques of providing atomicity can be used to ensure that all data blocks and the parity block are successfully written to the drive.

(16) As illustrated above, in some embodiments, the data block parity groups can be a fixed size. In some embodiments, the data block parity group can be based on the data being stored. For example, a parity block can be written every, e.g., 4 or 8 blocks if the data is being written a few blocks at a time. If long sequential writes are common, a parity block can be written every, e.g., 100 or 200 blocks. In some embodiments, the data block parity group can be variable sized. In some embodiments, a header can be written to the drive to indicate the data blocks included in the data block parity group for a particular parity block. In some embodiments, the header can be written to another drive and/or stored in other memory on a storage system.

(17) In some embodiments, the data blocks forming the data block parity group and the parity block are not stored consecutively and/or adjacently on the drive. For example, in some embodiments, there can be a logical mapping of the data block parity group to physical addresses on the drive. A particular data block can be mapped to one or more sectors on the drive and, in the case of multiple sectors, the sectors can be non-consecutive. Accordingly, data of the data block parity group can be distributed on the drive. Similarly, the parity block can be mapped to one or more sectors on the drive, and the sectors can be non-consecutive.

(18) Error Recovery

(19) The technology can facilitate data recovery after drive errors and/or failures. While magnetic disk drives can include ECC data to facilitate recovery from partial loss of a sector, in some cases the ECC data cannot be used to recover data when the entire sector is lost. The technology can facilitate recovering the data for a data block (e.g., the data for one or more failed sectors). The data stored on the failed sector can be recovered by performing an exclusive OR operation on the parity block and the remaining data blocks in the data block parity group. For example, with reference to FIG. 2C, one or more sector failures can cause data block A.sub.1 stored at address 210a to be unreadable. The value of data block A.sub.1 can be recovered by performing an Exclusive OR operation on the data blocks A.sub.2-A.sub.8 and parity block P.sub.1. For example, the nth bit of data block A.sub.1 can be calculated by performing an exclusive OR operation on the nth bits from each of data blocks A.sub.2-A.sub.8 and parity block P.sub.1. Beneficially, when the data block size is larger than the drive's sector size, the technology can facilitate recovery when multiple adjacent sectors fail.

(20) Applications

(21) The technology can be used in various drive configurations. FIG. 3 depicts drive 305 and drive 310, in accordance with the technology. Drive 305 and drive 310 are mirrored in a RAID 1 configuration. As illustrated, certain data written to drive 305 is also written to drive 310. In the illustrated application, data blocks A.sub.1-A.sub.4 are written to drive 305 at addresses 307a-307d. A drive controller module associated with drive 305 calculates parity block P.sub.1, stored at address 307e, based on the data block parity group containing data blocks A.sub.1-A.sub.4. Data blocks A.sub.1-A.sub.4 are mirrored to drive 310 and stored at addresses 312a-312d. A drive controller module associated with drive 310 calculates parity block P.sub.1, stored at address 312e, based on the data block parity group containing data blocks A.sub.1-A.sub.4. Data blocks B.sub.1-B.sub.4 are written to drive 305 at addresses 307f-307i. A drive controller module associated with drive 305 calculates parity block P.sub.2, stored at address 307j, based on the data block parity group containing data blocks B.sub.1-B.sub.4. Data blocks B.sub.1-B.sub.4 are mirrored to drive 310 and stored at addresses 312f-312i. A drive controller module associated with drive 310 calculates parity block P.sub.2, stored at address 312j, based on the data block parity group containing data blocks B.sub.1-B.sub.4.

(22) In some instances, a mirrored pair of drives in a conventional RAID 1 configuration can recover from one of the two drives failing. Data can be read from the remaining operational drive. However, in some instances the remaining operational drive cannot recover from additional errors on that drive, such as sector failures. Beneficially, the technology can improve error recovery. For example, if drive 305 fails, drive 310 can still recover from sector failures. If a sector failure makes address 312a unreadable, data block A1 can still be recovered using data blocks A.sub.2-A.sub.4 (stored at addresses 312b-312d) and parity block P.sub.1 (stored at address 312e), as described above.

(23) FIG. 4 depicts drive 405, drive 410, drive 415, and drive 420 in accordance with the technology. Drives 405, 410, 415, and 420 are in a RAID 5 configuration. As illustrated, data blocks A.sub.1-A.sub.3 are striped across drives 405, 410, and 415. Data block A.sub.1 is stored at address 407a of drive 405. Data block A2 is stored at address 412a of drive 410. Data block A.sub.3 is stored at address 417a of drive 415. Data block A.sub.P (e.g., a parity block calculated based on data blocks A.sub.1-A.sub.3) is stored at address 422a of drive 420. Data blocks B.sub.1-B.sub.3 are striped across drives 405, 410, and 420. Data block B.sub.1 is stored at address 407b of drive 405. Data block B.sub.2 is stored at address 412b of drive 410. Data block B.sub.3 is stored at address 422b of drive 420. Data block B.sub.P (e.g., a parity block calculated based on data blocks B.sub.1-B.sub.3) is stored at address 417b of drive 415. Data blocks C.sub.1-C.sub.3 and data block C.sub.P, data blocks D.sub.1-D.sub.3 and data block D.sub.P, data blocks E.sub.1-E.sub.3 and data block E.sub.P, data blocks F.sub.1-F.sub.3 and data block F.sub.P, data blocks G.sub.1-G.sub.3 and data block G.sub.P, and data blocks H.sub.1-H.sub.3 and data block H.sub.P are similarly striped across drives 405, 410, 415, and 420 as illustrated.

(24) In accordance with the technology, parity block P.sub.1, stored at address 407e of drive 405, is calculated based on the data block parity group containing data blocks A.sub.1, B.sub.1, C.sub.1, and D.sub.P. Parity block P.sub.2, stored at address 412e of drive 410, is calculated based on the data block parity group containing data blocks A.sub.2, B.sub.2, C.sub.P, and D.sub.1. Parity block P.sub.3, stored at address 417e of drive 415, is calculated based on the data block parity group containing data blocks A.sub.3, B.sub.P, C.sub.2, and D.sub.2. Parity block P.sub.4, stored at address 422e of drive 420, is calculated based on the data block parity group containing data blocks A.sub.P, B.sub.3, C.sub.3, and D.sub.3. Parity block P.sub.5, stored at address 407j of drive 405, is calculated based on the data block parity group containing data blocks E.sub.1, F.sub.1, G.sub.1, and H.sub.P. Parity block P.sub.6, stored at address 412j of drive 410, is calculated based on the data block parity group containing data blocks E.sub.2, F.sub.2, G.sub.P, and H.sub.1. Parity block P.sub.7, stored at address 417j of drive 415, is calculated based on the data block parity group containing data blocks E.sub.3, F.sub.P, G.sub.2, and H.sub.2. Parity block P.sub.8, stored at address 422j of drive 420, is calculated based on the data block parity group containing data blocks E.sub.P, F.sub.3, G.sub.3, and H.sub.3.

(25) The technology can facilitate recovery of data in some instances where a conventional RAID 5 configuration could not recover the data. For example, if drive 405 fails and a sector failure causes data block A.sub.2 to be unreadable, the technology can facilitate recovery of data block A.sub.1 and data block A.sub.2. As described above, data block A.sub.2 can be recovered using data blocks B.sub.2, C.sub.P, D.sub.1, and P.sub.2. Data block A.sub.1 can then be recovered using data blocks A.sub.2, A.sub.3, and A.sub.P. More generally, the technology can facilitate data recovery of similar drive and sector failures.

(26) FIG. 5 depicts a flowchart illustrating a method of enhancing error recovery for a data storage drive. At step 510, a first one or more data blocks are received to write to a first drive. For example, as described above with reference to FIG. 2A, data blocks A.sub.1-A.sub.8 can be received. At step 520, a first parity block is calculated based on a first data block parity group. The data block parity group can contain the one or more data blocks. For example, as described above with reference to FIG. 2A, parity block P.sub.1 can be calculated based on data blocks A.sub.1-A.sub.8. At step 530, the one or more data blocks are written to the first drive. For example, as described above with reference to FIG. 2A, data blocks A.sub.1-A.sub.8 are written to addresses 210a-210h of drive 205. At step 540, the first parity block is written to the first drive. For example, as described above with reference to FIG. 2A, parity block P.sub.1 is written to address 210i of drive 205. It should be appreciated that the steps of the described method can occur in other orders and/or in parallel. For example, steps 520 and 540 can be performed before, after, or in parallel with step 530.

(27) FIG. 6 depicts a flowchart illustrating a method of recovering a data block on a drive. At step 610, it is determined that a first data block on a first drive is unreadable, where the first data block is a member of a first data block parity group. For example, as discussed above, a first data block can be unreadable due to a sector failure on the first drive. At step 620, a second one or more data blocks are read from the first drive, where the second one or more data blocks are from the first data block parity group. For example, as described above, the remaining data blocks in the data block parity group associated with the unreadable data block can be read from the drive. At step 630, a first parity block associated with the first data block parity group is read from the drive. For example, as described above, the parity block associated with the unreadable block can be read from the drive. At 640, a value of the first data block is calculated based on the second one or more data blocks and the first parity block. For example, as described above, the unreadable data block can be recovered based on the associated parity block and the remaining data blocks from the data block parity group associated with the unreadable data block.

(28) The above-described techniques can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The implementation can be as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable storage device or in a propagated signal, for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers. A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

(29) Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices or drives for storing data, e.g., magnetic, magneto-optical disks, or optical disks. Data transmission and instructions can also occur over a communications network. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in special purpose logic circuitry.

(30) To provide for interaction with a user, the above described techniques can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer (e.g., interact with a user interface element). Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.

(31) The above described techniques can be implemented in a distributed computing system that includes a back-end component, e.g., as a data server, and/or a middleware component, e.g., an application server, and/or a front-end component, e.g., a client computer having a graphical user interface and/or a Web browser through which a user can interact with an example implementation, or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet, and include both wired and wireless networks.

(32) The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

(33) The technology has been described in terms of particular embodiments. The alternatives described herein are examples for illustration only and not to limit the alternatives in any way. The steps of the technology can be performed in a different order and still achieve desirable results. Other embodiments are within the scope of the following claims.