SYSTEMS AND METHODS FOR USING PERSISTENT, PASSIVE, ELECTRONIC INFORMATION CAPTURING DEVICES

Abstract

Personal companions crowd-source and/or crowd-share characterizations, and optionally raw data, from real-world, virtual and/or mixed-reality experiences. Characterizations can advantageously be stored in one or more self-evolving, structured databases, and can be organized according to objects, actions, events and thoughts. Characterizations can be weighted differently for different users, and “forgotten” over time, especially in favor of maintaining higher level characterizations. Personal companions can be used to obtain additional information, and conduct interpersonal, commercial, or other interactions or transactions.

Claims

1. A method of using a computer system and first and second persistent, passive, electronic information capturing devices (PPIC devices) to provide guidance to a first person, comprising: using the first PPIC device to gather information (first person information) about an environment encountered by the first person; using the computer system to identify a circumstance based upon abstractions from the first person information; using the second PPIC device to gather information (second person information) about an environment encountered by a second person, wherein the first one of the PPIC devices is different from the second one of the PPIC devices, and the first person is different from the second person; and using the computer system to utilize (a) the circumstance and (b) the second person information to automatically provide the guidance to the first person.

2. The method of claim 1, wherein the first persistent, passive information capturing device is activated by a question and answer conversation with the first person.

3. The method of claim 1, wherein the second persistent, passive information capturing device is activated by a question and answer conversation with the second person.

4. The method of claim 1, wherein the first persistent, passive information capturing device and the second persistent, passive information capturing device comprises an artificial intelligence system that is trained using crowd-sourced question and answer conversations.

5. The method of claim 1, wherein the first persistent, passive information capturing device captures an identification of a circumstance, wherein the identification is captured from passive comments by one or more people.

6. The method of claim 1, wherein the first persistent, passive information capturing device captures an incidental identification, wherein the incidental identification is captured from one or more visual cues in the physical environment.

7. The method of claim 6, wherein the one or more visual cues comprise text.

8. The method of claim 6, wherein the one or more visual cues comprise an image.

9. The method of claim 1, wherein at least one of the first and second PPIC devices comprises a body-worn microphone.

10. The method of claim 1, wherein at least one of the first and second PPIC devices comprises a body-worn camera.

11. The method of claim 1, wherein the circumstance comprises a medical issue and the guidance comprises a suggested course of action regarding one or more treatment options.

12. The method of claim 1, wherein the circumstance comprises a time-based issue and the guidance comprises a suggested course of action to minimize an amount of time associated with the issue.

13. The method of claim 1, wherein the circumstance comprises finance-based issue and the guidance comprises a suggested course of action based on one or more financial priorities.

14. The method of claim 1, wherein the first PPIC device and the second PPIC device are vehicle-mounted.

15. The method of claim 1, wherein the guidance is provided audibly.

16. The method of claim 1, wherein the guidance is provided visually.

16. The method of claim 1, wherein the guidance is provided by directly controlling a device.

Description

BRIEF DESCRIPTION OF THE DRAWING

[0110] FIG. 1 is a high-level conceptual overview of how images have historically been recorded, used and stored

[0111] FIG. 2 is the overview of FIG. 1, annotated to include various aspects of crowd-sourcing and crowd-sharing of information.

[0112] FIG. 3A is a schematic of a person using a personal companion, where the camera is included in a sensor module affixed to clothing.

[0113] FIG. 3B is a schematic of a person using a personal companion, where the camera is included in a sensor module worn as an earpiece.

[0114] FIG. 4 is a schematic showing preferred correlations among image recognition and characterization facilities.

[0115] FIGS. 5A, 5B and 5C are part of an objects characterization facility. FIGS. 5A and 5C show an objects table, and FIG. 5B shows a corresponding object position-keys table.

[0116] FIG. 6 is a schematic of an actions characterization facility, and includes various actions-related data tables.

[0117] FIG. 7 is a schematic of an events characterization facility, and includes various events-related data tables.

[0118] FIG. 8 is a schematic of several people interacting through their personal companions with a canopy and with each other.

[0119] FIG. 9 is a sample User Preferences table.

DETAILED DESCRIPTION

[0120] The following discussion provides many example embodiments of the inventive subject matter. Although each embodiment represents a single combination of inventive elements, the inventive subject matter is considered to include all possible combinations of the disclosed elements. Thus if one embodiment comprises elements A, B, and C, and a second embodiment comprises elements B and D, then the inventive subject matter is also considered to include other remaining combinations of A, B, C, or D, even if not explicitly disclosed.

[0121] All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The use of any and all examples, or exemplary language (e.g. “such as”) provided with respect to certain embodiments herein is intended merely to better illuminate the invention, and does not pose a limitation on the scope of the invention otherwise claimed. No language in the specification should be construed as indicating any non-claimed element essential to the practice of the invention.

[0122] Throughout the discussion, references may be made regarding servers, services, interfaces, portals, platforms, or other systems formed from computing devices. It should be appreciated that the use of such terms is deemed to represent one or more computing devices having at least one processor configured to execute software instructions stored on a computer readable tangible, non-transitory medium. For example, a server can include one or more computers operating as a web server, database server, or other type of computer server in a manner to fulfill described roles, responsibilities, or functions.

[0123] In FIG. 2, systems, methods and apparatus are contemplated in which objects, actions, events, and thoughts are abstracted from the image data using crowd-sourced characterizations and characterization parameters 32C, additional information is procured 42C, transactions are executed 52C, and information is stored 62C, using crowd-sourced strategies. The term “parameters” is used herein to refer characteristics that can be used in combination with values to describe an object, action, event or thought.

[0124] In FIG. 3A, a personal companion 300A generally comprises (a) a sensor module 320A, (b) a networkable electronics module 330 including processor 332, software 334 and data storage 336 in the electronics module 330, (c) a speaker 370, and (d) a distally operated system (distal service) 360 including processor 362, software 364 and data storage 366. In this particular example, a person 310 is wearing the sensor module 320A on his shirt 311, and carries the electronics module 330 in a pocket (not shown), or a hip carrier (not shown), or perhaps a purse (not shown). The electronics module 330 has a power and data link (not shown) with the sensor module 320A and a wired or wireless link (not shown) with the speaker 370A, and is at least intermittently networked with a distally operated system (distal service) 360.

[0125] The sensor module 320A can comprise a single housing or be separated into multiple housings, and can include any practical sensor or combination of sensors, including especially a camera 322A and a microphone 324A. Other contemplated sensors can be included in the sensor module 320A, or located elsewhere on or near the body, and include those that measure pulse, blood pressure, pO.sub.2 (partial pressure of Oxygen), body temperature, chemicals in the sweat, or other biometrics, and ambient temperature, humidity, movement, proximity or other environmental characteristics. In this particular example, the person 410 is wearing a pulse monitor 328 on his wrist, which is wirelessly coupled with the electronics module 330. Multiple sensors can be housed together, or be separated by several millimeters, centimeters, or any other suitable distance. Duplicate sensors can be included, as for example a backup or stereo camera (not shown).

[0126] Sensor modules are preferably small enough to be comfortably carried for hours at a time. In the case of a clothing-worn sensor module, the outwardly facing (away from the wearer) surface area could advantageously be only a few square centimeters. Sensor modules can be coupled to the clothing or wearer using a pin, band, snap, lanyard, necklace or any other suitable connector. In FIG. 3 the sensor module 320 is coupled to the upper chest area of the shirt 311 using a safety pin (not shown). Another option is to put the camera and/or microphone on an in-ear Bluetooth headset.

[0127] Sensor modules preferably have an electronically operated coating that changes color or in some other manner designates to the wearer or others that the sensor is in or out of operation. Thus, for example, a camera or other sensor that collect still, video or other images might a coating overlay on the lens that changes to red, blue, grey or some other color or pattern to designate that the sensor is off line. Similarly, a microphone or other sound sensor might have a blinking or steady light to designate that it is off line. The sensor in the images in this application should be interpreted as having such coatings or lights. In fact, it is contemplated that Google Glass™, which might or might not be utilized with abstracting systems and methods contemplated herein, could also have such coatings or lights.

[0128] As used herein, and unless the context indicates otherwise, the term “coupled to” is intended to include both direct coupling (in which two elements that are directly connected, logically or physically) and indirect coupling (in which at least one additional element is logically or physically located between the two elements). The terms “coupled to” and “coupled with” are used synonymously.

[0129] Data collected by the sensor module(s) is preferably sent to the electronics module 330, and the data flow can optionally be bidirectional. In FIG. 3 the camera 322A send images to the electronics module 330, and the networkable electronics module 330 sends focus, blackout or other operating commands to the camera 322A. A similar situation independently occurs for sounds, in which the microphone 324A sends sound data to the electronics module 330, and the electronics module 330 might send filtering or other commands to the microphone 324A. Electronics module 330 also sends sounds to the speaker 370A, which in this instance is an ear bud.

[0130] Power source(s) for the sensor module 320A, and possibly even for the electronics module 330, could be located in any suitable location, as for example, within the electronics module, or even separately from both of those devices, perhaps even in the fabric of a worn garment. In FIG. 3A, some power is provided by a battery woven into the fabric of the shirt 311, and power is carried to the networkable electronics module 330 via power cable (not shown). Batteries woven into clothing are described in US 2011/0271424 (Revol-Cavalier, Publ November 2011).

[0131] Optional glasses 380 can be equipped with an electronic display 384, which is preferably operated by electronics module 330 to show video, text or other visually perceivable information to the wearer. It is contemplated that a sensor module could be incorporated into, or attached to a pair of glasses.

[0132] It is contemplated that distally operated system (distal service) 360 can comprise any suitable computer implementation, can employ various computing devices including servers, services, interfaces, systems, databases, agents, peers, facilities, controllers, or other types of computing devices, operating individually or collectively, locally or in a distributed fashion. Interaction between (a) a local device of a given user, for example networkable electronics module 330, and (b) the distally operated system (distal service) 360, is preferably conducted as a client/server fashion, although peer-to-peer and all other suitable configurations are contemplated. Computing devices can comprise a processor configured to execute software instructions stored on a tangible, non-transitory computer readable storage medium (e.g., hard drive, solid state drive, RAM, flash, ROM, etc.). Software instructions operating on computing devices preferably configure the computing device to provide the roles, responsibilities, or other functionality as discussed herein with respect to the disclosed apparatus. In especially preferred embodiments, the various servers, systems, databases, or interfaces exchange data using standardized protocols or algorithms, possibly based on HTTP, HTTPS, AES, public-private key exchanges, web service APIs, known financial transaction protocols, or other electronic information exchanging methods. Data exchanges preferably are conducted over a packet-switched network, the Internet, LAN, WAN, VPN, or other type of packet switched network.

[0133] As used herein, the term “personal companion” refers to the entire system as viewed from the perspective of an individual user. Thus, there can be aspects of multiple personal companions that share resources at distally operated system (distal service) 360. From time to time herein, reference is made to a local portions of the system, which refers to portions of the system worn or otherwise carried by an individual user.

[0134] FIG. 3B is similar to FIG. 3A, except that the personal companion 300B has a sensor module 320B worn as an earpiece, in a manner similar to how people currently wear Bluetooth headsets. In this instance sensor module 320B includes a camera 322B, a microphone 324B and a speaker 370B.

[0135] In FIGS. 3A/3B the camera 332A/332B receives images of a table 381, which supports an orange 382 and an apple 383. The images are sent to the electronics module 330, which relays at least portions of the image data to the distally operated system (distal service)360. The distally operated system (distal service) 360 uses software 362 to identify visual features in whatever image data was received, and then apply those visual features against a crowd-sourced database to identify one or more objects within the image(s). As discussed in greater detail below, contemplated systems and methods can be used not only to identify objects, but also to identify actions, events, and thoughts.

[0136] FIG. 4 depicts a high level overview of preferred correlations among data structures. Currently preferred embodiments use a feature recognition facility 400, an objects characterization facility 500, an actions characterization facility 600, an events characterization facility 700, and a thoughts facility 800. As used herein, the term “facility” means a physical device comprising electronics programmed or otherwise configured to (a) perform a desired function or (b) provide access to a device, system or service that performs the desired function. Facilities can be stand-alone, networked and/or implemented in cloud computing or other distributed environments.

[0137] Feature recognition facility 400 can use any suitable algorithms, software and/or hardware, and it is considered to be within the skill of ordinary persons in the art to implement feature recognition facility 400 using known products.

[0138] For example, identification of objects and visual features using image data can be accomplished using a Scale-Invariant Feature Transform (SIFT) or Speeded Up Robust Feature (SURF). Information on these technologies can be found at the corresponding Wikipedia articles, http://en.wikipedia.org/wiki/Scale-invariant_feature_transform and http://en.wikipedia.org/wiki/SURF, respectively. Such technologies are already used in conjunction with a single camera SLAM system. See for example, Active Vision Laboratory's web page at http://www.robots.ox.ac.uk/˜bob/research/research_objectslam.html.

[0139] Identification of persons and facial features using facial recognition software products can be accomplished using products such as iOmniscient™ (see http://iomniscient.com/index.php?option=com_content&view=article&id=58&Itemid=54).

[0140] Identification of objects and features using sound data can also be accomplished with available software. For example, content of human speech can be recognized using Dragon™ NaturallySpeaking software from Nuance™ (see www.nuance.com), and birdsong, music and other non-speech sounds can be recognized using programs Raven™ (see www.birds.comell.edu/brp/raven/ravenoverview.html).

[0141] Identification of objects and features using odor data can be accomplished using zNose™ from Electronic Sensor Technology, or the S&T™ (Smell and Taste) software from F&F Consulting (see http://www.fid-tech.com/eng/technol/data/electronics/1.449.html).

[0142] It should be appreciated that personal companions and their respective sensors can participate in incidental identification. Incidental identification, as used herein and applied to the present invention, comprises any identification made by a personal companion and associated sensors in a passive manner. For example, a body-worn camera can incidentally identify the text on a sign and determine that the sign requires a user to stop when he or she is in a car. In another example, a body-worn microphone can incidentally capture passive comments from one or more people regarding a subject, such as, for example, a statement from a passerby commenting on the presence of broken glass on a sidewalk. However, the invention described herein can passively identify, process, and record any information collected by one or more sensors associated with the personal companion.

[0143] Additionally, it is contemplated that incidental identification allows a personal companion to collect and process information without user input, thereby allowing the personal companion to learn passively without prompting by the user (e.g., initiating identification of an object by asking the personal companion a question).

[0144] It should be appreciated that personal companions need not conduct image or sound recognition per se. that is, there is no necessarily any database of images against which a captured image is compared. Personal companions preferably store pattern characteristics and associate those patterns with objects, ideas, behaviors and other things. For example personal companions could store information that oranges are bumpy, orange colored, usually have no stem attached, and about the size of a fist. In that instance, determining that an object is an orange does not require image recognition software, but instead pattern recognition software, preferably using crowd sourcing of the patterns.

[0145] It is contemplated that at least some feature recognition could take place locally, to the extent permitted by the sophistication of the local hardware in electronics module 330. For example, the S&T™ system discussed above is specifically designed for use on cell phones or other mobile devices. Nevertheless, it is far more likely that feature recognition facility 400 will operate by the local hardware sending image, sound, odor or other data to a distal service for interpretation.

[0146] Outputs of the feature recognition facility 400 are preferably stored in database tables that correlate specific features with objects and actions. And unlike the prior art, such storage preferably takes place using crowd-sourced characterizations and parameters.

[0147] Objects characterization facility 500 is implemented in part in objects table 510 shown in FIGS. 5A and 5C, and an objects position-keys table 520 shown in FIG. 5B. The corresponding database manager(s), communications manager(s), hardware, etc can be located anywhere, including for example the electronics module 330 (FIGS. 3A and 3B) and the Canopy (FIG. 8), and those figures should be interpreted in that manner.

[0148] Objects table 510 is used to correlate names of objects, which could be things, people, animals, buildings, trees, toys, and so forth, with characteristics. In this particular example the first two columns 511.sub.1, 511.sub.2 are used for record number and position key, respectively, the third column 511.sub.3 is used for a name of the object, and the remaining columns 511.sub.4-511.sub.y are used for the characteristics. In this example, the data used for position key is the name of an object category (fruit, vehicle, person, animal, etc), which is used in conjunction with table 520 to interpret the data in some of the remaining columns.

[0149] Object table 510 could be very large, having many millions of records. For searching efficiency, however, object table could be split into many different tables. One way of doing that would be to have different tables for different object categories, e.g., a table for fruit, a table for vehicle, and a table for person.

[0150] Object position-keys table 520 uses the first two columns 521.sub.1, 521.sub.2 for record number and category, respectively, and the remaining columns 521.sub.3-521.sub.n for data that determines what information cells of a given record in the object table 510 will store.

[0151] From a linguistic perspective, one can think of objects table 510 as correlating nouns in column 511.sub.3 with corresponding adjectives in columns 511.sub.4-511.sub.y for a particular object. For example, the first record 512.sub.1 in objects table 510 correlates the noun “orange” with the adjectives “round”, “orange” and “dimpled” in corresponding cells of columns 511.sub.4, 511.sub.5 and 511.sub.6. The first record 522.sub.1 in the objects position-keys table 520 correlates the adjective “round” with the category “shape”, the adjective “orange” with the category “color”, and the adjective “dimpled” with the category “texture”.

[0152] The second record 512.sub.2 in the objects position-keys table 520 correlates the noun “truck” with the adjectives “monster”, “white”, “expensive”, and “big wheels”. The reader should note that the term “adjective” is used herein in a relatively broad sense to describe an aspect of the object, and the words or characters used to store that need not be grammatically correct in any particular language. Thus, the second record 512.sub.2 stores the words “big wheels”, even though in English the correct adjective might be “big wheeled”.

[0153] The reader will hopefully should notice that not all cells of every record need to be utilized. To the contrary, each of the various tables in this application will very likely have many cells with no data. Also, instead of including literals in the objects table 510, it should be appreciated that one could alternatively or additionally use an adjectives table (not shown) that would simply list adjectives, which could then be pointed to from objects table 510.

[0154] Object records can also store symbols. In records 512.sub.7 and 512.sub.8 the objects table 510 stores images of one of the Apple logos, and two different contexts in which that logo was found. Record 512.sub.9 stores a bar code for the xyz product, which was identified as being found on packaging. Record 512.sub.10 stores a QR code for Wikipedia, which was identified as being found on a t-shirt. Of particular interest is record 512.sub.11, included here to demonstrate that virtually anything can be used as a symbol. In this case the image of George Washington standing up in a boat is stored as a symbol.

[0155] FIG. 5C shows some of the columns of Table 510 that were not depicted in FIG. 5A because of space limitations. In this example, column 511.sub.7 identifies the personal companions (or perhaps that of users operating the personal companions) that provided the information in the record.

[0156] Column 511.sub.8 identifies the confidentiality level accorded the information in the record. Typically the default would be shareable, but users might instruct their personal companions to restrict the information to only the personal companion (or person), to family members, and so forth.

[0157] Column 511.sub.9 identifies a date or perhaps a date/time stamp in which the information was loaded or last updated.

[0158] Column 511.sub.10 identifies a date or perhaps a date/time stamp that the record will expire. A default might be for the record to never expire, or perhaps for some time period, such as 5 or 10 years from the load date.

[0159] Column 511.sub.11 identifies a weighting that should be applied to this record. This could be on any suitable scale, as for example 1 to 5, or 1 to 100. Weightings might correspond to the confidence that the person providing the information has with respect to the correlations in the records. For example, a person providing information might be almost completely sure that a given QR code is associated with a given product, but only 1% sure that a person standing up in a boat in a painting is George Washington. The weighting might be determined by a personal companion asking for a weighting, or in other instances by comparing one person's characterizations with those of others, giving relatively heavy weightings to characterizations upon which people tend to agree.

[0160] Column 511.sub.12 identifies a persona to which this record relates. Personas used in connection with crowd-sourced databases are described in another one of my patent applications, US2008/0097849 (Ramsaier et al., Publ April 2008). In the context of the current application, personas could be used to modify the weighting given to specific records depending on whether the persona for that record is consistent or inconsistent with the current persona. Current persona could be expressly told to one's personal companion, or more preferably inferred by place, people, etc, and could be stored in a User Preferences table such as row 8 of table 900 of FIG. 9.

[0161] Of course the data presented in all of the tables in this application are exemplary only, included to indicate the gist of the ideas presented herein. Similarly, although records and columns are depicted as having fixed lengths, which is currently thought to be desirable to provide quick processing speed, records and columns could instead have variable lengths, and indeed other structures besides flat tables could be used. One should also appreciate that even though the letters “n” and “m” are used repeatedly to designate the number of columns and records in multiple tables, the various tables can have, and almost certainly will have, different numbers of columns and records. Still further, the use of any particular logo in this application should not be construed as indicating any legal association with, or endorsement by, any entity associated with the logo.

[0162] It should also be appreciated that although the tables and other exemplifications herein use English language words, any language could be used, including non-phonetic languages such as Japanese, Mandarin and Korean. In addition, it is contemplated to use translation tables or other strategies to translate data from one language into another language.

[0163] In FIG. 6, actions characterization facility 600 generally includes an actions table 610 and an actions position-keys table 620, with the database manager, communications manager, hardware, etc not shown. Actions table 610 is used to correlate aspects of movements, walking, riding, driving, eating, speaking, flying, sinking, playing, and so forth, with names of the actions. In this particular example the first two columns 611.sub.1, 611.sub.2 used for record number and position key, respectively, the third column 611.sub.3 is used for the name of the action, and the remaining columns 611.sub.3-611.sub.n are used for other characterizations of the actions.

[0164] Action position-keys table 620 uses the first two columns 621.sub.1, 621.sub.2 for record number and category, respectively, and the remaining columns 621.sub.3-621.sub.n for data that determines what information cells of a given record in the action table 610 will store.

[0165] Actions table 610 should be interpreted as having corresponding columns for source identification, confidentiality, time stamps, weighting and personas as discussed with respect to FIG. 5C.

[0166] From a linguistic perspective, one can think of actions table 610 as correlating verbs in column 611.sub.3 with corresponding adverbs in columns 611.sub.4-611.sub.n for a particular action. In the example shown record 622.sub.1 is a key to interpreting data in records 611.sub.1 and 611.sub.2 in actions table 610, where speed is “fast” or “slowly” and “manner” is “recklessly” or “haltingly”.

[0167] The reader may notice at this point that the categories depicted in the figures for different types of objects and actions are examples only, and are therefore somewhat arbitrary for the purpose of demonstration. Indeed, categories are intended to evolve over time depending on what users of the system tend to utilize. It may also be that some implementations use subcategories, or no categories at all for one or more of the objects characterization facility 500, actions characterization facility 600, events characterization facility 700, or thoughts facility 800. In other instances the category can be the same as the object, action, etc. See for example records 612.sub.3, and 612.sub.4 in FIG. 6.

[0168] One could alternatively or additionally use an adverbs table (not shown) that would simply list adverbs, which could then be pointed to in objects table 610.

[0169] In FIG. 7, events characterization facility 700 generally includes an events table 710 that correlates (a) objects from the objects table 510 with (b) actions from the actions table 610 and (c) outcomes. In this particular example the first two columns 711.sub.1, 711.sub.2 are used for record number and name of the event, respectively, and the remaining columns 711.sub.4-711.sub.n are used for characteristics of the events.

[0170] It is contemplated that one could use an events-position key table analogous to tables 520 and 620 for objects and events, respectively. But that is considered disadvantageous because it introduces unnecessary complexity.

[0171] From a linguistic perspective, one can think of actions table 710 as a collection of stored sentences, where the data in cells of the various columns correspond to the subjects, verbs, objects, and other parts of speech. For example: [0172] In events record no. 1, the observation that “Martha Smith reads” is recorded as Martha Smith (object record 5) reads (action record 5). [0173] In events record no. 2, Martha Smith (object record 5) reads (action record 5) a Harry Potter book (object record 6). [0174] In events record no. 3 Martha Smith (object record 5) reads (action record 3) a Harry Potter book (object record 6) to an unknown young adult male (object record 3). [0175] In events record no 4., the unknown young adult male (object record 2) is walking a labrador dog (object record 4), with an outcome that someone (perhaps someone identified in a column of the table that is not visible), is “happy”. Here the literal “happy” is inserted into the table, but one could have an adjectives table as discussed above (not shown), and record only links to the adjectives. One could also describe outcomes in terms of adverbs, and include either literals of the adverbs, or use links to an adverbs in an adverbs table (not shown).

[0176] The system could be made more complicated than simply using nouns, adjective, verb and adverbs. For example, in table 710, column 711.sub.6 is used for preposition objects.

[0177] The terms “events” and “outcomes” should be construed herein in a sufficiently broad manner to include correlations of symbols with objects, actions, events, and thoughts. Links are considered addresses of data objects. In record 712.sub.6, for example, one of the Apple logos is associated with an outcome of linking to the main web page URL for the Apple company. Similarly, a hypothetical bar code for XYZ company is associated with a link to its website, and a Wikipedia QR code is associated with a link to its website.

[0178] Of course, since this is a bottom-up, self evolving database, individuals can make whatever associations they desire. To demonstrate that aspect, the hypothetical bar code for XYZ company is also associated with a link to a competitor's ABC website. Similarly, the image of George Washington standing in a boat is stored as a symbol that represents multiple different things, a concept (pride in America), a historical event (revolutionary war), a web page, and a book, Washington: A Life, by Ron Chernow).

[0179] Thoughts can be handled in a manner analogous to events, with the main difference that thoughts are not necessarily correlated with specific real-life events. For example, whereas Table 710 of FIG. 7 might store information about a specific accident, a record of a thoughts table (not shown) might store the generalization that a car (noun) moving (verb) fast (adverb) and erratically (adverb) tends to result in accidents (outcome). Another record might store the generalization that music (noun) played (verb) loudly (adverb) over a long time (adverb) tends to result in deafness (outcome). A third example is that a record could store the generalization that regular (adverb) flossing (verb) tends to result in gum disease (outcome).

[0180] It is contemplated that generalizations or other thoughts could be stored more or less directly from parsing of communications with users, and/or result from automatic review of records in events table(s).

[0181] It is also contemplated that any characterizations and/or feature recognition facilities can be associated with one or more artificial intelligence systems. Artificial intelligence can include, but is not limited to, neural nets, machine learning algorithms, supervised learning classifiers, time-series forecasting, regression analyses, predictive analytics algorithms, and fuzzy logic.

[0182] For example, features recognition facility 400 can use a supervised learning classifier to identify different feature inputs and outputs and infer a function from labeled training data. It is contemplated that the labeled training data can be input manually by a user of features characterization facility 400. It is further contemplated that the features characterization facility 400 can preferably also receive crowd-sourced labeled training data, thereby quickening the pace and the depth of supervised learning.

[0183] In another example, actions characterization facility 600 can use a time-series forecasting to predict when one or more actions will likely occur. It is contemplated that actions characterization facility 600 can predict future events by observing the temporal ordering of a user to determine when the user typically takes an action. It is further contemplated that action characterization facility 600 can crowd-source time-series data to come to more accurate conclusions about typical behavior patterns in particular contexts, such as, for example, the fact that many individuals typically arrive at work around 9:30 AM in a designated geographic area.

[0184] In yet another example, objects characterization facility 500 can use a linear regression analysis to predict the relationship between multiple variables. It is contemplated that objects characterization facility 500 can predict that a user's keys are highly related to a car parked in the driveway. The object characterization facility 500 can further correlate the car parked in the driveway with a particular house and come to a conclusion that the car is highly related to the house. To take this example a step further, objects characterization facility 500 can determine that the car is associated with the house most frequently before 8:00 AM and after 5:00 PM on weekdays using time-series forecasting in conjunction with a regression analysis. As with the previous examples, objects characterization facility 500 can crowd-source data to draw more accurate conclusions about the relationship between two objects.

[0185] The present invention is not limited to the embodiments and examples discussed herein and can comprise any facility paired to any one or more artificial intelligence systems to assist in identifying and predicting the occurrence of objects, events, features, events, and any other measurable factor.

[0186] Real-World, Virtual World, and Mixed-Realities

[0187] FIGS. 3A and 3B should be interpreted as exemplifying any of real-world, virtual or mixed-reality environments. Thus, the orange 382 and apple 383 could be real or virtual, and independently the person 310 could be real or virtual. As used herein, the term mixed-reality should be interpreted to include augmented reality. Similarly, the tables shown in FIGS. 5A, 5B, 6 and 7 should all be construed as storing information that can have any combination of real-world, virtual or mixed-reality components.

[0188] Examples of Crowd-Sourcing/Sharing of Parameters and Identifications

[0189] Hopefully, one can now understand how the apparatus of FIGS. 3 through 8 can implement step 32C of FIG. 2. In a preferred embodiment, a person wears or carries equipment such as the camera 322, microphone 324, electronics module 330, and speaker 370 of FIG. 3. The camera images some object, as for example, orange 382. The image, or at least data derived from the image, is sent to feature recognition facility 400, which either identifies the object as an orange, or identifies features (round, shaped, smallish, orange) from which other software can associate the object with an orange. Unlike the prior art, however, where a company has trained an image recognition facility to identify all such objects as oranges in a top-down manner, the training here is contemplated to be accomplished by multiple users. Thus, upon first encountering an orange by any of the users, the system and user might use microphone 324 and speaker 370 to engage in a dialogue as follows:

[0190] System—What is this?

[0191] User—An orange.

[0192] System—How do you know?

[0193] User—It's round and orange and dimpled.

[0194] System—What kind of object is it?

[0195] User—It's a fruit.

[0196] That data would be entered into a table such as objects table 510, as shown in row 512.sub.1. If the system didn't already know that the adjective “round” referred to a shape, or that “red” referred to a color, the system might initiate a dialogue such as the following:

[0197] System—What sort of characteristic is “round”?

[0198] User—Round is a shape.

[0199] System—What sort of characteristic is “orange ”?

[0200] User—Orange is a color.

[0201] System—What sort of characteristic is “dimpled”?

[0202] User—Dimpled is a texture.

[0203] That data could be stored in a record such as 522.sub.1 of table 520. Of course, as time went on, the system would learn the words of characteristics, objects, and so forth. One key is that the image recognition facility is trained in a crowd-sourced and/or crowd-shared manner. Another key is that the image recognition facility identifies both object names (e.g., the name “orange”), characteristics (e.g., round, orange, dimpled), and parameters (e.g., shape, color, texture, etc).

[0204] Readers will appreciate that analysis of the words spoken by the user, and generation of queries and comments by the system would likely utilize at least rudimentary language parsing and creation facilities, preferably ones that consider both syntax and semantics. Such facilities are known in the prior art, and are not by themselves considered to be inventive herein.

[0205] In the case of actions, the image recognition facility would identify actions such as “walking” and “riding”, and also identify characteristics of those actions, including for example, speed. That information would be stored in tables such as 610 and 620. An exemplary dialogue might take place as follows:

[0206] System—What is the boy doing?

[0207] User—He is standing.

[0208] System—How do you know?

[0209] User—His feet are on the ground.

[0210] System—How is he standing?

[0211] User—Quietly.

[0212] Events can be stored in an analogous manner, using a table such as table 710 of FIG. 7. To store that information, an exemplary dialogue might take place as follows:

[0213] System—What is happening?

[0214] User—Martha Smith is reading a book.

[0215] Assuming the system could already recognize Martha Smith, the activity of reading, and the object as a book, the system might store the information as in record 712.sub.2 of table 710.

[0216] Readers might be skeptical that objects, actions, and events can be characterized with only a handful of characteristics sourced by question and answer interactions. But upon reflection, readers should appreciate that the process is analogous to how items, actions, events and thoughts can be characterized in the old twenty questions guessing game (“Is it bigger than a bread box?”).

[0217] It is also contemplated that a question and answer conversation can be initiated by a person. In one embodiment, a user can tell a personal companion to initiate a dialog in order to add characterizations about a particular object of interest. For example, a user can initiate a conversation about the high sentimental value of a picture drawn by the user's child that would otherwise be considered of little worth by the personal companion based on past conversations with the user.

[0218] In another embodiment, the user subject to a condition preventing speech (e.g., a user that is unable to speak, an environment where talking is not allowed, and a temporary inability to speak) can initiate a question and answer conversation using nonverbal cues, such as, for example, a hand signal. In this embodiment, it is contemplated that the personal companion can converse with the user verbally (e.g., through a speaker or headphones) or non-verbally (e.g., through a series of vibrations felt on the user's arm).

[0219] It is contemplated that a personal companion is not limited to purely verbal conversations and can, instead, converse with a user in any manner known in the art.

[0220] Crowd-Sourcing/Sharing Of Additional Information

[0221] Element 42C of FIG. 2 refers to crowd-sourcing/sharing of additional information. In a crowd-sourced example, a person might say to his personal companion something along the lines of “This ornament breaks easily”, or the personal companion might see the user break the ornament with little or no provocation. The characterization of breaking easily would be uploaded to the canopy, and made available to others. In a crowd-shared example, the personal companion of another person would see that the other person had picked up one of the ornaments in a store, and based upon passively received characterizations of others, the personal companion might say to the user “Be careful. That ornament breaks easily”.

[0222] FIG. 7 provides a specific example of how the characterizations could be stored. There, outcomes in column n are, or provide links to, the additional information. For example, in record 712.sub.7, the XYZ barcode is associated with a link to the URL for the XYZ company, and may well have been added to the system by an employee for the XYZ company. The association of the XYZ barcode with a competing company in record 712.sub.8, or with the literal “breaks easily” in record 712.sub.14, however, would have almost certainly been added to the system by someone else.

[0223] In another crowd-sourced example of obtaining additional information, a user might say to his personal companion, “Please tell others that the route 5 freeway is jammed at Katella”, or the personal companion might see that the traffic jam using its camera. If that information is made generally available to others, then the personal companion of another person might passively receive the information, see that its person was about to enter the freeway, and say something along the lines of “Are you sure you want to go this way? The freeway is jammed at Katella.”

[0224] It is important to note that although most of the examples herein are focused on deriving information from a microphone, camera or other sensor carried by a person, information could also be crowd-sourced/shared from stored materials, including for example Internet web pages, and county or other government records. For example, comments about a particular product could be crowd-sourced from blogs discussing the product. Thus, it is contemplated that a web crawler could be used to scour the Internet for information, using the canopy to provide context and suggestions as to how to characterize the information it finds, to establish links, and possibly even to conduct or take part in transactions.

[0225] Crowd-Sourcing/Sharing Of Transactions

[0226] Element 52C of FIG. 2 refers to transactions, which can be accomplished in a crowd-sourced or crowd shared manner.

[0227] In a crowd-sourced example, a person might ask his personal companion to find out whether any others are interested in purchasing a monster truck. And example of how this could be recorded is in FIG. 7. There, record 712.sub.15, in association with object record 510.sub.2 and action record 610.sub.6, shows an opportunity for 10 people to cooperate to each purchase a monster truck for $88,000. Many other examples are contemplated, including those where multiple individuals collectively purchase the same object, or perhaps collectively fund development of a company, commercialization of an invention, and so forth.

[0228] In a crowd-shared example, a personal companion might see that its person's vehicle is getting old, has high mileage, and is starting to break down. The personal companion might also have recorded oohs and ahs when the person viewed monster trucks. Based on those correlations, and correlations from that person or others that a new item is often purchased when an old item is wearing out, the personal companion might mention to its person that a given vehicle is available, or that others are banding together to make a multi-vehicle purchase. The personal companion would have known about the opportunity because it passively received that information from the personal companions of others, possibly funneled through a canopy.

[0229] In another situation, if a group of friends were planning a vacation together, and three of the friends just bought airline tickets, that information would be made available to a fourth friend in a crowd-sourced or crowd-shared process as discussed above. In a crowd-sourced aspect of a transaction, a person might ask his personal companion “What's going on with the vacation? If others purchased tickets, then please purchase one for me.” Or perhaps a person might say to his personal companion “Tell the others that I already bought the tickets.”

[0230] In a crowd-sourced example of the transaction, a personal companion might tell its person “Three others bought tickets already. You should buy one too.” Here again he personal companion would have known about the opportunity because it passively received that information from the personal companions of others, possibly funneled through a canopy. The personal companion might even not say anything to its person, and simply purchase a ticket.

[0231] Since personal companions can be context-aware, such interactions would preferably occur during an appropriate moment for the person, or perhaps triggered by something in the person's environment, such as a quiet moment, or perhaps when the person is looking at his calendar.

[0232] Storing Characterizations In The Canopy

[0233] Element 62C of FIG. 2 refers to storing characterizations in the canopy. This refers to the fact that records in the various tables can be generated by many different people, and yet everyone can have access to them. Although not readily apparent in this application, the ellipsis indicating other columns in the various table should be read to include, among other things, indicators of the source of the data, a time/date stamp, geographic location, social groups, and privacy indicators. In that manner the system can weight correlations properly to different users.

[0234] For example, an events table such as table 710 could store many instances of different people correlating the XYZ barcode or logo with Internet links and comment literals. Most or all of those correlations would be made available to everyone, and would preferably be weighted according to frequency, time/date stamp and so forth. Those associations would remain, as part of the canopy, at least during their various decay periods, even if the persons storing them died, or otherwise stopped using the system.

[0235] It is contemplated, however, that associations would be weighted. Weighting could be done in any suitable manner, but would preferably reflect who made the associations, how old the associations were, and the frequency with which a given association was made. Thus, associations made one day ago would be given greater weight those made a year ago, and associations made by hundreds of people would be given greater weight than those made by only one or two people. Also, associations made by a given individual would preferably be given greater weight for that individual than for others. This is how an apple to most people could be considered an apple, but the same apple to a produce manager might be considered a Macintosh.

[0236] FIG. 8 depicts a thoughts facility 800 in which multiple users 810a-810n of personal companions 820a820n, and their connections 830a-830n with a processing network 840, all in accordance with the canopy concept discussed above. At any point in time some of the personal companions will likely be passively observing their local environments, using previously crowd-sourced information to derive characterizations from those environments, but not interacting with their corresponding users at all. Other personal companions will likely be interacting with their corresponding users (sometimes in a question and answer mode), in effect training the system to crowd-source how objects, actions, events and thoughts should be abstracted, both in the present and in the future. Others will likely be using the data and/or characterizations from other personal companions, in a crowd-shared manner that does not require any action from their corresponding users to receive the information. Still others will likely be interacting with their corresponding users, making suggestions, providing warnings, and so forth, based upon characterizations about the their past or contemporaneous local environments, past or contemporaneous interactions with their users, and data and characterizations of others. All of that can be based at least in part on data and characterizations stored in the canopy.

[0237] As indicated elsewhere, depending on memory, computer power, and so forth, none, some or all of the steps listed above (recording, abstracting, and providing circumstance-relevant information) for any given personal companion can be accomplished locally, with the remainder accomplished distally (e.g., by a distal server or network), or even by recruiting processing power from other personal companions.

[0238] From the perspective of a given one of the users, thoughts facility 800 should be interpreted as comprising software operable by a user 810a in cooperation with a device 820a having a portable housing, within which is contained electronics configured by the software to: receive ambient data corresponding to an item 801 within an environment of the user 810a; and utilize (a) information derived from the ambient data and (b) an audible, at least partially computer generated, conversation with the person to assist in identifying the item.

[0239] Device 820a can be and/or include a cell phone, or at least sufficient electronics to conduct a phone call, and in other aspects can comprise a personal companion such as 300A in FIG. 3A or 300B of FIG. 3B.

[0240] The item can be a car, person, animal, building, coffee cup, or substantially any cognizable physical object, in the real world, a virtual world or a mixed reality world. Thus, for example, the item could comprise an electronic rendering of an object, such as an image on a TV screen or other electronic display.

[0241] The ambient data can include image data produced by a CCD, CMOS or other image sensor, and since the image sensor can be remote from the housing of the electronics, the ambient data can include image data received from a source external to the housing. Ambient data can additionally or alternatively include sound, biometric, olfactory or other types of data.

[0242] The electronics of device 820a is preferably configured to send information derived from the ambient data to a service distal (part of canopy 840) from the device. The information sent from the device 820a can comprise at least a portion of the ambient data, a matrix manipulation or other derivation of at least a portion of the ambient data, and/or textual or other characterizations of the ambient data. Characterizations could, for example, include color, size, movement, relative position, type of object (e.g. face), loudness, or rhythm. Additionally or alternatively, the information sent from device 820a could comprise a Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Feature (SURF), or other non-textual characterization of a feature of at least a portion of the object.

[0243] The electronics of device 820a can further be configured to receive a proposed identification from the distal service, and to relate the proposed identification to the person as part of the audible conversation. Sending and or receiving herein can be accomplished directly, by point to point communications, or indirectly via intermediate points. Relating of the proposed identification can occur in any suitable manner including for example, stating or suggesting.

[0244] The electronics of device 820a can further be configured to receive an address from the distal service, and to obtain additional information about the item by contacting the address. All manner of physical and electronic addresses are contemplated, including for example, a telephone number, an email address, a SMS address, and a social networking address. Similarly, all manner of additional information are contemplated, including for example, price, availability, options, physical characteristics, and consumer comments.

[0245] The electronics of device 820a can further be configured to receive a comment from the distal service, and render the comment as part of the audible conversation.

[0246] Device 820a can optionally include a screen, projector or other display apparatus 802a, and where the electronics is configured to receive image data from the distal service, the device can render the image data using the display apparatus.

[0247] The electronics of device 820a can further be configured to receive a command from the distal service, and to use the command to operate one of the other personal companions, or another device as for example a television, radio or a 3D printer.

[0248] The audible conversation can be of any practical length, but preferably includes a sequence of at least a first second utterance by the person, a first computer generated utterance, a second utterance by the person, and a second computer generated utterance.

[0249] The electronics of device 820a can further be configured to utilize a database (as for example in the canopy) to provide a characterization of the item, wherein the database associates parameters with values, and at least some of the parameters are crowd-sourced or crowd-shared, and/or where data within the database is crowd-shared.

[0250] Devices 820b through 820n should be thought of a similar to device 820a, and for example, could have corresponding display apparatus 802b through 802n.

[0251] Another set of novel concepts herein involves a personal companion or other electronic device being programmed to make use of an item of stored information on behalf of its user in a manner that requires no affirmative action by the user.

[0252] This could be accomplished further to the electronic device soliciting information from its user using an audible human-understandable conversation that includes at least some computer-generated speech, communicating the solicited information for storage in a hardware-based data structure; and obtaining the item of information from the data structure. In some embodiments the obtained item will be derived from (1) at least some of the solicited information and (2) other information solicited by, and uploaded to, the data structure by another person's electronic device.

[0253] The solicited information could comprises raw data and/or one or more characterizations of raw data.

[0254] There are many ways in which stored information can be used by a personal companion or other electronic device on behalf of its user person, in a manner that requires no affirmative action by the user. For example, the personal companion could use the stored information to recognize at least one of an object, an action and an event, in an environment about the person. Another contemplated use is for the electronic device to refrain from saying something in response to the personal companion recognizing at least one of an object, an action and an event, in an environment about the person. Other contemplated uses are for the electronic device to send an operating command to a piece of equipment, or perhaps an email or other form of communication to a person or other entity. Another contemplated use is for the electronic device to make a purchase, or conduct some other transaction, especially a transaction involving money.

[0255] Uploading, Storage and Deletion of Data

[0256] An events table such as table 710 could store millions of instances of different people seeing various fruits, or even many instances of the same person seeing a particular fruit. Most or all of those instances have no significance whatsoever, and should not waste space in storage. To facilitate that deletion function, it is contemplated that raw data, such as raw video or other image, audio, and biometric data will be saved for only a relatively short period of time, such as a few minutes or perhaps a few hours. The particular amount of time that raw data is saved is preferably controlled by the person wearing the local portions of the system, by the amount of local memory, by connectivity to Internet or upload path(s), and so forth. It is, however, also contemplated that the local portion of the system could upload raw data, so that, for example, the raw data of a audio/video clip of a car accident, or perhaps a particularly interesting portion of a sporting event, could be saved and shared with others.

[0257] Ideally, the local portion of the system, e.g., networkable electronics module 330, would have enough processing power and memory to store characterizations in local portions of the various characterization facility tables, such that for the most part only characterizations are sent to any distal facility for merging with characterizations of others. Records in both local and distal facilities are preferably subject to deletion based upon suitable factors, including for example redundancy and a date/time stamp. Where generalization or other thoughts records are abstracted from collections of events records, it is contemplated that at least some of the underlying events records can be deleted.

[0258] Persistency and Background Operation

[0259] One aspect of the inventive subject matter is that personal companions can have a high degree of persistency. In preferred embodiments a person's personal companion is active for at least 50%, 60%, 70%, 80% or even up to 100% of that person's waking day. It basically records a significant portion of the person's life, the sights that the person would see and the sounds that the person would hear. The personal sensor module would thus typically record encounters with friends and co-workers, driving experiences, cooking dinner, and so forth. Depending on the capability of the camera and microphone, the system could even record images and sounds outside of the person's native senses, such as ultraviolet or infrared light, or ultrasonic sounds. The networked electronics module could advantageously include a GPS or other geolocation facility, and record the person's location on an ongoing basis.

[0260] One advantage of both persistency and background operation is that information could be collected from many different people, and not only be almost immediately available to others, but actually be used by others without anyone intentionally storing or retrieving the information. For example, in the prior art it is known that a driver driving along a freeway might see that the road is jammed up with traffic, and post an image or video of the traffic jam to Facebook™, or send information regarding the traffic jam to Waze™ or Twitter™. Thirty people might see that post, and five write back with thanks for the warning. All of that, however, takes conscious effort on the part of those involved.

[0261] In contrast, a contemplated personal companion could have a camera that would persistently and automatically observe everything in its view, including the traffic. In preferred embodiments there is no need to repeatedly “point and click” or take other affirmative actions to capture data. The system could automatically abstract that the congestion constitutes a traffic jam, and upload that abstraction (perhaps along with still images or video) to the collective events facility. The personal companions of some others could also have observed their surroundings and actions, and might have generalized that they are about to take that same freeway, and would likely hit the same traffic jam. Their personal companions could then warn them, saying something along the lines of “The 57 freeway is jammed at Anaheim stadium.” Thus, unlike the prior art, the personal companion observations can be persistent, and abstraction, sharing and application of those observations can all be automatic.

[0262] Speed of Operation

[0263] It is contemplated that personal companions could provide near-contemporaneous characterization and/or abstraction of observed information. As used herein, the term “near-contemporaneous characterization and/or abstraction of observed information” with respect to a feature, object, action, or an event condition means that the characterization and/or abstraction takes place within 15 minutes of the sensor sensing the feature, object, action, or event being characterized or abstracted. As used herein, the term “contemporaneous characterization and/or abstraction of observed information” with respect to a feature, object, action, or an event condition means that the characterization and/or abstraction takes place within 5 minutes of the sensor sensing the feature, object, action, or event being characterized or abstracted. These terms are intended thereby to be distinguished from concurrent, simultaneous, coincident, and real-time, which are referred to herein as occurring within than five seconds of an occurrence.

[0264] Providing Information from Multiple Sources

[0265] It is contemplated that personal companions could associate locally derived information with information from any other available source or sources, including for example, information available on the Internet, identity of nearby business associates or personal friends, and information from books, blogs and other Internet sources. Thus, if a person asked for driving directions, the system might generate a query, which is then submitted to Google.com, Yahoo.com, Ask.com or Bing.com to determine the answer. One benefit of accessing search engines using a person companion is that the returned information could be stripped of advertising, and/or could be re-ranked according to according to individual's preferences and history.

[0266] The concept of asking a question or stating a command in natural language, and sending the question or command to a distal service to retrieve an answer or operate a device, has recently been embodied in Apple's Siri™. But those concepts were disclosed much earlier in the priority filing, PCT/US00/25613 to what is now US issued '867 patent, and pending applications US2012/0178432 (Fish, Publ July 2012), US2012/0179452 (Fish, Publ July 2012), US2012/0208600 (Fish, Publ Aug 2012) and US2012/0231843 (Fish, Publ September 2012).

[0267] It is contemplated that information stored by personal companions, whether raw data or characterizations, could be automatically posted to social networking and/or other sites. And personal companions might also monitor websites and other resources in the background, observing and abstracting from those resources.

[0268] Other Aspects of the Inventive Subject Matter

[0269] The following paragraphs describe some specific contemplated aspects of personal companions. Unless the context requires a contrary interpretation, each of the other aspects should be interpreted as capable of being independently implemented. Thus, even where a sentence says that a personal companion could do “x”, and the next sentence says a personal companion could also do “y”, the reader should interpret “x” and “y” as being independently implementable, not that the personal companion need do both “x” and “y”.

[0270] It should also be appreciated that each of the other aspects could be implemented along with one, or independently of crowd-sourcing and/or crowd-sharing.

[0271] Inconsistent or Incorrect Information

[0272] The use of self-evolving databases contemplated herein will naturally tend to store information that is inconsistent and/or incorrect. In addition, personal companions should be able to relearn or correct previous learning. For example, if a personal companion confuses dogs and cats, because both are furry mammals with tails and four legs, a user should be able to talk back and forth with his personal companion to add distinguishing features. Or if a personal companion is storing data the correlates particular characteristics with the name Mary, when in fact those characteristics refer to Jane, a user should be able to tell his personal companion to replace the name Mary with the name Jane. Still further, if a personal companion has stored characterizations, whether from its user or from others, that a poisonous plant is safe to eat, a user should be able to insist that his personal companion identifies that plant as poisonous. This can be accomplished using a weighting factor in the database for this particular user (see e.g., column 511.sub.11 in FIG. 5C).

[0273] Another consequence of using self-evolving databases as contemplated herein is that such databases can accommodate nuanced views of different users. For example, most people would view a supermarket orange as just an orange. But the produce manager would likely distinguish among a Valencia orange, a navel orange, and a blood orange. In another example, one person might view a plastic orange as an orange, while another might view it as a toy or a prop. Some people might characterize a book a being “fantastic” while another might characterize it as “awful”. And indeed, it may be that someone instructs his/her personal computer that a car is called a bicycle. Preferred apparatus, systems and methods herein can accommodate all of those situations because each personal companion could have its own data tables, and where records from multiple personal companions are stored together, the tables can include a source identifier (e.g. column 511.sub.7 of FIG. 5C).

[0274] Privacy

[0275] Users likely won't want the local instances of their personal companions to be observing their environment all the time. Among other things, users might want privacy in their homes, and might want to avoid recording images of copyright subject matter. It is contemplated therefore that local instances of personal companions, or at least one or more sensors of the personal companions, could be place into a sleep mode. This could be done, for example, by expressly telling a personal companion to go to sleep, or by pushing a button on the electronics module 330. Another option is for a user to tell his personal companion to sleep automatically, such as when the personal companion perceives the event of “getting into bed”, see row 612.sub.9 of FIG. 610 or between 9 pm and 6 am. Such preferences could be kept in a User Preferences table (see row 1 of table 900 FIG. 9), and could all be triggered by perception of objects corresponding to rows in table 510 and/or events corresponding to rows in table 610.

[0276] In some instances the user might want his personal companion to “wake up” after a given period of time, or perhaps upon sensing a keyword, sound or motion. In other instances, a user might want a sensor to be awake, but the local instance of their personal companion to preclude uploads or retrievals. Alternatively the user might want the personal companion to retrieve information, but to remain silent. These preferences could all be kept in a User Preferences table (see row 2 of table 900 of FIG. 9), and could all be triggered by perception of objects corresponding to rows in table 510 and/or events corresponding to rows in table 610.

[0277] In other instances personal companions could be programmed to anonymize information. That could be accomplished in many ways, including expressly instructing a personal companion to upload raw data and observations to the canopy without source identification tags, or uploading with a code that instructs the canopy to delete any uploaded source identification tags. For example, records in table 510 or 610 could be loaded with source identification data (see e.g. column 511.sub.7 in FIG. 5C) blocked out. In addition, a personal companion could provide assurances to its user that private data about their habits, day-to-day activities, personal thoughts, opinions, etc. cannot be traced back to them.

[0278] Dealing with Different Users

[0279] Based upon previous questions and answers with its user, a personal companion could automatically implement various preferences. For example, a personal companion could activate a blackout coating on its camera, or stop recording image and/or audio data at night, or during a movie. (See above discussion re privacy). As another example, one person's personal companion might focus on automobiles, abstracting what types and years of automobiles were viewed during the day, while another person's personal companion might substantially ignore automobiles and instead focus on identifying people he meets during the day. This can all be accommodated by personal companions each having their own data tables as discussed above, and where records from multiple personal companions are stored together, the tables can include a source identifier (e.g. column 511.sub.7 of FIG. 5C).

[0280] Focus could additionally or alternatively be implemented in a User Preferences table, as in row 3 of table 900 of FIG. 9, where various focuses include fruit, friends, co-workers, and events characterized as danger. The ellipses in column 6 should be interpreted as indicating that the table can have more than just seven columns. Another possibility for focus is that a user might instruct his personal companion to interpret the signs along a roadway, and notify the user when a certain exit is coming up, or perhaps when the speed limit changes.

[0281] Different personal companions could interact with their users according to different personality traits. Thus, one personal companion might interact in a strict manner, while another personal companion might interact in a very warm, kind manner. A personal companion might also change the way it interacts depending on circumstance, so that in a situation that is perceived as being dangerous, the personal companion interacts in a very terse, quick spoken manner, but while reading on a blanket at the beach, the same personal companion might interact in a slower, more lyrical manner. Similarly, personal companions could adjust the language (e.g. from Spanish to English) or the speech level (from adult to child) depending on the perceived characteristics of the person being spoken to. For example, a personal companion could be instructed to provide an infant with appropriate music or sounds in a caretaker mode, and an adult with adult music. Such preferences could be implemented using a User Preferences table, as in rows 4 and 5 of table 900 of FIG. 9 where Interaction Level is adult, and Language is English.

[0282] It is further contemplated that personal companions could have restrictions that accommodate demographics, preferences, or sensibilities of the users. For example, there could be a kids version that restricts the type of information, websites, etc., that can be accessed by a child user. As an example, personal companions could be instructed to automatically filter out adult images or other types of information.

[0283] Personal companions could also be designed to utilize an inside or outside service to effect translations from one language to another. Such preferences could be implemented using a User Preferences table, as in row 6 of table 900 of FIG. 9 where Translation Service is recorded as Google Translator.

[0284] Initializing a Personal Companion

[0285] Personal companions do not have to start from scratch. When a person buys a new personal companion, and uses it for the first time, it could already benefit from characterizations, preferences and so forth stored in the canopy, or otherwise previously stored by other users. The new personal companion could, for example, start out with records from other users that have the same Interaction Level and Language, and one or more of the same focuses. See User Preferences table, rows 4 and 5 of table 900 of FIG. 9.

[0286] Moreover, it might be very useful during initial setup or later modification for a personal companion to guide its user as to preferences by asking about preferences utilized by others. For example, a personal companion might say something along the lines of “Many other users have reminders to do something upon leaving the house. Do you want to set a similar reminder?”

[0287] Monitoring

[0288] A personal companion could monitor a child, older person or indeed anyone for signs of sickness, and report back with crowd-sourced phrases such as “She looks flushed”, or “She stopped breathing” based upon matching up perceived characteristics with crowd-sourced characteristics. Such characterizations could be maintained in the appropriate tables, e.g., objects table for “She looks flushed”, and an actions table for “She stopped breathing”. Personal companions could also monitor a child for some other condition, such as happiness or sleepiness.

[0289] A personal companion could similarly be used to monitor what goes on in a child's life at school, or at a park, or during a social interaction. In that case the parent's personal companion could direct the child's personal companion to look for certain things, and report back in specified ways, such as through the canopy, or perhaps directly by placing a phone call or sending an email. See for example, row 7 in the User Preferences table in FIG. 9.

[0290] A personal companion could also monitor a non-human object for some relevant condition, including for example, water overflowing from a bathtub, a stove being left on, a garage door being left open, or a package being left at the front door.

[0291] A personal companion could be used to record ideas, such as when a person is talking out load while taking a walk or driving a car. In such instances the user might expressly instruct the personal companion to retain raw data of audio and/or video for the next x minutes, or until further notice. The user might also specify that the personal companion should send the audio to a local device or service to be rendered as text.

[0292] A personal companion could also assist in short term or long term memory enhancement. For example, in cases where a user missed hearing something important in a conversation, or on TV or the radio, he could ask his personal companion “What did he say?”

[0293] A personal companion could be used to read pre-recorded text, such as that from an email, or a book. See for example, row 9 in the User Preferences table 900 in FIG. 9.

[0294] Even though preferred personal companions would likely be used to acquire data in a persistent or continuous manner, it is contemplated that they could instead, or from time to time, acquire data in an occasional or frequent manner. As used herein, the term “occasional” means that something occurs or is sampled at a rate of at least once a week, or at least a cumulative 2% of the time over a one day period. In contrast, the term “frequent” as used herein means that something occurs or is sampled at a rate of at least once a day during a five day period, or at least a cumulative 10% of the time over a one day period.

[0295] Most or all of personal companion interactions could be time shifted rather than done in real time or near real time. For example, one could retrospectively process input from a personal companion or even a simple wearable camera, to abstract and identify objects and events experienced on a skiing run. Where the camera was not part of a personal companion, the video could be fed to the personal companion, and then processed after the event by conversing with the personal companion. Similarly, at a conference where a camera is not appropriate, a personal companion could be used to record the live commentary, and interact with a user after later in the day. Where a personal companion is not available during the conference, a simple microphone and recording device could be used to record the commentary, which could later be analyzed using a personal companion.

[0296] Helping with Day to Day Tasks

[0297] A personal companion could couch individuals in memorizing a poem, a story, speech and so forth. It could listen to its user talk, compare the language with a correct or at least a preferred version, and then report back with crowd-sourced phrases such as “That was really good”, or “That still needs a lot more work” based upon matching up characteristics such as accuracy, speed, clarity, and so forth.

[0298] A personal companion could remind a user, or someone else to follow certain protocols, such as putting on a seat belt, taking off shoes when coming into the house or going upstairs, or taking vitamins or prescription drugs.

[0299] A person companion could assist fashion-challenged or color blind user with matching colors on clothes, or choosing wall paper or paints.

[0300] Aspects of Physical Devices

[0301] A person companion could have multiple cameras pointing in different directions, for example a back facing camera so that a user can be warned when he is about to step into a dangerous area, or about to sit on a whoopee cushion. Multiple cameras could also provide an extremely good stereo perspective, with better distance resolution than human eyes. Cameras could also capture images using wide angle, zoom and other views. Cameras could also have filters for different frequency bands, and could detect ultraviolet, infra-red or other frequencies outside human perception.

[0302] A person companion could have multiple microphones to assist in detecting the direction from which a sound is coming, or to zoom in on specific speech or other sound. Here again, data from microphones could be filtered for different frequencies, and could be used to detect frequencies outside of human hearing range.

[0303] As pointed out above, personal companions could be intimately associated with a cell phone or other communication device, and could use that device to receive and/or make calls. Thus, for example, a personal companion could automatically get on the Internet to order flowers or a Christmas present based on an action preference (see e.g., row 13 of the User Preferences table 900 of FIG. 9). Similarly, a personal companion could automatically get on the phone to call for an ambulance based upon an event such as Perla fell and didn't get up. As yet another example, a movie star or other person might set his personal companion to automatically send out tweets with characterization of events during the last few hours, thereby creating another way of “following” a person. That could even be used by law enforcement, parents, or others to keep track of given individuals without completely intruding on their privacy.

[0304] Personal companions could be electronically, auditory or in some other manner coupled to mechanical or electronic effectors, such as automobiles, robots, wheelchairs and so forth. Thus, for example, if a personal companion characterized it's user as being drunk, it might prevent the user's car from starting. Such actions could be set as preferences, as for example in row 13 of User Preferences table 900 in FIG. 9. As another example, a personal companion could characterize it's user as too hot, and turn on a cooling fan.

[0305] Personal companions would preferably be worn in a manner that obscures their presence, or at least makes them non-obvious. Thus, for example, the sensor module 320A in FIG. 3A is made to appear as an ornamental pin.

[0306] Answering Specific Questions

[0307] Since personal companions inherently utilize crowd-sourcing to obtain information, they can readily be used to summarize what correlations others have made. For example, a user might ask his personal companion a question such as “How many people associate xyz product with breaking easily?” That would be simple query to put against a database table such as table 710 in FIG. 7 (see especially row 14), and could be enhanced using a semantic equivalence dictionary.

[0308] Since personal companions could persistently or even continuously characterize object and actions, they might well be able to answer questions such as “Where did I leave my watch?” or “Where is Julie”? That functionality could be especially facilitated by appropriate entries in a User Preferences table, such as row 11 of the table of table 900 of FIG. 9. As with other data in the User Preferences table, the designation to keep track of the wristwatch could most efficiently be entered into the table using a verbal question and answer exchange between the user and his personal companion.

[0309] Personal companions could also be of great assistance to people with faulty eyesight or hearing. For example, a blind person could pick up an orange, and ask his personal companion what it is. A deaf person might ask questions of his personal companion, and get the answers written out on a display screen.

[0310] Reminders

[0311] Although the world already has several excellent calendaring and reminder programs, one advantage of a personal companion is that it could be context-aware, and trigger reminders accordingly. For example, a personal companion might remind its user to turn out the lights, heat, air conditioning etc upon leaving the house. An example is in row 12 of the User Preferences table 900 of FIG. 9. As another example, a personal companion might ask its user if he wants to open the garage door when the user's car is about a block from home.

[0312] A personal companion could also remind its user of annual or monthly events, as for example birthdays or holidays. Prior art calendars can do that as well, but only if they are specifically instructed by the user or someone else to do so. A personal companion could infer what reminders to calendar, as for example by observing that there is a birthday around June 9.sup.th of every year.

[0313] Recommendations and Warnings

[0314] There are many systems currently in place that allow individuals to make recommendations to others. For example, many web sites solicit and make available recommendations regarding plumbers, doctors and other workers, many other sites that do similar thing with respect to restaurants, vacations, and so forth.

[0315] Personal companions can go one step further, by virtue of their being context-aware. Thus, if a person tells his personal companion that he is going to go to xyz restaurant, the personal companion could respond that the line there is at least a half hour, based upon observations of other personal companions of people standing in line. Then based on similar information in other restaurants, the personal companion might say “Would you consider going to ABC restaurant instead. The wait there is only 15 minutes.”

[0316] A personal companion might also make a vacation recommendation, such as saying “Don't go to Julien today. It's too crowded and overcast” based on the user's prior characterizations of vacations as being “too crowded” or “overcast”, and based upon crowd-sourced characterizations from other personal companions that are already in Julien.

[0317] Along similar lines, a personal computer might make purchase recommendations based on the characterizations of others. For example it might say “There are 20% negative characterizations of the xyz product, and 80 positive or neutral characterizations.” As another example, a person shopping in a store might pick up a jar of peanut butter, only to have his personal companion say something along the lines of “52% of people characterized that product as being delicious”, or “Ralphs is selling that same product today for $2.39.”

[0318] Similarly, based upon its user characterizing many books as being “too hard”, a personal companion might advise that a book being held by the user at a bookstore “has been characterized by many others as being too hard.” A similar situation could exist with respect to a personal companion advising that a TV show or movie is “too violent” because others have made that characterization.

[0319] A personal companion could focus on recording and characterizing biometrics of its user, for example, blood pressure, heart rate, temperature, etc., and warn when they get out of whack. In a similar manner, a personal companion could recognize different colors of sputum or other body fluids, and/or interpret coughing sounds, and then ask appropriate questions regarding additional signs or symptoms, and make appropriate recommendations for treatment or further evaluation. The recommendations could be for individual doctors or other professionals, for hospitals or other practices, or even for telemedicine individuals or teams that the user might never visit in person. The recommendations could be based at least in part on characteristics and preferences of the personal companion user, including for example, age of doctor, gender, estimated cost and so forth. Of course, as with most everything else about personal companions, determination of situations for which warnings and recommendations are appropriate, and determining what warnings and recommendations to make are preferably crowd-sourced.

[0320] Once a user has gone to a doctor or some other professional, personal companions could crowd-source feedback as to those experiences, and in particular could prompt for feedback based upon feedback given by others. For example, a personal companion might say something along the lines of “Dr. Smith was rated by many others as being very thorough. Do you agree?”

[0321] The crowd-sourced data discussed herein could certainly be mined by researchers to help correlate signs and symptoms with diseases, and treatments with outcomes. Similarly, crowd-sourced data could be mined by police or other authorities to track down criminals.

[0322] A personal companion might also point out non-medical aspects of the user's appearance, and warn where there a likely problem, such as a shirt inside out, lipstick on wrong, hair needs cutting, too much perfume, or excessive or unusual body odor. The latter items would of course require either an odor sensor, or stored characterizations as to crinkling of nose or other features that might be seen in the faces or actions of others. A personal companion could also warn of current situations, such as low gas in an automobile, or past occurrences such as leaving home with the stove still on. See for example, row 10 in the User Preferences table in FIG. 9.

[0323] Still other contemplated examples that rely on a personal companion's ability to be context-aware include warning that plugging in the TV could overload the circuit, warning that furniture in a showroom would be too large to fit through the doorway at home, or too large or wrong color for the room, warn of potential drug interactions when a person is opening several different pill bottles, warning that lifting a given weight in a gym is too much, or warning that a given food contains too many calories, or may well cause an allergic reaction.

[0324] Other contemplated examples, which would involve characterizations from personal companions of other people, include warning that a baby at home has been crying for more than an hour, a son or daughter is leaving a party with four other kids, or with alcohol in the car.

[0325] A personal companion could view food that a person is eating, or is being served, and advise a user with respect to allergies, and special diets such as sugar free or gluten free diets.

[0326] A personal companion could project future events, such as projecting future traffic patterns based upon historical trends, and estimates probabilities of events occurring. For example, a person companion might say there is a 50% chance of an adverse occurrence if the user walks alone, at night, down a given street.

[0327] A personal companion could infer rules, or crowd-facilitate them from others. Thus, a driver new to a given country might not know a particular rule, such as a default speed limit. But his personal companion could discover that information based upon observation of the user's environment (e.g., driving a car on neighborhood streets) and inputs of others through their personal companions.

[0328] A personal companion could control equipment or make recommendations based on inference about a person. Thus, even though a house thermostat is set to 72° F., a personal companion might cause the setting to be raised, or recommend that the setting be raised, based upon an inference that a person in the room in older, and tends to get cold. Similarly a person companion might recommend that an older person eat dinner earlier in the evening based upon correlations made by personal companions of others that older persons do better when they eat dinner at an earlier time.

[0329] A personal companion could see that its user is about to send a nasty email or text, or post a mean comment, and then point out to its user that the user appears to be in a bad mood, and might want to change the content or delay sending the message. A personal companion could also advise the recipient's personal companion that the sender was in a bad mood.

[0330] Commercializing Personal Companions

[0331] One potentially lucrative avenue for making money from personal companions is to charge their users, directly or indirectly, for access to video, audio or other content. Thus, a user might ask his personal companion to listen to a song or view a video, and the personal companion might report back that the media or other information is accessible, but only upon payment of a fee.

[0332] Another avenue for making money is to charge for preventing data from getting onto any of the crowd-sourced/shared databases contemplated herein, or for removing such data.

[0333] Another avenue is to gain revenue by pushing advertisements or recommendations to users through their personal companions. Or from the other side of the coin, revenue could be realized by charging users to not receive advertisements or recommendations.

[0334] Non-Human Applications of Companions

[0335] The systems, methods and apparatus discussed herein are not necessarily directed to human users. One could have an animal companion, which might be affixed to a collar or other clothing of a dog, cat, horse, monkey, bird etc. Although the animal would have far less sophisticated communication capability that most humans, an animal companion could still observe the world about the animal, monitor biometrics, and talk to the animal in a human or animal language. For example, if a dog were trained to drop things that he put in his mouth using the phrase “drop it”, the animal companion could verbally instruct the dog to drop something that he shouldn't be chewing on. The dog wouldn't be able to effectively teach the animal companion, but people in the company of the dog could talk to the animal companion to provide training. Moreover, crowd-sharing could provide an ever increasing wealth of characterizations made by others (human or otherwise).

[0336] One could also have companions for inanimate objects, as for example cars, boats, planes, trains, buildings, and robots. In those instances the companions could monitor the environment about object, and also possibly temperature, voltage, and other conditions of the object. Companions could also interact with people nearby the object. For example, a car companion might be taught to recognize people leaning on the car, and warn them to stay away. Some of the interactions with nearby people could provide training to the companions.

[0337] Epilog

[0338] It should be apparent to those skilled in the art that many more modifications besides those already described are possible without departing from the inventive concepts herein. The inventive subject matter, therefore, is not to be restricted except in the scope of the appended claims. Moreover, in interpreting both the specification and the claims, all terms should be interpreted in the broadest possible manner consistent with the context. In particular, the terms “comprises” and “comprising” should be interpreted as referring to elements, components, or steps in a non-exclusive manner, indicating that the referenced elements, components, or steps may be present, or utilized, or combined with other elements, components, or steps that are not expressly referenced. Where the specification claims refers to at least one of something selected from the group consisting of A, B, C . . . and N, the text should be interpreted as requiring only one element from the group, not A plus N, or B plus N, etc.