Gestures including motions performed in the air to control a mobile device

Abstract

A cellular telephone or mobile device with several methods of touch, voice, and gesture based input is described. A user is able to interact with a touch screen display on the device to select one or more keys, expand keys, and customize legends or toolbars. A user may deliver spoken audio to the device and actuate a series of commands on the device including search on the device, search on the Internet, accessing an Internet resource, or downloading a document. A user may be able to execute one or more macros on the device. The device may further have a plurality of authentication methods based on the activities on the device. Authentication requests may be made for specific applications or the device. An authentication request may include identifying pictures taken on the device and engaging in activities with the photos.

Claims

1. A method including a mobile device comprising: capturing, by the mobile device, at least one gesture performed in the air; identifying the at least one gesture; linking the at least one gesture to a command for the mobile device to perform; providing a customized input entry upon viewing a message with a contact, the customized input entry comprising of one or more suggested frequently used phrases specific to a location of the mobile device; providing a plurality of keyboards, wherein at least one of the plurality of keyboards is configurable to be in a fixed position and at least one of the plurality of keyboards is configurable to be in a movable position; and wherein the mobile device is enabled to accept a user request to drag and drop at least one of the plurality of keyboards to a desired position.

2. The method of claim 1, wherein the at least one gesture is performed by a physical contact of a finger of the user with a touch screen display of the mobile device.

3. The method of claim 2, wherein the at least one gesture includes moving a current window in a foreground to a background position and a background window to the foreground position.

4. The method of claim 1, a plurality of shapes are enabled to be configured by the user and coupled to one or more actions.

5. The method of 1, wherein the at least one gesture comprises of a left swipe motion, and wherein said at least one gesture is configured to shift a window to be displayed out of a user view.

6. The method of claim 5, wherein the at least one gesture comprises further of a right swipe motion and wherein said right swipe motion is configured to position a window to be displayed in a predefined portion of the right side of a view.

7. The method of claim 6, wherein a plurality of windows are displayed at the same time.

8. The method of claim 7, wherein the plurality of windows are displayed in roundtable like format.

9. A non-transitory computer readable medium that when executed by a processor of a mobile device is configured to: capture by the mobile device at least one gesture; identify the at least one gesture; link the at least one gesture to at least one instruction for the mobile device to perform; provide a suggested input entry upon viewing a message with a contact, the suggested input entry comprising of one or more frequently used phrases specific to the a location of the mobile device; wherein a first portion of a keyboard is configured to be displayed in a first position and a second portion of the keyboard is configured to displayed in a second position simultaneously on the mobile device; a plurality of keyboards, wherein at least one of the plurality of keyboards is configurable to be in a fixed and at least one of the plurality of keyboards is configurable to be in a movable position; and wherein a user is enabled to drag and drop at least one of the plurality of keyboards to a desired position.

10. The non-transitory computer readable medium of claim 9, wherein the at least one gesture comprises of a left swipe motion and wherein said at least one gesture is configured to shift at least one of the plurality of windows of the display to the left.

11. The non-transitory computer readable medium of claim 10, wherein the at least one gesture further comprises of a right swipe motion such that the right swipe motion is configured to position a display of at least one of the plurality of windows to a right side.

12. The non-transitory computer readable medium of claim 11, wherein at least two or more of the plurality of windows are displayed at the same time.

13. The non-transitory computer readable medium of claim 12, wherein a plurality of windows are displayed in roundtable like format.

14. The non-transitory computer readable medium of claim 12, wherein the gesture comprises of physical contact with a display of the mobile device.

15. The non-transitory computer readable medium of claim 12, wherein the at least one of the plurality of keyboards is configurable to be in a fixed or movable position is a floating keyboard.

Description

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

(1) The accompanying drawings, being incorporated in and forming a part of this specification, illustrate embodiments of the disclosure and, together with the description, serve to explain the principles of the present disclosure:

(2) FIG. 1 is an embodiment showing an example of an authentication request on the screen of the device using a history of the interaction of a camera application.

(3) FIG. 2 is an embodiment showing an example of an authentication request on the screen of the device based on the call history.

(4) FIG. 3 is an embodiment showing an example of a multi-level authentication request.

(5) FIG. 4 is an embodiment showing an example of an authentication request in conjunction with a CT/MD and server.

(6) FIG. 5 is an embodiment showing an example of sending data to the CT/MD and receiving data from the CT/MD.

(7) FIG. 6 is an embodiment illustrating an example of collaboration across multiple devices using text, signature, and graphics.

(8) FIG. 7 is an embodiment showing an example of CT/MD with multiple keyboards spaced at different positions on a touch screen display.

DETAILED DESCRIPTION OF THE INVENTION

(9) The present disclosure enables a key on the device may be expanded based on a user actuating a region of the touch screen CT/MD. An application on the CT/MD may expand a key on the CT/MD in order to highlight a specific key that is depressed or should be depressed. A key may be expanded in size by a configuration setting on the CT/MD. The full keyboard may be expanded or compressed when a first key is actuated. A second key may be expanded in response to a first key being expanded to facilitate easier actuation of another key. The second key may be an expanded version of the first key and display simultaneously with the first key being displayed. A second key may be expanded if it is a suggested next character.

(10) The present disclosure further allows users to see a variety of compressed or expanded keys based on configuration settings. A user may enter a desired key expansion amount or set a percent expansion of a keyboard.

(11) The present disclosure further allows individuals to use a variety of compressed or expanded keys based on contextual or situational factors. A key may be expanded based on the ambient light detected by a light sensor on the device. In periods of high intensity light, keys may be expanded to take up a greater area of the viewable screen to facilitate easier viewing. A key may also be expanded in response in response to moisture, which may result from perspiration by the individual or humidity.

(12) The present disclosure further allows a user's attention to be directed to certain activities and actions. A key, button, or icon on the CT/MD may flash to attract a user's attention. A key, button, or icon may be highlighted in a first or second color to attract a user's attention and serve as a notification. Notification messages may be flashed on the screen or appear translucent on the screen. A selected key may appear in a first color and the full keyboard may appear in a second color, appear in a color of less intensity or appear translucent.

(13) It is an aspect of the present disclosure to enable the server to influence or control the keyboards, menu items, icons, notifications, and toolbars on the device. The device may be configured to change an icon, keyboard, menu item, or legend in response to a message, data, or other information received from a server.

(14) There may be a hierarchy of icons or macros configured in the device. Icons may be group together into one or more menus. For example, two or three icons may be associated together and placed into a group or menu. This group or menu may be given a new name or icon for actuation. This set may then be grouped with additional icons. Icons may therefore be cascaded in a hierarchy of use on the device. A user may create a new icon using a drawing program. A user may use a photo as an icon. A photo may also be used as a macro function. For example, a first photo of a contact may be placed on the desktop of the device. Actuation of this first photo may call the contact's home number. A second photo of the contact may be placed on the desktop of the device. Actuation of this second photo may call the contact's mobile phone number. These photos may be similarly associated with a hierarchy. Selecting the first photo may run a macro to execute a series of functions. For example, actuation of the first photo may call the home number and if there is no answer may subsequently call the contact's mobile number, send a text message or compose an email.

(15) Authentication:

(16) It is an aspect of the present disclosure to enable the device or a network server to use multiple authentication methods in conjunction with one another to authenticate a user to a device. A user may be required to satisfy a plurality of input and output methods or pass a percent threshold of authentication methods in order to satisfy an authentication requirement. A server may track the number of authentication methods used in order to grant a match between the inputted data and stored values. The server may require authentication responses on multiple devices in order to authenticate a new device or existing device.

(17) The device may also accept voice based input to a macro. The macro function may execute a series of commands. A macro may be spoken to the device. The device in response may convert the spoken audio to text or a data command. The command may execute an application on the device. The application on the device may contact a server to process the request, query another server, query an Internet service, search the Internet, download data from a database, or process a calculation. The server may return the raw data to the device or a synthesized response to the device. The device may accept a spoken command to access a data file on a server. The device may accept a command to control other devices such as a garage door opener.

(18) The escrow server may be a identity management server, authentication server, digital certificate server, public key/private key encryption server, policy server, or other server.

(19) FIG. 1 illustrates the display 701 of a CT/MD 700. The CT/MD 700 presents an authentication request on the screen of the device. The device uses the history of the interaction of a camera application to determine the authentication request 702 to the device. The CT/MD or a server may record in a table, database or other storage means which picture was taken using the camera on the device. Additional meta data about the image taken may be accessed from the image itself which may be encoded with location, GPS coordinates, time of day, device used for image capture, or other information. The CT/MD or server may determine which image is received by storing in a data table meta data about received images. Alternatively, a user may enter in this data in an authentication setting. Alternatively, a server administrator may configure images for authentication. A user may be asked to identify which image was taken 703 using the camera 704 on the device and which image was received 705. The user may indicate a response by spoken words using phrases such as ‘image on the left’ or by touching an area of the display 706 corresponding to the correct response using a finger or pen.

(20) FIG. 2 illustrates the display 801 of a CT/MD 800. The CT/MD 800 presents an authentication request 803 based on the call history. The call history may be stored in memory on the device or a database 804 located on the device or on the server. The CT/MD or server may present a series of questions to the user seeking access to the device around what calls were made from the device. As an example, a user may be asked which of the last 3 numbers or contacts was called or when a call was initiated to these numbers. This would correspond to an lower security level since the CT/MD or server is providing information to the user. A more stringent security setting may require a user to enter in a phone number called recently, enter the name of a contact recently called, or select a contact listed as a favorite or family member. Alternatively, a user may be asked questions around the frequency with which they call a specific number. This information may be provided in a range (daily, weekly, monthly, rarely) to make it easier for the user to answer the question. The call history may also be linked to a location. For example, a user may be asked to answer where they often call their home number. One of the options may require a user to type in the street name or city or identify a location on a map. A user may often call home from the office but may be unlikely to call home from their home location. These settings may be modified for higher or lower security levels. Questions may further be modified so as to provide information as part of the authentication response or not provide information based on the degree of security required. Similarly, the authentication requests may be a secondary authentication request after a higher or lower security authentication request. Alternatively, the authentication request may for a specific application. A corporate application, email, or credit card related application may necessitate a higher level of security while a media application may not. A phone number itself may be associated with an authentication item. As an example, various contacts may be associated with high or low security levels or various restrictions. Individuals may often need a single device instead of a work and personal device. Correspondingly, the work and personal contacts and numbers may end up on the same device. A user may wish to disable contacts associated work during non business hours. The CT/MD may allow for a user to enter a separate authentication method for certain contacts such as office contacts based on the time of day or other setting.

(21) FIG. 3 shows a multi-level authentication request on a CT/MD 900. An application may be executed in response to contact or selection from a finger, pen, or other object made with the icon 901 on the display. The authentication request may grant access to both the device 900 and the application 904, only the application 904, or only the device 900. The authentication request may be based on a history item related to the application 904 for which access is requested or for another application such as messaging 903. A history item may include any activity that occurred on the device. As an example, the name, date and time an application was installed on the CT/MD may represent a historical item. Alternatively, the last website accessed on the browser may be used as a history item.

(22) FIG. 4 illustrates an authentication request in conjunction with a CT/MD 1000 and server 1001. The server may request a type of authentication request of the device. An application that requests a streaming music service from a server may be required to satisfy an authentication request 1003 defined by the server 1001. The streaming music service may be initiated by making contact with the icon 1004. The device may present the authentication request in a textual or graphical form 1005 on the CT/MD 1000. The authentication request 1003 may require the use of historical usage data of device 1000, server 1001 or escrow server 1006. The authentication request 1003 may contain information that requires a user to enter information on the CT/MD 1000 to confirm physical possession of the CT/MD 1000. The server 1001 or escrow server 1006 may define the number of authentication requests required. The authentication request may further require a plurality of authentication responses across a second electronic device 1007.

(23) FIG. 5 shows the server 1101 sending data to the CT/MD 1100 and receiving data from the CT/MD 1100. The icons 1102, 1103, 1104, or macros 1105 may be configured standalone or in conjunction with the server 1101. These icons may be modified in real time based on data from the device or server. The macros, icons, legends or other items may change in response to data from the server 1101. After receipt of the data 1106, the CT/MD 1100 may show different icons 1107 or different macros 1108 with pictures, numbers, shapes, highlights, or blinking indicators. Names of icons 1101 may also change. A physical key 1111 may also be redefined based on a configuration setting on the device 1106 or on the server 1101.

(24) FIG. 6 illustrates collaboration across multiple devices using text, signature, and graphics. The server 1200 facilitates the transfer of data between multiple CT/MDs 1201, 1202, and 1203. The server 1200 receives a message 1204 from a device 1202. The server sends the message 1204 to another device 1203. The user may respond to the message on the same device 1203 or another device 1201 where the response is sent first to the server 1200 and then to other CT/MDs 1202 and 1203. The server 1200 stores the data for the messages in the database 1205. The message content may also include video or images captured using the camera 1206.

(25) Using the camera, the participants may engage in peer to peer video conferencing or server mediated video conferencing. A fingerprint image may be acquired by a fingerprint scanner 1207 located on the device or by a camera located on the device to authenticate a person or new device. The CT/MD or server may store digitized samples of finger print data in a look up table for a person. The CT/MD may also acquire an ink based image of a fingerprint using the camera and process the image for fingerprint recognition either on the CT/MD or on a server. A new device may be synchronized with the server and download the shared data after providing a positive response to an authentication query.

(26) The device may be further configured with the ability for a user on a first device to send a signature request for a document 1209 to a contact in the contact list of the device. The document 1209 may also include purchase or transaction requests. For example, a purchase order may need to be electronically signed. The request may be sent to a specific device using, e.g., a text message. The document 1209 may also be sent to a specific email address.

(27) A user may set a hierarchy of signatures required to complete the approval of a document. For example, the document may be first required to be signed by a first individual. A second or third individual may be required next. Lastly, a fourth individual may be required to complete final approval of the document 1209.

(28) A user may define a time period under which a signature 1208 should be completed.

(29) The document 1209 to be signed electronically and supporting documents may be stored on the device or on a server. A request including an Internet accessible link to the document 1209 may be sent from a user on a first device to other users on other devices. A user on a second device may download the document 1209 and select an area of the document 1209 to annotate or sign. The document 1209 may be annotated with markers to identify areas for the signature. The document 1209 may be annotated with notes. The CT/MD, Server, or other electronic device may collect the responses, process the signatures including for handwriting recognition, and notify the other devices when the process is complete. The finalized signed document may not be published to the other devices until all the signatures are collected.

(30) The signature may come from the user actuating with the screen using strokes of the finger or a stylus on a specified region for a signature.

(31) A document may require a second authentication method in addition to a signature 1208. The second authentication method may include a picture of the individual obtained by a camera 1206 on the device, a fingerprint reading 1207, a GPS location, a phone call to an online system, a response to a text message with a password or identifier, an email to a specific address, or another method.

(32) The users may be notified using email, text message, voice message, or another method when all the signatures are complete.

(33) The CT/MD with the server can enable a plurality of functions including document viewing, sharing, annotating, modifying, editing the text document, graphics, image, video, audio, sound and data and or combinations thereof as a collaborative product 1210. The collaborative product 1210 between users may be inclusive of text, voice, sound, audio, image, video, graphics and data; As an example, a collaborative product may include all the pictures taken on a given day and further annotated with comments on each picture.

(34) The collaborative product 1210 may be accessed in a contemporaneous manner and/or at a later time by one or more users. The collaborative product 1210 may be enabled for maintaining revision control by the central server or a designated intelligent keyboard user. The collaborative product 1210 may be enabled for time stamping the access and actions of one or more parties to the collaborative work product in progress. The server may maintain electronically valid signatures images and authentication files for each user that is party to the transaction or signature event.

(35) The CT/MD may feature a signature window and enable witnesses to corroborate the event. Certain witnesses that have obtained appropriate approvals may notarize the document electronically. Other witnesses may have associated with them various credibility ratings. A witness may also sign as to observing the signature in either the physical presence or by watching the signature in real time in an online fashion. The witness may also be recorded using the camera or video camera on the CT/MD. As an example, the video camera on the CT/MD may record or stream the signing event to another CT/MD or server. The other CT/MD or server may store the signing event for future playback, auditing, or other purposes. Alternatively, the camera on the CT/MD may take snapshots while the signature event is taking place. An audio recorder integrated into the CT/MD may record sounds that are taking place at this time. The CT/MD may further record the GPS location of the signature event. The signature and associated meta-data related to the event may be archived to the signature document and stored locally on the CT/MD or on the server.

(36) The collaboration event may further be enabled to record negotiation and requests by one or more parties. The collaboration product 1210 may be marked as pending or as a final version. Parties may state by selecting a touch box, an accept button, or other marker to indicate agreement.

(37) The parties privy to the collaboration event may be required to agree to the event at a specified time, in real time or simultaneously.

(38) The server may compare the signatures, signature images, signature strokes, time to sign, signature movement across the touch screen, to stored samples located on the device or a server. The CT/MD or server may perform handwriting recognition using a software program.

(39) There may exist a plurality of servers enabled to stored one or more samples of the data. A server may function as an escrow server whereby the server holds authentication information or collaboration information for a specified time or rule.

(40) The authentication system may require a plurality of authentication methods simultaneously, sequentially, temporally spaced, or geographically spaced using GPS information. One authentication method may require a voice reading while a second authentication method may require a password being entered on the device. A server may require authentication to a service by requiring a user to enter a password on the CT/MD and a code delivered by the server to the CT/MD. The server may also require authentication by requiring a user to enter a password on the CT/MD and then a password obtained from the CT/MD while the password is entered. An image may compared against a stored sample on the device or on the server. The image may also include a retinal scan that is compared against a stored sample.

(41) The server or CT/MD may also require and obtain GPS location information for the phone during an authentication event. The server may correlate a plurality of authentication information to determine whether the collaborator is in proximity to another collaborator. The server may use an IP address to determine location. The server may also store and correlate IP addresses for authentication purposes.

(42) FIG. 7 illustrates a CT/MD 1300 with multiple keyboards 1301, 1302, 1303, 1304, 1305, 1306, 1307, 1308 spaced at different positions on a touch screen display. A user may define the ideal spacing in a configuration file. Alternatively the CT/MD or server may contain ideal spacing information. An individual's hand size and finger size may be used to create an ideal spacing of the keyboard. The finger size may be measured during a calibration process where the user places their fingers on the touch screen and the CT/MD determines the points of contact. Alternatively, a user may describe their hand type such as small, medium, or large and the system may use the associated keyboard spacing. The CT/MD may also download various keyboard spacing from a server. This spacing allows for greater vertical separation between the rows of a keypad. A horizontal separation may also be enabled or different combinations of split keyboards groups of keys, or specific keys may be configured. Keyboards 1302 and 1308 feature greater horizontal and vertical separation than a full numeric keypad organized as a row. Similarly macros or icons can be split across different areas. The location of the keyboards may be configurable by a user. The keyboards may allow for easy access to keys to prevent inadvertent selection of the wrong character. A first keyboard such as the numeric keyboard 1302 may be located at the top of the display and may appear or disappear based on the application the CT/MD. The application 1309 as shown is a spreadsheet application where character entry and numeric entry may both be required by a user.

(43) The keys in the keyboard typically could be LCDs for displaying the respective legends, and desirably are touch sensitive.

(44) The foregoing descriptions of specific embodiments of the present disclosure have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the disclosure to the precise forms disclosed, and it should be understood that many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the present disclosure and its practical application, to thereby enable others skilled in the art to best utilize the present disclosure and various embodiments, with various modifications, as are suited to the particular use contemplated. It is intended that the scope of the disclosure be defined by the Claims appended hereto and their equivalents.