Interactive control device and method for operating the interactive control device
09671867 · 2017-06-06
Assignee
Inventors
- Katharina Bachfischer (Düsseldorf, DE)
- Christoph Waeller (Braunschweig, DE)
- Volkmar Wagner (Berlin, DE)
- Rainer Dehmann (Berlin, DE)
Cpc classification
G06F2203/04805
PHYSICS
G06F3/017
PHYSICS
B60K35/50
PERFORMING OPERATIONS; TRANSPORTING
G06F3/04812
PHYSICS
B60K2360/143
PERFORMING OPERATIONS; TRANSPORTING
B60K35/60
PERFORMING OPERATIONS; TRANSPORTING
G06F2203/04108
PHYSICS
B60K2360/141
PERFORMING OPERATIONS; TRANSPORTING
B60W50/10
PERFORMING OPERATIONS; TRANSPORTING
B60K35/00
PERFORMING OPERATIONS; TRANSPORTING
B60K35/10
PERFORMING OPERATIONS; TRANSPORTING
International classification
B60K35/00
PERFORMING OPERATIONS; TRANSPORTING
G06F3/0488
PHYSICS
G06F3/0481
PHYSICS
Abstract
An interactive control device includes a display device, and a method is for operating the interactive control device. The method includes: displaying graphical information on the display device; receiving sensor information; activating a control action if on the basis of the sensor information it is ascertained that a body part of a user is located within an activation region that is spatially defined relative to a display region of a control element on the display device with which the control action is associated; the received sensor information including user information that is evaluated prior to an activation of the control action in order to ascertain a control intention for the at least one control element; and the information represented on the display device being adapted as a function of the ascertained control intention such that the at least one control element is represented in a manner optimized for the activation of the control action associated with the control element. The control device may be arranged as a component of a motor vehicle console so as to be able to implement the control method.
Claims
1. A method for operating an interactive control device, the control device including a display device configured to represent information including control elements, comprising: ascertaining a control probability based on an ascertained control intention for each of a plurality of control elements represented on the display device, from a movement of a user's body part in relation to the plurality of control elements that does not activate the control elements, wherein the control intention is ascertained based on the movement of the user's body part in relation to a respective control element; ascertaining a control spotlight region corresponding to a portion of the display device encompassing an expected touch position of the user's body part on the display device, wherein the location and size of the control spotlight region is ascertained based on the ascertained control probability for each control element, the size of the control spotlight region being greater than the user's body part and smaller than the display device; and selectively adapting display of the control elements such that (i) each control element which is at least partially within the control spotlight region is proportionally optimized for activation according to the respective ascertained control probability, and (ii) control elements outside the control spotlight region are not proportionally optimized for activation; wherein the control probability for each control element is ascertained prior to activating a control action; wherein each control element which is at least partially within the control spotlight region is scaled as a function of at least one of (i) distance from the center of the control spotlight region and (ii) amount of planar overlap with the control spotlight region.
2. The method according to claim 1, wherein the ascertaining of the control spotlight region includes ascertaining whether the user's body part is located within an activation region that is spatially defined relative to a display region of at least one of the control elements on the display device.
3. The method according to claim 1, wherein the ascertaining of the control intention and the adapting of the represented information are performed at least one of (a) iteratively and (b) continuously.
4. The method according to claim 2, wherein the ascertaining of the control spotlight region is based on information relating to at least one of (a) a bodily action, (b) a direction of movement of a body part, and (c) a viewing direction of a user.
5. The method according to claim 1, further comprising ascertaining user information at least partly by at least one of (a) a camera system and (b) an ultrasonic sensor system.
6. The method according to claim 1, wherein the ascertaining of the control intention is based on information from surroundings about a driving situation, which is taken into account when adapting the at least one of the control elements.
7. The method according to claim 1, wherein the adapting includes changing at least one of (a) a transparency, (b) the size of the control element, (c) a distance from adjacent additional control elements and (d) an animation of at least one of (i) the control element and (ii) several control elements.
8. The method according to claim 1, wherein the ascertaining of the control intention includes detecting and evaluating gestures executed by the body part, the adapting of the represented information being performed in accordance with an adaptation function assigned to the detected gesture.
9. The method according to claim 8, wherein the gestures include at least one static gesture, which is detected on the basis of a predefined body part attitude.
10. The method according to claim 8, wherein the gestures include at least one dynamic gesture, which is detected on the basis of a predefined path line traversed by the body part.
11. The method according to claim 8, wherein the gestures include at least one complex gesture, which is detected on the basis of at least one of (a) a transition between predefined static gestures and (b) a static gesture, which traverses a predefined path line.
12. The method according to claim 1, wherein the information represented on the display device is adapted for a representation that is optimized for a visual communication of information if no control intention is ascertained.
13. An interactive control device, comprising: a display device adapted to represent information that includes control elements; and a control device having at least one processor adapted to ascertain a control probability based on an ascertained control intention for each of a plurality of control elements represented on the display device, from a movement of a user's body part in relation to the plurality of control elements that does not activate the control elements, the control intention ascertained based on the movement of the user's body part, ascertain a control spotlight region corresponding to a portion of the display device encompassing an expected touch position of the user's body part on the display device, wherein the location and size of the control spotlight region is ascertained based on the ascertained control probability for each control element, the size of the control spotlight region being greater than the user's body part and smaller than the display device, and to selectively adapt display of the control elements such that (i) each control element which is at least partially within the control spotlight region is proportionally optimized for activation according to the respective ascertained control probability, and (ii) control elements outside the control spotlight region are not proportionally optimized for activation; wherein the control probability for each control element is ascertained prior to activating a control action; wherein each control element which is at least partially within the control spotlight region is scaled as a function of at least one of (i) distance from the center of the control spotlight region and (ii) amount of planar overlap with the control spotlight region.
14. The device according to claim 13, further comprising at least one sensor device adapted to detect the user's body part within an activation region that is spatially defined relative to a display region of at least one of the control elements on the display device in order to ascertain a control intention for at least one of the control elements represented on the display device.
15. The device according to claim 13, wherein the control device is adapted to ascertain the control intention and to adapt the information at least one of (a) iteratively and (b) continuously.
16. The device according to claim 14, wherein sensor information includes information about a bodily action of the user.
17. The device according to claim 13, further comprising at least one of (a) a camera system and (b) an ultrasonic sensor system adapted to ascertain user information.
18. The device according to claim 14, wherein sensor information includes information relating to at least one of (a) a viewing direction of the user and (b) a direction of movement of the user's body part.
19. The device according to claim 14, wherein the control device is adapted to ascertain at least one of (a) a direction of movement of the body part and (b) a viewing direction of the user in accordance with sensor information.
20. The device according to claim 13, wherein the control device is configured to adapt the at least one of the control elements in accordance with information from surroundings about a driving situation.
21. The device according to claim 13, wherein the control device is configured to adapt a transparency by modification at least one of (a) a distance from adjacent additional control elements and (b) an animation of at least one control element.
22. The device according to claim 14, wherein the at least one sensor device is adapted to ascertain sensor information on the basis of at least one of (a) high-frequency signals transmitted via the body of the user and (b) a position of the body part.
23. The device according to claim 14, wherein the at least one sensor device includes a position-resolving touch-sensitive device.
24. The device according to claim 14, wherein at least one sensor device is adapted to detect and evaluate gestures performed by the body part, the control device configured to adapt the represented information in accordance with an adaptation function assigned to the detected gesture.
25. The device according to claim 24, wherein the gestures include at least one static gesture that is detectable by the at least one sensor device as a predefined body part attitude.
26. The device according to claim 24, wherein the gestures include at least one dynamic gesture, which is detectable by the at least one sensor device on the basis of a predefined path line traversed by the body part.
27. The device according to claim 24, wherein the gestures include at least one complex gesture, which is detectable by the at least one sensor device on the basis of at least one of (a) a transition between predefined static gestures and/(b) a static gesture that traverses a predefined path line.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
DETAILED DESCRIPTION
(12)
(13) The control device detects sensor information 12. The sensor information on the one hand includes information about one or several user(s). This information is called user information. The latter includes in particular information about bodily actions of the user(s), which are detected by sensor units. The bodily actions include for example an arm movement, which may be ascertained for example by a sensor unit which detects using cameras and/or a sensor unit based on ultrasound. Furthermore, using the sensor unit equipped with cameras it is possible to detect a viewing direction of the user(s). Directing one's view to the display device represents a particularly important bodily action since a control operation is usually preceded by a look onto the display device of the interactive control device. Additionally, the detected sensor information may include information about a driving situation or a surrounding environment of the motor vehicle. Certain control actions may frequently be preceded by certain driving events.
(14) On the basis of the detected sensor information, a control intention of the user(s) is subsequently ascertained 14. In order to ascertain a control intention of the user, the various information contained in the sensor information, especially the user information, is evaluated. For this purpose, particularly the behavior of the user(s), i.e. the bodily action(s) of the user(s), are evaluated and assessed. If a change in the viewing direction onto the display device of the interactive control device is established, for example, and if correlated in time a movement of the arm or the hand, which is spatially nearest to the interactive control element, is detected, then a control intention may be deduced. Many different combinations of individual information are possible, which may be evaluated and assessed. For example, a targeted movement of a body part in the direction of the control unit may be a precondition for a control intention to count as detected.
(15) With the aid of a query, a check is performed as to whether the control intention has changed 16. If this is not the case, then the representation of the information on the display device is continued without change. If a change in the control intention has occurred, i.e. if a control intention has been recognized or if it is established that a control intention no longer exists, then the information that is represented on the display device is adapted in accordance with the control intention or the nonexistent or no longer existing control intention 18. A change exists even if the control intention has become (more) concrete.
(16) If the change of the control intention lies in the fact that a control intention of a user is detected that did not exist previously, then the information to be displayed is changed such that the layout on the display device is optimized for a haptic control operation. Example embodiments may provide for no control elements to be graphically represented in the visual layout. In the haptic layout, the control elements are inserted, i.e. their transparency level of 100% is reduced. There may likewise be a provision to switch from a pictorial representation to a text representation. In example embodiments, in which small control elements are graphically represented in the visual layout, the control elements in the haptic layout may be magnified. The adaptation may further include a change in the design of the control elements, for example an animation of the control element (widget). In a display device in the form of an autostereoscopic three-dimensional display device, a spatial protrusion of the control elements or a spatially anterior superposition of the representation of the information may be provided.
(17) The ascertainment and adaptation of the represented information may occur in steps that may transition into one another continuously. First, a control intention is ascertained using sensor information of a first sensor unit. The first sensor unit may include an imaging sensor system, for example a camera system. If this general control intention has been detected, then the visual layout is switched to a haptic layout. Control elements are inserted for example, which were previously not visible.
(18) When ascertaining the control intention, example embodiments furthermore provide for a position and a surrounding region on the display device to be ascertained, which is intended for a control action, i.e. an activation of a control element. This process is called a determination of a control spotlight 20.
(19) Various sensor information may be evaluated for ascertaining the control spotlight. In particular, a direction of movement of a body part of the user and a speed of movement, a viewing direction onto the display device and information about the driving situation are evaluated individually or jointly. As the result of ascertaining the control spotlight, one obtains a position on the display device, which indicates the intended target position, and a radius, which is a measure for the uncertainty of the ascertained intended target position. Exemplary embodiments may provide for the shape of the control spotlight to be noncircular. This is advantageous in particular if the control direction deviates greatly from a direction of a surface normal of the display surface of the display device.
(20) The information that characterizes the control spotlight (position/radius and/or other indication of region) is taken into account when adapting the information to be represented. For example, the control elements are scaled as a function of the distance from the position and a planar overlap with the control spotlight. This means that control elements situated near the position of the control spotlight, which is at the center of the control spotlight, are scaled larger than those elements that are further removed from the position of the control spotlight. Furthermore, those control elements are scaled larger that have a large planar overlap with the control spotlight when projecting the control spotlight onto the display device.
(21) The described method represents a determination of a control probability for the individual control elements 22. These control probabilities may also be determined according to another method on the basis of the user information, possibly by taking into account other information about situational influences, for example the driving situation, hitherto existing habits of the user, etc.
(22) The control spotlight is continuously adapted. In a further step, the movements of the body part and possible additional properties of the body part are ascertained more precisely using a second sensor unit.
(23) In example embodiments, in which transducers generating high-frequency signals are situated near the body of the user and receivers are situated on or around the display device in order to receive high-frequency signals transmitted via the body of the user and to determine from this a position of the hand of the user near the display device, the adaptation of the represented information may be finely adjusted particularly well. Body parts of users may be detected by such sensor systems at a distance of approximately 30 cm from the display device. If multiple users are within the range of the interactive control device in a vehicle, for example the driver and the front passenger of the motor vehicle, these may be distinguished on the basis of different high-frequency signals that are coupled into the body of the user via different transducers, which are integrated for example in a driver seat and in a front passenger seat. A scaling or general adaptation of the represented information may thus be additionally adapted to the function (driver/front passenger) of the user. For example, it is practical to represent fewer, but instead larger control elements on the display device if a driver wants to activate control elements on the display device at a high speed of the motor vehicle. A front passenger, who does not have to concentrate on driving the motor vehicle, may operate smaller control elements, for example, of which instead more are representable on the display device. In order to perform the fine adjustment, a provision may be made to ascertain a distance of the user's body part from the display device when detecting the control intention. Additionally, the extension of the body part is ascertained. This extension is also taken into account when scaling and/or refining the control element(s). For a user who has large and wide fingers, the control elements must be displayed larger than for a user who has small and narrow fingers. A finer adaptation is thus possible in the additional step. More than two steps may be provided. The second sensor unit may also be an sensor unit based on ultrasound.
(24)
(25)
(26)
(27)
(28) If the body part, in this case finger 71, is located in the activation region, then the control action associated with the control element is activated.
(29)
(30) If a finger 155 approaches display device 150, control elements 151 in one of the two groups 152, 153 will be magnified. If a control intention is ascertained for one of control elements 151 of the first group 152 developed as a list, then the list is magnified, as shown in
(31) While in the exemplary embodiment described in connection with
(32) The following will describe, with reference to
(33)
(34) The focus may be shifted in that the user moves his finger 171 to an edge of display device 170. In
(35) Alternatively, there may be provision for the focus to remain at the center of display device 170 and for control elements 161-169 to be shifted relative to the focus. That is, the control fields move to the left, as is indicated by an arrow 173. This means that the focus is next assigned to control field 166 having the number 6, control field 166 being magnified at the center of display device 170.
(36) In both exemplary embodiments, the focus scrolls or leafs through the list of control elements.
(37) This process may be terminated in that finger 171 is moved across the center of display device 170. Subsequently, the control element to which the focus is assigned may be activated in order to trigger the associated control action.
(38) The scrolling or leafing speed may vary as a function of the position of finger 171. That is, the further the finger is moved from the center, the faster the scrolling/leafing occurs.
(39) The leafing/scrolling direction may coincide with the direction in which the finger is moved or may be opposite. In the case of a list there may be a further provision for the list to be considered endless. This means that a final list element is again followed by the first list element.
(40)
(41) It is not necessary for the focus to remain fixed in one position on the display device. Rather, it may move along with the control element to which it is assigned. If this control element is further away from a predefined setpoint position of the focus than the subsequent control element of the list, then the focus jumps to this subsequent control element.
(42) Example embodiments may provide for multiple control elements to be magnified. For example, a primary focus and two secondary focuses may be provided. The control elements assigned to the secondary focuses are magnified, for example, but somewhat smaller than the control element assigned to the primary focus.
(43) Example embodiments may provide for a control element, for which the highest control probability is ascertained (for example, the control element that is located in the fixed focus or the center of the control spotlight), to be activated if the body part of the user, for example the finger, abides unchanged for a predefined dwell time without an adaptation of the represented information occurring. This applies particularly also if the finger is not located in the actual activation region of the control element. This allows for a quasi activationless control of a control element.
(44)
(45) The control unit further includes an activation module 134, which triggers or activates a control action if a body part, for example a finger, is located in an activation region that is predefined relative to the representation of the control element on display device 102. In this connection, signals may be transmitted via the vehicle bus that influence other control devices of motor vehicle 101.
(46) Control unit 120 and individually included modules 122, 128, 134 may take the form of hardware as well as software.
(47) The display device may take the form of a projection display in which the represented information is projected onto a surface. In such a case, the distance or the movement of the body part relative to this surface or a look onto this surface, etc. are relevant.
(48) In the sensor units that transmit high-frequency signals via the human body, frequencies in the range of approximately 80 kHz to 150 kHz may be particularly suitable. The sensor units, however, may also be operated at frequencies outside of this indicated frequency range.
(49) In addition or alternatively to a sensor unit that detects a body part on the basis of the transmission of high-frequency signals via the human body, other contactless sensor units may be used such as sensor units based on ultrasound or even sensor units that use optical methods. Such a sensor unit may be designed, for example, according to the following principle. A transmitting LED radiates a rectangularly amplitude-modulated light signal in the optical or infrared wavelength range. The light signal reflected on an object is detected by a photodiode. A compensation LED sends a 180 phase-shifted, likewise rectangularly amplitude-modulated reference light signal to the photodiode via a constant light path. The compensation LED is controlled via a control loop using a control signal such that the received reflected light signal of the send LED and the received reference light signal of the compensation LED cancel out at the photodiode, and an equisignal is detected. A change in the control signal is a measure of the distance of the object. A sensor unit designed according to this principle is largely independent of temperature fluctuations and brightness fluctuations.
(50) At least one sensor unit or several sensor units may be designed so as to be able to detect a planar extension of the body part (possibly by interacting). This makes it possible to detect gestures that are performed by the body part, for example a hand, and to interpret them as a control intention. Gestures that depend only on the body part attitude, in particular a hand attitude, are called rigid or static gestures. A hand 140 held flat in front of the display device, as shown in
(51) If the movement of the body part in a certain spatial region is compared to predefined path lines, then dynamic gestures may be detected. Examples of path lines are shown by way of example in a two-dimensional plane in
(52) If it is possible to resolve multiple parts of the hand separately and/or to detect their relative speeds, then complex gestures or hand attitudes and sequences of movements, for example an extension and closing of the fingers of a flat hand or a clenching of the fingers to form a first (
(53) The gestures shown are only exemplary gestures. Various control intentions may be assigned to the individual gestures, which effect a corresponding adaptation of the represented information.
(54) It may be provided for the interactive control device to be capable of being operated in a learning mode and thus be able to learn specific gestures in the individual specificity by different persons. For example, a driver may be prompted to perform specific gestures. An individual specificity of the gesture may be learned on the basis of the measurement data detected during the performance. Learning advantageously occurs in relation to individual persons. Persons may be identified by the specificity of the gestures themselves or by a user or driver identification encoded in a vehicle key for example.
(55) Some exemplary embodiments may be provided such that gestures of a driver are distinguished from those of a front passenger. If the interactive control device is situated in a center console of the vehicle, then the driver's gestures may be distinguished from those of a front passenger by whether the gestures are performed using a right or a left hand, it being assumed that the driver and the front passenger respectively use the hand closest to the center console for operation. In sensor units that allow for a differentiation between the driver and the front passenger, for example in that different high-frequency signals are transmitted via the body of the driver and of the front passenger, which are used by the sensor units for detecting the position of the body part, a distinction of gestures of the front passenger and of the driver is possible on the basis of the information of the sensor unit. If a distinction is possible, then different control intentions or different representation functions or adaptation functions for the driver and for the front passenger may be assigned to the same gesture.