METHOD AND APPARATUS FOR SPATIO-DATA COORDINATION
20190310760 ยท 2019-10-10
Inventors
- Maxime CORDEIL (Clayton, AU)
- Tim DWYER (Clayton, AU)
- Yongchao LI (Clayton, AU)
- Benjamin BACH (Clayton, AU)
- Elliott WILSON (Clayton, AU)
- Jon MCCORMACK (Clayton, AU)
Cpc classification
G06F3/04847
PHYSICS
G06F3/04815
PHYSICS
G06T19/20
PHYSICS
G06F3/011
PHYSICS
G06F3/0346
PHYSICS
G06T19/00
PHYSICS
International classification
G06F3/0481
PHYSICS
G06F3/0484
PHYSICS
Abstract
The invention is directed to spatio-data coordination (SD coordination) which defines the mapping of user actions in physical space into the space of data in a visualisation. SD coordination is intended to lower the user's cognitive load when exploring complex multi-dimensional data such as biomedical data, multiple data attributes vs time in a space-time-cube visualisation, or three-dimensional projections of three-or-higher-dimensional data sets.
Claims
1. A display adapted to show a virtual object in a defined area, comprising: a first dimension and a second dimension defining a 2D display area, each dimension interfacing with a corresponding portion of the 2D display, each dimension selectively defining an area to be displayed corresponding to the respective dimension, the display providing a visual representation of a physical object, the display providing a mutual interaction between the 2D display area and the visual representation in the first and second dimensions.
2. The display of claim 1, further comprising a third dimension enabling a 3D display area, each dimension interfacing with a corresponding portion of the 3D display area.
3. The display of claim 2, wherein each of the first, second and third dimensions correspond to an x axis, y axis, and z axis.
4. The display of claim 2, wherein a mutual feedback path enables the virtual object shown and the 3D display area to be linked.
5. The display of claim 1, wherein at least one dimension is associated with a programmable fader or a fine motor control.
6. A method of displaying a virtual object in 2D, namely in a first and second dimension, the method comprising the steps of: generating with a processor a 2D display area, bounded by the first and second dimensions; displaying with a display device, at least a portion of an object visually within the 2D display area; and selectively controlling via a data axis the portion of the object displayed by defining, in each dimension, a display range, the display range corresponding to the portion of the object being displayed in the 2D display area.
7. The method as claimed in claim 6, further comprising generating with the processor a third dimension, and a 3D display area, bounded by the first, second and third dimensions; displaying with the display device at least a portion of an object visually within the 3D display area; and selectively controlling via the data axis the portion of the object displayed by defining, in each dimension, a display range, the display range corresponding to the portion of the object being displayed in the 3D display area.
8. The method as claimed in claim 6, wherein a user selectively controls the portion of the object displayed by defining the display range in each dimension and/or the orientation of the object in the display area.
9. The method as claimed in claim 6, wherein the portion of the object displayed is selectively controlled via a slider and/or rotatable knob.
10. The method as claimed in claim 7, wherein each of the first, second and third dimensions correspond to an x axis, y axis, and z axis.
11. The method as claimed in claim 6, further comprising providing a pointer in the virtual environment that operates with a physical control associated with at least one dimension.
12. The method as claimed in claim 6, wherein the 2D display area is Euclidean.
13. The method as claimed in claim 6, wherein a selective control for selectively controlling the object displayed is orientation-congruent.
14. A method of displaying a virtual object in 2D, namely in a first dimension and a second dimension, the method comprising: a physical 2D display area, bounded by the first dimension and second dimension, displaying, with a display device, at least a portion of an object visually within the physical 2D display area; wherein, in a first mode, the portion of the object displayed by the display device is selectively controlled by defining, in each dimension, a display range, the display range corresponding to the portion of the object being displayed in the 2D display area wherein the selective control is provided by a user defining the display range in each dimension and/or the orientation of the object in the display area; and wherein, in a second mode, the portion of the object displayed by the display device is selectively controlled by defining, in each dimension, a display range, the display range corresponding to the portion of the object being displayed in the 2D display area wherein the selective control is provided by a user interacting with the virtual display to control the physical 2D display area bounded by the first, second and third dimensions.
15. The method as claimed in claim 14, further comprising providing a third dimension, and a physical 3D display area, bounded by the first, second and third dimensions, displaying via the display device at least a portion of an object visually within the 3D display area; wherein in a first mode, the portion of the object displayed by the display device is selectively controlled by defining, in each dimension, a display range, the display range corresponding to the portion of the object being displayed in the 3D display area wherein the selective control is provided by a user defining the display range in each dimension and the orientation of the object in the display area; and wherein in a second mode, the portion of the object displayed by the display device is selectively controlled by defining, in each dimension, a display range, the display range corresponding to the portion of the object being displayed in the 3D display area wherein the selective control is provided by a user interacting with the virtual display to control the physical 3D display area bounded by the first, second and third dimensions.
16. The method as claimed in claim 14, wherein the selective control is provided by defining the display range in each dimension and the orientation of the object in the display area.
17. The method as claimed in claim 14, further comprising actuating a slider and/or rotatable knob to selectively control the object displayed by the display device.
18. A method as claimed in claim 15, wherein each of the first, second and third dimensions correspond to an x axis, y axis, and z axis.
19. The method as claimed in claim 14, further comprising providing a pointer in the virtual environment that mutually operates with a physical control associated with at least one dimension.
20. The method as claimed in claim 14, wherein a selective control for selectively controlling the portion of the object displayed by the display device is orientation-congruent.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Further disclosure, objects, advantages and aspects of preferred and other embodiments of the present application may be better understood by those skilled in the relevant art by reference to the following description of embodiments taken in conjunction with the accompanying drawings, which are given by way of illustration only, and thus are not limitative of the disclosure herein.
[0051] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
[0052]
[0053]
[0054]
[0055]
[0056]
[0057]
[0058]
[0059]
DETAILED DESCRIPTION
[0060] For purposes of description herein, the terms upper, lower, right, left, rear, front, vertical, horizontal, interior, exterior, and derivatives thereof shall relate to the invention as oriented in
[0061] In this specification, we coin the term Spatio-Data (SD) coordination, referring to interactions from a physical interaction-space into a digital visualisation-space, and visa-versa.
[0062] Referring to
[0063] For example, for a visualization of a multi-dimensional scaling in three dimensions, the data-space may contain hundreds of dimensions associated with the individual data points. Through multi-dimensional scaling, the number of dimensions associated to each data point gets reduced to three. These three dimensions are then mapped to three orthogonal spatial dimensions in the visualisation space, for example axis x, y and z. Eventually, this three-dimensional Euclidean space is rendered, e.g. into a VR environment using an HMD.
[0064] In order to allow for spatio-interaction, we assume the interaction space has the following three characteristics:
[0065] 1. It has to be of Euclidean nature and occupy a well-defined, bounded part of the user's physical environment;
[0066] 2. The mapping between interaction and visualisation space has to be orientation-congruent, i.e. it has to preserve the space's orientation. For example, any position or movement towards the right of the user in the interaction space results in a movement to the right of the user in the visualisation space. The same holds true for all spatial dimensions (up-down/top-bottom, right-left, and towards-away from the user); and
[0067] 3. We assume a computer-in-the-loop, i.e. a computer that processes input and generates an output in the form of a visualisation.
[0068] SD coordinated interaction devices are any devices and systems that use such a direct mapping between interaction and visualisation space and that satisfy the three conditions above. We believe that designing interaction systems for SD coordination decreases a user's cognitive load when exploring the data. Ideally, display and interaction would hence be the same in the user's physical environment, i.e. the user interacts with the visualisation where the visualisation is perceptually situated.
[0069]
[0070] The Fader Axes uses the original concept of physical and virtual selection coupling: there is a 1 to 1 mapping or mutual interaction between the selections parameters in the virtual model and the selection parameters on the slider, being in the physical environment. This has three implications:
[0071] Physical to virtual mapping. Referring to
[0072] Virtual to physical mapping. Referring to
[0073] Coupling physical/virtual actions. Referring to
[0074] Possible physical design and implementations and interactions of the fader axes implementation include the following:
[0075] The number of physical and virtual sliders per axis:
[0076] one physical and motorized slider per axis and N virtual slidersthe physical axes is only equipped with one physical motorised slider, and range selection on this axis is performed by a coupled physical/interaction action with N additional virtual sliders. The virtual slider may be one or more icon on a screen.
[0077] N physical sliders and N virtual sliders (N>=1)The physical axes is equipped with N (more than one) physical motorised slider, and range selection on this axis is performed by a coupled physical/interaction action with N additional virtual sliders.
[0078] The number of physical axes: possible implementations of physical and virtual interactions for value and range selection can involve:
[0079] one single axis (for one-dimensional selections only);
[0080] two orthogonal axes (for 1, 2 dimensional selections);
[0081] three orthogonal axes (for 1, 2, 3 dimensional selection); and
[0082] N axes, for N-dimensional selections (out of 3D Euclidian space, e.g. 1 axis for time).
[0083] The dimensions of the display and selections:
[0084] Two-dimensional displaye.g. 2D computer screen 610, touch screen, phone 660examples are illustrated in
[0085] 3D immersive display 210e.g. Virtual Reality head mounted display or Augmented Reality head mounted displaysee
[0086]
[0087] The embodiment may use the Unity 3D (https://unity3d.com/) game engine to display the 3D immersive visualisations. 3D game engines facilitate the software creation/implementation of high quality 3D interactive visualisations with head mounted displays such as head mounted display 210. Unity and other game engines also facilitate the integration of the tracking technology used to track fingers and hand positions in a 3D space. Other libraries/toolkits can be used to produce similar 3D immersive visualisations.
[0088] The 3D Fader Axes (
[0089] 3 physical axes 220 that represents a 3D orthonormal referential (X, Y, Z);
[0090] 2 actuated faders 240 are attached to each axis to select a range of values (a minimum and a maximum value); and
[0091] (optionally) 1 Rotary push button 230 is attached to each axis to rotate a model, and, e.g. validate a selection operation.
[0092] Embodiments of the present invention utilise a number of further features, one of which is that the faders or sliders are programmable. The sliders or faders are programmable in as much as the position of the cursor that indicates the value in the range can slide automatically thanks to a controlled motor. Specifically, the motor is controlled for example by a micro controller (here an Arduino board). Accordingly, the cursor on the controlled faders can jump rapidly to a specific position (for example, jump rapidly from value 0.1 to 0.75) and/or slide continuously along the axis (for example, while continuously matching the projected position on the axis of a tracked finger in 3D space).
[0093] In addition, or alternatively, a range slider may be used with only one physical sliding cursor, as long as one cursor position can be virtually set with a hand tracker. In this way, only 1 slider is need and a border position can also be defined on the physical axis 220.
[0094] In embodiments of the present invention, the sliders give fine motor controls. Physical motorised potentiometer sliders can yield high precision. The motors can give very precise controls over the cursor position on the axis and therefore match precise user selections.
[0095] In spatio-data coordination embodiment, a pointer is, for example:
[0096] Virtual: Moved in 3D space with a hand-tracking device (such as the leapmotion, a 6DOF controller, kinnect . . . ).
[0097] Touch: Moved in 2D space for example on a touch screen.
[0098] Touch: Moved on a touch sensitive surface (e.g. on the faces of a cube 710).
[0099] Physical: Moved along sliding axis (e.g. on the fader cursors).
[0100] There may also be provided a pointer in the virtual environment that mutually operates the physical axis controls. A hand tracking device (e.g. a leap motion as manufactured by Leap Motion, Inc. San Francisco, Calif., USA) may be used to attach a virtual pointer on the tip of a finger. That is, like a mouse pointer on a screen can be moved with a hand movement by operating a mouse, a virtual pointer can be placed in a 2D or 3D space with a hand tracking device. This pointer has 2 or 3 coordinates (x, y, and/or z respectively) in the eucledian space and the orthonormal space: X, Y and/or Z. A virtual pointer (or several virtual pointers attached to several fingers on both hands) may be used to select values on the X, Y and/or Z axis. The physical sliders may use those (x, y and/or z) coordinate values to match the selection on the physical axis. Hence, a consequence is that a user can perform a rough selection using a virtual pointer in the virtual environment and then use the physical sliders that are now matching the selected values, to perform a more precise selection.
[0101] Preferably, the 3D space is Euclidean. A Eucledian orthonormal space is necessary to achieve proper 2D and 3D data visualisation. It ensures that distances between data points are comparable.
[0102] Preferably, that the interaction is orientation-congruent. This is important because in the spatio-data coordination concept, a user action (e.g. a selection) should be aligned with/should respect the rotation of the model, to make actions predictable and easy to operate.
[0103] The present invention may be put to various possible uses, such as:
[0104] 3D CT scan analysis, e.g. to guide the exploration and the selection of potential tumours (
[0105] 3D Building modelling, e.g. to help the navigation of 3D CAD models (
[0106] 3D engineering simulation, e.g. to enhance the selection of 3D disconnected components.
[0107] In embodiments of the present invention, for example, a three-dimensional scatterplot visualisation may be a cube-shaped visualisation space, which can be decomposed into three visualisation components: edges, cube faces, and interior volume with data points.
[0108] Touch-sensitive Cube 710: a hand-sized tangible cube with rigid faces and edges (
[0109] Physical Axes 220see
[0110] Virtual Mid-air: Operated by unconstrained mid-air gestures (
[0111] While this invention has been described in connection with specific embodiments thereof, it will be understood that it is capable of further modification(s). This application is intended to cover any variations uses or adaptations of the invention following in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains and as may be applied to the essential features hereinbefore set forth.
[0112] As the present invention may be embodied in several forms without departing from the spirit of the essential characteristics of the invention, it should be understood that the above described embodiments are not to limit the present invention unless otherwise specified, but rather should be construed broadly within the spirit and scope of the invention as defined in the appended claims. The described embodiments are to be considered in all respects as illustrative only and not restrictive.
[0113] Various modifications and equivalent arrangements are intended to be included within the spirit and scope of the invention and appended claims. Therefore, the specific embodiments are to be understood to be illustrative of the many ways in which the principles of the present invention may be practiced. In the following claims, means-plus-function clauses are intended to cover structures as performing the defined function and not only structural equivalents, but also equivalent structures. For example, although a nail and a screw may not be structural equivalents in that a nail employs a cylindrical surface to secure wooden parts together, whereas a screw employs a helical surface to secure wooden parts together, in the environment of fastening wooden parts, a nail and a screw are equivalent structures.
[0114] Various embodiments of the invention may be embodied in many different forms, including computer program logic for use with a processor (e.g., a microprocessor, microcontroller, digital signal processor, or general purpose computer and for that matter, any commercial processor may be used to implement the embodiments of the invention either as a single processor, serial or parallel set of processors in the system and, as such, examples of commercial processors include, but are not limited to Merced, Pentium, Pentium II, Xeon, Celeron, Pentium Pro, Efficeon, Athlon, AMD and the like), programmable logic for use with a programmable logic device (e.g., a Field Programmable Gate Array (FPGA) or other PLD), discrete components, integrated circuitry (e.g., an Application Specific Integrated Circuit (ASIC)), or any other means including any combination thereof. In an exemplary embodiment of the present invention, predominantly all of the communication between users and the server is implemented as a set of computer program instructions that is converted into a computer executable form, stored as such in a computer readable medium, and executed by a microprocessor under the control of an operating system. For example, in some aspects, a computer program stored in non-transitory computer readable storage medium, when executed by a processor of a computer, causes the computer to execute the steps of displaying a virtual object in 2D, namely in a first and second dimension, and providing a 2D display area, bounded by the first and second dimensions displaying at least a portion of an object visually within the 2D display area selectively controlling the portion of the object displayed by defining, in each dimension, a display range, the display range corresponding to the portion of the object being displayed in the 2D display area. In further aspects, the computer program may execute the step of providing a third dimension, and to providing a 3D display area, bounded by the first, second and third dimensions, displaying at least a portion of an object visually within the 3D display area, and selectively controlling the portion of the object displayed by defining, in each dimension, a display range, the display range corresponding to the portion of the object being displayed in the 3D display area. In some aspects, the computer program stored in non-transitory computer readable storage media may be in the form of an application.
[0115] Computer program logic implementing all or part of the functionality where described herein may be embodied in various forms, including a source code form, a computer executable form, and various intermediate forms (e.g., forms generated by an assembler, compiler, linker, or locator). Source code may include a series of computer program instructions implemented in any of various programming languages (e.g., an object code, an assembly language, or a high-level language such as Fortran, C, C++, JAVA, or HTML. Moreover, there are hundreds of available computer languages that may be used to implement embodiments of the invention, among the more common being Ada; Algol; APL; awk; Basic; C; C++; Conol; Delphi; Eiffel; Euphoria; Forth; Fortran; HTML; Icon; Java; Javascript; Lisp; Logo; Mathematica; MatLab; Miranda; Modula-2; Oberon; Pascal; Perl; PL/I; Prolog; Python; Rexx; SAS; Scheme; sed; Simula; Smalltalk; Snobol; SQL; Visual Basic; Visual C++; Linux and XML.) for use with various operating systems or operating environments. The source code may define and use various data structures and communication messages. The source code may be in a computer executable form (e.g., via an interpreter), or the source code may be converted (e.g., via a translator, assembler, or compiler) into a computer executable form.
[0116] The computer program may be fixed in any form (e.g., source code form, computer executable form, or an intermediate form) either permanently or transitorily in a tangible storage medium, such as a semiconductor memory device (e.g, a RAM, ROM, PROM, EEPROM, or Flash-Programmable RAM), a magnetic memory device (e.g., a diskette or fixed disk), an optical memory device (e.g., a CD-ROM or DVD-ROM), a PC card (e.g., PCMCIA card), or other memory device. The computer program may be fixed in any form in a signal that is transmittable to a computer using any of various communication technologies, including, but in no way limited to, analog technologies, digital technologies, optical technologies, wireless technologies (e.g., Bluetooth), networking technologies, and inter-networking technologies. The computer program may be distributed in any form as a removable storage medium with accompanying printed or electronic documentation (e.g., shrink wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the communication system (e.g., the Internet or World Wide Web).
[0117] Hardware logic (including programmable logic for use with a programmable logic device) implementing all or part of the functionality where described herein may be designed using traditional manual methods, or may be designed, captured, simulated, or documented electronically using various tools, such as Computer Aided Design (CAD), a hardware description language (e.g., VHDL or AHDL), or a PLD programming language (e.g., PALASM, ABEL, or CUPL). Hardware logic may also be incorporated into display screens for implementing embodiments of the invention and which may be segmented display screens, analogue display screens, digital display screens, CRTs, LED screens, Plasma screens, liquid crystal diode screen, and the like.
[0118] Programmable logic may be fixed either permanently or transitorily in a tangible storage medium, such as a semiconductor memory device (e.g., a RAM, ROM, PROM, EEPROM, or Flash-Programmable RAM), a magnetic memory device (e.g., a diskette or fixed disk), an optical memory device (e.g., a CD-ROM or DVD-ROM), or other memory device. The programmable logic may be fixed in a signal that is transmittable to a computer using any of various communication technologies, including, but in no way limited to, analog technologies, digital technologies, optical technologies, wireless technologies (e.g., Bluetooth), networking technologies, and internetworking technologies. The programmable logic may be distributed as a removable storage medium with accompanying printed or electronic documentation (e.g., shrink wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the communication system (e.g., the Internet or World Wide Web).
[0119] Comprises/comprising and includes/including when used in this specification is taken to specify the presence of stated features, integers, steps or components but does not preclude the presence or addition of one or more other features, integers, steps, components or groups thereof. Thus, unless the context clearly requires otherwise, throughout the description and the claims, the words comprise, comprising, includes, including and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in the sense of including, but not limited to.