Proximity based cross-screen experience App framework for use between an industrial automation console server and smart mobile devices

Abstract

A system and method for a Graphic User Interface used as the human machine interface (HMI) for Industrial Automation (IA) systems. The system and method use a peer-to-peer session framework between a console with one or more physically connected displays within a control room environment and a companion device. One or more embodiments of the disclosure relate to control of a console, or first device, in a control room by means of a companion device, or second device, through a peer to peer session framework. The use of the second device may provide additional capabilities beyond the pre-purposed capability provided by a dedicated Operator keyboard of the console server.

Claims

1. A method for exchanging information in a peer-to-peer session framework for industrial automation, the method comprising: searching for a proximity-based availability of one of a first device and a second device by another of the first device or the second device, the first device having a first input tool; establishing the peer-to-peer session framework between the first device and the second device, the first device having a first operating state and the second device having a second operating state; changing the configuration of the second device from a second operating state to a third operating state upon establishing the session with the first device; and configuring the second device in the third operating state as a second input tool to the first device, wherein the first device comprises an industrial automation console server configured to monitor an industrial automation process.

2. The method for exchanging information in a peer-to-peer session framework for industrial automation of claim 1, wherein the first device is connectable to the first input tool having a set of input functions or input methods, the first device is connectable to a first display, and the second device is connectable to a second display, the method further comprising: configuring the second device with at least the same set of input functions or input methods as the first input tool to the first device.

3. The method for exchanging information in a peer-to-peer session framework for industrial automation of claim 2, further comprising, configuring the second display to display information regarding an input function or input method of the second input tool.

4. The method for exchanging information in a peer-to-peer session framework for industrial automation of claim 3, wherein the second display is a touchscreen display configured to display information regarding the input function of the second input tool when a contact is detected.

5. The method for exchanging information in a peer-to-peer session framework for industrial automation of claim 1, wherein the first device is connectable to a first display and the second device is connectable to a second display, further comprising: displaying at least a first subject and a second subject on the first display in the first operating state of the first device, and changing the configuration of the first device from a first operating state to a fourth operating state upon establishing the session with the second device, wherein the second subject is displayed on the second display in the third operating state.

6. The method for exchanging information in a peer-to-peer session framework for industrial automation of claim 1, wherein the second device is a mobile device and a sensor of the mobile device is configured to send data to the first device.

7. The method for exchanging information in a peer-to-peer session framework for industrial automation of claim 6, wherein the second device has at least one of a microphone or camera configurable to send information.

8. The method for exchanging information in a peer-to-peer session framework for industrial automation of claim 7, further comprising: controlling the first device through voice recognition means of the second device.

9. The method for exchanging information in a peer-to-peer session framework for industrial automation of claim 1, wherein the first device is connectable to a first display and the second device has a second display, further comprising: displaying a subject on the first display in the first operating state; and transferring the subject from the first display to the second display after establishing the peer-to-peer session framework.

10. The method for exchanging information in a peer-to-peer session framework for industrial automation of claim 1, further comprising: transmitting data from the first device to the second device; and transmitting the data from the second device to a third device through another communication method.

11. The method for exchanging information in a peer-to-peer session framework for industrial automation of claim 1, further comprising: configuring the first device based on settings stored in the second device.

12. The method for exchanging information in a peer-to-peer session framework for industrial automation of claim 1, further comprising: configuring the first device and the second device; storing a configuration of the first device and the second device in the second device; establishing a second peer-to-peer session framework between the second device and a third device; and configuring an operating state of the third device similar to the first device based on the stored configuration in the second device.

13. The method for exchanging information in a peer-to-peer session framework for industrial automation of claim 1, wherein the first device is connectable to a first display and a second display, further comprising: displaying a subject on at least one of the first display and the second display based on an input to the second device.

14. The method for exchanging information in a peer-to-peer session framework for industrial automation of claim 1, further comprising: searching for a proximity-based availability of a third device by a second device; establishing the peer-to-peer session framework between the third device and the second device, and configuring the second device to communicate with both the first and third device.

15. The method for exchanging information in a peer-to-peer session framework for industrial automation of claim 1, wherein the second device as a second input tool is configured to provide inputs corresponding to preprogrammed function keys of the first device.

16. An industrial automation console server for monitoring an industrial automation process, the industrial automation console server comprising: at least one non-transitory computer readable medium operable to store program code; at least one processor operable to read said program code and operate as instructed by the program code, the program code comprising: searching for a proximity-based availability of a second device; establishing a peer-to-peer session framework between the industrial automation console server and the second device, the industrial automation console server having a first operating state and the second device having a second operating state; changing the configuration of the second device from a second operating state to a third operating state upon establishing the session with the industrial automation console server; and configuring the second device in the third operating state as a second input tool to the industrial automation console server.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1A illustrates a related art dedicated Operator keyboard.

(2) FIG. 1B illustrates a screen shot of a related art on-screen Browser Bar user interface for an IA console.

(3) FIG. 2 illustrates screen shot of an exemplary system layout according to one preferred embodiment.

(4) FIG. 3 illustrates screen shot of an exemplary Tool Box View according to an embodiment.

(5) FIG. 4 illustrates screen shot of an exemplary Preset Builder View according to an embodiment.

(6) FIG. 5 illustrates screen shot of an exemplary Navigation View according to an embodiment.

(7) FIG. 6 illustrates screen shot of an exemplary Touch Mouse View according to an embodiment.

DETAILED DESCRIPTION

(8) Embodiments will be described below in more detail with reference to the accompanying drawings. The following detailed descriptions are provided to assist the reader in gaining a comprehensive understanding of the methods, apparatuses, and/or systems described herein, and equivalent modifications. Accordingly, various changes, modifications, and equivalents of the systems, apparatuses and/or methods described herein will be suggested to those of ordinary skill in the art. Also, descriptions of well-known functions and constructions may be omitted for increased clarity and conciseness.

(9) The terms used in the description are intended to describe embodiments only, and shall by no means be restrictive. Unless clearly used otherwise, expressions in a singular form include a meaning of a plural form. In the present description, an expression such as comprising or including is intended to designate a characteristic, a number, a step, an operation, an element, a part or combinations thereof, and shall not be construed to preclude any presence or possibility of one or more other characteristics, numbers, steps, operations, elements, parts or combinations thereof.

(10) In contrast to the existing method of a wired dedicated Operator keyboard, interfacing with a console by a wireless method provides benefits to an Operator. By using a second device rather than a dedicated Operator Keyboard, which is wired to the console, the Operator would be able to freely move around a control room. In the past, prior to the keyboard, Operators felt more engaged walking up and down the control room wall with their clip board to make notes and create logs instead of sitting in front of consoles. Enabling an operator to walk around and in front of the display screens of a console while also using a smart tablet may allow him to interact more closely with what is happening in the process.

(11) Also, offloading or removing interaction UIs from the console displays provides the ability to display more meaningful content rather than menus. As displays are of a finite size, the presence of the interaction UIs overlay the critical content of the view itself. FIG. 1B demonstrates this shortcoming where the interaction UIs take up space on the display. On screen interaction UIs overlay the meaningful content or reduces the amount of usable screen space for display of the meaningful content. The UI itself obscures the active screen content. With a dedicated Operator keyboard, the interaction UIs are needed to achieve the required functionality of the console. However, by moving the interaction UIs to a second device, it would be possible to display more of the critical content of the view on the at least one console display.

(12) Exemplary embodiments address the issues and problems raised by previous methods in the related art.

(13) FIG. 2 shows an exemplary system architecture 20 of the present disclosure.

(14) The exemplary system architecture 20 includes an industrial automation (IA) console server 101. The console server 101 is operated by a control room user operator. The console server 101 may support one or more locally connected monitor displays 109A-109D for display output of information. The IA console functions 102 are software components installed on the console server 101. The console functions 102 are used to support the human machine interface (HMI). The console server 101 includes at least one cross-screen experience (CSE) server 105 that exposes an implementation of the cross-screen experience App interface App I/F 104 for interaction with some aspect of the console functions 102. The console server 101 also includes a peer-to-peer (P2P) daemon 103 and is capable of supporting at least one wireless peer-to-peer protocol, e.g. [BLUETOOTH], [WI-FI DIRECT], [NFC], or the like. The P2P daemon 103 provides for a peer-to-peer service that provides the bus connection and session management function for the peer-to-peer messaging framework used over a proximity based wireless protocol. In addition to a proximity based protocol, additional special purpose protocols may be implemented. Each CSE server 105 can also support simultaneous sessions from multiple CSE Apps.

(15) An exemplary embodiment of a CSE server is the Human Interface Device (HID) server 108. The HID server 108 is interacts with the console functions 102. The HID server 108 is implemented with an App interface 107 and P2P daemon 106.

(16) The exemplary system architecture further includes a smart mobile device, or companion device 201. The companion device includes a peer-to-peer P2P daemon 203 and is capable of supporting at least one wireless peer-to-peer protocol, e.g. [BLUETOOTH], [WI-FI DIRECT], [NFC], or the like.

(17) The companion device 201 includes a CSE App 205, the CSE App being a CSE client that encompasses an implementation of the cross-screen experience App interface for interaction with some aspect of the console functions 102. The console server 101 and the smart mobile device 201 can establish a CSE peer session 401, that is, an active peer session between a CSE server and CSE App. With this configuration, either device can initiate transmission of a message to the other device. Additionally, multiple CSE Apps can be installed and used on the companion device with the ability to share context between the CSE Apps. A CSE App 205 can connect to any available CSE server 105 within proximity. An exemplary embodiment of a CSE App 205 is the intelligent Human Interface Device (HID) App 206.

(18) An exemplary embodiment of the system architecture of FIG. 2 establishes a CSE peer session 402 between the HID App 206 and the HID server 108.

(19) The exemplary system architecture also can include connection by the smart mobile device to other web services 301. The web service can be deployed either to the Cloud or an on-premise web server that facilitates additional content access for a CSE App.

(20) Although FIG. 2 illustrates an exemplary system with numerous components, one of ordinary skill would understand that the presence of each and every component is not required in the various embodiments.

(21) Non-limiting examples of the smart mobile device 201, or companion device, include cellular phones, tablets, phablets, laptops, personal digital assistants, and the like.

(22) By using a secured proximity based peer-to-peer networking technology (e.g. [BLUETOOTH], [WI-FI DIRECT], [NFC], or the like) a user operator can utilize the mobile device to search, locate, and connect to an available console server (CSE Server) within the near vicinity. Alternatively, the console server may establish the connection to the mobile device.

(23) Once the mobile device 201 is connected to the CSE Server 101, a CSE Peer Session 401, 402 is established availing the mobile device as an intelligent companion device. Within the session, an installed Cross-Screen experience App on the mobile device is used to enable the user operator to interact with the content currently being displayed on the connected control room operation console. Non-limiting examples of the content may include Graphic, Trend, Alarm, Application, etc.

(24) The console and companion device may also provide for additional security protocols in order to prevent unauthorized access.

(25) A peer-to-peer messaging App framework exists to support display of extra data on the mobile screen that is synchronized with the content being viewed on the operation console. The unique capabilities of the mobile device (Touch, Voice, Camera, Location, Remote Networking, etc.) are used within the CSE App to enhance the Console Server's primary screen experience or to simply extend additional pertinent content to the mobile device's local screen.

(26) According to one embodiment, the CSE App 205 on the companion device may be used to provide a human interface device for the user operator. The companion device 201 is connected to a console server 101 via a proximity based peer-to-peer service. Once connected, the HID App 206 is displayed on the companion device 201 providing access to the tools or inputs for utilizing cross screen features.

(27) According to one embodiment, the features of the on-screen UI of the console displays may be removed from the displays in order to maximize display space of the desired system information or illustration 13. Instead, the features of the on-screen UI are displayed on a touchscreen of the companion device.

(28) In one embodiment, the features of the on-screen UI displayed on the touchscreen of the companion device may be in multiple navigable menus including a tool box view, a preset builder view, a navigation view, and a touch mouse view.

(29) Tool Box View

(30) FIG. 3 shows a screen shot of an adaptive UI 300 displayed by the companion device 201 in a tool box view 331 after establishing a peer-to-peer session framework. By offloading the chrome of the UI to the companion device 201, the system information displayed by the displays of the console 101 are not obscured. The UI provides for selectable tabs for at least a tool box view 331, preset builder view 332, navigation view 333, and touch mouse view 334.

(31) The tool box view 331 provides for selection of one or more of plural destination displays 31, 32 for key target requests. The layout of the displays in the UI 31, 32 is dynamically built based on the context of the connected console with regards to the number, layout, and orientation of the physical displays. The information for the layout may be collected by the console from the settings in the operating system of the console or by manual input and configuration in the settings, or the like.

(32) In the exemplary layout illustrated in the screen shot of FIG. 3, the UI shows that the connected console has two physical displays 31, 32 oriented in a side-by-side layout. A target display can be selected by a simple tap on the corresponding display target. Screen content can be swapped or moved between displays through the use of touch gestures between the corresponding display targets. In one embodiment, pressing on one display element and sliding to another display element will swap the content between the displays. Alternatively, there can be selection of one display 32 as a primary display for the displaying of content.

(33) While the exemplary UI in FIG. 3 has two displays, other numbers of displays may be used.

(34) In one embodiment, as also illustrated in FIG. 3, the UI replicates and enhances the Operator keyboard experience for Window Call functions. In addition to the function keys 33, the UI may include toolbar options that would otherwise occupy space on a display. The UI may provide for Call View icons 34 which may provide for displaying particular graphics to provide information on the system. The Call View icons 34 may provide for graphics that show system status or hardware connected, an operations guide of notes for the operator, an overview graphic of the system, or the like. View Operation icons 35 may provide for navigation between desired graphics. For example, the View Operation icons 35 may provide for viewing of other sections of the system. Window Control icons 37 may provide for rearranging the graphical information displayed on the monitors. For example, the Window Control icons 37 may provide for cycling between views on a display. Builder icons 37 may provide for making and modifying the graphics by which system information is displayed. There may be text identifying the respective icons displayed.

(35) In the embodiment of FIG. 3, the UI replicates and enhances the Operator keyboard experience for programmable function keys. The function key targets are dynamically labeled to provide an indication of which keys are programmed and for what purpose, as seen in keys 33. A tap & hold discover method is supported to enable the user operator to discern the specific programmed function associated with the key target prior to executing it.

(36) The visibility of individual key elements 33 is contextually controlled so that the user operator is not confused with key elements options that are not usable based on the console settings and the current screen context. The visibility of the individual elements may be altered by means of changing the border, transparency of the elements, or the like.

(37) In one embodiment, the UI may provide a touch-based pan or zoom experience for a selected console window. As opposed to typical on-console pan/zoom functions, the chrome of the pan/zoom control panel is completely offloaded to the companion device so as not to obscure content on the primary screens. The native touch gestures on the tablet can be used to control the zoom and pan interaction with the screen. The UI provided pan or zoom may be provided through a pop-up window from the tool box view on the companion device.

(38) In one embodiment, the UI may provide a prioritized recommendation system in the tool box view of FIG. 3. Based on the context of the displays or previous actions by the user operator, the UI may generate a pop up window with a selection of likely to be useful functions. Selection of one of the selection of functions can result in execution on a display of the console and the generation of a subsequent selection of likely to be useful functions and so on.

(39) Preset Builder View

(40) FIG. 4 shows a screen shot of an adaptive UI displayed by the companion device in a preset builder view after establishing a peer-to-peer session framework. By offloading the chrome of the UI to the companion device 201, the system information displayed by the displays of the console 101 are not obscured.

(41) In one embodiment, the UI replicates and enhances the preset menu builder experience for entering custom commands and building and executing preset menu entries for Frame and View commands in a multi-display console environment.

(42) The layout of the preset builder view 40 adapts to the console context in terms of number and layout of displays, view mode (windowed or framed), and available elements (graphics, trends, panels, etc.). The preset builder provides a walkthrough method of saving context to the companion device.

(43) Through the layout of the preset builder view 40, the operator is able to select the default display destination for the preset with the Monitor selection 41. The Monitor selection 41 represents the potential display configurations. With each potential display configuration, there is at least one smaller icon and a larger icon. The larger icon represents the default primary display in that configuration. The user selected display configuration may be indicated by being moved to the top of the preset builder view and have its graphic appearance different from the other possible configurations.

(44) Based on the Frame selection column 42, the operator is able to select the frame style which defines how many commands are needed to be entered to populate the frames in the view. An exemplary configuration of the preset builder view provides for the selection of a display and the number of frames for display within the display. In FIG. 4, the exemplary configuration has selected a single frame on the display.

(45) Using the Window Hierarchy list 46, the operator can browse and locate the desired elements or objects to populate into each frame when the preset is executed. The Preview list 45 shows each constructed command and its target destination by display and frame.

(46) Once constructed, the preset can be executed 44 and/or saved as context 43 on the companion device for future re-use.

(47) Navigation View

(48) FIG. 5 shows a screen shot of an adaptive UI displayed by the companion device in a navigation view 500 after establishing a peer-to-peer session framework. The navigation view provides a consolidated UI for exploring all elements that are available to send view content to a console display. At a top level, these elements are organized as presets 51A, 51B, function key definitions 51C, window hierarchy 51D, and previously saved context 51E. As each element type is explored, the description and resulting command associated with the element can be reviewed prior to executing the request. That is, a tap and hold or tap and release gesture to select one of the elements will show a next level of elements associated with the selected element. For example, selection of the function key definitions 51C may show a next level showing available function keys that a user may select.

(49) The Presets and Context menus provide for exploring saved display arrangements. The function keys and Window hierarchy are for identifying elements to populate into a view.

(50) In one embodiment, the UI is provided to enable the user operator to select the desired display as the target destination for the request.

(51) In one embodiment, when exploring previously saved context elements for a console, the user operator is able to review and restore individual view elements or restore all active View elements from the saved context. This approach is useful to enable a user operator to quickly establish a working console scenario. Non-limiting considerations for establishing a working console based on saved context may include basis on time of day (e.g. operator shift change) or process condition (e.g. Boiler startup).

(52) Since the saved context is stored on the companion device, the saving of context may be useful to quickly restore the working console context onto a different console altogether for alternate operating scenarios (multiple control rooms, war room investigation, loading pre-defined Operator Training Scenarios).

(53) Touch Mouse View

(54) FIG. 6 shows a screen shot of an adaptive UI displayed by the companion device in a touch mouse view after establishing a peer-to-peer session framework. The UI provides a touch gesture experience, in lieu of a mouse, for window navigation and selection on the console displays. The UI is organized in coordinates based on the number and layout of the displays in the connected console. As shown in FIG. 6, the Touch Mouse View is connected to a console with two physical displays oriented in a side-by-side layout 61, 62. Touch swipes across the display coordinates can be used to control the movement of the cursor within and across displays. Mouse key presses may be achieved by at least double tapping on a cursor location or tapping an element for left 63A, middle 63B, or right 63C mouse keys.

(55) In one embodiment, Multi-touch gestures are used to request mouse key (left, middle, right) presses at the current cursor location.

(56) Use of Enhanced Sensors from the Companion Device

(57) According to one embodiment of using the cross-screen experience App between a console server and a connected companion device in a control room environment, the sensors of the companion device can be used to provide additional functionality to the console.

(58) In one embodiment, the companion device can provide addition support to the control room user operator's interaction with an offsite remote support center engineer. The connected companion device can be used by the control room user operator to gather the context of the current problem incident environment from the console. The context can be packaged with other ancillary data gathered via other sensors (Voice, Camera) and efficiently shared with the remote support engineer via an alternate wireless connection mechanism on the companion device (e.g. 3G/4G) that is not directly available from the console server itself.

(59) In one embodiment, the companion device can record audio from the control room and send it to an offsite remote support center engineer. Or the companion device can record audio and store it for future analysis.

(60) In one embodiment, the companion device can take a photograph or snapshot of the console and send it to an offsite remote support center engineer.

(61) According to one embodiment, the advanced sensors available in the off the shelf mobile device hardware can be leveraged to seamlessly provide advanced alternate Operator interaction models with the IA console applications without impacting the integrity of the pre-existing console hardware and software. In one embodiment, the companion device can take a photograph or snapshot and transmit the image for display on the console displays. In one embodiment, the companion device can record audio and transmit the audio recording to the console.

(62) According to one embodiment, the companion device may be used to support the launching of other context aware CSE Apps installed on the companion device. In one embodiment, a specialized CSE Trending App on the device can be launched by passing the current processing scope as context for the App. In its execution, the CSE Trending App can further aggregate content from both the CSE console environment and other data sources (e.g. Cloud-based data store) via an alternate wireless connection mechanism on the companion device. The CSE Apps may thus provide analytics and historical trends of the IA process operation that may be unavailable to the console alone. The CSE Apps may be downloadable from an App store.

(63) According to one embodiment, the system may support scenario specific Operator training exercises. The companion device can be used by the Instructor to initialize the training scenario context onto the console. In one embodiment, specific context or information may be prepared for display on a console. The companion device would be able to connect with at least one console and set up a display of the at least one console with the training scenario. In this way, it may be possible to set up multiple consoles to simultaneously train multiple user operators.

(64) Although this specification has been described above with respect to the exemplary embodiments, it shall be appreciated that there can be a variety of permutations and modifications of the described exemplary features by those who are ordinarily skilled in the art without departing from the technical ideas and scope of the features, which shall be defined by the appended claims.

(65) A method of one or more exemplary embodiments may be recorded as computer-readable program codes in non-transitory computer-readable media (CD ROM, random access memory (RAM), read-only memory (ROM), floppy disks, hard disks, magneto-optical disks, and the like) including program instructions to implement various operations embodied by a computer.

(66) While this specification contains many features, the features should not be construed as limitations on the scope of the disclosure or of the appended claims. Certain features described in the context of separate embodiments can also be implemented in combination. Conversely, various features described in the context of a single exemplary embodiment can also be implemented in multiple exemplary embodiments separately or in any suitable sub-combination.

(67) Although the drawings describe the UI views in a specific order, one should not interpret that the UI views are performed in a specific order as shown in the drawings or successively performed in a continuous order, or all the UI views are necessary to obtain a desired result. Also, it should be noted that all embodiments do not require the distinction of various system components made in this description. The device components and systems may be generally implemented as a single software product or multiple software product packages.

(68) A number of examples have been described above. Nevertheless, it is noted that various modifications may be made. For example, suitable results may be achieved if the described techniques are performed in a different order and/or if components in a described system, architecture, device, or circuit are combined in a different manner and/or replaced or supplemented by other components or their equivalents. Accordingly, other implementations are within the scope of the following claims.