Uninterruptable verification and control upgrade for real-time control system

Abstract

Exemplified system and method facilitates update/upgrade and validation of controls systems in an uninterrupted manner. An update/upgrade agent operates with a multiple-processor or multi-processor core system to simultaneously and concurrently implement two real-time embedded controllers in which a first controller executes a current firmware or control application installed at the site while a second controller executes an updated controller firmware or control application.

Claims

1. A method for operating an embedded control system for controlling an asset, the embedded control system comprising a first processor core and a second processor core, the method comprising: executing, by the first processor core, a first set of instructions to generate a first stream of control parameters; receiving, by the second processor core, a second set of instructions; executing, by the second processor core, the second set of instructions to generate a second stream of control parameters, wherein the second set of instructions are executed concurrently to the first set of instructions; monitoring a portion of the first stream and a corresponding portion of the second stream, or one or more parameters derived therefrom, for deviations based on a set of validation criteria and/or for instability conditions; validating the second stream in response to the monitoring; receiving a selection command associated with a selection of the second stream in response to the validation; and causing the second stream to be outputted to an input-output interface of the embedded control system to control the asset in response to the selection command.

2. The method of claim 1, wherein the first processor core continues to generate the first stream after the second stream generated by the second processor core is selected to be outputted to the input-output interface.

3. The method of claim 1, comprising: monitoring the second stream to determine if one or more parameters of the second stream meets one or more pre-defined conditions; and in response to a condition of the pre-defined conditions being met, causing the first stream to be outputted to the input-output interface of the embedded control system to control the asset.

4. The method of claim 1, wherein the portion of the first stream and the corresponding portion of the second stream, or the one or more parameters derived therefrom, are graphically presented at a user computing device, the presentation being made available from a cloud control infrastructure.

5. The method of claim 4, wherein comparison parameters derived from the portion of the first stream and the corresponding portion of the second stream are graphically displayed at the user computing device.

6. The method of claim 1, wherein the second set of instructions are received at an update/upgrade agent executing on the embedded control system.

7. The method of claim 1, wherein the second set of instructions are received at an update/upgrade agent executing on a field device operatively coupled to the embedded control system.

8. The method of claim 1, wherein the second set of instructions are received at an update/upgrade agent executing on a computing device located in a cloud control infrastructure.

9. The method of claim 1, wherein the first processor core and the second processor core are located on a same processor, wherein the first set of instructions are executed on a first hypervised instance executing on the first processor core, and wherein the second set of instructions are executed on a second hypervised instance executing on the second processor core.

10. The method of claim 1, wherein the first processor core is located on a first processor, and wherein the second processor core is located on a second processor.

11. The method of claim 1, wherein the first set of instructions and the second set of instructions each includes firmware instructions and operating system instructions.

12. The method of claim 1, wherein the first set of instructions and the second set of instructions each includes deterministic real-time control instructions.

13. The method of claim 1, comprising: validating, for a pre-defined period, the portion of the first stream and the corresponding portion of the second stream, or the one or more parameters derived therefrom; and generating a report, wherein the report includes differences between the portion of the first stream and the corresponding portion of the second stream.

14. The method of claim 1, wherein the deviations are established based on a set of historical data associated with the asset or a set thereof.

15. The method of claim 1, comprising: receiving a validation command that causes the second stream to be outputted to the input-output interface for a pre-defined period of time, wherein after the pre-defined period, the first stream is outputted to the input-output interface.

16. The method of claim 1, comprising: receiving a validation sequence command that comprises a sequence of validation commands, wherein with each successive validation command within the sequence, a given pre-defined period of time associated with a current validation command is lengthen compared to that of a previous validation command, wherein each validation command includes a pre-defined period that the second stream is outputted to the input-output interface such that, after the pre-defined period, the first stream is outputted to the input-output interface.

17. The method of claim 1, comprising: in response to receiving the second set of instructions, stopping execution of third instructions executing on the second processor core.

18. An embedded control system for controlling an asset, the apparatus comprising a first processor core and a second processor core in communication with a memory, the apparatus programmed to: execute, on the first processor core, a first set of instructions to generate a first stream of control parameters; receive a second set of instructions; execute, on the second processor core, the second set of instructions to generate a second stream of control parameters, wherein the second set of instructions are executed concurrently to the first set of instructions; monitor a portion of the first stream and a corresponding portion of the second stream, or one or more parameters derived therefrom, for deviations based on a set of validation criteria and/or for instability conditions; validate the second stream in response to the monitoring; receive a selection command associated with a selection of the second stream in response to the validation; and cause the second stream to be outputted to an input-output interface of the embedded control system to control the asset in response to the selection command.

19. A non-transitory computer readable medium having instructions stored thereon, wherein execution of the instructions, cause at least one processor to: execute, on a first processor core, a first set of instructions to generate a first stream of control parameters for controlling an asset; receive a second set of instructions; execute, by a second processor core, the second set of instructions to generate a second stream of control parameters, wherein the second set of instructions are executed concurrently to the first set of instructions; monitor a portion of the first stream and a corresponding portion of the second stream, or one or more parameters derived therefrom, for deviations based on a set of validation criteria and/or for instability conditions; validate the second stream in response to the monitoring; receive a selection command associated with a selection of the second stream in response to the validation; and cause the second stream to be outputted to an input-output interface to control the asset in response to the selection command.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments and together with the description, serve to explain the principles of the methods and systems:

(2) FIG. 1 illustrates an example update/upgrade agent configured to provide update/upgrade instructions for a control system in an uninterrupted manner and to provide in-situ validation of the update/upgrade instructions executing at the control system, in accordance with an illustrative embodiment.

(3) FIG. 2 is a diagram of an example state machine for the in-situ validation operation by a validation controller that operates with the update/upgrade agent, in accordance with an illustrative embodiment

(4) FIG. 3 is a diagram showing timing of the validation sequences of FIG. 2, in accordance with an illustrative embodiment.

(5) FIG. 4 is a flow diagram of a process of operating an update/upgrade agent to provide update/upgrade instructions for a control system in an uninterrupted manner and to provide in-situ validation of the update/upgrade instructions executing at the control system, in accordance with an illustrative embodiment.

(6) FIG. 5 is a diagram of an example hypervised real-time embedded control system, in accordance with an illustrative embodiment.

(7) FIG. 6 is a flow diagram of a process of operating an update/upgrade agent to provide update/upgrade instructions for a hypervised multi-core control system in an uninterrupted manner and to provide in-situ validation of the update/upgrade instructions executing at the control system, in accordance with an illustrative embodiment.

(8) FIG. 7 is a diagram of a local embedded control device with a hypervised platform, e.g., as described in relation to FIG. 5, in accordance with an illustrative embodiment.

DETAILED SPECIFICATION

(9) Before the present methods and systems are disclosed and described, it is to be understood that the methods and systems are not limited to specific methods or to specific components. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.

(10) As used in the specification and the appended claims, the singular forms a, an and the include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from about one particular value, and/or to about another particular value. When such a range is expressed, another embodiment includes from the one particular value and/or to the other particular value. Similarly, when values are expressed as approximations, by use of the antecedent about, it will be understood that the particular value forms another embodiment. It will be further understood that the endpoints of each of the ranges are significant both in relation to the other endpoint, and independently of the other endpoint.

(11) Throughout the description and claims of this specification, the word comprise and variations of the word, such as comprising and comprises, means including but not limited to, and is not intended to exclude, for example, other components or steps. Exemplary means an example of and is not intended to convey an indication of a preferred or ideal embodiment. Such as is not used in a restrictive sense, but for explanatory purposes. Disclosed are components that may be used to perform the disclosed methods and systems. These and other components are disclosed herein, and it is understood that when combinations, subsets, interactions, groups, etc. of these components are disclosed that while specific reference of each various individual and collective combinations and permutation of these may not be explicitly disclosed, each is specifically contemplated and described herein, for all methods and systems. This applies to all aspects of this application including, but not limited to, steps in disclosed methods. Thus, if there are a variety of additional steps that may be performed it is understood that each of these additional steps may be performed with any specific embodiment or combination of embodiments of the disclosed methods.

(12) FIG. 1 illustrates an example update/upgrade agent 100 configured to provide update/upgrade instructions for a real-time embedded control system 102 in an uninterrupted manner and to provide in-situ validation of the update/upgrade instructions executing at the control system 102, in accordance with an illustrative embodiment. As shown in FIG. 1, the real-time embedded control system 102 is configured to provide deterministic real-time control processes for an industrial asset 104. These deterministic real-time control processes, in some embodiments, are performed in sub-millisecond time period. A deterministic real-time control process for a motor or servo drive, for example, can include closed-loop PWM (pulse-width modulated) outputs and can be executed in the order of hundreds of microseconds (s) or less.

(13) Examples of other industrial assets 104 include plant assets such as, for example, but not limited to: equipment and machines used in power generation (e.g., gas turbines, wind turbines, solar power converters, stream turbine, nuclear reactors, plant controls, etc.), in power distribution (e.g., sub-station equipment, transmission equipment, etc.), in water and waste treatment (e.g., pumps, motors, relays, etc.), in recycling, in garbage processing (e.g., furnaces, conveyor system, sorters, etc.), in manufacturing (e.g., automation equipment, warehouse equipment, etc.), in healthcare (e.g., diagnostics equipment, imaging systems, lab equipment, etc.), in transportation (e.g., jet engines, locomotive engines, vehicles, cargo containers, cargo vessels, cargo trucks, freight transport, road vehicles, etc.), in refining, in mining (e.g., excavation equipment, etc.), in oil exploration (e.g., well equipment), in natural gas exploration (e.g., well equipment), in pipeline infrastructure (e.g., pumps, valves, monitoring equipment, etc.), in data centers (e.g., cooling equipment, etc.), among others. Further examples of industrial assets 104 are described in U.S. Prov. Appl. No. 62/435,610, filed Dec. 16, 2016, title INDUSTRIAL INTERNET CONNECTED CONTROL SYSTEM, which is incorporated by reference herein in its entirety.

(14) Referring to FIG. 1, the industrial asset 104 is operatively coupled to an IO interface 106 (input-output interface) of the real-time embedded control system 102. In some embodiments, the IO interface 106 is configured to receive control inputs (e.g., current or voltage inputs 106a) from sensors associated with the industrial asset 104 and to generate control outputs (e.g., PWM outputs 106b, protection outputs 106c, and etc.) to regulate the industrial asset 104, as well as ports for serial or parallel communication 106d.

(15) Referring still to FIG. 1, the real-time embedded control system 102 includes a first set of processor cores 108 and a second set of processor cores 110. Here, a set refers to one or more processor cores. In some embodiments, the first set of processor cores 108 and the second set of processor cores 110 are part of a same, single multi-core processor component (e.g., chipset). A processor core is an independent processing unit that read and execute program instruction and can be located with other cores within a single computing component (e.g., a multi-core CPU chipset). In other embodiments, the first set of processor cores 108 is part of a first processor or first multi-core processor component (e.g., a first multi-core CPU chipset) and the second set of processor cores 110 is part of a second processor or multi-core processor component (e.g., a second multi-core CPU chipset).

(16) The processor cores 108 and 110 are coupled to memory 112 configured to store, at least, first instructions 114 and second instructions 116. In some embodiments, the first and second instructions 114, 116 each includes controller firmware instructions, operating system instructions, and control application instructions. Control application are operational instruction relating to the control of the industrial asset (e.g., control loop) and include read-only portion that fixed and write portion that may include control state information. Firmware are read-only instructions that includes underlying instructions for device control, device monitoring, and data manipulation, among others. Firmware can also include instructions to protect the real-time embedded control system from unauthorized intrusion, hacking, and the like and to monitor for malicious instructions to which the real-time embedded control system may be exposed. The second instructions, which is intended to replace the first instructions, preferably include updated instructions to firmware, operating system, or control application, and addresses an existing vulnerability or issue in such instructions. These vulnerability may be unrelated to the operation of the real-time embedded control system but makes the real-time embedded control system vulnerable to corruption, unintended manipulation in some manner. In some embodiments, the second instructions include control state information for the controller firmware instructions, operating system instructions, and control application instructions. These control state information may include setpoint and operational values relating to the execution of controller firmware instructions, operating system instructions, and control application instructions. In some embodiments, the control state information are preserved and stored as an image capture of execution of controller firmware instructions, operating system instructions, and control application instructions on a first set of hardware/device and is replicated for execution on a second set of hardware/device.

(17) Referring still to FIG. 1, the real-time embedded control system 102 is commutatively coupled to the update/upgrade agent to receive the second instructions 116 from a local or remote controller (shown as Control Management Infrastructure 117). The update/upgrade agent 100 serves a relay/gateway for updatable or upgradable instructions for a given real-time embedded control system 102 or a set of real-time embedded control system (in FIG. 1, a second real-time embedded control system 102b is shown). In some embodiments, the control management infrastructure 117 is a local controller configured to communicatively couple to the update/upgrade agent 100. In some embodiments, the user interface to the control management infrastructure 117 is mobile friendly and cloud aware such that an end-user can access, via said interface (e.g., on a user computing device 130), validation data, generate reports, and perform the upgrade from a remote location.

(18) In some embodiments, the control management infrastructure 117 is a cloud-based control infrastructure. That is, storage, networking, processing computing resources in a network are used to archive and push the various instructions and versions thereof to the update/upgrade agent 100. In some embodiments, the cloud-based control infrastructure provides notifications/alerts of errors and validation issues detected by the update/upgrade agent 100 during the update/upgrade event. In some embodiments, the cloud-based control infrastructure provides a user portal to access reports with summary of the validation status and results.

(19) Referring still to FIG. 1, the first set of processor cores 108 execute the first instructions 114 to provide deterministic control outputs (shown as First computed outputs 118) to control the industrial asset 104 (e.g., via open-loop control; or via closed-loop control using feedback from control inputs received at the TO interface 106). The first computed outputs 118 form a first stream of control parameters over a number of control cycles. In some embodiments, the first computed outputs 118 are stored, for each control cycle, in a first location in memory 112 to be accessed by digital logic circuitries (e.g., a field-programmable gate array (FPGA), complex programmable logic device (CPLD), application-specific integrated circuit (ASIC), or combination logics) that implements a selector circuit 120. In addition, the first set of processor cores 108 stores updated control state information (e.g., setpoint and model parameters), for example, within a shared memory space to be accessed by the second set of processor cores 110.

(20) The second set of processor cores 110 uses the control state information when executing elements of the second instructions 116 to provide a same deterministic control outputs (shown as Second computed outputs 122) that can also be used to also control the industrial asset 104. The second computed outputs 122 are concurrently stored (with the first computed outputs 118), for each control cycle, in a second location in memory 112. The second computed outputs 122 form a second stream of control parameters over a number of control cycles. In some embodiments, the control state information generated from execution of the second instructions by the second set of processor cores are stored in a second memory location to be evaluated against the control state information generated by the first set of processor cores.

(21) The selector circuit 120 is configured to access, e.g., via a local bus (not shown), the first or second computed outputs 118, 122 stored in the memory 112 and to provide a selected computed outputs (e.g., 118 or 122) to an output circuit 124 (e.g., a digital-to-analog converter (DAC), a PWM output) that outputs a control output signal at the IO interface 106 for regulating and/or controlling the industrial asset 104.

(22) Referring still to FIG. 1, the first computed outputs 118 and the second computed outputs 122, or a parameter derived therefrom, and generated control state information associated with execution of each of the set of cores, are made locally available to the update/upgrade agent 100 (i.e., a local or remote agent) over a bus or communication interface shown as interface 126. In some embodiments, the bus or communication interface 126 is a high-speed data or communication bus that operatively couples to the update/upgrade agent 100 which is implemented as part of the real-time embedded control system 102. In other embodiments, the bus or communication interface 126 is a high-speed communication channel that operatively couples to the update/upgrade agent 100 which is implemented in a standalone controller or a remote controller.

(23) In some embodiments, the real-time embedded control system 102 includes real-time code that directs the first or second set of processor cores, or a portion thereof, to combine the first computed outputs 118 and the second computed outputs 122 (e.g., via an XOR operator), and the two sets of generated control state information, to generate a combined output to be transmitted to the update/upgrade agent 100 over a bus or communication interface 126. In some embodiments, the real-time embedded control system 102 includes real-time code that directs the first or second set of processor cores, or a portion thereof, to compare the first computed outputs 118 and the second computed outputs 122 and to compare the different set of control state information to generate a difference output to be transmitted to the update/upgrade agent 100, e.g., over a bus or communication interface 126. The validation assures that differences between the first controller and second controller of the real-time embedded control system 102 are expected. In some embodiments, the update/upgrade agent 100 is configured to generate reports for evaluation when it observes differences that are not expected.

(24) In-Situ Control Validation

(25) As discussed in relation to FIG. 1, the selector circuit 120 is configured to access, e.g., via a local bus, the first or second computed outputs 118, 122 stored in the memory 112 and to provide a selected computed outputs (e.g., 118 or 122) to an output circuit 124 (e.g., a digital-to-analog converter (DAC), a PWM output) that outputs a control output signal at the IO interface 106 for regulating and/or controlling the industrial asset 104. The selector circuit 120 and operation therewith facilitates uninterrupted and live operation of the real-time embedded control system 102 and the industrial asset 104 during the control update and validation processes.

(26) In some embodiments, other validation tools to evaluate the firmware and operating system are also executed on the real-time embedded control system. For example, various memory locations of the real-time embedded control system may be scanned and off-loaded to be analyzed to assess expected or unexpected operations of various components of the firmware and operating system.

(27) In some embodiments, the selector circuit 120 is configured to select between the first computed outputs 118 or the second computed outputs 122 via a selection command (also referred to as an in-situ switch-over command) transmitted from the update/upgrade agent 100. In some embodiments, the selector circuit 120 is configured to temporarily select the second computed outputs 122 for a single or finite number of control cycles and then to switch the selection back to the first computed outputs 118. In some embodiments, a validation controller 128 in conjunction with the update/upgrade agent 100 is configured to validate a single or finite number of sample periods of the control cycle. The in-situ switch-over command also serves as persistent switch-over command, e.g., after validation is completed and as a switch-back command in the event of detected fault condition associated with execution of the updated instructionsneither of which would result in an interruption to the operation of the industrial asset 104 or the real-time embedded control system 102.

(28) In some embodiments, the update/upgrade agent 100 directs a state machine in the real-time embedded control system 102 to perform the validation across the single or multiple sample periods. Because the real-time embedded control system has multiple states of operation, multiple sample periods facilitate a complete validation by confirming outputs and internal states at each of the multiple states of operations.

(29) In some embodiments, following the initial single sample period validation being completed, the update/upgrade agent 100 enters a mode that validates the control outputs over multiple sample periods.

(30) In some embodiments, the number of sample periods that the new system is configured to control is incrementally increased following each successful validation period. For example, the validation sequence may start with a 10-sample validation period and then incrementally increase the validation period to 20 samples in which control is transferred back to the original control system during periods that validation is not occurring. Other number of samples and validation sequences may be used.

(31) FIG. 2 is a diagram of an example state machine 200 for the in-situ validation operation by a validation controller (e.g., real-time embedded control system 102) that operates with the update/upgrade agent 100, in accordance with an illustrative embodiment. In some embodiments, the state machine 200 is executed on a validation controller 128 (see FIG. 1).

(32) In some embodiments, the validation controller 128 is implemented as digital logic circuitries, or via one of the sets of processor cores, that forms a local controller in the real-time embedded control system 102. In other embodiments, the validation controller 128 is a remote controller commutatively coupled to the real-time embedded control system 102 over a high-speed and deterministic link. In some embodiments, the validation controller 128 is a part of the update/upgrade agent 100.

(33) Referring still to FIG. 2, the state machine 200 includes an initial waiting state (shown as State 0 202) as the validation controller 128 waits for a command from the update/upgrade agent 100 to initiate a validation sequence for the control of the industrial asset 104. Validation of the firmware operations and operating system operations may be performed prior to validation of the control operations, e.g., while the state machine 200 is in the waiting state. In the waiting state, the state machine 200, in some embodiments, directs the validation controller 128 to compare the first computed outputs 118 (shown as computed_output_1) with the second computed outputs 122 (shown as computed_output_2). In some embodiments, the comparison is performed by one or more of the first or second set of processor cores 108, 110 or by digital logic circuits. The first and second computed outputs 118, 122 may include bit-wide register data, multi-bit wide register data, byte-wide register data, and multiple-byte wide register data associated with digital outputs and analog outputs (e.g., integers and floating numbers) of the controls. In some embodiments, a digital circuit (e.g., in a FPGA, CPLD, ASIC, and etc.) is configured to compare individual register values to detect any differences therebetween. In some embodiments, the memory locations, and memory blocks thereof, associated with the first and second computed outputs 118, 122 are compared to detect any differences in the register values therein.

(34) In some embodiments, the test program is used to change inputs to the controls and to observe whether an expected outcome is generated.

(35) In FIG. 2, two example validation sequences 204 and 206 are shown. In the first validation sequence 204 corresponding to State 1 208, the state machine 200 facilitates the initial single sample period validation. The first validation sequence 204, in some embodiments, includes initializing (210) a validation window period n to an initial state (shown in FIG. 2 as 0). The state machine 200 directs then the real-time embedded control system 102, via the selector circuit 120, to select (212) the second computed outputs 122. The state machine 200 then increments the validation window period n. The state machine 200 then directs the real-time embedded control system 102 to select (214) the first computed outputs 118. The first validation sequence 204 confirms proper connections among the hardware and software circuits of the real-time embedded control system 102.

(36) Referring still to FIG. 2, in the second validation sequence 206 corresponding to State 2 218, the state machine 200 facilitates multiple sample period validations of incrementally varying periods. The second validation sequence 206, in some embodiments, includes initializing (220) the validation window period n to an initial state. The state machine 200 then directs the real-time embedded control system 102, via the selector circuit 120, to select (222) the second computed outputs 122 and to switch back (226) to the first computed outputs 118 after the validation window period n has passed. During each switch over, the state machine 200 directs (224) the validation controller 128 to compare the first computed outputs 118 with the second computed outputs 122 and to direct the real-time embedded control system 102 to switch-back to the first computed outputs 122 upon a mismatched being detected. The state machine 200 then directs the real-time embedded control system 102 to increment (230) the validation window period n for a next validation sequence following a pre-defined waiting state (228). The second validation sequence 206 increases confidence in the second instructions in properly and correctly generating outputs for the real-time embedded control system 102. Because of the control speed of the real-time embedded control system 102, the validation sequences 204, 206 (as shown in FIG. 2) may take only a few seconds or minutes to perform.

(37) FIG. 3 is a diagram showing timing of the validation sequences of FIG. 2, in accordance with an illustrative embodiment. As shown in FIG. 3, the real-time embedded control system 102 is operating live in generating the first computed outputs 118 from the control inputs and providing, based on the first computed outputs 118, control outputs (302) to regulate the industrial assets 104. Upon the second instructions 116 being provided to the real-time embedded control system 102 by the update/upgrade agent 100, the second set of processor cores 110 execute the second instructions to generate (shown as time 304) the second computed outputs 122. The second computed outputs 122 are generated for some time 306 until a validation sequence is executed. As discussed in relation to FIG. 2, validation of the firmware and operating system operations may be performed prior to validation of the control operations. The validation sequences may be initiated by a technician or engineer.

(38) A first validation sequence 310, as described in relation to FIG. 2, is performed at after time 306 in which the selector circuit 120 selects the second computed outputs 122 for one of the control cycle and then switches back to the first computed outputs 118. As shown in FIG. 3, the second computed output 122 for a given control cycle n (312) is selected to be outputted in a next control cycle (314).

(39) Following the first validation sequence 310, a second validation sequence comprising of multiple sample period validations of incrementally varying periods (shown as sequences 316a and 316b), as described in relation to FIG. 2, is then performed in which the selector circuit 120 selects the second computed outputs 122 for an increasing number of the control cycles and switch back to the first computed outputs 118. As shown in FIG. 3, the second computed output 122 for a given control cycle n (312) is selected to be outputted in a next control cycle (314). Although two sequences 316a, 316b are illustrated, other number of repetitions and varying of the periods may be used.

(40) After the validation is complete, at some point, the update/upgrade agent 100 provides a persistent switch-over command to the real-time embedded control system 102 to select the second computed outputs 122. Notably, the first set of processor cores 108 continue to generate (318) the first computed outputs 118 for an extended period of time after the validation is completed. In some embodiments, this period can be days or months after the switch-over and validation period has been completed. The first set of processor core continues to generate the first computed outputs, which serves as a backup stream to the second computed outputs. To this end, if issues are observed (e.g., by the operator or via notification from the update/upgrade agent), the end user or operator can switch the real-time embedded control system to using the outputs from the older firmware version (i.e., the first instructions). In some embodiments, the update/upgrade agent is configured to continue to generate periodic reports of the evaluation of the newer firmware version (i.e., the second instructions). In some embodiments, the first set of processor cores 108 continues to generate (318) the first computed outputs 118 until a next update or upgrade event is performed.

(41) The in-situ validation process, described herein, facilitates earlier and more frequency updates and/or upgrades of lower level control operations, firmware operations, and operating system operations, of the real-time embedded control systems; many of these lower-level control operations not necessary being related to the control loops associated with control of the industrial asset. As discussed, these updates and/or upgrades can address flaws and vulnerabilities of the real-time embedded control system to corruption and/or undesired manipulation.

(42) In some embodiments, the update/upgrade agent 100 includes validation modeling tools configured to monitor the real-time control application for significant deviations among the controllers (i.e., the first set and second set of processor cores) and for significant instabilities in the operation of the real-time control and the underlying firmware and operating system. In some embodiments, the update/upgrade agent 100 is configured to establish, using historical data (e.g., associated with the first computed outputs), output bounds in the controls associated with nominal behavior. In some embodiments, the output bounds are established based on a model generated from the historical data in which the model defines an outside bound for an automatic switch back to the original control system (e.g., the first computed outputs). In some embodiments, the control output bounds is selectable as a manual input that is provided by an end user.

(43) Example In-Situ Update/Upgrade

(44) FIG. 4 is a flow diagram of a process of operating an update/upgrade agent to provide update/upgrade instructions for a control system in an uninterrupted manner and to provide in-situ validation of the update/upgrade instructions executing at the control system, in accordance with an illustrative embodiment. In FIG. 4, the first set of processor cores 108 is shown executing a number of real-time control loops (shown as 402a, 402b) with the industrial asset 104. For each control loop (e.g., 402a and 402b), the real-time embedded control system 102, via execution of the first instructions 114, receives (404) feedback control inputs from the industrial asset 104, execute (406) the run time control code to generate the first computed outputs 118, which is selected (408) and used to generate (410) control outputs to the industrial asset. In some embodiments, the feedback control inputs 404 and computed outputs 118 are generated in a given cycle and the control outputs to the industrial asset are provided at a next control cycle. To this end, events shown in FIG. 4 may occur over more than one control cycles.

(45) Referring still to FIG. 4, the update/upgrade agent 100 is shown transmitting (412) the second instructions 116 (with update/upgraded firmware, operating system, and/or control application code) to the real-time embedded control system 102 to be executed by the second set of processor cores 110. Upon receiving the second instructions 116, the real-time embedded control system 102 may store the second instructions 116 and direct the second set of processor cores 110 to execute (414) the stored firmware, operating system, and control operations instructions.

(46) In some embodiments, the first and second set of processor cores 108, 110 are part of a single processor unit (e.g., a multi-processor core chipset). In such embodiments, the real-time embedded control system 102 is configured to instantiate, via a hypervisor, a second instance of the real-time control, operating system, and firmware to be executed by the second set of processor core 110 using the second instructions 116.

(47) Once operating, the first set of processor cores 108 and second set of processor cores 110 each concurrently executes, for a given set of control cycle, the real-time control loops (shown as 416a, 416b, 416c) with the industrial asset 104. As shown in FIG. 4, for each control loop (e.g., 416a, 416b, 416c, respectively), the real-time embedded control system 102, via execution of the first instructions 114 by the first set of processor cores 108, receives (418a, 418b, 420c, respectively) feedback control inputs from the industrial asset 104, execute (420a, 420b, and 420c, respectively) the run time control code to generate the first computed outputs 118, which is selected (422a, 422b, and 422c, respectively) and used to generate (424a, 424b, 424c, respectively) control outputs to the industrial asset 104. In addition, for each of the control loops (e.g., 416a, 416b, 416c, respectively), the real-time embedded control system 102, via execution of the second instructions 116, accesses (426a, 426b, 426c, respectively) feedback control inputs (418a, 418b, 418c respectively) provided by the industrial asset 104 and control state information generated from execution of the first set of processor cores 108, to generate the second computed outputs 122.

(48) Referring still to FIG. 4, validation sequences, as described in relation to FIGS. 2 and 3 may be performed to validate expected operations of the second instructions 116 by the second set of processor cores 110. During the validation period, the firmware, operating system, and underlying control application is evaluated. In addition, temporary in-situ switch-over (e.g., for a single or finite number of control cycles) to operation via the second instructions as described in relation to FIGS. 2 and 3 may also be performed. In addition, application changes can be made online via test modes in which programmer or end user can make a change to the running system and monitor the effects. In such test modes, if the change produces an undesired effect, the changes can be backed out of the system.

(49) In some embodiments, as the first and second computed outputs 118, 122 are generated, in some embodiments, the validation controller 128 aggregates and/or summarizes and transmits (430, 432) the computed outputs to the update/upgrade agent 100 to be included in a validation report. In some embodiments, the validation controller 128 compares differences of the first and second computed outputs 118, 122 and generate an alert, notification, or report to the update/upgrade agent 100. In some embodiments, the update/upgrade agent 100 is configured to generate and transmit (434) a corresponding alert, notification, or report to the control management infrastructure 117 to present to the end user (e.g., technician or engineer). In some embodiments, the user interface of the control management infrastructure is mobile-friendly and cloud aware such that an end-user can access validation data, generate reports, and perform the upgrade from a remote location.

(50) Referring still to FIG. 4, once validation is completed, an end user can send (438) an in-situ switch-over command to the update/upgrade agent 100 to direct (440a, 440b) the real-time embedded control system 102 to use the second computed output 122 to regulate the industrial asset 104. In some embodiments, a command (440a, 440b) is sent to each of the first set of processor cores 108 and the second set of processor cores 110, e.g., to direct the second set of processor cores to generate the control state information to share with the first set of processor cores.

(51) The switch-over may be performed in stages. In some embodiments, in the first switch-over stage, each of the first set of processor cores 108 and the second set of processor cores 110 may execute control instructions that access data associated with the feedback inputs (e.g., voltage, current, and digital inputs corresponding to power levels, current levels, temperature levels, pressure levels, volume levels, and etc.) of the industrial asset 104 while the underlying processing of the feedback inputs is performed by hardware executing the first instructions 114. These underlying processes may include the instructions that access hardware registers (e.g., ADC (analog-to-digital convertor) registers, counters registers, and etc.). In some embodiments, in a second switch-over stage, the underlying processing of the feedback inputs is performed by hardware executing on the second instructions 114. In some embodiments, the switch-over includes both the control applications and the underlying processes that directly interfaces with hardware registers.

(52) In FIG. 4, each of the first set of processor cores 108 and the second set of processor cores 110 is shown (in control loops 442a and 442b) to continue to execute control instructions that independently access data associated with the feedback inputs. To this end, performance of the updated controllers can continually be compared in-situ in an extended validation sequence (shown as 446) while the older controller control system continues to run. During the extended validation sequence, the operation may be actively monitored by the validation controller and/or update/upgrade agent, but not by an end user or technician.

(53) Example Hypervised Real-Time Embedded Control System

(54) FIG. 5 is a diagram of an example hypervised real-time embedded control system 500 with a multiple core processor, in accordance with an illustrative embodiment. The hypervised real-time embedded control system 500 facilitates execution of two real-time controllers simultaneously within a same platform. Notably, the hypervised real-time embedded control system 500 provides for a miniaturized real-time embedded control device with a small-footprint board design that allows for smaller control cabinets to be used. Particularly for industrial environment, smaller controller cabinets is desired.

(55) As shown in FIG. 5, the hypervised real-time embedded control system 500 includes a multi-core processor 504 (shown as CPU) with two or more independent processing units (shown as Core #1 506a, Core #2 506b, Core #3 506c, and Core #4 506d). In some embodiments, and as shown in FIG. 5, each independent processing unit (506a-506d) operates with a corresponding L1 Cache (shown as 408a, 408b, 408c, and 408d) and to a shared L2 Cache 510 and a shared L3 Cache 512.

(56) In this embodiment, and as shown in FIG. 5, the hypervisor 502 segments the CPU (e.g., shown as a quad-core processor) into two independent types of virtualized environment shown as a privileged environment 514 (shown as O/S #2Privileged) that executes the real-time control engine(s) and as a virtualized environment 516 (shown as O/S #1Virtualized). Each CPU (e.g., 506c and 506d) allocated to the privileged environment 514 may be configured to operate an instance of the real-time embedded control system 102. That is, CPU 506c may be configured to execute the first instructions 114 and CPU 506d may be configured to execute the second instructions 116, as described in relation to FIGS. 1-4.

(57) In some embodiments, the virtualized environment 516 is configured to operate a second type of deterministic control loop (referred to as a near-real-time control loop) with the privileged environment 514. This second deterministic control loop operates concurrently with the real-time control loops (e.g., 402a-402b, 416a-416c, as described in relation to FIG. 4) to provide controllable data from a cloud-based infrastructure in the control of the real-time embedded control system 102. Additional description of near-real-time control loop is provided in U.S. Prov. Appl. No. 62/435,610, filed Dec. 16, 2016, title INDUSTRIAL INTERNET CONNECTED CONTROL SYSTEM, which has been incorporated by reference herein in its entirety.

(58) Example In-Situ Update/Upgrade for Multi-Hypervised Environment

(59) In some embodiments, the second instructions 116 includes firmware, operating system, and control applications for both the privileged environment 514 and the virtualized environment 516 to facilitate update or upgrade of the real-time control loop and the near-real-time control loop.

(60) FIG. 6 is a flow diagram of a process of operating an update/upgrade agent to provide update/upgrade instructions for a hypervised multi-core control system 500 in an uninterrupted manner and to provide in-situ validation of the update/upgrade instructions executing at the control system, in accordance with an illustrative embodiment.

(61) In one aspect, FIG. 6 shows the update/upgrade of second instruction 116 via the hypervised multi-core control system 500. Similar to FIG. 4, a first instance 602 (e.g., executing via CPU 506c) of the real-time controller is shown executing a number of real-time control loops (shown as 402a, 402b) with the industrial asset 104. For each control loop (e.g., 402a and 402b), the real-time embedded control system 102, via execution of the first instructions 114, receives (404) feedback control inputs from the industrial asset 104, execute (406) the run time control code to generate the first computed outputs 118, which is selected (408) and used to generate (410) control outputs to the industrial asset. Then, the update/upgrade agent 100 is shown transmitting (412) the second instructions 116 (with update/upgraded firmware, operating system, and/or control application code) to the real-time embedded control system 102 to be executed by the CPU 506d.

(62) Upon receiving the second instructions 116, the real-time embedded control system 102 may store the second instructions 116 and direct the hypervisor 502 to instantiate (604) and execute (606) a second instance 608 (e.g., via the CPU 506d) of the real-time controller using the second instructions 116 in which the second instance 608 includes updated firmware, updated operating system (and/or components), and/or updated control operation instructions. Once operating, the first instance 602 and second instance 608 of the real-time controller each concurrently executes, for a given set of control cycle, the real-time control loops (shown as 416a, 416b) with the industrial asset 104, for example, as described in relation to FIG. 4.

(63) In another aspect, FIG. 6 shows the update/upgrade of second instruction 116 for controls of the real-time control loop (shown as 402a-402b and 416a-416b) and the near-real-time control loop (shown as 610a and 610b). As shown in FIG. 6, the near-real-time control loop 610a provides closed-loop deterministic control between the real-time embedded control system 102 and a field agent 612. The field agent 610 may be executed in the virtualized environment 516, as described in relation to FIG. 5. In this example shown in FIG. 6, the field agent 612 transmits output feedback (614) to the first instance 602 of the real-time controller. The first instance 602 of the real-time controller uses the transmitted output feedback in its real-time control loops (e.g., 402a, 402b) to regulate the industrial asset 104 and provide feedback (616) to the field agent 612.

(64) As shown in FIG. 6, upon receiving the second instructions 116, the hypervisor 502 also instantiates (618) and executes a second instance 620 (e.g., via the CPU 506d) of the field agent. The second instance 620 executes another instance of the near-real-time control (shown as 622) by transmitting output feedback (624) to the second instance 608 of the real-time controller and receives (626) feedback therefrom.

(65) Validation (628) is performed for both the second instance 608 of the real-time controller and the second instance 620 of the field agent, e.g., via various embodiments discussed in relation to FIG. 4.

(66) Example Local Embedded Control Device with Multi-Core Hypervised Platform

(67) FIG. 7 is a diagram of a local embedded control device with a hypervised platform 502, e.g., as described in relation to FIG. 5, in accordance with an illustrative embodiment. In some embodiments, and as shown in FIG. 7, the local embedded control device may have a small footprint, e.g., to maximize limited cabinet space, while also providing mounting support (e.g., for DIN rails) to facilitate ease of installation of the local embedded control device. The local embedded control device is preferably built for harsh environment, e.g., using COM (computer-on-module) Express architecture, and that can operate in extended temperature ranges for demanding industrial applications. It is contemplated that other ruggedized standard may be used. In some embodiments, and as shown in FIG. 7, the local embedded control device includes a number of Gigabit connections to provide connections for controller I/O, direct messaging, or handling of data. In some embodiments, the local embedded control device executes Technology Support Net (TSN) protocols to allow multiple protocols to operate over a single set of wires. In some embodiments, and as shown in FIG. 7, the local embedded control device includes a quad-core processor with embedded hypervisor 502 and cloud gateway environment to facilitate execution and update/upgrade of field agent control applications that can execute in parallel to the main control (e.g., real-time control applications). In some embodiments, the processor can include a number of processing cores selected from the group consisting of 2 cores, 3 cores, 4 cores, 6 cores, 8 cores, 10 cores, 12 cores, 14 cores, and 16 cores. The field agent control applications may include service apps, e.g., for analysis and action. In some embodiments, and as shown in FIG. 7, the local embedded control device includes advanced security and is, e.g., Achilles certified for critical communication robustness testing of the Virtual Machine (VM) against service attacks. In some embodiments, secure boot operations is used to prevent malicious application and unauthorized operating systems services from being executed. In some embodiments, secure boot operations include validating the firmware and validating the image and boot instructions, e.g., for the hypervisor, the real-time operating system, for the kernel instructions, and etc. In some embodiments, a Trusted Platform Module comprising a secure crypto-processor, which is a dedicated microcontroller designed to secure hardware by integrating cryptographic keys into devices, operates with the CPU (e.g., 504a-504d), and operates in accordance with the Trusted Platform Module (TPM) standard. In some embodiments, and as shown in FIG. 7, the local embedded control device includes IO interface, e.g., that operates Profinet, to provide high-speed data exchange with the real-time embedded control system 102.

(68) It will be apparent to those skilled in the art that various modifications and variations can be made without departing from the scope or spirit. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit being indicated by the following claims.

(69) In some embodiments, the exemplified system and method is implemented in a duplex control systems comprising two separate controllers each having its own set of controller hardware.