SYSTEM FOR REMOTE ASSISTANCE OF A FIELD OPERATOR
20230221792 · 2023-07-13
Assignee
Inventors
Cpc classification
G05B2219/32014
PHYSICS
G06F3/011
PHYSICS
International classification
Abstract
A system (1) for assisting a field operator, in particular a maintenance field operator, by a remote assistant (51) equipped with a viewer (50) comprises a first group of components configured to be connected to a support (10) wearable on the operator's head, including a first video camera (11), a local viewer (13) arranged to be watched by the operator, a first control unit (15) and a first wireless connection interface (19). The system also comprises a handpiece (20) providing a visual sensor (21) such as a video camera or a thermal imaging camera, a serial port (24) for connecting a peripheral device (30) such as an instrument or a sensor or a video source, a second control unit (25) and a second connection interface (26). The connection interfaces (15,25) are configured for mutually changing, locally or through a remote server (99) available through a data network (3), data streams (12,22,32) coming from the video camera (11), from the visual sensor (21) or from the peripheral device (30), respectively, intended to be displayed by the local viewer (13). Moreover, the connection interfaces are configured for exchanging the data streams with the assistants (51) remote viewer (50) through the remote server (99). The system (1) also comprises a scenario-switching device (80) for displaying a same data stream of interest (42) at the same time to the local viewer (13) and to the remote viewer (50), said data stream of interest selected among data streams (12,22,32) related to the scenarios (11′,21′) framed by the video camera (11) and by the visual sensor (21), respectively, and to the data obtained from the peripheral device (30).
Claims
1. A system (1,1′, 2,2, 2″) for assisting a field operator, in particular a maintenance field operator, by a remote assistant (51), said system comprising: a first video camera (11) configured: to be fixed on a support (10) wearable on said field operator's head; to frame a first scenario (11′) from said support (10); to generate a first data stream (12) related to said first scenario (11′); a local viewer (13) configured to be fixed to said wearable support (10) and to be positioned along a line of sight of said field operator; a first microprocessor processing unit (15) configured to be fixed on said wearable support (10); a first wireless connection interface (19) to a global data network (3), configured to be fixed to said wearable support (10) and to exchange data with a remote server (99) through said global data network (3); a handpiece (20) comprising: a visual sensor (21) configured: to frame a second scenario (21′) from said handpiece (20); to generate a second data stream (22) related to said second scenario (21′); a serial communication port (24) to connect an instrument (30), said serial communication port (24) configured to receive an additional data stream (32) from said instrument (30); a second microprocessor processing unit (25); a second connection interface (26,29) selected between: an inner second connection interface (26) between said second microprocessor processing unit (25) and said first microprocessor processing unit (15); an outer second connection interface (29) to said global data network (3), configured to exchange data with said remote server (99) through said global data network (3), wherein said second microprocessor processing unit (25) is functionally connected: with said visual sensor (21), so as to receive said second data stream (22); with said second connection interface (26,29), so as to emit said second data stream (22), with said serial communication port (24), so as to receive said additional data stream (32), such that said outer second connection interface (29) can also emit said additional data stream (32), wherein said first microprocessor processing unit (15) is functionally connected: with said first video camera (11), so as to receive said first data stream (12); with said inner second connection interface (26) or with said outer second connection interface (29) of said handpiece (20) through said remote server (99) and through said global data network (3), so as to receive said second data stream (22) or said additional data stream (32); with said local viewer (13), so as to transfer and remotely display said second data stream (22) or said additional data stream (32) to/by said local viewer (13); with said first wireless connection interface (19), so as to exchange said first data stream (12), or said second data stream (22), or said additional data stream (32) with said remote server (99) or with said second inner or outer connection interface (26,29); a scenario-switching device (80) functionally connected with said first microprocessor processing unit (15) or with said second microprocessor processing unit (25), wherein said scenario-switching device (80) is configured to select a data stream of interest (42) selected between said first data stream (12) and said second data stream (22), wherein said scenario-switching device (80) is also configured to select said additional data stream (32) as said data stream of interest (42), and to generate a scenario-switching signal (81) corresponding to said data stream of interest (42), which include either said first data stream (12), or said second data stream (22), or said additional data stream (32), wherein a computer device selected between said first microprocessor processing unit (15) and said server (99) is configured to receive said scenario-switching signal (81) and to selectively transfer said data stream of interest (42) responsive to said scenario-switching signal (81) both to said local viewer (13); and to a remote viewer (50), so that said field operator, on said local viewer (13), and said remote assistant (51), on said remote viewer (50), both display: either an image of said first scenario (11′); or, as an alternative, an image of said second scenario (21′); or, as an alternative, said additional data stream.
2. The system according to claim 1, wherein said scenario-switching device (80) comprises a scenario-switching control means (20d) provided in a device selected from the group consisting of: said handpiece (20); a housing (18) of said first microprocessor processing unit (15); a remote control device; a terminal associated with said remote viewer (50).
3. The system according to claim 1, wherein said scenario-switching control means is a push button (20d), and said scenario-switching device is configured in such a way to select said first data stream (12), said second data stream (22) and said additional data stream (32) as said data stream of interest by a predetermined number of strokes given to said push button (20d).
4. The system (1′) according to claim 1, wherein: said first wireless connection interface (19) comprises: an external communication WiFi card (191) configured to connect with an ambient WiFi network (5) providing an access point (6) to said global data network (3), in particular to the Internet; an internal communication WiFi card (192) configured to create a local WiFi network (7); said inner second connection interface (26) comprises a further WiFi card (261) configured to connect with said local WiFi network (7) created by said internal communication WiFi card (192).
5. The system (1′) according to claim 4, wherein said ambient WiFi network (5) and said local WiFi network (7) are configured to use different frequency bands, in particular said ambient WiFi network (5) is a 2.4 GHz network, while said local WiFi network (7) is a 5 GHz network.
6. The system (1′) according to claim 4, wherein said first wireless connection interface (19) comprises a connection kit provided with a sim-card for connection to said global data network (3) and configured to create said ambient WiFi network (5).
7. The system according to claim 1, wherein said first connection interface (19) and said inner second connection interface (26) are configured to exchange data with each other through a Bluetooth network.
8. The system (2′) according to claim 1, wherein said first wireless connection interface (19) and said outer second connection interface (29) each comprise a respective device selected from the group consisting of: a WiFi card configured to connect with an ambient WiFi network (5) providing an access point (6) to said global data network (3), in particular to the Internet; a connection kit provided with a sim-card for connection to said global data network (3).
9. The system according to claim 1, wherein said visual sensor (21) is selected from the group consisting of: a second video camera; a thermal imaging camera; an integrated visual sensor selectively providing the operations of a video camera and of a thermal imaging camera.
10. The system according to claim 1, wherein said second microprocessor processing unit (25) is configured to suspend an emission of said second data stream (22) when it receives/emits said additional data stream (32).
11. The system according to claim 1, wherein said serial communication port (24) is a USB-type port.
12. The system according to claim 1, further comprising said instrument (30), which is selected from the group consisting of: a microscope; a borescope; a thermometer; a force sensor; a stroke sensor; a concentration sensor for a chemical compound; a hygrometric sensor; an illuminance sensor; an electromagnetic radiation sensor; at least one video camera of a closed-circuit environmental imaging system; a combination thereof.
13. The system according to claim 1, wherein said handpiece 20 comprises a video-to-data conversion device associated with said serial communication port (24), such that said second microprocessor processing unit (25) can receive data from said instrument (30) configured as a video source, in particular from a borescope or from a microscope.
14. The system according to claim 1, wherein said handpiece (20) has an elongated shape and comprises: a central handle portion (27) configured to be grasped by said field operator; two opposite end portions (26,28), wherein said visual sensor (21) is arranged in a first end portion (23) of said opposite end portions (23,28).
15. The system according to claim 14, wherein said serial communication port (24) is arranged in a second end portion (28) of said opposite end portions (23,28).
16. The system (2″) according to claim 1, wherein said first wireless connection interface (19) is configured to: receive an augmented reality data stream (37) from said remote assistant (51) through said global data network (3), said augmented reality data stream (37) comprising graphic elements related to a scenario image selected between said image of said first scenario (11′) and said image of said second scenario (21′); superimpose said graphic elements on said scenario image on said local viewer (13).
17. The system according to claim 16, wherein said first microprocessor processing unit (15) is configured to transfer said first data stream (12) to said local viewer (13), said local viewer (13) configured to display an image of said first scenario (11′) to said field operator starting from said first data stream (12), so as to provide a see-through augmented reality video mode.
18. The system according to claim 1, comprising said support in the form of a helmet (10) or protective helmet, wherein said helmet (10) or protective helmet and at least one element selected among said first video camera (11), said local viewer (13), said first microprocessor processing unit (15) and said first wireless connection interface (19) are provided in the form of a mounting kit.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0065] The invention will be now shown with the description of a few exemplary embodiments and modifications, exemplifying but not limitative, with reference to the attached drawings, in which:
[0066]
[0067]
[0068]
[0069]
[0070]
[0071]
[0072]
[0073]
[0074]
DESCRIPTION OF PREFERRED EXEMPLARY EMBODIMENTS
[0075]
[0076] Systems 1 and 2 allow a field operator to send visual data to at least one remote assistant 51 through a global data network 3, typically through the Internet, and to ask assistant(s) 51 about the actions to be performed. Similarly, systems 1 and 2 allow assistant(s) 51 to provide instructions to a field operator in real time. The field operator and remote assistant 51 can be located everywhere, provided a terminal is available including a viewer 50 and enabling a terrestrial or satellite connection to the global data network.
[0077] Each of systems 1 and 2 comprises a first video camera 11 equipped with all that is required to be fixed to an operator's wearable support 10, in particular, a support intended to be worn on the operator's head. For this reason, video camera 11 is also indicated as “headset” 11. As shown in
[0078] Each of systems 1 and 2 also comprises a local viewer or local display 13 configured to be positioned along an operator's line of sight, at a place where he/she can see the images on the viewer without hindering the natural vision. In particular, also local viewer 13 includes a fastening means to be fixed to wearable support 10, in a front position laterally shifted with respect to the operator's eyes, as shown, for instance, still in
[0079] In order to direct data stream 12 from first video camera 11 towards local viewer 13, both system 1 and system 2 basically include a first microprocessor processing unit 15. A first wireless connection interface 19 to global data network 3 is also provided, through which first microprocessor processing unit 15 can exchange data with a remote server 99 via global data network 3. In particular, first microprocessor processing unit 15 can send first data stream 12 generated by first video camera 11 to remote server 99. The latter, in turn, is configured to direct data stream 12 to one or more remote viewers 50 available to respective remote assistants 51.
[0080] First microprocessor processing unit 15 and/or wireless connection interface 19 are advantageously housed in a same housing or box 18, which is also preferably configured to be connected to support 10. If support 10 is a protective helmet, as shown in
[0081] Remote server 99 can be a part of system 1 or 2, or it can be a server of a remote assistance service provider, configured to operate according to the invention.
[0082] A basic component of both systems 1 and 2 is a handpiece 20, otherwise indicated as a “handcam” or “hand torch”. This can have, for instance, the shape shown in
[0083] Visual sensor 21 can also be a video camera, indicated as second video camera, or a thermal imaging camera, but this is not a list including all possible visual sensors 21 of handpiece 20. The second video camera and thermal imaging camera 21 can be interchangeable devices for handpiece 20, or they can be integrated in a same device, of known type, which can selectively operate as a video camera or as a thermal imaging camera, according to the needs.
[0084] In any case, visual sensor 21 can frame a second scenario 21′ from handpiece 20, which is practically the scenario the operator can frame by visual sensor 21 by handling handpiece 20 and introducing his/her arm into an otherwise inaccessible environment. Starting from this second scenario 21′, which necessarily changes as the operator moves his/her arm and hand by which he/she grips handpiece 20, visual sensor 21 generates a second data stream 22.
[0085] Another use of handpiece 20 can consist in arranging it in a remote position from the field operator, in order to frame a scenario and to take a picture thereof in an environment where the operator same it is not allowed or safe to stay for a long time, or to frame the field operator him/herself while he/she is operating as required, in order to be visible to remote assistant 51 during this operation.
[0086] According to a first exemplary embodiment of the invention, in order to manage second data stream 22 generated by visual sensor 21, system 1 and system 2 provide a second microprocessor processing unit 25 and an inner second connection interface 26 or an outer second connection interface 29.
[0087] In the case of system 1 of
[0088] Second microprocessor processing unit 25 and inner or outer second connection interface 26,29 are also configured to receive and forward, besides data stream 22, an additional data stream 32 generated by a further possible peripheral device 30 connected to handpiece 20. This peripheral device 30 can comprise various instruments and sensors, or it can be an outlet device of a further closed-circuit video camera or of a video camera closed-circuit environmental imaging system, in particular, to take pictures of an apparatus to be supervised.
[0089] To this purpose, handpiece 20 can comprise a serial communication port 24, for example in the form of a USB port or the like, as shown, for instance, in
[0090] Examples of possible instruments 30 that can be connected to handpiece 20 through serial communication port 24 include microscopes, borescopes, thermometers, force sensors, stroke sensors, chemical concentration sensors such as hygrometric sensors, illuminance sensors, electromagnetic radiation sensors of various type such as radar sensors, or instruments providing these functions together. Instruments 30 can be a part of the system.
[0091] Handpiece 20 can also comprise a video-to-data conversion device associated with serial communication port 24, so as to receive data from sensor 30, typically from a borescope or from a microscope, or from a different peripheral device configured as a video source, for example, a terminal of a further closed-circuit video camera or of a closed-circuit video camera network.
[0092] Each instrument or sensor 30 is normally configured to measure a physical quantity, in the case of microscopes and borescopes the physical quantities being image parameters, and to generate additional data stream 32 starting from the measured quantity.
[0093] Therefore, second microprocessor processing unit 25 is functionally connected to serial communication port 24 through which it can receive additional data stream 32. This way, second microprocessor processing unit 25 can transfer additional data stream 32 in the same way it transfers second data stream 22, through second connection interface 26 or 29: [0094] in the case of system 1 according to the first exemplary embodiment, to first wireless connection interface 19, and from here [0095] to local viewer 13 and, [0096] to remote viewer(s) 50, through global data network 3 and remote server 99; [0097] in the case of system 2 according to the second exemplary embodiment, directly to server 99 through global data network 3, and from here [0098] to local viewer 13 through global data network 3 and first wireless connection interface 19, and from here [0099] to remote viewer(s) 50 through global data network 3 and remote server 99.
[0100] Second microprocessor processing unit 25 is preferably configured to suspend the transmission of second data stream 22 coming from visual sensor 21 when, during or immediately after a measure carried out by instrument 30, it receives/emits additional data stream 32, and vice-versa.
[0101] More in detail, in the case of system 1 of
[0102] Owing to its connection with inner second connection interface 26 of handpiece 20, first microprocessor processing unit 15 is also configured to transfer second data stream 22 and additional data stream 32 to server 99 via global data network 3, as well as first data stream 12. As in the case of first data stream 12, server 99 is configured to direct second data stream 22 or additional data stream 32 to at least one remote viewer 50. This makes it possible to display second scenario 21′, as well as the data obtained from any peripheral device 30, to remote assistant(s) 51.
[0103] Instead, as anticipated, in the case of system 2 according to the second exemplary embodiment of the invention, in order to manage second data stream 22 generated by visual sensor 21 and additional data stream 32 generated by any peripheral device 30 connected to handpiece 20, system 2 of
[0104] First data stream 12 can be sent to viewer 50 of remote assistant(s) 51 in a way similar to system 1 of the first exemplary embodiment of
[0105] System 1 or 2 also includes a scenario-switching device 80 diagrammatically shown in
[0106] More in detail, in the first exemplary embodiment of
[0107] Instead, in the second exemplary embodiment of
[0108] In an exemplary embodiment, scenario-switching device 80 comprises a push button 20d or an equivalent control means, preferably arranged on handpiece 20, as shown in
[0109] In particular, as shown in
[0110] As an alternative, scenario-switching device 80 can be configured to be operated through an external device such as a smartphone or a tablet.
[0111] Similarly, as shown in
[0112] The network architecture used by system 1′ according to a preferred modification of the first exemplary embodiment of
[0113] In this modification, first wireless connection interface 19 comprises an external communication WiFi card 191 configured to connect with an ambient WiFi network 5, in particular a 2.4 GHz network, which provides an access point 6 to global data network 3, in particular, to the Internet. First wireless connection interface 19 also comprises an internal communication WiFi card 192 configured to create a local WiFi network 7, in particular a 5 GHz network. This way, first data stream 12 can be directed from first wireless connection interface 19 to global data network 3 through external communication WiFi card 191, ambient WiFi network 5 and access point 6.
[0114] In the same modification, inner second connection interface 26 comprises a further WiFi card 261 configured to connect with local WiFi network 7 created by internal communication WiFi card 192. This way, second data stream 22 and additional data stream 32 can reach first wireless connection interface 19 through local WiFi network 7. From first wireless connection interface 19, second data stream 22 and additional data stream 32 can be sent to global data network 3 through external communication WiFi card 191, ambient WiFi network 5 and access point 6, similarly to first data stream 12. Moreover, second data stream 22 and additional data stream 32 can reach viewer 13 through a cable data connection between first wireless connection interface 19 and viewer 13.
[0115] The network architecture used by system 2′ according to a preferred modification of the second exemplary embodiment of
[0116] In this modification, first wireless connection interface 19 and outer second wireless connection interface 29 each comprise a WiFi card, not shown, configured to connect with an ambient WiFi network 5 providing an access point 6 to global data network 3, in particular, to the Internet.
[0117] Both in the case of system 1′ according to the first exemplary embodiment and in the case of system 2′ according to the second exemplary embodiment, ambient WiFi network 5 can be a corporate WiFi network or a public WiFi network. As an alternative, as shown in
[0118] In a further modification, not shown, first wireless connection interface 19 and/or outer second wireless connection interface 29 comprises a connection kit provided with a sim-card for connection to global data network 3.
[0119] With reference to
[0120] To include components 21,25 and 26 or 29, handpiece 20 can comprise a box consisting of a container 20 and a cover 20b, preferably separable from each other and fixed to each other by means of fastening elements 20c preferably arranged along both sides of handpiece 20.
[0121] As anticipated, handpiece 20 advantageously comprises a control element of the scenario-switching device to select the image, i.e. the data, to be displayed on local viewer 13 and on remote viewer(s) 50, in particular to select which data stream has to be used to form this image, among first data stream 12 related to the scenario framed by first video camera 11, second data stream 22 related to the scenario framed by visual sensor 21, and additional data stream 32 coming from further peripheral device or instrument 30 via serial connection port 24 of handpiece 20. In a handpiece advantageously including a thermal imaging camera or a video camera as the visual sensor 21, the switching device can also be configured to select the data stream to form the image among first data stream 12 generated by first video camera 11, second data stream 22 generated by a thermal imaging camera provided as the visual sensor 21, second data stream 22 generated by a thermal imaging camera provided as the visual sensor 21, and additional data stream 32.
[0122] As anticipated, a push button 20d can be provided on the outer surface of handpiece 20 as the control element of scenario-switching device 80, more in particular, push button 20d can protrude from cover 20b, as shown in
[0123] As shown in
[0124] As shown in
[0125] System 2″ shown in
[0126] To this purpose, first wireless connection interface 19 is configured to receive the augmented reality data stream 37 and to transfer it to first microprocessor processing unit 15, which manages the data streams directed to viewer 13. In the above-mentioned modification, not shown, of system 1 of
[0127] In particular, in a further modification of systems 1 and 2, first microprocessor processing unit 15 is configured to transfer also first data stream 12 to local viewer 13, related to the image of first scenario 11′ framed by first video camera 11, and first microprocessor processing unit 15 and/or server 99 are configured to conventionally associate the augmented reality data stream 37 also to first data stream 12 directed to viewer 13, superimposing the augmented reality graphic signs to the image of first scenario 11′ framed by first video camera 11, implementing a see-through augmented reality video mode.
[0128] Both system 1 of
[0129] The foregoing description of exemplary embodiments and modifications of the invention will so fully reveal the invention according to the conceptual point of view, so that others, by applying current knowledge, will be able to modify and/or adapt in various applications these exemplary embodiments and modifications without further research and without parting from the invention, and, accordingly, it is meant that such adaptations and modifications will have to be considered as equivalent to an exemplary embodiments of the invention and of the respective modifications. The means and the materials to put into practice the different functions described herein could have a different nature without, for this reason, departing from the field of the invention. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation.