System integrating video communication and physical sign analysis

Abstract

The present invention discloses a system integrating video communication and physical sign analysis, comprising at least one front-end device. The front-end device comprises a camera device, a display device, an audio device, a button device and a processor. The camera device, the display device, the audio device and the button device are all connected to the processor, and the processor can connect to the Internet network and the mobile device via wired or wireless means. The front-end device and the mobile device can perform video communication, and the front-end device can perform physical sign analysis according to the images collected by the camera device. Based on some applications in the prior art, the present invention combines video collection technology and human face analysis technology to perform physical sign analysis and obtain related indexes.

Claims

1. A system integrating video communication and physical sign analysis, comprising: at least one front-end device for receiving information and/or data that may exist at the front end; a mobile device, capable for interconnecting with the front-end device and establishing data communication; the front-end device and the mobile device can perform video communication, and the front-end device can perform physical sign analysis according to the images collected by a camera device; wherein during video communication, the front-end device or mobile device is the client of the video communication, and the client registers and maintains the registration status on the video communication; wherein the registration process of the client comprises: (1) assigning each client a unique client identification number; (2) initiating a registration application to the video communication service cloud when the client is running, and the registration application including registration information such as the client identification number and the client IP address of the current client; (3) generating a client status table by the video communication service cloud according to the registration information; (4) periodically sending heartbeat information to the video communication service cloud by the client, and refreshing the client status table by the video communication service cloud after receiving the heartbeat information; wherein the process of initiating video communication by the client comprises: (1) configuring a contact list on the client and binding a button to the contact persons in the contact list; (2) initiating a video communication connection request to another client through the video communication service cloud by the current client; (3) feeding back the current status information by another client, and feeding back the information of connection failure if the current status is in the call or during the call; and feeding back the information such as the IP address if the current status is no answer.

2. The system integrating video communication and physical sign analysis according to claim 1, wherein the front-end device comprises a camera device, a display device, an audio device, a button device and a processor, and the camera device, the display device, the audio device, and the button device are all connected to the processor and may perform data interaction with the processor.

3. The system integrating video communication and physical sign analysis according to claim 1, the processor can connect to the Internet network and perform data interaction with other devices via wired or wireless means.

4. The system integrating video communication and physical sign analysis according to claim 2, wherein the button device comprises a preset button and a physical sign analysis trigger button.

5. The system integrating video communication and physical sign analysis according to claim 2, wherein the camera device is a conventional camera having a certain resolution.

6. The system integrating video communication and physical sign analysis according to claim 2, wherein the display screen is an ordinary display screen or a touch display screen.

7. The system integrating video communication and physical sign analysis according to claim 2, wherein the process for the front-end device to perform physical sign analysis comprises: (1) capturing the video by the camera device and sending to the processor, and displaying on the display device; (2) performing human face detection analysis of the captured video frame by frame by the processor, and acquiring the largest human face close to the middle position of the video only; (3) positioning the coordinates of several feature points for the human face detected by each frame by algorithm; (4) extracting pixel RGB values of the area enclosed by several feature points for the human face image of each frame of the face in a time slice and performing time series analysis to form physical sign analysis waveform data; (5) sending the physical sign analysis waveform data and the human face images to the physical sign analysis service to obtain the relevant physical sign indexes of the person.

8. The system integrating video communication and physical sign analysis according to claim 7, further comprising a process of recording the physical sign indexes, specifically, comparing the current human face with the customer group face database to obtain the customer number, and recording the physical sign indexes obtained in the step (5) in the physical sign data record sheet of the customer number.

9. The system integrating video communication and physical sign analysis according to claim 1, wherein the initiator of the video communication is a front-end device or a mobile device.

10. The system integrating video communication and physical sign analysis according to claim 2, the processor can connect to the Internet network and perform data interaction with other devices via wired or wireless means.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is an overall structural view of the present invention.

(2) FIG. 2 is a schematic diagram of a front-end device.

(3) FIG. 3 is a schematic diagram of human face acquisition.

(4) FIG. 4 is a schematic diagram of a client status table.

(5) FIG. 5 is a flow diagram showing the registration of a client on a video communication service cloud.

(6) FIG. 6 is a flow diagram showing initiating a video communication by a client.

(7) FIG. 7 is a flow diagram showing the registration of a client for physical sign analysis.

(8) FIG. 8 is a flow diagram showing the physical sign analysis.

(9) FIG. 9 is a flow chart for query.

DETAILED DESCRIPTION

(10) The technical solutions of the present invention are further described in detail below with reference to the accompanying drawings. It should be noted that the embodiments are merely illustrative of the invention and are not intended to limit the invention.

(11) Referring to FIG. 1, the present invention provides a system integrating video communication and physical sign analysis, comprising: at least one front-end device and at least one mobile device. Two front-end devices and two mobile devices are shown in the figure. The front-end devices are respectively connected to the physical sign analysis service, and the front-end device may collect video and perform corresponding physical sign analysis according to the face recognition in the video. The front-end devices are further connected to the video communication service, and the mobile device is also connected to the video communication service respectively, that is, the front-end device and the mobile device can perform video communication through the video communication service.

(12) As shown in FIG. 2, the front-end device comprises a camera device, a display device, an audio device, a button device and a processor. The camera device, the display device, the audio device and the button device are all connected to the processor, and the processor can connect to the Internet network via wired or wireless means, of which, the processor is used for physical sign analysis and also for signal interaction with a video communication.

(13) In a front-end device, the button device comprises a preset button and a physical sign analysis trigger button. The button can be a physical button, or a virtual button set on the touch display screen. The preset button can be used to set a parameter value, to correspond to the parameter value directly by pressing the button, and the physical sign analysis trigger button is set. Physical sign analysis can be performed by pressing or touching this button, which makes the physical sign analysis to become an optional mode. The camera device is a conventional camera with a certain resolution (for example, 720p, 1080p or higher resolution). The display screen may be an ordinary display screen with different resolutions, or a display screen that can be displayed only, or a touch display screen that can be touch-operated.

(14) As shown in FIG. 3, it shows a process in which a front-end device captures video through a camera and divides the video.

(15) Specifically, the front-end device captures video through the camera and displays it on the display screen, and the program running on the processor of the front-end device performs face detection on the captured video frame by frame. If only one face is analyzed, the face image is collected as the basis for analysis of human face; and if multiple faces are detected, the largest human face close to the middle position of the video is taken, and the coordinates of several feature points for the human face detected by each frame are positioned by algorithm, and then the pixel RGB values of the area enclosed by several feature points for the human face image of each frame of the face in a time slice are extracted and time series analysis is performed to form physical sign analysis waveform data. The time slice can be arbitrarily set, for example, 10 s, 30 s, several minutes, the number of video frames per second can be 25 or 30, in an average distribution. The feature points of the human face are determined according to the physical sign analysis needs. The selection of feature points can be determined as required, for example, some feature points shown in the face part in FIG. 3, which may include feature points of eyes, mouth and ears, etc. Different feature points can be used to detect different physical sign indexes as needed. According to the physical sign analysis waveform data, the related physical sign indexes of the person are obtained. The physical index indexes include heart rate, respiratory rate, psychological pressure and blood pressure, etc.

(16) In some preferred embodiments, the color of the pixel is RGB color, in some preferred embodiments, the pixels of the images can be represented by red, green, and blue or a combination thereof respectively. In some preferred embodiments, in terms of numbers, each color may be represented by 8 bit or 16 bit respectively.

(17) In some preferred embodiments, physical sign indexes may further include a certain part of a human body that may be different from other human bodies, for example, the head, the legs, in some preferred embodiments, physical sign indexes may include an element of a human body that may be different from other human bodies, for example, the size of the head, the length of the legs, the height of human body, etc. In some preferred embodiments, physical sign indexes may be elements that are easily identified in image recognition; in some preferred embodiments, physical sign indexes may be a single element or a combination of elements. In some preferred embodiments, the recognition priority of the physical sign indexes may be set, for example, recognizing some elements in priority, for example, the eye size. When a human body cannot be distinguished from other human bodies by these elements, other elements may be further recognized, and so forth.

(18) The sign analysis service has a deep learning algorithm inside, and relevant deep learning modeling and a large number of sample learning are carried out according to the physical sign indexes that need to be detected in advance.

(19) The video communication service is responsible for video communication bridging between a front-end device and a front-end device or between a front-end device and a mobile app.

(20) The initiator of the video communication service may be a front-end device or a mobile APP. The front-end device and the mobile app need to register and remain registration status in the video communication service in real time. Video communication can be initiated in both directions between the front-end device and the front-end device or between the front-end device and the mobile app through the video communication service cloud.

(21) Both the front-end device and the mobile app can be used as the client. FIG. 5 shows the registration process of the client on the video communication service cloud, comprising:

(22) (1) assigning each client a unique client identification number;

(23) (2) initiating a registration application to the video communication service cloud when the client is running (the front-end device is energized or the mobile app is running), and the registration application including registration information such as the client identification number and the client IP address of the current client;

(24) (3) generating a client status table as shown in FIG. 4 by the video communication service cloud according to the registration information;

(25) (4) periodically sending heartbeat information through which the video communication service cloud indicates that the client is always in a registration status, to the video communication service cloud by the client, and refreshing the client status table by the video communication service cloud after receiving the heartbeat information.

(26) FIG. 6 shows the process of the client to initiate video communication, comprising:

(27) (1) configuring a contact list on the client and binding a button to the contact persons for the front-end device, that is, pressing a button to initiate a video communication request to the contact preset by the button.

(28) (2) initiating a video communication connection request to another client through the video communication service cloud by the current client. The video communication connection request is transmitted through the video communication service cloud;

(29) (3) feeding back the current status information by another client, and if the current status is in the call or during the call, feeding back the information of connection failure to the video communication service cloud, and the information of connection failure being sent to the client that initiates the video communication request by the video communication service cloud; and feeding back the information such as the IP address if the current status is no answer, and the information such as the IP address being sent to the client that initiates the video communication request by the video communication service cloud.

(30) FIG. 7 shows the preparatory process for the front-end device to perform the physical sign analysis. This preparatory process is a registration process of the physical sign analysis by the front-end device, comprising:

(31) (1) turning on the camera;

(32) (2) displaying the video captured by the camera by the display screen;

(33) (3) detecting whether there is a human face on the video frame by frame;

(34) (4) obtaining a human face photo if a face is detected;

(35) (5) sending a registration request to the physical sign analysis service by the client and attaching the obtained face photo;

(36) (6) including the human face photo in the customer group face database by the physical sign analysis service, and returning the corresponding customer number to the client, to complete the registration successfully.

(37) FIG. 8 shows the process of performing the physical sign analysis of the present invention, comprising:

(38) (1) turning on the camera;

(39) (2) capturing the video by the camera and sending to the processor, and displaying on the display device;

(40) (3) performing human face detection analysis of the captured video frame by frame by the processor to detecting whether a human face exists, and acquiring the largest human face close to the middle position of the video only when detecting a human face;

(41) (4) arranging frame-by-frame human face image sequences in a chronological order;

(42) (5) extracting pixel RGB values of the area enclosed by several feature points in a chronological order and performing time series analysis to form physical sign analysis waveform data;

(43) (6) sending a physical sign analysis request, attached with the physical sign analysis waveform data and a human face photo for analysis; sending the sign analysis waveform data and the human face images to the physical sign analysis service for deep learning of the relevant physical sign indexes to obtain relevant physical sign indexes of the person.

(44) FIG. 9 shows the query process in physical sign analysis service, comprising:

(45) (1) turning on the camera;

(46) (2) displaying, by the display screen, the video captured by the camera;

(47) (3) detecting whether a human face exists frame by frame by the processor;

(48) (4) obtaining a human face photo in the test result;

(49) (5) sending a query request and attaching the obtained human face photo;

(50) (6) comparing the human face photo with the customer group face database, to obtain the customer number;

(51) (7) querying the physical sign analysis record of the customer number;

(52) (8) returning the query result.