Embedded system for video processing with hardware means

Abstract

An embedded system for video processing comprises hardware means for realizing the following functions: a) a server function, wherein for realizing the server function the following functions are realized: a1) receiving of at least one image-, at least one audio-data stream; a2) compression of at least one of the image- and audio-data streams yielding in each case a combined data stream respectively; a3) output of at least one of the combined data streams via a protocol based network; b) a client function, wherein for realizing the client function the following functions are realized: b1) receiving of at least one combined data stream from the protocol based network; b2) decompression of the at least one combined data stream; b3) output of the combined decompressed image- and audio-data stream; c) wherein the hardware means are designed in such a way that they c1) compress and decompress a continuous combined data stream and c2) conduct an individual security measure by means of a hardware serial number saved in the system and/or an individual activation code and c3) comprise activation code monitoring means, which conduct a monitoring and comparison of activation codes.

Claims

1. An embedded system for video processing with hardware for realising the following functions: a) a server function, wherein for realising the server function the following functions are realised: a1) receiving of at least one image and at least one audio-data stream; a2) compression of the at least one of the image and audio-data streams yielding in each case a combined image and audio-data stream; a3) output of at least one of the combined image and audio-data streams via a protocol based network; b) a client function, wherein for realising the client function the following functions are realised: b1) receiving of the at least one combined data stream from the protocol based network; b2) decompression of the at least one combined data stream; b3) output of the combined decompressed image- and audio-data stream; c) wherein the hardware is designed in such a way to c1) check individual security utilizing a hardware serial number saved in the system and an individual activation code and c2) monitor the individual activation code by monitoring and comparison of the individual activation code to a list of activation codes wherein the hardware is adapted in such a way to compress and decompress the at least one combined image- and audio-data stream with at least 60 frames per second and a resolution of at least 19201080 pixels without data jam; wherein only a single type of said embedded system is required physically for security with respect to the at least one combined data stream, but also with respect to product piracy; wherein the hardware for realising the server function and the client function is arranged on a joint circuit board, where the joint circuit board is designed as a PC plug-in card and is displayable in a PC operating system environment as a network card; wherein the hardware further comprises a Field Programmable Gate Array (FPGA) that is individually encrypted, wherein the following functions are provided by the FPGA: a) connection to the protocol based network; b) firewall that filters IP, Port, and Mac addresses according to a rule table; c) encryption of the combined data stream to be output, if the embedded system operates as a server; and d) decryption of the at least one combined data stream, if the embedded system operates as a client; and wherein the at least one image data stream received if having been recorded by interlaced scanning is output as a progressive scan.

2. An embedded system according to claim 1, characterised in that the hardware monitoring the individual activation code is adapted to emit an activation code warning signal or an activation code approval signal.

3. An embedded system according to claim 1, characterised by: a) receiving more than one combined data stream via the protocol based network; b) decompression of more than one combined data stream; c) scaling and/or rotating and/or merging and/or cross-fading of more than one decompressed image- and audio-data stream; d) joint output of more than one decompressed image- and audio-data stream for simultaneous display on a monitor; and e) separate output of one or more combined data streams on several monitors, which are arranged as a video wall.

4. An embedded system according to claim 1, characterised by output of control signals for remote control of connected devices.

5. An embedded system according to claim 1, characterised in that the hardware comprise a Digital Signal Processor (DSP), wherein the following functions are provided by the DSP: a) compression of at least one of the image- and audio-data streams yielding in each case the at least one combined data stream, if the embedded system operates as a server; b) decompression of the at least one combined data stream, if the embedded system operates as a client; and c) scaling and output of the decompressed image- and audio-data stream via an interface, if the embedded system operates as a client.

Description

(1) Hereafter, the invention is illustrated further based on examples referring to the respective Figures.

(2) It is shown in

(3) FIG. 1 the schematic set-up of an embodiments of the video distribution system according to the present invention for application in a betting agency;

(4) FIG. 2 the set-up of a simple embodiment of the video distribution system according to the present invention;

(5) FIG. 3 a more complex embodiment of the video distribution system according to the present invention;

(6) FIG. 4 the schematic set-up of an embedded system according to the present invention;

(7) FIG. 5 the schematic set-up of an embedded system according to the present invention with a server functionality;

(8) FIG. 6 the schematic set-up of an embedded system according to the present invention with a client functionality;

(9) FIG. 7 an illustration of an embedded system according to the present invention as a module;

(10) FIG. 8 a schematic representation of the video data processing of an embedded system according to the present invention operating as a server;

(11) FIG. 9 a schematic representation of the video data processing of an embedded system according to the present invention operating as a client; and

(12) FIG. 10 a selection of display modes attainable with an embedded system according to the present invention operating as a client; and

(13) FIG. 11 a software-technological set-up of a branch; and

(14) FIG. 12 a schematic representation of the network security module; and

(15) FIG. 13 a schematic representation of the driver decoupling of a VideoWizard module in a PC operating system environment; and

(16) FIG. 14 an application of the system according to the present invention as a roulette transmission within a casino; and

(17) FIG. 15 an application of the system as a live game transmission outside of a casino; and

(18) FIG. 16 a schematic representation of the security of the network transmission; and

(19) FIG. 17 a schematic overview of the communication structure of the software units.

(20) The invention comprises a modern video distribution system on the basis of network based multiple H.264 data streams. Thereby, different video sources are converted into H.265 data streams by embedded systems operating as a server and decoded by embedded systems operating as a client and displayed on monitors.

(21) The application of the video distribution system is intended for example for betting agencies, where the image sources are normally provided by satellite receivers and are individually distributed to the available TV monitors by shop masters/users.

(22) The typical set-up of a video distribution system according to the present invention in a betting agency is depicted in FIG. 1 and consists of the four functional units receiving, distribution, display, and control. The receiving function is realised by a variable number of satellite receivers 100, which possess allocated embedded systems 110, which are operating as a server. The encoded video streams of the satellite receivers 100 are distributed via a network 120 (with a switch/distributer, VLAN). The display occurs via a variable number of monitors 130, which receive their images from allocated embedded systems 140, which are operating as a client, wherein the clients 140 individually provide from the supply of the receiver servers 110 selected image data streams.

(23) Thereby, the selection of the image data streams occurs at local control PCs 150, at which the shop master/user can dynamically create the allocation between satellite program and monitor 130 (control). To this end, the TV monitors 130 and the satellite receivers 100 are visualised graphically.

(24) Each embedded system 110 or 140, which is connected to a Sat-receiver 100 or TV monitor 130, additionally possesses an infrared control function, which allows to remotely operating the allocated device. Every satellite receiver 100 is also optionally connected to a network connection, via which the satellite channel listings can be updated and changed.

(25) The control of the embedded systems 110 or 140 and, therefore, the allocated devices 100 or 130 occurs physically via network 120 and structurally via the operator PC software. If necessary, several such operator stations 150 allow the control of the devices.

(26) The system according to the present invention is designed in a much more general way compared to known similar systems and possesses the following core properties: Flexible hardware based multi-function video gaming and visualization system for digital television and digital media content, combination of a freely selectable global hardware based video stream network with game visualisations and interactive live content display. Nearly arbitrary scalable by modular set-up (see also FIG. 3). A client can access several hundred server streams. The only limitation is the capacity of the used network. Division in clients and servers. Servers receive image data and feed them into the network. Clients receive the network streams and display them in very different manners. Interactive control via network (for example PC, IPad, network compatible device). Mixed use as client and server simultaneously is possible. Different sources, which can be fed via satellite or network locally or remotely (globally worldwide!) (live cams, digital and analogue TV sources, hard disk, magnetic disk data, media player etc.). Resolution up to full HD receiving of the image source and real time encoding according to state of the art H.264-methods in hardware (data reduction for sending via network) Decoding of several H.264 source data streams per client module. Different visualisation modes by the client (see also FIG. 10): 1 stream 2 simultaneous streams 4 simultaneous streams per TV 3 rotated streams 6 streams etc., video wall mode, wherein one or several streams can be distributed onto a wall consisting of many monitors cross-fading mode of different streams (blue box), Fading/cross-fading of live streams in game application. Interactive control of the mode by the user replay function of video streams on demand encryption and decryption of data streams in real time in hardware interactive possibility to show video contents anew or to change data streams and display modes interactively. Transfer via network and distribution via switch system (i.e. Gigabit-Ethernet, UDP protocol etc.) Use of combined FPGA and DSP/special chip technology (real time hardware)

(27) The basic concept of the video distribution system and the embedded systems for video processing is depicted schematically in FIG. 2 and is based on two functional units in combination: the H.264 video functions for compression and decompression and the functions for network security with hardware firewall and real time data encryption.

(28) H.264 Video Unit

(29) The video unit of the embedded system consists of one or several H.264 encoders, which can compress image data in real time using the state of the art H.264 method, and one or several H.264-decoders, which can decode and display several compressed image data streams simultaneously. Image and sound data are received via a camera system or a HDMI/PAL source with up to 60 frames per second and transmitted to an embedded system, which is operating as a server, and which processes, records, and compresses these data to a H.264 data format. The image data streams can then either be transferred uncompressed to a PC or a monitor or can be transmitted via a network connection. This is depicted schematically in FIG. 9.

(30) GigE vision compatible cameras, satellite receivers with HDMI output, DVD players or other devices adapted for HDMI output serve as image data source.

(31) The embedded system additionally possesses the opposite functionality (client). It is possible to feed H.264 as well as MPEG2 data streams, which are decompressed in this embedded system in real time and can either be recoded or directly outputted via HDMI. This is depicted schematically in FIG. 10. Thereby, it is possible to play videos in progressive and interlaced data format (interlaced scanning) and to deinterlace them as required.

(32) FIG. 10.1a, 10.2a shows a single video data stream, which is displayed scaled and positioned on a display (LCD, LED, OLED, TV etc. monitor) and underlayed with a background colour.

(33) FIG. 10.1b, 10.2b shows two single video data streams, which are scaled and positioned and underlayed with a background colour. One of the two videos is scaled smaller and is in the foreground.

(34) FIG. 10c shows two separate video data streams, which are both scaled and positioned and underplayed with a background colour. One of the two videos is scaled smaller and shown in a blending with the other video (blending).

(35) FIG. 10d shows four separate video data streams, all four of which are scaled and in a fixed position (non-overlapping).

(36) FIG. 10e shows three separate video data streams, all three of which are scaled and rotated by 90, and in a fixed position (non-overlapping).

(37) FIG. 10f shows six separate video data streams, all six of which are scaled and have a fixed position (non-overlapping).

(38) FIG. 10g shows a video data stream on a Video Wall realised with 4 monitors, wherein each of the 4 monitors displays a respective section of the complete image to be shown.

(39) The whole system is constructed in such a way that only a single type of embedded system (VideoWizard module) is required physically. This embedded system is designed for security with respect to the data streams, but also with respect to product piracy.

(40) Thereby, the video distribution system is devised in such a way that systems with over 100 servers and clients can be realised in a single gigabit network stream (this is dependent on the resolution of the data streams). Alternatively the data stream can be cascaded in a tree like fashion. A system with many components is schematically depicted in FIG. 3.

(41) The second simultaneous functionality of the embedded system consists of a network security functionality, which massively increases data transmission security via a hardware implemented firewall and AES real time data encryption. Thereby, the core functions in the present case are a 256 bit AES encryption of the network traffic as well as a hardware firewall with port activation and logging function.

(42) General Set-Up of the Embedded System:

(43) Conceptually, the embedded system 400 consists of a single component or a module (VideoWizard hardware), which is supplemented by a motherboard 710, which contains the additional functions for standalone operation.

(44) Functionally, the module is able to realise the server functions, the client functions, and the network security functions. These are described in more detail below.

(45) By setup, the embedded system (see FIG. 5 and FIG. 8) consists of a combination of FPGA and embedded processor, GPU and special hardware technology (all three functions in the utilised DSP). Thereby, both units share different functions. The functionality of the FPGA 410 is necessary for real time computing and processing of the GigE data of the camera 210, the processing of HDMI input data, and particularly PC connection and security aspects, the DSP realises the DVI output and scaling as well as the H.264 data processing as an encoder/decoder.

(46) Advantageously, a driver decoupling of the VideoWizard modules (for example for different operating systems such as Windows XP Embedded and Linux) is possible by means of the embedded system, wherein the VideoWizard module, which is designed as a PC plugin card, is faded into the operating system environment as a network card. Thereby, the connection is a network driver, which allows the use of all standardised network programmes of the operation system. The second access interface is realised via a low level user library, which provides the necessary control, image generating and DMA functions. The FPGA can provide network functions in combination with the TI co-processor. For example, an image data and control driver 133 is provided in conjunction with a so-called application-programming interface API 134 as well as a separate network driver 135 (compare FIG. 13 and FIG. 12).

(47) Server Functionality

(48) The server functionality of the embedded system is depicted schematically in FIG. 5 and partially in FIG. 12 (left of the network) and consists of the following detail functions: receiving of HDMI data streams up to 19201080 p60 (progressive, 60 Hz) or camera data streams; compression of the image- and audio-data streams in H.264 format; packaging of control and audio data into the H.264 data stream; standalone capability with access (via server PC 510) via Gigabit-Ethernet to internal registers of the embedded system; individual FPGA security against product piracy; AES encryption of the data stream; sending the compressed data via GigE network
Client Functionality

(49) The client functionality of the embedded system is depicted schematically in FIG. 6 as well as partially in FIG. 12 (right of the network) and consists of the following detail functions: receiving of the compressed data or a data stream, respectively, via GigE network; AES decryption of the data stream; decompression of the image- and audio-data from the H.264 format or extraction of the control data from the data stream, respectively; decompression of the MPEG2 image data streams where applicable; deinterlace function with high quality for realisation of progressive image material; image scaling; restoration of an image data stream with 60 frames per second from the H.264 data streams; standalone capability with access (via client PC 610) via Gigabit-Ethernet to internal registers of the embedded system; individual FPGA security against product piracy; output of video audio and control data;

(50) The embedded system (the VideoWizard) can preferably be utilised in combined modes. For example, a mix of client and server (with a selection of partial functions) or a recoding of several input data stream to new output streams is possible.

(51) Network Function (Firewall and Data Encryption)

(52) The network functionality of the embedded system consists of a hardware implemented firewall and AES real time data encryption, whereby the data transmission security is massively increased. Thereby, the core functions in the present case are a 256 bit AES encryption of the network traffic and a hardware firewall 131 with port activation and logging function. In detail the FPGA 410 predominantly realises the network functions (FIG. 13). Thereby, the FPGA contains a 256 bit AES real time encryption unit 132, which encrypts or decrypts, respectively, the incoming and outgoing network data. It is possible to choose between different H.264 data streams, which are transmitted into/out of the embedded system via network. The firewall 131 allows filtering of IP, port, and Mac addresses via a rule table. An adjustable logging function of the firewall 131 allows recording of the valid/invalid access attempts onto the embedded system. Optionally unlocked ports/IPs/Mac addresses can be blocked automatically after a certain amount of failed attempts or an alarm can be triggered if too many unallowed accesses to the embedded system occur per second. Regarding this matter, failed attempt monitoring means, which monitor the occurrence of failed attempts exceeding a defined threshold, are provided in the system.
Security and Stability
Security Against Product Piracy

(53) By means of an individual hardware serial number saved in the embedded system, it can be ensured that the embedded system can only be operated with an individual activation code. Therefore, a copy of the module for duplication is pointless. In the system, activation code monitoring means 161 can be provided, which are able to perform a monitoring or comparison of activation codes with a database and, if necessary, can emit an activation code warning signal if no match of the activation code or codes occurs or can emit an activation code approval signal if a match of the activation code or codes occurs.

(54) Security During Image and Data Transmission

(55) Optionally, the image data streams can be secured via the network with an AES encryption method, whereby the data is protected from an attacker in the transmission path. This concerns data transmission via Internet as well as internal transmission, for example in a casino.

(56) High stability and safeguard against failure during image data transmission and display. By the utilisation of two independent real time capable computing components a high operating safety with high output reserves is realised.

(57) Stability by Independence

(58) By means of the independence of the embedded system from a PC control, the stability of the overall application is increased. In case of a failure of a part of the overall system, the embedded systems continue to function independently.

(59) Network Security by a Hardware Firewall

(60) By use of a hardware firewall the unauthorised access from outside via not permitted ports, not activated IP addresses, or not authorised Mac addresses can be prevented. Moreover, the logging allows a recording of permitted and non-permitted accesses. By utilisation of a hardware firewall, the firewall cannot let accidentally, in case of capacity overload, not permitted packages pass, as is the case for software-implemented firewalls.

(61) Numerous modifications and developments of the described embodiments are realisable. For instance, an embedded system according to the present invention can also be used without a network, wherein it provides the server and client functionality simultaneously. Furthermore, different coding and encryption standards or methods can be utilised, and also according to the desired area of application, different interfaces, network protocols and network architectures can be used without changing the essence of the invention.

(62) Glossary

(63) Image data stream, audio data stream, video data stream, combined data stream; We consider these terms to mean data streams with a respective channel of the respective content (images, audio, video). A combined data stream contains image as well as audio data. Therefore, a video data stream can be an image data stream or a combined data stream.

(64) Client/Server

(65) We consider the terms client and server to mean so-called streaming clients or streaming servers. The term streaming client designates a special client for streaming media, which can be either software or hardware. Typical streaming clients support special streaming protocols such as RTP, RTCP, and/or RSVP. The term streaming server designates a decided server for distribution of streaming media data via a network. Typical streaming servers support special streaming protocols such as RTP, RTSP, RTCP, and RSVP (auxiliary protocol for QoS-method IntServ). Thereby, the term streaming media designates from a computer network received and simultaneously played audio and video data. The process of data transmission itself is termed streaming, and transmitted (streamed) programmes are termed live stream or shortly stream. (According to http://de.wikipedia.org/wiki/Streaming-Client, http://de.wikipedia.org/wiki/Streaming-Server and http://de.wikipedia.org/wiki/Streaming_Media)

(66) Embedded System

(67) The term embedded system (Eng. embedded system) describes an electronic data processor or computer, which is embedded (embedded) into a technical context. Thereby, the data processor assumes either monitoring, control, or regulatory functions or is responsible for a form of data or signal processing, for example during encryption or decryption, encoding or decoding, or filtering. (Cited according to http://de.wikipedia.org/wiki/Eingebettetes_System)

(68) Hardware Means

(69) We consider the term hardware means to mean electronic components, which are optimised for certain purposes and execute their function mainly in real time.

(70) Typically, DSP, programmable hardware such as FPGAs, special ASICs (application-specific integrated circuit), or combinations thereof are used for such purposes. These can also contain embedded microprocessors (i.e. ARM processors). In the context of this application the term hardware means does not comprise multi-purpose microprocessors, which are not optimised for the purposes named herein. Multi-purpose microprocessors (i.e. by Intel or AMD) are typically found in personal computer as central processing units.

(71) DSP

(72) A digital signal processor or DSP serves the purpose of continuous processing of digital signals (i.e. audio or video signals). For processing of analogue signals, the DSP is used in conjunction with analogue-digital converters and digital-analogue converters. Compared to multi-purpose microprocessors DSPs contain a processor, which is speed-optimised for frequently required mathematical operations. Some DSPs already contain the necessary ND and D/A converters at their input and output.

(73) A DSP has to be able to safely process a certain amount of data per unit of time. This results from the demand of a mostly fixed and externally given data rate with which the input data is entering the DSP or the processed data has to be written again. A kind of handshake or stopping in time during data processing is not possible in most cases. For example, the following measures can serve to increase the processing speed: special synchronous, serial interfaces for input and output so-called MAC commands (multiplier-accumulator) for simultaneous multiplication and addition in a machine cycle address generators for the implementation of loops and circular buffer structures without software overhead Harvard-architecture (command storage is logically and physically separated from data storage) decided hardware stacks. Repeated access to the memory in a cycle (Cited according to http://de.wikipedia.org/wiki/Digitaler_Signalprozessor)
FPGA

(74) A field programmable gate array or FPGA is an integrated circuit, into which a logic circuit can be programmed.

(75) However, the term programming in this context is to be distinguished from the term describing the design of software for a conventional processor: In a FPGA circuit structures are generated by means of hardware description languages or in the form of wiring diagrams and these data are transmitted into the component for configuration. Thereby, certain switch positions are activated or deactivated, resulting in a concretely implemented digital circuit. Therefore, instead of the term programming the term configuration of a FPGA is also used. By the specific configuration of internal structures different circuits can be realised in a FPGA, culminating in highly complex structures, such as i.e. microprocessors.

(76) FPGAs are mostly used, where fast signal processing and flexible changes in the circuit are of importance, for instance in order to introduce subsequent improvements in the implemented functions without having to directly change the physical hardware.

(77) The configuration typically takes place once before each use, whereby the FPGA is configured for a specific function, which it loses again upon switching off of the operating voltage. Therefore, a non-volatile memory, which stores the configuration, whose content on itself is also updatable, is allocated to the FPGA in most cases.

(78) For example, FPGAs are used for real time processing of simple to complex algorithms, for digital signal processing within the scope of digital filters, or for fast fourier transformation, but also for protocol implementations such as parts of the Ethernet MAC layer, coding of digital video signals, encryption of data and error correction methods, and for digital storage oscilloscopes.

(79) Ethernet, GigE

(80) Ethernet is a technology, which specifies software (protocols etc.) and hardware (cables, splitters, network cards etc.) for tethered data networks. It facilitates data exchange in the form of data packages between the devices connected to a local network (LAN). Currently, transfer rates of 10 Megabit/s, 100 Megabit/s (fast Ethernet), 1000 Megabit/s (Gigabit Ethernet, GigE) up to 100 Gigabit/s are specified. In its traditional embodiment, the LAN merely extends as far as one; Ethernet via fibreglass also has a limited range.

(81) Firewall

(82) A firewall is a security system, which protects a network or a single computer from unwanted network accesses. The firewall serves to restrict the network access based on sender or destination address and used services. It monitors data traffic and decides according to specified rules, where certain network packages can pass through or not. The simple filtering of data packages according to network addresses is the basic function of all firewalls. The task of a so-called packet filter firewall comprises blocking network packages or allowing them to pass according to their network address. To this end, it evaluates the header information of network packages. Simple packet filtering works with static rules and observes every network package separately, and therefore does not establish relations to previous network packages. In contrast, an extended kind of (state controlled) package filtering exists, which records such relations. In this manner, access to the source system, which has requested a communication, is further restricted.

(83) HD

(84) This abbreviation stands for high definition, (Eng.) wherein in our case image resolutions of 1280720 or 19201080 pixels are meant, wherein the higher resolution is also termed full HD.

(85) Video resolution comprises the same parameters as image resolution (lines and columns or pixel number, aspect ratio) and extends these by the temporal aspect of frame rate. Thereby, it has to be differentiated between the repetition of partial (mostly half images with interlaced scanning, interlaced), and full images (frames, progressive scan). Common frame repetition rates are 24, 25, 50, or 60 Hz. In the HDTV area 720p and 1080i are common. One speaks of full HD from 1080p25, which means 19201080 pixels, progressive, 25 Hz.

(86) Video

(87) We consider video to mean moved images or image streams, with or without sound.

(88) Interlaced Scanning, Progressive Scan

(89) Interlaced scanning serves to reduce flicker in television engineering. It was developed with the intention to display signals flicker free with a minimal bandwidth. Thereby, a complete image (frame) is constructed from two different half images. During image formation only uneven lines of the output device are displayed for the first half image; if this is complete, the second half image is constructed from the even lines. The human eye perceives the single lines not as flickering, but as travelling, and in the ideal case integrates the line jumps into a complete image.

(90) The progressive scan (Eng. progressive scan) designates a technique in image construction, wherein the output devicein contrast to interlaced scanningdoes not receive line interlaced half images, but real full images.