INSTRUCTION FOR A SIGN LANGUAGE

Abstract

A system for instruction of a sign language, the system comprising a display device configured to display video depicting an object, and display information relating to a sign language sign associated with the object.

Claims

1. A system for instruction of a sign language, the system comprising a display device configured to: display video depicting an object, and display information relating to a sign language sign associated with the object, wherein the object is a subject of the sign-language sign.

2. A system as claimed in claim 1, wherein the display device is configured to display the video depicting the object before or simultaneously with the information relating to the sign language sign.

3. A system as claimed in claim 1, wherein the display device is configured to display at a first time the video depicting the object at a first size and the information relating to the sign language sign at a second, smaller, size, and to display at a second time the information relating to the sign language sign at a size greater than the second size.

4. A system as claimed in claim 3, wherein the display device is further configured to display at the second time the video depicting the object at a size less than the first size.

5. A system as claimed in claim 3, wherein the display device comprises a human-machine-interface device receptive to a user input, wherein the display device is configured to display the information relating to the sign language sign at a size greater than the second size in response to a user input via the human-machine interface device.

6. A system as claimed in claim 1, wherein the information comprises video depicting the sign language sign associated with the object.

7. A system as claimed in claim 6, wherein the video depicting the sign language sign associated with the object comprises video of a human signing the sign language sign.

8. A system as claimed in claims 1, wherein the display device comprises an imaging device for imaging printed graphics.

9. A system as claimed in claim 8, configured to, in response to an imaging event in which the imaging device is used to image printed graphics depicting an object, analyse image data from the imaging event to identify characteristics of the image data, compare the identified characteristics of the image data to image data characteristics stored in memory and indexed to video depicting objects and to information relating to sign language signs associated with objects, retrieve for display video depicting objects and information relating to sign language signs associated with objects that is indexed to the image data characteristics, and display the retrieved video depicting objects and information relating to sign language signs associated with objects.

10. A system as claimed in claim 9, wherein the display device is configured to display the video depicting the object overlaid onto image data from the imaging event.

11. A system as claimed in claim 10, wherein the display device is configured to display the overlaid video depicting the object such that the video appears anchored to a position of the image data corresponding to the printed graphics.

12. A system as claimed in claim 9, wherein the video depicting the object corresponds to a three-dimensional model depicting the object, the electronic device comprises an accelerometer for detecting an orientation of the electronic device, and the electronic device is configured to vary the displayed video in dependence on the orientation of the electronic device.

13. A system as claimed in claim 10 wherein the display device is configured to display the overlaid video depicting the object such that the video appears anchored to a position of the image data corresponding to the printed graphics in a first mode of operation, and display the overlaid video depicting the object such that the video appears not anchored to any position of the image data in a second mode of operation.

14. A system as claimed in claim 13, wherein the display device comprises an accelerometer for detecting the orientation of the display device, and the display device is configured to operate in the first mode of operation in a first orientation of the display device and in a second mode of operation in a second orientation of the display device.

15. A system as claimed in claim 13, wherein the display device comprises a human-machine-interface device receptive to a user input, and the display device is configured to operate in the second mode of operation in response to .a user input via the human-machine-interface device.

16. A system as claimed in claim 8, further comprising a substrate having printed thereon a free-hand monochrome illustration depicting an object for imaging by the imaging device.

17. A system as claimed in claim 16, comprising a plurality of substrates, each substrate having printed thereon a free-hand monochrome illustration depicting an object for imaging by the imaging device, wherein the illustrations printed on the plurality of substrates depict mutually different objects.

18. A system as claimed in claim 1, wherein the display device is adapted to be hand-held or wearable.

19. A computer-readable non-transitory data carrier comprising instructions which, when executed by a computer, cause the computer to carry out the method of: displaying video depicting an object, and displaying information relating to a sign language sign associated with the object.

20. An augmented reality system comprising: a substrate having printed thereon a free-hand monochrome illustration, a computing device having stored in memory data defining characteristics of the illustration indexed to video data, wherein the computing device is configured to receive image data, analyse the image data to identify characteristics of the image data, compare the identified characteristics of the image data to the characteristics of the illustration stored in the memory, and determine whether a match exists between the identified characteristics of the image data and the characteristics of the illustration.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order that the present invention may be more readily understood, embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings, in which

[0073] FIG. 1 shows schematically a system for instruction of sign language embodying an aspect of the present invention;

[0074] FIG. 2 shows a hand held electronic device of the system being used to image a graphical representation of an object printed on a substrate;

[0075] FIG. 3 shows the hand held electronic device being used in a first mode operation to display video depicting the object graphically represented on the substrate;

[0076] FIG. 4 shows the hand held electronic device being used in a second mode of operation to display video depicting the object graphically represented on the substrate;

[0077] FIG. 5 shows the hand held electronic device being used to display video relating to a sign language sign associated with the object;

[0078] FIG. 6 shows a substrate embodying an aspect of the present invention having graphics printed thereon;

[0079] FIG. 7 is a block diagram showing schematically stages of a process for displaying video depicting an object and information relating to a sign language sign associated with the object in response to imaging of printed graphics depicting an object;

[0080] FIG. 8 is a block diagram showing schematically stages of a process for analysing an image to identify image characteristics;

[0081] FIG. 9 shows schematically a computer-implemented technique for analysing an image to identify image characteristics;

[0082] FIG. 10 shows schematically a computer-implemented technique for comparing identified image characteristics to reference image characteristics;

[0083] FIG. 11 shows schematically a computer-generated three-dimensional model of an object;

[0084] FIG. 12 shows schematically representations of the surfaces of the three-dimensional model shown in FIG. 11 printed on a substrate;

[0085] FIG. 13 shows hand illustration applied onto the substrate over the representations of the surfaces of the model; and

[0086] FIG. 14 shows image data of the illustrated substrate mapped onto the three-dimensional model.

DETAILED DESCRIPTION OF THE INVENTION

[0087] A system for instruction of sign language comprises a hand-held electronic device 101, backend computing system 102, and a substrate 103 having printed thereon graphics 104 depicting an object, in the example, a football.

[0088] Hand-held electronic device 101 is a cellular telephone handset having a transceiver for communicating wirelessly with remote devices via a cellular network, for example, via a wireless network utilising the Long-Term-Evolution (LTE) telecommunications standard . Handset 101 comprises a liquid-crystal display screen 106 visible on a front of the handset, and further comprises an imaging device 107 for optical imaging, for example, a CCD image sensor, on a rear of the handset for imaging a region behind the handset. In the example, the screen 106 is configured to be ‘touch-sensitive’, for example, as a capacitive touch screen, so as to be receptive to a user input and thereby function as a human-machine-interface between application software operating on the handset 101 and a user. The handset 101 comprises computer processing functionality and is capable of running application software. As will be described, the handset 101 is configured to run application software, stored in an internal memory of the handset, for the instruction of a sign language, for example, for the instruction of British Sign Language. It will be appreciated by the person skilled in the art, that for the purpose of the present invention, the handset 101 may be a conventional ‘smartphone’ handset, which will typically comprise all the necessary capabilities to implement the invention.

[0089] Backend computing system 102 is configured as a ‘cloud’ based computing system, and comprises a computing device 108 located remotely from the handset 101 in communication with the handset 101 via the wireless network 105. For example, the wireless network 105 could be an LTE compliant wireless network in which signals are transmitted between the computing device 108 and the handset 101 via intermediate wireless transceivers

[0090] Substrate 103, in this example, is a sheet of paper having the graphics 104 printed on a surface of the paper, for example, using an inkjet printer. The graphic 104 is a representation of a free-hand illustration of a football.

[0091] Referring in particular to FIG. 2, handset 101 is operated to run application software, which causes the imaging device 107 of the handset 101 to continuously image a region behind the handset. Handset 101 may thus be located in front of substrate 103 to thereby image the graphic 104 printed on the substrate 103. Handset 101 is configured to transmit image data in real time to the backend computing system 102 via the wireless network 105 for processing by the computing device 108.

[0092] The backend computing system 102 is configured to receive the image data and process the image data to detect characteristics of the imagery. As will be described in detail with reference to later Figures, the backend computing system 102 is configured to analyse the received image data to detect whether a graphic depicting an object corresponding to a predefined object data set stored in memory of the computing device 108 is being imaged. In the example of FIG. 2, the backend computing system 102 analyses the graphic 104 depicting a football printed on substrate 103, and matches this to image to video data depicting a football and to video data relating to a sign language sign associated with a football, that is stored in memory of the computing device 108. In response to the match, the backend computing system 102 is configured to transit the video data depicting a football and the video data relating to a sign language sign associated with a football to the handset 101 via the wireless network 105.

[0093] As an alternative to backend computing system 102 for processing of image data captured by the imaging device 107 of the handset 101, the handset could comprise on-board image processing functionality for processing the image, thus negating the requirement to transmit image data to the backend computing system. This may advantageously reduce latency in processing of the image, for example resulting from delays in transmission, but disadvantageously may increase the cost, complexity, mass, and/or power-consumption of the handset 101.

[0094] Referring next in particular to FIG. 3, the handset 101 is configured to display the received video depicting the football and also display the video depicting the sign language sign associated with a football on the screen 106 simultaneously on regions 301, 302 of the screen respectively. Display video depicting the object which is the subject of the sign language sign may advantageously aid understanding of the nature of the object to be signed by the user. In the example, the video depicting the object is an animation of a football bouncing up and down on real-time video imagery of the substrate 103. In the example, the video depicting the object thus takes the form of ‘augmented reality’ imagery, in which video data depicting a football that is received from the backend computing system 102 is overlaid onto real-time imagery imaged by the imaging device 107 of the handset 101. Augmented reality imagery of this type may be particularly effective in aiding a user’s understanding of an object to be signed.

[0095] Referring still to FIG. 3, the system is configured to firstly display on the screen 106 the video depicting the object, in the example a football bouncing up and down, on a large area of the screen 301, i.e. in a ‘fullscreen’ mode, and to display the video relating to the sign language sign on a smaller area of the screen 302, i.e. as a ‘thumbnail’. This configuration may best allow a user to understand the nature of the object depicted in the video, whilst also providing a preview of the sign language sign associated with the object to be signed.

[0096] The application software running on handset 101 allows for switching between ‘anchored’ and ‘non-anchored’ modes of viewing the videos. In a first, ‘anchored’, mode of operation, depicted in FIG. 3, the video data depicting the object football is overlaid onto real-time imagery captured by the imaging device 107 of the handset 101 in a way that the video data depicting the object football remains positionally locked relative to the position of the imagery of the graphic 104 on the substrate 103. Thus, in this mode of operation the positions of the video depicting the object, e.g. the bouncing football, and the real-time imagery of the graphic printed on the substrate adapt relative to the area of screen to accommodate for movement of the handset 101. This may provide a realistic visual which may best engage the user’s interest and attention. In a second, ‘non-anchored’ mode of operation, depicted in FIG. 4, a static snapshot from imagery of the graphic printed on the substrate may be displayed on the screen 106 over which the video depicting the object, i.e. the bouncing ball, is overlaid. Thus, in this mode of operation the user is not required to continuously point the imaging device 107 of the handset 101 at the graphic 104 printed on the substrate 103, rather the user may hold the handset in any desired position whilst the video depicting the object and the imagery of the printed graphic remain visible. This second mode of operation may allow a user to relax and move positions whilst maintaining use of the application software to view the object video and the image of the printed graphic. For the purpose of controlling the mode of operation the application software presents an icon 303 on the screen 106. In response to a user touching the icon 303 the application software is configured to switch between the anchored and non-anchored modes of operation of FIGS. 3 and 4 respectively.

[0097] Referring next in particular to FIG. 5, the application software running on the handset 101 is configured, after firstly displaying the video depicting the object in ‘fullscreen’ mode, as illustrated in FIGS. 3 and 4, to change the display such that secondly the video relating to the sign language sign is displayed on a large area of the screen 501, i.e. in ‘fullscreen’, and the video depicting the object is displayed on a small area of the screen 502, i.e. as a ‘thumbnail’. In this order of display, a user having first seen the video depicting the object, and thus hopefully having fully understood the nature of the object, may subsequently view and learn the sign language sign associated with the object. Maintaining the video depicting the object as a thumbnail may usefully serve as an aide memoire to the user as to the nature of the object associated with the sign language sign.

[0098] Referring in particular to FIG. 6, the graphic 104 printed on the substrate 103 is a monochrome representation of a free-hand illustration of an object, in the example, a football. Thus, for example, the process of producing the printed substrate may comprise firstly creating a free-hand illustration of the object football, uploading a scan of the free-hand illustration to a computer running a printer control program, and using the computer to control a printer, for example, a lithographic printer, to apply an ink to the substrate 103. The process could optionally comprise an intermediate image editing process implemented on the computer where the scanned image of the illustration could be edited, for example, to add additional features or to delete features of the illustration from the image.

[0099] It has been observed that a free-hand illustration provides a particularly effective means of representing an object to be imaged by the imaging device. This is thought to be because of the natural variability in features of the illustration that result from free-hand illustration. Referring in this regard still to FIG. 6, it will be noted that the free-hand illustration of the football comprises a large number of different line features, such as edges 501, 502, each of which features may serve as a reference point in an ‘automated’ computer-implemented process of feature detection, for example, in an edge detection technique. This relatively great number of potential image reference features advantageously increases the identifiable variations between illustrations of different objects, thereby reducing the risk of mis-identification of an object by the system.

[0100] In contrast, illustrations created using a computer in a line vector format, where each point of the illustration is defined by common co-ordinates and relationships between points by a line and curve definitions from a finite array of possible definitions, tend to exhibit lesser variation between illustrations of different objects. It has been observed that this undesirably increases the risk of mis-identification of an object by the system.

[0101] Moreover, it has been found that the illustrations should preferably be presented in monochrome. Monochrome colouring provides a maximal contrast between line features of the illustration. This has been found to advantageously improve feature detection in a computer implemented feature analysis technique, for example, an edge detection technique. This reduces the risk of mis-identification of the illustration by the system.

[0102] In the specific example, the substrate 103 is paper. Paper advantageously provides a desirably flat and uniform structure for graphics 104, which may improve imaging of the graphics by the imaging device 107. However, the graphics 104 could be printed onto an alternative substrate, for example, onto a fabric. This may be desirable, for example, where the graphic is to be printed onto an item of clothing, for example, onto a shirt.

[0103] Certain difficulties have been observed however in printing graphics onto fabric for the purpose of using the graphics in a computer-implemented image analysis technique. In particular, it has been observed that with certain fabrics, for example, coarsely woven cotton such as hessian, image resolution is lost when the graphics are printed onto the fabric due to the large spacing between threads. Problems associated with lost resolution are particularly exacerbated for graphics having relatively small dimensions. Preferred fabrics for this application are cotton, silk, bamboo and linen. Types of suitable cotton include: Poplin cotton, ringspun cotton, combed cotton and cotton flannel.

[0104] A preferred fabric for the application is ringspun cotton-style weave having a weight of 100 grams per square-metre, or greater, preferably at least 150 grams per square-metre, and even more preferably at least 180 per square-metre.

[0105] A number of particularly suitable fabric and printing techniques have been identified, including: (1) Muslin cloth comprising 100% cotton and having a minimum weight of 100 grams per square-metre, where graphics are printed onto the fabric using screen-printing or direct-to-garment techniques, with a graphic size of at least 5 square-centimetres; (2) Ringspun cotton comprising 100% cotton having a minimum weight of 180 grams per square-metre, where graphics are printed using screen-printing with a minimum graphic size of 4 square-centimetre, or direct-to-garment techniques with a minimum graphics size of 2 square-centimetre: (3) Heavyweight cotton, having a weight of at least 170 grams per square-metre, graphics printed using a screen printing technique with a minimum graphic size of 4 square-centimetres, or printed using a direct-to-garment technique with a minimum graphic size of 2 square-centimetres: (4) Denim, having a weight of at least 220 grams per square-metre, graphics printed using a screen printing technique with a minimum graphic size of 5 square-centimetres, or printed using a direct-to-garment technique with a minimum graphic size of 3 square-centimetres: and (5) Curtain, having a weight in the range of 250 grams per square-metre to 300 grams per square-metre, graphics printed using a screen printing technique with a minimum graphic size of 5 square-centimetres, or printed using a direct-to-garment technique with a minimum graphic size of 3 square-centimetres.

[0106] Suitable fabrics may comprise cotton and synthetic fibre mixes, for example, polyester synthetic fibres in a 60% cotton, 40% polyester mix, or acrylic synthetic fibres in a 70% cotton, 30% polyester mix. It has been observed in this respect that synthetic fibre additions may improve the print resolution for printed graphics. Further cotton-synthetic mixes that have been observed to form a suitable substrate for printing of the graphics, including Spandex, Elastane and Lycra, although for these fibres a relatively greater percentage of cotton should be used in the mix, for example, a 90% cotton, 10% synthetic fibre mix.

[0107] Fabrics laminated to paper, for example, bookbinding cloth, have additionally been observed to form suitable substrates for printing of the graphics. It has been observed in this regard that laminating fabrics to paper improves the flatness of the printing surface of the material, thereby reducing distortion of the graphic resulting from creasing of the fabric. Suitable print techniques for printing onto laminated fabric include screen-printing, offset litho-printing, and direct-to-garment printing. A preferred minimum graphic size for offset-printing onto fabric laminated to paper is 5 square-centimeter. Foil stamping is a further known suitable printing technique for printing graphics onto fabrics laminated to paper, in which technique lines of graphics should be at least 1 millimetre in width, a graphics should have a minimum size of 5 square-centimetres.

[0108] Where graphics are screen-printed onto fabric, it has been observed that a silkscreen printing weave of at least 120 thread per centimetre (T) should ideally be used. Larger images may however be acceptably printed using a silkscreen printing weave with a lower thread count, although the thread count should ideally be at least 77T.

[0109] Referring to FIG. 7, a process for imaging a graphic depicting an object printed on a substrate and displaying video depicting that object and sign language information relating to a sign associated with the object is shown.

[0110] At step 701 an imaging event is initiated, whereby the imaging device 107 of the handset 101 begins to image its field of view. The imaging event could for example be initiated automatically by the application software.

[0111] At step 702 image data captured by the imaging device 107 of the handset 101 is stored in computer readable memory. In the specific example, where image analysis and comparison is performed by a computing device 108 located remotely from the handset 101, the step of storing the image data is preceded by an intermediate step of firstly transmitting the image data from the handset to the backend computing system 102 for storage on memory of the computing device 108. In an alternative implementation however, image analysis and comparison could be performed locally on the handset 101, in which case storing the image date could comprise storing the image data on local memory of the handset 101.

[0112] At step 703 a computer implemented image analysis process is implemented to identify characteristics of the stored imagery. Data defining image characteristics may then be stored in memory of the computing device undertaking the image analysis, in this exampie in the memory of the remote computing device 108. The image analysis process is described in further detail with reference to FIGS. 8 and 9.

[0113] At step 704 a computer implemented image comparison process is implemented, whereby the identified characteristics of the captured imagery are compared to data sets stored in memory of the computing device 108, which data sets are indexed to video files depicting an object corresponding to the identified image characteristics and to video files relating to a sign language sign associated with the corresponding object. The image comparison process is described in further detail with reference to FIG. 10.

[0114] At step 705 the video files depicting an object corresponding to the identified image characteristics and video files relating to a sign language sign associated with the corresponding object are retrieved from memory of the computing device 108, and transmitted using the wireless network 105 to the handset 101.

[0115] At step 706 the retrieved video files are displayed on the screen 106 of the handset 101 in accordance with the implementation described with reference to FIGS. 3 to 5.

[0116] Procedures of the image analysis step 703 are shown schematically in FIG. 8 In a first step 801 the imagery imaged by the imaging device 107 of the handset 101 is pixelated, such that the image is represented by an array of discrete pixels having colour characteristics corresponding to the colouring of the original image. A simplified image pixelation technique is shown in FIG. 9, whereby the captured imagery is divided into an array 901 of pixels.

[0117] At step 802 a conventional edge detection process is implemented by the computing device 108 The edge detection process may address each pixel of the array in turn. For example, the edge detection process could assign a value to each pixel in dependence on the colour contrast between the pixel and a neighbouring pixel This measure of colour contrast may be used as a proxy for detection of a boundary of a line feature of the illustration. The result would thus be an array of values corresponding in size to the number of pixels forming the pixelated image.

[0118] At step 803 the detected image characteristics are stored in memory of the computing device 108.

[0119] A simplification of the image comparison step 704 is shown schematically in FIG. 10. Referring to the Figure, a data set 1001 defining characteristics of the captured image is retrieved from memory of the computing device 108. In the example, the data set comprises a 3x3 array, and thus defines a 9 pixel image. In the simplified example, each pixel of the array is assigned a value of either 0 or 1 in dependence on the degree of colour contrast between the pixel and an immediately adjacent pixel. For example, where the colour contrast exceeds a threshold a value of 1 is assigned, whereas where the colour contrast is less than a threshold a value of 0 is assigned. Thus, the dataset defining the image characteristics may be compared to datasets 1002, 1003, 1004 stored in memory of the computing device 108 that are indexed to video depicting an object and video relating to a sign language sign associated with the object. Where a match 1005 in the datasets is identified, it may be inferred that the captured image is of a particular object, and video indexed to the dataset 1004, and information relating to a sign language sign indexed to dataset 1004, may be retrieved for display.

[0120] Processes relating to a method of generating a computer model for an augmented reality system as shown in FIGS. 11 to 14.

[0121] The method involves a first step of generating using a computer a three-dimensional model 1101 of an object, in the example a football, comprised of a plurality of constituent three-dimensional blocks, such as blocks 1102, 1103. In the example, the model 1101 is defined by a plurality of polygons. The model is analysed to identify surfaces of the blocks that define a visible surface of the three-dimensional model, such as surfaces 1104 and 1105.

[0122] Referring in particular to FIG. 12, representations 1201 of the shapes of the visible surfaces of the blocks of the model 1101 are then printed, for example, using a computer-controlled printer, onto a substrate 1201, for example, onto paper or fabric.

[0123] Referring next in particular to FIG. 13, the method then involves hand-illustrating onto the substrate 1201 over the representations of the visible surfaces of the blocks with desired graphics, in the example, graphics depicting surface markings of a football.

[0124] Referring next to FIG. 14, following an intermediate step of imaging the illustrated substrate 1201 and uploading the image data to a computer, the method involves mapping the image data on to the three-dimensional model 1101, such that image date depicting the hand-illustrated surfaces of the blocks is assigned to its corresponding position on the visible surface of the model.