Fitting System

Abstract

A method and apparatus to scan and measure the surface of objects using a radiation source, such as a laser. When combined with bio-data from bio-sensors to create a profile, the invention is particularly useful for measuring the internal surface of a prosthetic socket and improving the fit between the socket and the residual limb.

Claims

1. A method of identifying the differences in shape and the physical contact characteristics between an object and a body part which is engageable with the object comprising: scanning the object with radiation in order to produce a surface map of the object; attaching a plurality of bio-sensors to at least one of a surface of the object or to a surface of the body part at locations which are known relative to a reference point, engaging the body part with the object; collecting bio-sensor data from the bio-sensors to record information on the engagement between the body part and the object over the surface of the object; and superimposing the data from the bio-sensors over the surface map of the object in order to identify areas of the object which need to be adjusted in order to improve the fit of the body part with the object.

2. The method of claim 1, wherein the object is one of a prosthetic socket, an orthotic article, an article of furniture, or a wheelchair, and the body part is one of a stump of an amputated limb, stump of an amputated toot, a complete limb, a skin of a patient, a bottom sitting on a wheelchair, a back lying on a bed, or a liner covering at least part of the body part.

3. (canceled)

4. The method according to claim 1, wherein the scanning comprises: projecting radiation pattern onto the surface of the object with a radiation source at a first distance from the surface of the object; taking as image data an image of the radiation projected onto the surface using at least one capturing element which is in a fixed and known position relative to the radiation source; analyzing the image data from the capturing element to identify a position in three dimensions of each point illuminated by the projected radiation; varying the distance of the radiation source from the surface in order to change the parts of the surface illuminated by the projected radiation; using the data from the capturing element in order to identify the position of each new point illuminated by the projected radiation; and repeating until all points on the surface have been scanned.

5. The method according to claim 2, wherein locations of the plurality of biosensors is determined by illuminating the plurality of biosensors and capturing image data.

6. The method according to claim 2, wherein the scanning is carried out by a radiation pattern as a projected laser pattern.

7. The method according to claim 2, wherein the collecting of data comprises one of data relating to pressure or temperature at the locations.

8. The method according to claim 2, further comprising displaying a 3D model of the mapped surface to a user with the bio-sensor data superimposed thereon.

9. The method according to claim 1, further comprising projecting onto the surface of the object a pattern which is calibrated to the surface map generated by a computer so as to enable a user easily to correlate the actual surface with the virtual surface and hence easily identify on the real surface areas of the object which need adjusting based on the bio-sensor information.

10. The method according to claim 8, wherein the object is a prosthetic socket and the adjusting comprises forming an interior surface of the prosthetic socket.

11. The method according to claim 1, further comprising using data from at least one motion unit to identify areas of the object which need to be adjusted in order to improve the fit of the body part with the object during movement.

12. An apparatus for identifying the differences in shape and physical contact characteristics between an object and a body part which is engageable with the object, the apparatus comprising: a radiation source for scanning the surface of the object in order to produce a surface map thereof; an adjuster for varying at least one of a distance or orientation of the radiation source from the surface of the object; a plurality of bio-sensors attachable to at least one of the surface of the object, to the surface of the body parts or to liners covering body parts at locations which are known relative to a reference point, data collectors connected to the plurality of bio-sensors for collecting bio-sensor data from the plurality of bio-sensors; data processing device adapted for superimposing the bio-sensor data onto the surface map to produce a bio-data profile map of the object; and a display for displaying the bio-data profile map to a technician.

13. The apparatus of claim 12, wherein the plurality of bio-sensors are arranged as a bio-sensor strip comprising plastic films, powder leads and data leads, a power and data connector, wherein the plurality of bio-sensors are disposed on the bio-sensor strip (821, 921) and at least one power lead and data lead is connected to one or more of the plurality of bio-sensors and the at least one power lead and data lead are placed on the bio-sensor strip and are in contact with an interlace component itself connected with a power supply and the data processing device.

14. The apparatus according to claim 12, wherein the radiation source is a conical laser assembly which comprises a single point laser and an optical element which converts the single laser beam into a two-dimensional

15. (canceled)

16. (canceled)

17. An apparatus according to claim 12, further comprising a capturing element associated with, and in a known position relative to pattern on the surface and measuring the distance between a plurality of points illuminated by the radiation source and the capturing element, and a data processing device for processing the data from the capturing element and converting the data into a map of the surface.

18. An apparatus according to claim 17, wherein the capturing element is arranged in a fixed position relative to the radiation source, the distance from the laser being known, and moves with the radiation source towards and away from the surface.

19. (canceled)

20. The apparatus according to claim 12, further comprising a plurality of light sources for illuminating the surface of the object to identify locations of the plurality of biosensors.

21. (canceled)

22. (canceled)

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order that the invention may be well understood, there will now be described some embodiments thereof, given by way of example, reference being made to the accompanying drawings, in which:

[0041] FIGS. 1A to 1E are a conceptual representation of the invention where it relates to socket surface acquisition, measurement and rendering.

[0042] FIG. 2 is a view of the first aspect of the laser and camera system.

[0043] FIG. 3A is a section view of a second aspect of the laser and camera system.

[0044] FIG. 3B is a section view of a third aspect of the laser and camera system.

[0045] FIG. 4 is a representation of the laser plane and the camera field of view of the first aspect.

[0046] FIG. 5A is a representation of the laser plane and the camera field of view of the second aspect.

[0047] FIG. 5B is a representation of the laser plane and the camera field of view of the third aspect.

[0048] FIG. 6 is a top view of the laser plane and the camera field of view of the first aspect.

[0049] FIG. 7A is a top view of the laser plane and the camera field of view of the second aspect.

[0050] FIG. 7B is a top view of the laser plane and the camera field of view of the third aspect.

[0051] FIGS. 8A, 8B and 8C are representations of the bio-sensors strip

[0052] FIG. 9 is a conceptual representation of the invention combining the bio-sensor data with the socket surface map, resulting in a superimposed bio-data and virtual socket surface map.

[0053] FIGS. 10A to 10F describe the invention in use, showing a prosthetic technician and a patient with an amputated leg, where the technician is using virtual reality vision system to better observe areas to adjust in the socket.

[0054] FIGS. 11A to 11F describe the invention in use, showing a prosthetic technician and a patient with an amputated leg, where the technician is using an augmented reality device to better observe areas to adjust in the socket.

[0055] FIGS. 12 and 13 show flow diagrams of the method

DETAILED DESCRIPTION OF THE INVENTION

[0056] The invention will now be described on the basis of the drawings. It will be understood that the embodiments and aspects of the invention described herein are only examples and do not limit the protective scope of the claims in any way. The invention is defined by the claims and their equivalents. It will be understood that features of one aspect or embodiment of the invention can be combined with a feature of a different aspect or aspects and/or embodiments of the invention.

[0057] Referring first to FIGS. 1A to 1E, there is shown a summary of the steps involved in mapping and modelling in three dimensions of a socket for an artificial limb. Reference is also made to FIGS. 2 to 4 which shows the apparatus used for the mapping and modelling of the socket in three dimensions. The description below assumes that a laser is used as the radiation source, but it will be appreciated that other beams of light could be used to scan the socket of the artificial limb and this application is not limited to laser scanning.

[0058] FIG. 1A shows a projected laser line 101, as it is projected on the surface of a scanned object. FIG. 1B shows the reference point 102 of the projected laser line 101. In this aspect, the projected laser line 101 is substantially circular in form and the reference point is the centre of the circle, as the centre is computed from data acquired by one or more cameras 411, 511 (see FIGS. 4, 5A and 5B) in a subsequent step. It will be appreciated that the projected laser line 101 may not be circular (or elliptical) in other aspects and, in this case, a suitable reference point needs to be found and used. FIG. 1C shows the projected laser line 101 segmented into discrete data points 103 following identification of the projected laser line 101 by the camera 211. This results in a complete conversion of the projected laser line 101 into a plurality of individual pixels 103, of which A, B and C are representations. By knowing the resolution of the camera 411 and its position relative to the laser beam, by calculating the position of all of the plurality of the line pixels 103, by calibrating the system to infer the real position of each point, the distance of each data point 103 of the line 101 to the reference point 102 may be calculated, and thus position and spatial coordinates of the data points 103 can be determined. These data points 103 will also represent the actual surface of the scanned object.

[0059] In FIG. 1D, as the camera 211 and laser assembly move together in incremental steps along the z axis, scanning a new area of the socket, variations in the surface dimensions of the socket will result in a corresponding change in the projected laser line 104a-g being projected thereon. As the projected laser line acquisition, segmentation, pixel conversion and distance-to-reference point process is repeated across the entire depth of the scanned object, this results in a plurality of virtual substantially projected laser lines 104, which are then joined, as shown in FIG. 1E, to generate a virtual surface map 106 of the scanned object. FIG. 1E also shows a circled x .Math. 105 which represents origin coordinates on the virtual surface map 106 of the scanned object. The circled x as the origin coordinates 105 will be used as a spatial reference point to allow correct alignment with the origin coordinates 105 of a bio-data map at the final stage of the process.

[0060] In FIG. 2, there is shown a schematic illustration of a conical laser assembly 200 according to a first aspect of the invention. There is a moving support assembly 240 which supports a devices support frame 210 and the devices support frame 210 moves with the moving support assembly 240. The devices support frame 210 supports a camera 211, a laser device 213 provided with a laser lens 214 and a conical mirror 215. The moving support assembly 240 is connected to a linear screw 208 by a bushing 209 so as to be moveable towards and away (more frequently vertically) along the longitudinal axis of the scanned socket 216. The linear screw 208 and moving support assembly 240 are secured to an anchor frame 239 which will be attached to a solid, immovable surface or point such as a wall or an appropriate apparatus housing. The linear screw 208 is attached to a motor 207 mounted on the anchor frame 239. The motor 207 rotates the linear screw 208, leading to a movement of the bushing 209 and consequently of all the moving support assembly 240 of all elements (210, 213, 211, 215) connected thereto.

[0061] The camera 211 is mounted above a single point laser device 213 which projects a conventional laser beam 236 onto a conical mirror 215. The laser 213 is arranged to focus the laser beam 236 on the vertex of the conical mirror 215, and the mirror surface of the conical mirror 215 reflects the laser beam 236 outwards from the plane of the base of the mirror 215 so as to project the laser line 201 extending from the plane of the base of the mirror 215. The scanned object, a prosthetic socket 216, is mounted on a fixing base 217 which does not move so that the scanned object remains stationary. A physical origin coordinate 212, identified by a circled cross and placed on the surface of the socket 216 provides a spatial reference point which will be useful to orient and align the physical socket 216 with the virtual 3D model of the socket 216 and with the 3D profile of the bio-data.

[0062] In use, the devices support frame 210 moves vertically, starting from a top position where the laser 213 focuses its laser beam 236 on the conical mirror 215 and begins to scan the top of the socket 216. A line of laser light, the perimeter of which is represented by points line 201 is projected on the internal area of the socket 216, whereupon the process previously described of laser line acquisition, segmentation, calibration, distance-to-reference point calculation and line coordinates calculation is performed. These data are stored in a computer (not shown) and the motor 207 turns the linear screw 208 which in turn moves the moving support assembly 240 to the next incremental position, thereby lowering the devices support frame 210 one unit of movement (typically in increments of 5 mm, but this is not limiting of the invention). The entire process is repeated again in a new data acquisition stage until the entire socket 216 internal surface is scanned and mapped. At the conclusion of the process the data corresponding to each slice of socket surface is joined and a full 3D map of the lines is formed, thus rendering a virtual image of the socket 216 as previously shown in FIG. 1F.

[0063] It will be inferred that there is a blind spot on socket 216 mapping caused by the devices support frame 210, the arms of which will block the field of view of camera 211. In order to acquire the hidden points on the socket 216 surface, this may be achieved by installing a decoupling mechanism (not shown) between the moving support assembly 240 and the devices support frame 210, which will allow the support arms of the devices support frame 210 to rotate sufficiently, for the previously hidden portion of laser plane 201 to become visible to the camera 211 while the camera 211 stays in the same position.

[0064] Strips of bio-sensors 219 are arranged on the inside of the socket 216 and will record various biomedical parameters, as explained below. These bio-sensors 219 are described in more detail in connections with FIGS. 8 and 9. Only two bio-sensors 219 are shown on FIG. 2 for simplicity, but in fact the inside surface of the socket 216 will have a much larger number of bio-sensors 219. It will also be realised that the bio-sensors 219 are shown much larger on this FIG. 2 than in real life, as the bio-sensors 219 should not affect the position of the limb in the socket 216. The bio-sensors 219 have target or reference markings on their top surface which are visible to the camera. A light source 220, such as an LED white light, illuminates the inside of the socket 216 and the camera 211 records the position of the bio-sensors 219 using the markings. The camera 211 is moved along the vertical axis and several images are captured. The shape of the markings is known and thus the position of the bio-sensors 219 relative to the vertical axis can be determined. It would also be possible to take images under ambient light.

[0065] FIG. 3A shows a second aspect of the invention which overcomes the blind spot issue discussed above. This second aspect uses the same laser 313 provided with a laser lens 314 which comprises a diffractive optical element 315. When the laser beam from the laser 313 is directed through diffractive optical element 315, the optical element 315 diffracts the projected laser beam 337 and produces a projected solid laser line 301. This is projected outward onto the surface of the scanned socket 316, the diameter and contour of the projected laser line 301 being dependent on the surface of the scanned socket 316.

[0066] The laser 313 and the camera 311 are mounted on the device supporting frame 310 that is attached to a moving support assembly 340. The moving support assembly 340 is connected to a linear screw 308 by a bushing 309 so as to be moveable towards and away (more frequently vertically) along the longitudinal axis of the scanned socket 316. The linear screw 308 is attached to the anchor frame 339 which will be attached to a solid, immovable surface or point such as a wall or an appropriate floor-standing apparatus housing 341. The linear screw 308 will be attached to a motor 307, which will rotate the linear screw 308 leading to a movement of the bushing 309 and consequently of the moving support assembly 340 and of all elements (310, 313, 311, 315) connected thereto.

[0067] The capturing element in the form of the camera 311 is mounted in a fixed position relative to the laser 313 but unlike the first aspect shown in FIG. 2, the camera 311 is slightly offset from the longitudinal axis of the laser 313. The camera 311 thus moves with the laser 313 towards and away along the longitudinal axis of the scanned socket 316. The scanned object, a prosthetic socket 316, is kept in place by a fixing base 317, which does not move so that the prosthetic socket 316 remains stationary during scanning. A physical origin coordinate 312, identified by a circled cross and virtually placed or actually drawn on the surface of the prosthetic socket 316 provides a spatial reference point which will be useful to orient and align the physical prosthetic socket 316 with the virtual 3D model of the prosthetic socket 316 and with the 3D model of the bio-data.

[0068] In use, the laser 313 and the camera 311 move together to scan and map the interior surface of socket 316. The optical element 315 diffracts the laser light so as to produce a projected laser cone 301 on the surface of the scanned socket 316, where the laser cone 301 is projected, whereupon the process previously described of line acquisition, segmentation, calibration, distance-to-reference point calculation and line coordinates calculation is performed. These data are stored in a computer (not shown) and the motor 307 moves to the next incremental position, thereby moving the devices support frame 310 one unit of movement (typically but not limiting of the invention, 5 mm), and the entire process is repeated again until the entire socket 316 surface is scanned and mapped. At the conclusion of the process the data corresponding to each slice of socket surface is joined and a full 3D map of the lines is formed, thus rendering a virtual image of the socket 316 as shown in FIG. 1F.

[0069] Unlike the first aspect shown in FIG. 2, the second aspect of FIG. 3A does not have obstacles on the path of the camera 311 or the projected laser line 301, and there are therefore substantially no hidden areas on the socket 316 surface. It should be noted that data from any hidden areas can be reconstructed mathematically.

[0070] A third aspect of the invention is shown in FIG. 3B which shows an arrangement with two (or more) cameras 311 positioned to the right and left of the laser 313. The other elements depicted in FIG. 3B are otherwise identical with those of FIG. 3A. The arrangement shown in FIG. 3B is able to scan more accurately the surface because more information from the two cameras 311 is gained and a stereographic picture can be formed.

[0071] In FIG. 4, there is shown a schematic representation of the first aspect of the invention, comprising a camera 411, a field of view 418 and a laser plane 401, created by the reflection of the laser beam from the laser 413 on the conic mirror 415. The camera field of view 418 has a centre C1, a length C3 and a width C2, so that the image which will be captured will have C3C2 pixels and this number of captured pixels will vary depending on camera resolution of the camera 411. The laser plane 401 will have a variable shape depending on the surface and shape of the scanned object, e.g. the prosthesis socket 216 shown in FIG. 2. When scanning the socket 216, the laser plane 401 will project a circle of laser light with a centre L1 of a shape which will most frequently be roughly elliptical. L3 and L4 are examples of the variable diameters of that elliptical shape. In this first aspect, the centre points C1 and L1 are in the same x, y, and z position and will be considered to the reference point in this first aspect. By keeping a fixed distance L2 between the camera 411 and the laser plane 401, it is possible to determine a calibration rule that relates the dimension in pixels in the virtual image to the dimension in millimetres of the real scanned object (e.g. socket 216) and with this rule calculate the position of each point of the scanned object surface. By joining all these points acquired at each acquisition stage, a full model of the scanned object may be obtained.

[0072] In FIGS. 5A and 5B, there are shown a schematic representation of the second aspect and the third aspect of the invention. These FIGS. 5A and 5B show a laser 513, a single camera 511 in FIG. 5A and two cameras 55 in FIG. 5B, a diffractive optical element 515, a single camera field of view 518 in FIG. 5A and two camera fields of view 518 in FIG. 5B and a projected laser beam 501. The projected laser beam 501 produces a pattern of know dimensions and shape. It could be for example, a cone or a grid pattern, but this is not limiting of the invention. It will be noted that the laser 513 and the camera(s) 511 are fixed in relation to each other. The diffractive element 515 creates a laser plane 501 with an angular opening L5. The laser plane 501 has a variable shape according to the scanned object shape and surface and the diffractive element 515. The camera field of view 518 depicted in FIG. 5A has a reference axis C1, a length C3 and a width C2. The projected laser beam produces a series of laser scans 501 along the reference axis will have a variable shape depending on the surface and shape of the scanned object. When scanning the socket 316 of FIG. 3, each successive laser scan 501 be imaged as a two-dimensional image in the camera(s) 511 will again have a reference point L1 on the reference axis and L3 and L4 are examples of the points of the scan.

[0073] However, in this second aspect shown in FIG. 5A, camera field of view reference point C1 and laser reference point L1 are in different positions and the distance D1 between the camera field of view reference point C1 and the laser plane reference point L1 is a function of the distance between the reference axis of the laser 513 and the reference axis of the camera 511. The field of view reference point C1 will always be on the same longitudinally oriented reference axis of the camera in successive ones of the (two-dimensional) images taken at each of the image acquisition stage. However, the laser plane reference point L1 in the camera field of view will vary as the surface of the scanned object 316 approaches the camera 511 or moves away from the camera 511. For a given point of the projection of the laser beam on the scanned object 316, it is possible to define a line of view from the point of projection to the camera(s) 511. The world coordinates of this given point will be the intersection of this line of view from the camera(s) 511 to the projected laser pattern. These variable distances and the changing of the laser reference point L1 require appropriate calculation, calibration and dynamic calibration methods, to relate the virtual dimensions of each of the series of images acquired by the camera 511 with the real dimensions in millimetres of the scanned object surface upon which the projected laser beam 501 is projected and thus to determine with precision the correct coordinates of each point of the projected laser beam 501 and therefore the correct surface measurement of the scanned object 316. This variable value of distance D1 only occurs in the images (virtual D1). In reality, the distance between the reference axis of the laser 513 and the reference axis of the camera 511 (real D1) is always substantially the same, thus allowing to compute the dynamic calibration method.

[0074] Similar issues occur with the third aspect of the invention shown in FIG. 3B, which includes two cameras 311. The third aspect is different from the second aspect in the sense that the two (or indeed more) cameras 311 require a variation of the method for achieving the 3D reconstruction. As described with respect to the first and second aspects, the cameras 311 and laser 313 are moved along the vertical axis and the projected laser beam 337 is captured by both of the cameras 311. This capture can be simultaneously performed or statically.

[0075] The relative orientation/position and origin of the field of views of the cameras 311 is known and thus by identifying a given physical reference (i.e. a point 301 of the projected laser beam) in the captured image by both cameras 311, it is possible to infer the relative position of the point 301 of the projected laser beam 337 to the reference axis. The reference axis has an origin between the cameras 311. This same physical reference of the point 301 captured by both of the cameras 311 is represent by different pairs of pixels in the two-dimensional images taken by the cameras 311 and it is this difference combined with the position and orientation between cameras 311 that enables the calculation of the three-dimensional position of the points 310 on the projected laser beam 337.

[0076] To find the relative position between both of the cameras 311 (after being placed in the camera mount), several images of a given reference figure should be captured simultaneously by both cameras 311, at different distances and orientation to the cameras 311. For example, the given reference figure could be a chessboard, but this is not limiting of the invention. By identifying key-points in this reference figure in both of the captured images (either by manually picking or automatic processing) and previously knowing their real/physical distances between key points in the reference figure, it is possible to mathematically determine the relative position between the field of view of both of the cameras 311.

[0077] Both the sockets 316 in FIGS. 3A and 3B have bio-sensors 319 as explained in connection with FIG. 2. A light source 320 illuminates the bio-sensors 319 and the camera(s) 311 record the position of the bio-sensors 319. In the case of FIG. 3B, there are two light sources 320.

[0078] FIG. 6 illustrates a top view of the first aspect, comprising the camera 611 and the laser 613, and their respective longitudinal axes are aligned along the same axis. The laser plane 601 is centred and inside the camera field of view 618.

[0079] FIG. 7A shows a top view of the camera 711 and the laser 713 in the second aspect, showing the two devices, the camera 711 and the laser 713 to be placed no longer along the same axis, but offset from each other, resulting in the centre of laser plane 701 to be different from the reference axis, i.e. centre of camera field of view 718. Similarly, FIG. 5B shows the same top view of the third aspect of the invention with two cameras 711.

[0080] It will be appreciated that the use of the two cameras 311, 511, 711 in the third aspect of the invention means that both cameras 311, 511, 711 need to be calibrated in order to know the precise relative position and pose between the two cameras 311, 511, 711 and the lens distortion in each of the two cameras. These parameters are always different due to manufacturing variability.

[0081] To calibrate the cameras 311, 511, 711 and to compensate for difference in the lens parameters of the cameras 311, 511 and 711, a method based on Zhang. A Flexible New Technique for Camera Calibration published in IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000, is used. This method requires a custom-designed camera mount which holds both cameras 311, 511, 711 and the laser 313, 513, 713 i.e. the same laser that will be used in the laser scanner system. In the calibration, a chessboard design placed in a flat surface is used as a reference figure. The dimensions of the squares of the chessboard is known. A number of pictures are taken simultaneously from both of the cameras 311, 511, 711 in which the chessboard is placed at different distances to the cameras 311, 511, 711 and at different orientations.

[0082] The corners of the chessboard of every pair of images taken from the cameras 311, 511, 711 are detected. The number of squares of the chessboard are known and thus it is simple to match the corners detected by both of the stereo images taken from the two cameras 311, 511, 711. The chessboard plane is considered to be at z=0, only leaving the problem to be solved only in this plane (with origin in one of the corners of the chessboard).

[0083] Each of the images is automatically processed in order to find the chessboard patterns, acquiring one conversion from the image corner pixels to the real 3D positions of the chessboard. This enables the computation of the intrinsic lens parameters for each of the cameras 311, 511, 711 (i.e. distortion coefficients), by trying to minimize the 2D<->3D re-projection error in all images. After carrying out this calculation, it is possible to use these 2D-3D correspondences to calculate the transformation matrix between the images from the two cameras 311, 511, 711.

[0084] This calibration enables the computation of the matrix which projects the images of both cameras 311, 511, 711 onto a common image plane, i.e., to rectify the images. This process makes it easier to find correspondences between the stereo images because the process aligns the image in such a way that theoretically it is only necessary to search along a single line (if the calibration is accurate, correspondent points are in the same row of the rectified images). After constructing the undistorted and coplanar image planes, the 3D reconstruction can be achieved by triangulation.

[0085] FIG. 8 now illustrate the bio-sensors. FIG. 8A shows a top view of bio-sensor strip 821 comprising bio-sensors 819 and power and data leads 820, which in turn connect to a power and data connector 823, itself connected to a transmitting device 822 that can be connected, preferably wirelessly, to a computer or handheld mobile smart device (not shown).

[0086] FIG. 8B shows a section view of the bio-sensor strip of FIG. 8a, comprising two strips of plastic film 824 and 825, which sandwich the bio-sensors 819. In one aspect, the bio-sensor 819 are pressure sensors, but as noted elsewhere, other types of sensors can be used. The bio-sensor comprises a pressure-sensitive resistor which resistance changes depending on the load or pressure applied to the bio-sensor 819. Measurement of the change of resistance is carried out using a bridge circuit.

[0087] The power leads and data leads 820 made, for example, of silver ink are not illustrated in this FIG. 8B. Side A1 of plastic film 824 will be in contact with the stump skin, i.e. the skin on the residual limb, or the liner covering the residual limb, and will preferably have a bio-compatible finish. A non-limiting example of the bio-compatible finish is polyurethane. Side A2 of the plastic film 824 faces the bio-sensors 819 and holds the bio-sensors 819 in place. Preferably, the side A2 has an adhesive finish from, for example, a medical grade acrylic adhesive, so that the bio-sensors 819 do not move.

[0088] Side B1 of polymer film 825 on FIG. 8B faces the bio-sensors 819 and holds the bio-sensors 819 in place. The side B1 may have an adhesive finish, e.g. from a medical grade acrylic adhesive, so that bio-sensors 819 do not move. The side B2 of the plastic film 825 is, for example, silicone and will be in contact with the prosthetic socket surface of the prosthetic surface 216, 316 (not shown in FIG. 8B). The side B2 will preferably have an adhesive finish, so that the bio-sensor strip 819 is firmly held in place on the socket surface.

[0089] The side A1 of the plastic film 824 will have one or more markings 830 on the surface. These markings are illuminated by the light source 220, 320 to locate the bio-sensors on the inside of the socket 216, 316 as explained previously. In one non-limiting aspect, the markings are multi-coloured roundels (concentric circles). The different colours are used to indicate differing positions or the different types of bio-sensors within the socket 216, 316. Currently, at least two markings per strip are required to uniquely identify the position of the bio-sensor strip 819, but a single marking could be acceptable.

[0090] FIG. 8C shows a section view of a simpler embodiment of the bio-sensor strip 819, comprising a single plastic film 825, to which the bio-sensors 819 with an adhesive finish 826 are applied to the B1 side of the plastic film 825. This side B1 faces the stump skin and has a bio-compatible finish, while side B2 of plastic film 825 will be in contact with the prosthetic socket surface of the prosthetic socket 216, 316 and will preferably have an adhesive finish, so that the bio-sensor strip 819 is firmly held in place on the socket surface. It will be appreciated that the sides B1 and B2 could be reversed so that B2 is in contact with the stump skin.

[0091] Before use, the bio-sensor strips 819 are covered on the adhesive side with a peelable cover if dispensed in pre-cut strips, or not covered if they are dispensed in rolls. In use, the peelable covers are removed, the bio-sensor strips 820 are cut to the right size, if needed, and the bio-sensor strips 820 are applied to the internal surface of the prosthetic socket 216, 316, in the orientation best determined by the prosthetic fitting technician's experience.

[0092] The biosensor strips 819 can measure different types of biodata, which include but are not limit to pressure between the stump skin and the prosthetic socket surface, or temperature. In one non-limiting example, the biosensor strips are formed of a force sensing resistor comprising at least one polymer layer whose resistance varies on application of a pressure. The change in the resistance can be measured, for example, by a bridge circuit. In one aspect of the bio-sensor a plurality of polymer layers is used with different characteristics to allow a wide range of different pressures to be measured.

[0093] FIG. 9 shows the elements of the complete system of the present invention. The bio-sensor strip 921 comprising the bio-sensors 919 acquires the bio-data which is transmitted to a computer, a hand-held device or similar data processing device 927. The bio-sensor strips 921 are applied to an internal surface of the prosthetic socket 916, resulting in a sensorized socket 928, which will receive a residual stump 932 of the residual limb. The bio-sensor strips 921 are positioned in relation to the real origin coordinate 912, which is known or which can be automatically determined by the light source 220, 320 scanning system as shown and described with respect to FIGS. 2 and 3. The data from each of the bio-sensors 919 can be overlaid on the socket surface map 906 generated by the light source scanning system by using a computing device 927, resulting in a 3D bio-data profile 930. The 3D bio-data profile 930 can be oriented by relating the virtual origin coordinate 905 with the real origin coordinate 912, allowing the accurate representation of the bio-data profile 929 with the socket surface map 906.

[0094] Furthermore, the spacing between the bio-sensor strips 921 can be adjusted to vary the resolution of the data obtainedthe bio-sensor strips 921 can be arranged closer together or even on top of each other with an offset in areas where greater data resolution is required. When evaluating pressure, a correct fit between the stump 932 and the socket 916 will produce uniform pressure distribution across certain areas of the surface of the socket 916, depending on the socket type, while a poor fit will produce areas of altered pressure which will be evidenced by more concentrated curves in the bio-data profile 929, in zones where this should not occur.

[0095] It will be appreciated that absolute values from the bio-sensors are not required. The values can be normalised or otherwise mathematically manipulated with respect to the maximum value recorded.

[0096] Artificial colour may be added to the bio-data profile 929 to create a heat map and thus illustrate the areas of pressure. Shifts in the colour may be used to differentiate between areas of equally uncomfortable areas of increased or reduced pressure, such as red for higher pressure, and blue for lower pressure. The prosthesis technician can therefore identify high pressure areas of the socket 916 which need fine tuning and shaping back as well as areas of lower pressure which indicate regions of the socket 916 which need building up. Other types of bio-data of interest may be represented using the same method.

[0097] The arrangement of the bio-sensors 919 on the bio-sensor strips 921 enables the system to be wearable, non-invasive, autonomous (with long battery time), modular, flexible (with different placement of sensors), scalable (more sensors as needed), and versatile (different type of modules/sensors).

[0098] The images of the bio-sensor strips 921 are drawn on the surface of the stump 932 to indicate their location in relation to the stump 932. The real origin coordinates 912 are actually or virtually drawn on the scanned socket 916 and the scanning and data acquisition apparatus produces the 3D image of the scanned socket 906.

[0099] An example can serve to illustrate this in more detail. Suppose the prosthesis is an artificial leg or an artificial arm. In use, the patient with the artificial leg is made to walk (in the case of a leg) or move (in case of an artificial arm) for a certain amount of time until sufficient bio-sensor data has been obtained from the bio-sensors 919 to produce the virtual 3D bio-data profile 930 comprising bio-data profile curves 929 of the pressure, temperature or any other bio-data of interest. The position of these bio-data profile curves 929 is known by reference to virtual origin coordinates 905 of the 3D bio-data profile 930.

[0100] It is also possible to combine the bio-sensor data with data from one of more inertial motion units which is carried by the patient and attached to the limb. The inertial motion unit will have three, six or nine axes and provide information about the changes in the data as the patient moves. This data can be used to characterise potential gait anomalies.

[0101] FIGS. 10A-C shows the fitting of a prosthetic leg using the components of the present invention, using a virtual reality or augmented reality vision system.

[0102] FIG. 10A shows a prosthesis fitting technician using a virtual reality or augmented reality vision system 1031 and the residual member 1032 of a user. It will be noted that some residual members 1032 are covered with liners or socks, part of which is shown as 1033 on the figure.

[0103] FIG. 10B shows what the prosthesis fitting technician sees through the virtual reality or augmented reality vision system 1031, which superimposes the 3D surface map 1006 of the socket 916, which has been obtained by the scanning, imaging and surface determination system of the present invention, with the virtual origin coordinate point 1005 of the socket 916, themselves precisely aligned with the user's residual limb 1032, in the same limb location where the socket 916 was worn during the testing and data acquisition phase (walking, moving). This allows the prosthesis fitting technician to correctly identify the problematic areas of the socket 916.

[0104] Furthermore, a light grid may be projected onto the patient's stump or over the surface of the actual socket 916 which is calibrated to the 3D socket model so as to help the prosthetic fitting technician to correlate the 3D image with the actual socket surface of the socket 916 and hence help identify the areas that need adjustment on the actual socket surface.

[0105] FIG. 10C shows the same image as FIG. 10B, but now the virtual reality or augmented reality vision system 1031 of the present disclosure adds the layer of the bio-data profile 1030 and the bio-data profile maps 1029 to FIG. 10B. At the virtual origin coordinate point 1005 two virtual origin coordinate points are precisely superimposed: the origin point of the 3D socket surface map 1006 and the origin point of the bio-data profile 1030, both precisely aligned with the same limb location where the socket 916 was worn during testing. The alignment of all of these origin coordinate points is needed so that bio-data obtained actually matches the appropriate area of the physical socket 916 and the corresponding area of the residual limb 1032 location.

[0106] The purpose of the layering of the various data maps is to evidence the areas where pressure, temperature or other data are more intense, and which may indicate areas of pain or discomfort for the person wearing the prosthesis, or less intense which may indicate areas which lack support.

[0107] In FIG. 10D, the prosthetic fitting technician is now shown holding the socket 1016 and a shaping tool 1038. The prosthetic fitting technician will now begin to shape the socket 1016, using the virtual reality or augmented reality vision system 1031, which is able to correctly display all of the data obtained by the virtual reality vision system 1031 and to referencing the data to the physical socket 1016 by identifying the real origin coordinate point 1012 on the socket 1016.

[0108] FIG. 10E shows the visualization of the 3D socket surface map 1006 over the actual prosthetic socket 1016, using the virtual reality or augmented reality vision system 1031. This displays the information of the 3D socket surface map 1006 over the corresponding zones of the prosthetic socket 1016, which can be achieved by using the virtual origin coordinate 1005.

[0109] FIG. 10F shows the visualization of the 3D bio-data profile 1030 over the physical prosthetic socket 1016, using the virtual reality vision system 1031. The display of the information of the 3D bio-data profile 1030 is overlaid onto the corresponding zones of the prosthetic socket 1016, which can be achieved using the virtual origin coordinate 1005 of three data sets (bio-data profile curves, socket surface model and physical stump) and overlapping the bio-data profile curves 1029 over the surface map 1006, allowing the prosthetics fitting technician to correctly and quickly identify the areas that require adjustment.

[0110] FIG. 11 shows a similar process to that described in FIGS. 10A-F, but now the prosthetics fitting technician is using an augmented reality device, such as a portable smart tablet equipped with augmented reality software and appropriate rendering and layering software. This avoids using the cumbersome virtual reality vision system and allows the technician to observe all of the necessary information on the same plane.

[0111] In FIG. 11A shows the prosthetic fitting technician using a handheld device 1134, such as a tablet or any other mobile device, mounted on a flexible arm support 1135 fixed to a solid surface such as a wall or table (not shown) to examine the patient's residual limb 1132.

[0112] FIG. 11B shows the image as seen by the technician of the socket surface map 1106 superimposed over the amputee residual limb 1132, using the handheld device 1134. This handheld device 1134 displays the information of the socket surface map 1106 over the corresponding zones of the residual limb 1132. Both images are aligned by means of both the virtual socket origin coordinate 1105 and the real stump origin coordinate 1112.

[0113] FIG. 11C now adds to the handheld device's display 1134 the visualization of the 3D bio-data profile 1130 over the amputee residual limb 1132, using the handheld device 1134. This displays the information of the 3D bio-data profile 1130 over the corresponding zones of the residual limb 1132, which can be achieved using both the virtual socket origin coordinate 1105 and the real stump origin coordinate 1112, and overlapping the bio-data profile curves 1129 over the socket surface map 1106, allowing the prosthetics fitting technician to correctly and easily identify the problematic areas and to shape the problematic areas.

[0114] In FIG. 11D, the prosthetics fitting technician goes to work with a shaping tool 1138 on the socket 1116 which is observed by means of the handheld device 1134 mounted on a flexible arm support 1135 and which is oriented by means of the real origin coordinate 1112.

[0115] FIG. 11E shows the image as seen by the technician of the surface map 1106 over the prosthetic socket 1116, using the handheld device 1134. This handheld device 1134 displays the information of the surface map 1106 over the corresponding zones of the physical prosthetic socket 1116, which can be achieved using both the virtual origin coordinate 1105 and the real origin coordinate 1112.

[0116] FIG. 11F now adds to the handheld device's display 1134 visualization of the 3D bio-data profile 1130 over the prosthetic socket 1116, using a handheld device 1134. This displays the information of the 3D bio-data profile 1130 over the corresponding zones of the prosthetic socket 1116, which can be achieved using both the virtual origin coordinate 1105 and the real origin coordinate 1112 and overlapping the data profile curves 1129 over the surface map 1106, allowing the prosthetics fitting technician to correctly and quickly identify the problematic areas.

[0117] The alignment of the socket surface map 1106 and/or the 3D bio-data profile 1130 over the object of interest can be achieved by using the same origin coordinate in the real object (real origin coordinate 1112), in the socket surface map 1106 (virtual origin coordinate 1105) and in the 3D bio-data profile 1130 (virtual origin coordinate 1105), or by resorting to a best match algorithm, that computes the best geometrical match between two 3-dimensional objects.

[0118] The method will now be described with respect to the flow diagrams shown in FIGS. 12 and 13. The method starts at step 1200 and the object 216, 316, for example the prosthesis 932, is scanned in step 1205. A plurality of the biosensors 919 are attached to either the surface of the object 216, 316 or the body part, for example the stump 919, in step 1210. The object 216, 316 is subsequently engaged with the body part 919 in step 1215 and data is collected in step 1220 from the bio sensors 919. The collected data is processed in step 1222 and superimposed in step 1225 over the surface map 1006 of the object 216, 316 to identified areas of the object 216, 316 that need to be adjusted. The superimposed data on the surface map 1006 is displayed as a 3D model to the user, for example the prosthetics fitting technician, in step 1227 before adjustments are made in step 1230.

[0119] The method for creating the surface map is shown in FIG. 13 which starts with scanning the object 216, 316 in step 1300 using a laser 213, 313 projecting a laser line 201, 301 on the surface of the object 216, 316. An image of the object 216, 316 with the laser line 201, 301 is taken in step 1310 and the image data analyses in step 1315. If all of the object 216, 316 has been scanned in step 1320, then the method is completed in step 1330 and the surface map of the object 216, 316 is created. Alternatively, the distance between the laser and the object 216, 316 is moved to scan a different part of the object 216, 316 in step 1325.

[0120] The present invention comprising the use of the bio-sensors, the mapping of the internal surface of the prosthetic socket, the generation of bio-data related to socket/stump fit and the identification of socket areas which require adjustment represents several economic and comfort benefits. The method is non-invasive, unobtrusive and does not require a clinician's attendance. It saves considerable time in the fitting process, thereby reducing cost and increasing the patient's quality of life.

REFERENCE NUMERALS

[0121] 101 Laser Line

[0122] 102 Centre

[0123] 103 Data Points

[0124] 104 Laser line

[0125] 105 Origin coordinates

[0126] 106 Surface map

[0127] 200 Conical laser assembly

[0128] 201 Projected radiation line

[0129] 207 Motor

[0130] 208 Linear screw

[0131] 209 Bushing

[0132] 210 Support frame

[0133] 211 Camera

[0134] 212 Physical origin coordinate

[0135] 213 Laser

[0136] 214 Laser lens

[0137] 215 Conical mirror

[0138] 216 Socket

[0139] 217 Fixing base

[0140] 219 Bio-Sensors

[0141] 220 Radiation source

[0142] 236 Radiation beam

[0143] 239 Anchor frame

[0144] 240 Support assembly

[0145] 241 Wall or housing

[0146] 300 Conical laser assembly

[0147] 301 Projected radiation pattern

[0148] 307 Motor

[0149] 308 Linear screw

[0150] 309 Bushing

[0151] 310 Device supporting frame

[0152] 311 Camera

[0153] 312 Original coordinate

[0154] 313 Laser

[0155] 314 Laser lens

[0156] 315 Optical element

[0157] 316 Scanned socket

[0158] 317 Fixing base

[0159] 319 Bio-sensors

[0160] 320 Radiation source

[0161] 339 Anchor frame

[0162] 340 Support Assembly

[0163] 341 Wall or housing

[0164] 401 Laser Plane

[0165] 411 Camera

[0166] 413 Laser

[0167] 415 Conic mirror

[0168] 418 Field of view

[0169] 501 Laser plane

[0170] 511 Camera

[0171] 513 Laser

[0172] 515 Optical Element

[0173] 518 Field of view

[0174] 601 Laser plane

[0175] 611 Camera

[0176] 613 Laser

[0177] 618 Field of view

[0178] 701 Laser plane

[0179] 711 Camera

[0180] 713 Laser

[0181] 719 Field of view

[0182] 819 Bio-sensors

[0183] 820 Data Leads

[0184] 821 Bio-sensor strip

[0185] 822 Transmitting device

[0186] 823 Power and data connector

[0187] 824 Plastic film

[0188] 825 Plastic film

[0189] 826 Adhesive finish

[0190] 830 Markings

[0191] 905 Virtual original coordinates

[0192] 906 Socket surface map

[0193] 912 Real origin coordinate

[0194] 916 Prosthetic socket

[0195] 919 Bio-sensors

[0196] 921 Bio-sensor strip

[0197] 927 Data processing device

[0198] 928 Sensorised socket

[0199] 929 Curves

[0200] 930 Bio data profile

[0201] 932 Stump

[0202] 1005 Virtual origin coordinate point

[0203] 1006 3D surface map

[0204] 1016 Socket

[0205] 1029 Bio-data profile maps

[0206] 1030 Bio-data profile

[0207] 1031 Virtual reality vision system

[0208] 1032 Residual member

[0209] 1033 Liner or sock

[0210] 1038 Shaping tools

[0211] 1105 Virtual origin coordinate point

[0212] 1106 Socket surface map

[0213] 1112 Real sump origin coordinate

[0214] 1129 Bio-data profile curves

[0215] 1130 Bio-data profile

[0216] 1132 Residual limb

[0217] 1134 Handheld device

[0218] 1135 Support

[0219] 1138 Shaping tool