Indoor visible light positioning method and system based on single LED lamp

Abstract

An indoor visible light positioning method and system based on a single LED lamp. The system includes an LED communication module and a smartphone module. The LED communication module sends its coordinates and shape data to the smartphone module. The smartphone module includes an inertial measurement unit (IMU) and a camera. The IMU is configured to obtain movement data of a smartphone. The camera is configured to shoot video streams of the LED lamp. Computing processing is performed on center point coordinates of the LED lamp and IMU data in the video streams, and constraints are provided by using a homography matrix of ellipses in the video streams, to obtain accurate location information of the smartphone, and provide location-based services such as navigation and query for a user.

Claims

1. An indoor visible light positioning method based on a single LED lamp, comprising: step 1: using the visible light communication technology to encode and modulate to-be-sent information, and transmitting the information through an LED lamp; step 2: shooting video streams of the LED lamp by using a camera of a smartphone, and recording data measured by an inertial measurement unit (IMU) of the smartphone; and measuring, by the IMU, acceleration (a.sub.x, a.sub.y, a.sub.z) and rotation angle information (α,β,γ) of the smartphone in all directions to calculate a movement direction and displacement (t.sub.x, t.sub.y, t.sub.z) of the smartphone; step 3: capturing LED lamp images from each frame of the video streams, decoding and demodulating the data transmitted by the LED lamp, extracting an LED lamp shape from the images, and collecting a location (X.sub.o, Y.sub.o, Z.sub.o) and a radius R.sub.o of the LED lamp in a world coordinate system; step 4: dividing the video streams into single image frames by timeslot, and for every two adjacent images I.sub.i and I.sub.i+1, calculating a homography matrix:
H.sub.i,i+1=K[R.sub.i,i+1+t.sub.i,i+1n.sub.i.sup.T/z.sub.i]K.sup.−1 wherein H.sub.i,i+1 is the homography matrix, K is an intrinsic matrix of the smartphone camera, n.sub.i is a normal vector of a lamp plane in an i.sup.th camera coordinate system, z.sub.i is a distance from a center of the camera to the lamp plane when the i.sup.th image is taken, R.sub.i,i+1 is a rotation matrix of the camera, and t.sub.i,i+1 is a displacement parameter of the camera; a general elliptic equation is:
ax.sup.2+bxy+cy.sup.2+dx+ey+f=0 for multi-view geometry, exporting the homography matrix H.sub.i,i+1 between I.sub.i and I.sub.i+1 through elliptical shapes projected onto the images, wherein an ellipse on each image is represented by a conic coefficient matrix C.sub.i: $C_i=\left[\begin{array}{ccc}{a}_i& 0& d_i/2\\ 0& c_i& e_i/2\\ d_i/2& e_i/2& f_i\end{array}\right]$ wherein [a.sub.i, b.sub.i, C.sub.i, d.sub.i, e.sub.i, f.sub.i] represent coefficients of a general equation of an ellipse projected on the i.sup.th image; under the homography matrix H.sub.i,i+1, converting C.sub.i into C.sub.i+1=H.sub.i,i+1.sup.−TC.sub.iH.sub.i,i+1.sup.−1 to establish an optimization model: $\begin{array}{c}\min \\ z_1,z_2,\mathrm{.Math.}{z}_n\end{array}{.Math.C_{i+1}-H_{i,i+1}^{-T}{C}_i{H}_{i,i+1}^{-1}.Math.}_2$ wherein z.sub.1, z.sub.2, . . . z.sub.n represents a distance from the center of the camera to the lamp plane when image 1, 2, . . . , n is taken, and n represents a total quantity of video frames; and calibrating the rotation matrix R.sub.i,i+1 through the optimization model; step 5: calculating intrinsic and extrinsic matrices of the camera based on the calibrated camera parameters, wherein the intrinsic matrix M.sub.1 is: $M_1=\left[\begin{array}{cccc}f/d_X& 0& {\mu }_0& 0\\ 0& f/d_Y& {\upsilon }_0& 0\\ 0& 0& 1& 0\end{array}\right]$ wherein f is a focal length of the camera, d.sub.X and d.sub.Y represent physical dimensions of each pixel in an image in the X axis and Y axis, and (u.sub.0, v.sub.0) represents an origin of an image coordinate system; and calculating the extrinsic matrix M.sub.2 based on the movement direction and displacement (t.sub.x, t.sub.y, t.sub.z) of the smartphone: $M_2=\left[\begin{array}{cc}R& t\\ 0^T& 1\end{array}\right]$ in the extrinsic matrix, 0.sup.T=(0,0,0).sup.T, R is the rotation matrix of the smartphone, and t is a three-dimensional translation variable that contains displacement data of the smartphone in the movement; step 6: obtaining multiple images of the LED lamp by shooting video streams, and obtaining a distance Z.sub.c from the LED lamp to the center of the camera based on triangle similarity theorems:
Z.sub.c=B×f/d wherein d is a length between the center points of the LED lamp in two images; B is an actual interval between the two images taken by the camera; B is calculated based on acceleration sensor data in the IMU; or the radius R.sub.o of the LED lamp is compared with a minor axis of an ellipse obtained from the image, to obtain a ratio parameter s, and then d is used to calculate B according to the same ratio parameter s; step 7: obtaining a projection matrix of the camera based on the intrinsic and extrinsic matrices of the camera, and for any image point P.sub.w(X.sub.w, Y.sub.w, Z.sub.w) in space, obtaining a location (μ, v) of an image point p.sub.i based on the projection matrix and the distance Z.sub.c from the LED lamp to the center of the camera: $Z_c\left[\begin{array}{c}\mu \\ v\\ 1\end{array}\right]=M_1{M}_2\left[\begin{array}{c}{X}_W\\ Y_W\\ Z_W\\ 1\end{array}\right]$ wherein Z.sub.c is the distance from the LED lamp to the center of the camera; step 6: when the location (μ, v) of the image point p.sub.i is known, using the intrinsic matrix M.sub.1 of the camera to convert the image point p.sub.i in the image coordinate system into an image point P.sub.c in a camera coordinate system, using the extrinsic matrix M.sub.2 to convert the image point P.sub.c into an image point P′.sub.w in the world coordinate system, and then adding specific coordinates (X.sub.o, Y.sub.o, Z.sub.o) of the LED lamp in the world coordinate system to P′.sub.w to obtain coordinates P.sub.w of the smartphone; and step 7: according to the coordinates P.sub.w of the smartphone, transmitting service data information related to the coordinates P.sub.w of the smartphone through the LED lamp; shooting video streams of the LED lamp by using the camera of the smartphone, and decoding and demodulating the data transmitted by the LED lamp to obtain the related service data information.

2. The indoor visible light positioning method based on a single LED lamp according to claim 1, wherein coding and modulation is performed for the LED lamp by using run length limited (RLL) coding and Raptor coding.

3. The indoor visible light positioning method based on a single LED lamp according to claim 2, wherein a video taken by the smartphone is 30 or 60 frames per second.

4. A system using the indoor visible light positioning method based on a single LED lamp according to claim 1, comprising: an LED communication module, a power module, and a smartphone module, wherein the power module supplies power to the LED communication module; the LED communication module comprises a coding and modulation module, a control module, and an LED lamp, wherein the control module controls the LED lamp according to the coding and modulation module; and the smartphone module comprises a camera, a decoding and demodulation module, an IMU module, a positioning module, and a location-based service module, wherein the camera, the decoding and demodulation module, and the positioning module are connected in turn, and the camera, the IMU module, and the location-based service module are all connected to the positioning module.

5. The system according to claim 4, wherein the location-based service module comprises a positioning and navigation module, an information service module, a tracking and identification module, and a security check module.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is a system framework design diagram according to an embodiment of the present invention.

(2) FIG. 2 is a system deployment diagram according to an embodiment of the present invention.

(3) FIG. 3 is a schematic diagram of visible light positioning according to an embodiment of the present invention.

(4) FIG. 4 is a schematic diagram of a location-based service according to an embodiment of the present invention.

(5) FIG. 5 is a schematic diagram of a single-LED scenario according to an embodiment of the present invention.

(6) In the figure, 1. circular LED lamp, 2. smartphone, 3. room with only one LED lamp, 4. encoding for the LED lamp, and 5. user with a smartphone in hand.

DETAILED DESCRIPTION

(7) The following further describes embodiments of the present invention with reference to the drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention but not intended to limit the scope of the present invention. Those skilled in the art should understand that any equivalent modifications to the present invention shall fall within the scope defined by the claims.

(8) As shown in FIG. 1 to FIG. 5, an indoor visible light positioning method based on a single LED lamp is used with visible light communication and computer vision technologies, to position smartphone users indoor when there is only a single LED lamp. Coding and modulation are performed for the LED lamp on the ceiling by using RLL coding and Raptor coding. The blink frequency of the LED lamp includes information such as a specific location (X.sub.o, Y.sub.o, Z.sub.o), a radius R.sub.o, and an ID of the lamp in a world coordinate system. The blink effect of the LED lamp is invisible to the naked eye, so it will not affect people's life. A camera uses a rolling shutter to capture images. When a CMOS imager uses a short exposure time to capture fast-modulated LED, a band effect occurs. Different LED frequencies or duty cycles change the widths of bright and dark bands in an image. After the camera shoots the LED image, it can perform decoding and demodulation according to the corresponding band effect for identification.

(9) In embodiments of the present invention, after coding and modulation are performed for the circular LED lamp on the ceiling, an IMU of a smartphone is used to collect movement data of the smartphone, and meanwhile, a camera is used to shoot videos of the LED lamp at the same time.

(10) The IMU measures acceleration (a.sub.x, a.sub.y, a.sub.z) and rotation angle information (α,β,γ) of the smartphone in all directions to calculate a movement direction and displacement (t.sub.x, t.sub.y, t.sub.z) of the smartphone. During actual operations, the smartphone must capture video streams and collect the IMU data at the same time, and the time and quantity units of the extracted data and video frames must be unified.

(11) As a circular LED lamp is used in various embodiments of the present invention, a changed elliptic shape is projected onto a video frame image, together with some feature point information in the ellipse. In such cases, it is necessary to obtain a focal length of the smartphone camera and a physical dimension of each pixel, and collect and preprocess all IMU data, so as to obtain values of a pitch angle α, a roll angle β, and a yaw angle γ, and relative location change data of the smartphone during the movement of the smartphone. Ideally, intrinsic and extrinsic matrices of the smartphone camera can be obtained based on such data. Generally, the intrinsic matrix M.sub.1 of a same smartphone does not change, while the extrinsic matrix M.sub.2 changes as the smartphone moves or rotates.

(12) For computer vision, basic knowledge of camera perspective projection models can be obtained from relevant literature, for example, a conversion relationship between an image coordinate system, a camera coordinate system, and a world coordinate system, and intrinsic and extrinsic parameters of the camera described in the pinhole imaging model (linear camera model). The intrinsic matrix M.sub.1 of the camera is responsible for converting the camera coordinate system to a pixel coordinate system:

(13) $M_1=\left[\begin{array}{ccc}f/d_X& 0& {\mu }_0\\ 0& f/d_Y& v_0\\ 0& 0& 1\end{array}\right]$

(14) f represents a focal length of the camera, d.sub.X and d.sub.Y represent physical dimensions of each pixel in an image in the X axis and Y axis, and (u.sub.0, v.sub.0) represents an origin of an image coordinate system, and defines an intersection of the camera's optical axis and the image plane, usually at the center of the image. The extrinsic matrix is generated by the camera during movement and rotation and keeps changing. The extrinsic matrix is generally expressed as follows:

(15) $M_2=\left[\begin{array}{cc}R& t\\ 0^T& 1\end{array}\right]$

(16) In the extrinsic matrix, 0=(0,0,0).sup.T; R is a rotation matrix of the smartphone, a matrix of orthogonal units with a size of 3×3; and t is a three-dimensional translation variable that contains displacement data of the smartphone during moving in three directions. A projection matrix of the camera can be obtained by obtaining the intrinsic and extrinsic matrices of the camera. Then, for any point P.sub.w in space, a location (μ, v) of an image point p.sub.i can be obtained through projection matrix conversion

(17) 0$Z_c\left[\begin{array}{c}\mu \\ v\\ 1\end{array}\right]=M1M2\left[\begin{array}{c}{X}_W\\ Y_W\\ Z_W\\ 1\end{array}\right]$

(18) Similarly, when the image point p.sub.i (μ, v) is known, a point p.sub.i in the image coordinate system can be converted to P.sub.c in the camera coordinate system by using the intrinsic matrix M.sub.1, P.sub.c is converted to P′.sub.w in the world coordinate system by using the extrinsic matrix M.sub.2, and then specific coordinates (X.sub.o, Y.sub.o, Z.sub.o) of the LED lamp in the world coordinate system are added to P′.sub.w, to obtain coordinates P.sub.w of the smartphone. During the conversion, a distance Z.sub.c from the LED lamp to the center of the camera needs to be obtained.

(19) In embodiments of the present invention, multiple images of the LED lamp are obtained by shooting video streams, to simulate a binocular visual ranging method, and obtain Z.sub.c based on triangle similarity theorems.
Z.sub.c=B×f/d

(20) d is a parallax between two images, that is, a length between the center points of the LED lamp in two images, and B is an actual interval between the two images taken by the camera. B can be calculated based on acceleration sensor data in the IMU. Alternatively, the radius R.sub.o of the LED lamp is compared with a minor axis of an ellipse obtained from the image, to obtain a ratio parameter s, and then d is used to calculate B according to the same ratio parameter s.

(21) Then, Z.sub.c is used for conversion between the pixel coordinate system and the world coordinate system to obtain the location of the smartphone in the world coordinate system. During the data measurement, however, a measured value of a yaw angle in a rotation angle is quite different from a true value because a magnet sensor is vulnerable to interference from the ambient environment. This reduces the positioning accuracy. Therefore, the following method is used to constrain the location of the smartphone.

(22) The video streams of the LED lamp shot by the camera are divided into single image frames by timeslot. For every two adjacent images I.sub.i and I.sub.i+1, a homography matrix can be calculated based on the knowledge of computer vision:
H.sub.i,i+1=K[R.sub.i,i+1+t.sub.i,i+1n.sub.i.sup.T/z.sub.i]K.sup.−1

(23) n.sub.i is a normal vector of a lamp plane in an i.sup.th camera coordinate system, z.sub.i is a distance from a center of the camera to the lamp plane when the i.sup.th image is taken. When image frames are extracted by timeslot, movement data of the smartphone is processed to obtain the intrinsic matrix M.sub.1, the extrinsic matrix M.sub.2, and the movement distance (t.sub.x, t.sub.y, t.sub.z) of the smartphone between every two image frames.

(24) For multi-view geometry, the homography matrix H.sub.i,i+1 between I.sub.i and I.sub.i+1 can be exported through elliptical shapes projected onto the images. An ellipse on each image can be represented by a conic coefficient matrix:

(25) $C_i=\left[\begin{array}{ccc}{a}_i& 0& d_i/2\\ 0& c_i& e_i/2\\ d_i/2& e_i/2& f_i\end{array}\right]$

(26) Under the homography matrix, H.sub.i,i+1, C.sub.i is converted into C.sub.i+1=H.sub.i,i+1.sup.−TC.sub.iH.sub.i,i+1.sup.−1. In this way, a precise constraint is defined for calibrating R.sub.i,i+1 and t.sub.i,i+1. Due to errors in the three rotation angles and each displacement change, accurate solutions may not be produced. In this case, the system uses the following method to solve the problem.

(27) $\begin{array}{c}\min \\ z_1,z_2,\mathrm{.Math.}{z}_n\end{array}{.Math.C_{i+1}-H_{i,i+1}^{-T}{C}_i{H}_{i,i+1}^{-1}.Math.}_2$

(28) Solutions to various data are obtained by using optimized methods, to calibrate R.sub.i,i+1 and t.sub.i,i+1, constrain a movement status of the smartphone, correct the IMU measurement data of the smartphone, reduce the error of the rotation angles in all directions, and improve the positioning accuracy of the LED lamp. In this way, the problem that the magnet sensor is vulnerable to interference is resolved, and the location of the smartphone is obtained.

(29) After the positioning is realized, at least one embodiment of the present invention uses a highly reliable modulation and coding technology to build a visible light communication network, and provides location-based information services to smartphones through the communication network. User authentication and authorization can be performed first. During provision of the location-based service, information that interests users most is filtered and sent to the users, and users are allowed to store personalized information. Embodiments of the present invention can provide users with various services such as positioning, navigation, query, identification, and event inspection in places like shopping malls, scenery spots, and underground garages.

(30) As shown in FIG. 1, an indoor visible light positioning system based on a single LED lamp includes an LED communication module, a power module, and a smartphone module. The power module supplies power to the LED communication module. The LED communication module includes a coding and modulation module, a control module, and an LED lamp, where the control module controls the LED lamp according to the coding and modulation module. The smartphone module includes a camera, a decoding and demodulation module, an IMU module, a positioning module, and a location-based service module, where the camera, decoding and demodulation module, and the positioning module are connected in turn; the camera, the IMU module, and the location-based service module are all connected to the positioning module; and the location-based service module includes a positioning navigation module, an information service module, a tracking and identification module, and a security check module. A series of materials obtained by the camera and IMU in the smartphone can provide accurate positioning for the system, which then provides location-based services. In the LED communication module, a LED lamp is set to circular. Through coding and modulation, visible light blinks at a specific frequency, to transmit information of the LED lamp. The smartphone module collects IMU data and uses the camera to shoot video streams of the LED lamp. The video streams and various data are extracted to accurately locate the smartphone and provide location-based services to the smartphone user.

(31) The smartphone module includes the camera and the IMU. The camera with a focal length of f is configured to shoot video streams of the LED lamp, and meanwhile, the IMU collects movement status data of the smartphone, such as rotation angles (α, β, γ) in all directions and acceleration (a.sub.x, a.sub.y, a.sub.z) in all directions, to obtain an intrinsic matrix M.sub.1 and an extrinsic matrix M.sub.2 of the smartphone. LED lamp images after short exposure are obtained by the camera, and data such as the specific frequency of the LED light, center point coordinates, and a radius of the LED lamp can be identified from ellipse images of the LED lamp.

(32) As shown in FIG. 2, a circular LED lamp with a diameter of 21 cm is disposed on the ceiling 3 or 4 meters from the ground. The rated voltage is 12 V. The LED lamp consists of nine small LED lamps, each being 0.2 W. The coding schemes (4) used for LED lamp communication is RLL coding and Raptor coding. The smartphone is Huawei TRT-AL00A, with a camera focal length of 35 mm. A user holds the smartphone at a height between 110 cm and 180 cm, to shoot video streams of the LED lamp. In actual deployment, various LED lamp and smartphone models can be used.

(33) Firstly, the LED lamp is deployed on the ceiling, and a location of the LED lamp, such as (0, 0, 0) is set. In more complex indoor scenarios, it is necessary to assign specific accurate coordinates to each LED lamp, and then perform coding and modulation to facilitate the management by a system positioning module. In addition, a focal length of the smartphone camera, and a ratio between a pixel distance and an actual distance for photography need to be extracted as basic inputs. Then, video streams of the LED lamp are shot, and IMU measurement data of the smartphone is recorded.

(34) As shown in FIG. 3, video streams are extracted frame by frame at fixed time intervals. In this design, a video captured by the smartphone is 30 frames per second. If the video is 30 or 60 frames per second, one image frame is extracted every one or two seconds. The IMU data at each time interval is compared with the frames extracted from the video streams. After the matching is completed, a specific rotation matrix R and a relative displacement vector t are generated between every two frames. For each image frame, a center point of an ellipse is extracted, and all parameters required for a general equation of the ellipse in each image frame is obtained, to construct a homography matrix of each ellipse. Movement data collected by the IMU of the smartphone is corrected by using the homography matrix, especially the rotation angles of the smartphone. Such a design can reduce IMU data drift errors.

(35) As shown in FIG. 4, location-based services mainly include positioning, navigation, information services, tracking and identification, and security check. Functions can also be added according to user needs. LED lamps are widely used in various places, such as shopping malls, scenery spots, and underground garages. Therefore, this system can be applied in various scenarios, without requiring extra devices. Single-LED scenarios shown in FIG. 5 are common in real life, and the system can better provide location-based services for smartphone users.

(36) The above descriptions are merely preferred implementations of the present invention. It should be noted that a person of ordinary skill in the art may further make several improvements and modifications without departing from the principle of the present invention, but such improvements and modifications shall also be deemed as falling within the protection scope of the present invention.