System for virtual display and method of use

Abstract

A preferred system and method for projecting a business information model at a construction site includes a network, a system administrator connected to the network, a database connected to the system administrator, a set of registration markers positioned in the construction site, and a set of user devices connected to the network. The system includes a hard hat, a set of headsets mounted to the hard hat, a set of display units movably connected to the set of headsets, a set of cameras connected to the set of headsets, and a wearable computer connected to the set of headsets and to the network. The cameras capture an image of the set of registration markers. A position of the user device is determined from the image and an orientation is determined from motion sensors. A BIM is downloaded and projected to a removable visor based on the position and orientation.

Claims

1. A visor for a construction hard hat capable of rendering semi-transparent information comprising: a pocket rigidly attached to the hard hat; a headset removably attached to the pocket; and, a display unit, operatively connected to the headset, for displaying the semi-transparent information on an inside surface of the visor; the display unit further comprising: a set of registration cameras positioned on the hard hat; and, a computer operatively connected to the set of registration cameras and the display unit and programmed to: receive position information from the set of registration cameras; calculate a building information model (BIM) display view based in the position information; send the BIM display view to the display unit as the semi-transparent information; wherein the display unit further comprises: a verbal command translator, operatively connected to the computer; wherein the computer is further programmed to: receive a verbal command from the verbal translator; and, change the BIM display view based on the verbal command.

2. The visor of claim 1, wherein the display unit further comprises: a projector; a light guide attached to the projector; and, a view camera operatively positioned adjacent the projector and the light guide.

3. The visor of claim 2, wherein the projector further comprises: a light source; and, a lens adjacent to the light source.

4. The visor of claim 3, wherein the light guide further comprises: a set of interior surfaces; an input surface attached to the set of interior surfaces and adjacent to the projector; and, an output surface attached to the set of interior surfaces opposite the input surface.

5. The visor of claim 4, wherein the input surface is positioned at a first angle to provide total internal reflection in the light guide.

6. The visor of claim 4, wherein the output surface is positioned at a second angle to project a rendering out of the light guide.

7. The visor of claim 1, wherein the set of registration cameras are positioned on the hard hat to view a 360° arc about a vertical axis of the hard hat.

8. The visor of claim 1, wherein the semi-transparent information is one or more of the group of construction plans, queuing information, tool locations, photographs, text, equipment manuals and parts lists.

9. The visor of claim 1, wherein the semi-transparent image is displayed by the display unit as a stereoscopic image.

10. A system for projecting a business information model at a construction site comprising: a set of registration markers positioned at the construction site; and, a user device comprising: a set of headset computation modules; a set of display units operatively connected to the set of headset computation modules; a set of registration cameras operatively connected to the set of headset computation modules and directed towards the set of registration markers; a computer connected to the set of headset computation modules and programmed to carry out the steps of: receiving the business information model; receiving a position image of at least one registration marker of the set of registration markers from the set of registration cameras; determining a position and an orientation of the user device based on the position image; rendering the business information model, based on the position and orientation, as a rendered business information model; displaying the business information model as a stereoscopic image; wherein the step of rendering further comprises the steps of: rotating a business information model segment based on the position and the orientation; determining a clipped business information model segment from the position and the business information model segment; and, rendering the clipped business information model segment as a stereoscopic image.

11. The system of claim 10, wherein each registration marker of the set of registration markers further comprises: a unique position code; and, wherein the computer is further programmed to: associate the unique position code with a unique position of each registration marker of the set of registration markers.

12. The system of claim 10, further comprising: a movement sensor operatively connected to at least one headset computation module of the set of headset computation modules; and, wherein the computer is further programmed to: receive headset movement data from the movement sensor; and, determine the position and the orientation of the user device from the headset movement data.

13. The system of claim 12, wherein the computer is further programmed to: dynamically calculate a transition position of the user device from the headset movement data when the position image is unavailable.

14. The system of claim 10, wherein the system further comprises: a remote sensor; a network operatively connected to the computer; and, wherein the computer is further programmed to carry out the steps of: capturing a virtual image of a virtual object and an actual image of a physical object; and, sending the captured image and the actual image to the network.

15. The system of claim 10, wherein the computer is further programmed to carry out the steps of: capturing a first image of the rendered business information model; capturing a second image of a physical object; and, superimposing the first image and the second image.

16. In a system comprising a computer, a headset connected to the computer, a display unit connected to the headset and a set of registration cameras connected to the headset, a method comprising the steps of: receiving a business information model segment; receiving a registration image of a registration marker; determining a position of the headset based on the registration image; determining an orientation of the headset; rendering the business information model segment based on the position and the orientation as a rendered business information model segment; displaying the rendered business information model segment on the display unit; wherein the step of determining a position further comprises the steps of: locating a set of edges in the registration image; deskewing the registration image using the set of edges to determine a set of position angles to generate a deskewed image; receiving a set of dimensions; determining a distance from the set of dimensions and the deskewed image; determining the position from the set of position angles and the distance; wherein the display unit further comprises a temporal camera, and wherein the step of determining an orientation further comprises: retrieving a set of movement data from the temporal camera; and, determining a set of rotational movements from the set of movement data.

17. The method of claim 16, further comprising the steps of: reading a code from the deskewed image; locating a marker position of the registration marker; associating the marker position with the code; and, storing the marker position and the code.

18. The method of claim 16, wherein the step of deskewing further comprises the steps of: determining a set of reference lines for the set of edges; calculating a pair angle between each pair of the set of reference angles; calculating a skew angle for each pair of angles; and, rotating the registration image by the skew angle.

19. In a system comprising a computer, a headset connected to the computer, a display unit connected to the headset and a set of registration cameras connected to the headset, a method comprising the steps of: receiving a business information model segment; receiving a registration image of a registration marker; determining a position of the headset based on the registration image; determining an orientation of the headset; rendering the business information model segment based on the position and the orientation as a rendered business information model segment; displaying the rendered business information model segment on the display unit; wherein the step of rendering further comprises the steps of: rotating the business information model segment based on the position and the orientation; determining a clipped business information model segment from the position and the business information model segment; and, rendering the clipped business information model segment as a stereoscopic image.

20. In a system comprising a computer, a headset connected to the computer, a display unit connected to the headset and a set of registration cameras connected to the headset, a method comprising the steps of: receiving a business information model segment; receiving a registration image of a registration marker; determining a position of the headset based on the registration image; determining an orientation of the headset; rendering the business information model segment based on the position and the orientation as a rendered business information model segment; displaying the rendered business information model segment on the display unit; retrieving a set of data; receiving a selection of a subset of data from the set of data; receiving a position and a size for the subset of data; displaying the subset of data at the position and the size; wherein the step of rendering further comprises the steps of: rotating the business information model segment based on the position and the orientation; determining a clipped business information model segment from the position and the business information model segment; and, rendering the clipped business information model segment as a stereoscopic image.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) In the detailed description presented below, reference is made to the accompanying drawings.

(2) FIG. 1 is a schematic of a BIM distribution system of a preferred embodiment.

(3) FIG. 2 is a schematic diagram of a user device of a preferred embodiment.

(4) FIG. 3A is a side view of a user device of a preferred embodiment.

(5) FIG. 3B is a side view of a camera matrix of a preferred embodiment.

(6) FIG. 4A is a front view of a user device of a preferred embodiment.

(7) FIG. 4B is an isometric view of a user device of a preferred embodiment.

(8) FIG. 4C is an isometric view of a display unit of a preferred embodiment.

(9) FIG. 4D is an exploded isometric view of a connection for a display unit of a preferred embodiment.

(10) FIG. 5 is a top view of a camera matrix of a preferred embodiment.

(11) FIG. 6 is a top view schematic of a projector unit and ray diagram of a preferred embodiment.

(12) FIG. 7A is a top view schematic of a projector unit and a virtual image of a preferred embodiment.

(13) FIG. 7B is a point of view of an augmented image of a preferred embodiment.

(14) FIG. 8 is a plan view of a user device and a registration system of a preferred embodiment.

(15) FIG. 9A is a schematic of a registration marker of a preferred embodiment.

(16) FIG. 9B is a schematic of a registration marker of a preferred embodiment.

(17) FIG. 10 is a schematic of a data flow for an augmented reality application of a preferred embodiment.

(18) FIG. 11 is a command input menu for a user device of a preferred embodiment.

(19) FIG. 12 is a flow chart of a state machine method of a preferred embodiment.

(20) FIG. 13 is a flow chart of a method for registering a marker of a preferred embodiment.

(21) FIG. 14A is a flow chart of a method for calibrating a position of a user device of a preferred embodiment.

(22) FIG. 14B is a top view schematic of a camera position with respect to a registration marker of a preferred embodiment.

(23) FIG. 14C is a side view schematic of a camera position with respect to a marker of a preferred embodiment.

(24) FIG. 14D is a captured skewed image of a registration marker of a preferred embodiment.

(25) FIG. 14E is a flow chart of a method for deskewing an image of a preferred embodiment.

(26) FIG. 14F is a deskewed image of a preferred embodiment.

(27) FIG. 14G is a side view schematic of a camera and a registration marker of a preferred embodiment.

(28) FIG. 15 is a flow chart of a runtime process of a preferred embodiment.

(29) FIG. 16 is a flow chart for method of determining a position of a user device of a preferred embodiment.

(30) FIG. 17 is a flow chart for a method of rendering a stereoscopic overlay for a user device of a preferred embodiment.

(31) FIG. 18 is a flow chart of a method for updating a business information model of a preferred embodiment.

(32) FIG. 19 is a flow chart of a method for updating a business information model of a preferred embodiment.

DETAILED DESCRIPTION

(33) It will be appreciated by those skilled in the art that aspects of the present disclosure may be illustrated and described in any of a number of patentable classes or contexts including any new and useful process or machine or any new and useful improvement.

(34) Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, C#, .NET, Objective C, Ruby, Python SQL, or other modern and commercially available programming languages.

(35) Referring to FIG. 1, system 100 includes network 101, system administrator 102 connected to network 101, and a set of user devices 104 each of which is connected to network 101. System administrator 102 is further connected to BIM database 103 for storage of relevant data. For example, receiving data may include a business information model, engineering change orders, textual data, equipment manuals and operation instructions, images, photos, text messages, videos, emails, graphics, documents, 2-dimensional and 3-dimensional drawings, and sketches.

(36) In a preferred embodiment, each of user devices 104 communicates with system administrator 102 to access BIM database 103 to project a BIM as will be further described below.

(37) It will be appreciated by those skilled in the art that any type of three-dimensional rendering may be employed in the disclosed embodiment and that a BIM is just one example of such a three-dimensional rendering.

(38) Referring to FIG. 2, user device 200 includes wearable computer 201 connected to headset 202. Wearable computer 201 is further connected to replaceable battery 203, microphone 204, control input device 206, and speaker 205.

(39) Wearable computer 201 includes processor 207 and memory 209 connected to processor 207, and network interface 208 connected to processor 207. Augmented reality application 210, BIM 211, and a set of videos, images, and data 212 are stored in memory 209. In one embodiment, control input device 206 is connected to wearable computer 201. In preferred embodiment control input device 206, is a remote control having a navigation pad and a selection button. Any type of control input device known in the art may be employed.

(40) Headset 202 is further connected to display unit 213 and a set of cameras 214. Headset 202 includes processor 215, a set of sensors 216 connected to processor 215, and memory 217 connected to processor 215.

(41) Referring to FIG. 3A, a preferred implementation of user device 200 is described. Hard hat 302 is worn by user 301. Hard hat 302 has pocket 305 integrally formed in it. In a preferred embodiment, the hard hat includes a “pocket and clip” arrangement used to secure the headset as shown and described in U.S. Pat. No. 8,191,292 to Cummings, et al., which is incorporated herein by reference. Various types of head covers or helmets may be employed to support the headset. Headset 303 detachably mounts to hard hat 302 by flexible mounts 304 attached to case 350 and pocket 305. Headset 303 is further connected to wearable computer 313 via connection 314. Wearable computer 313 is preferably a portable computing device, such as a laptop or tablet computer, worn as a backpack by user 301. Connection 314 provides a data and power connection from wearable computer 313 to headset 303. Headset 303 includes processor 310, memory 311 connected to processor 310, and sensors 312 connected to processor 310. Mounting arm 306 is slidably inserted into headset 303 to allow forward and backward movement. In a preferred embodiment, the mounting arm is biased by a mechanical coil spring which enables it to retract into case 350. Display arm 307 is pivotably connected to mounting arm 306 for pivotal movement about axis 324. Display unit 308 is attached to display arm 307. Display unit 308 includes projector 325, camera 326, and display light guide 309. Camera 326 has field of view 328. In a preferred embodiment, field of view 328 is 90°. In other embodiments, other suitable field of view ranges may be employed. Display arm 307 is further connected to headset 303 with data and power connection 327.

(42) User 301 wears communication device 315. Communication device 315 includes earpiece speaker 316 and microphone 317. Communication device 315 is preferably connected to wearable computer 313 via a wireless connection such as a Bluetooth connection. In other embodiments, other wireless or wired connections are employed. Communication device 315 enables voice activation and voice control of an augmented reality application stored in the wearable computer 313 by user 301.

(43) In one embodiment, camera matrix 318 is detachably connected to headset 303. Camera matrix 318 includes halo 319 and halo 321, each of which is detachably connected to headset 303. A set of base cameras 320 is connected to halo 319 and in communication with headset 303. A set of angled cameras 322 is connected to halo 321 and in communication with headset 303.

(44) Referring to FIG. 3B in another embodiment, camera matrix 318 is attached inside surface 329 of hard hat 302. In this embodiment, halos 319 and 321 are attached to inside surface 329 of hard hat 302 with a suitable adhesive or fastener. Hole 334 is integrally formed in hard hat 302 adjacent to headset 303 for connection to camera matrix 318. In a preferred embodiment, connector 335 is a USB 3.0 connector, connected to a processor of headset 303 and positioned in hole 334 to connect to camera matrix 318. Other suitable data connections may be employed. Set of base cameras 320 is connected to halo 319, each of which is positioned in a hole of set of holes 330. Set of holes 330 is integrally formed in hard hat 302. In one embodiment, side cameras 333 of set of base cameras 320 are attached to headset 303 outside of hard hat 302. In another embodiment, side cameras 333 are eliminated. Set of angled cameras 322 is connected to halo 321, each of which is positioned in a hole of set of holes 331. Set of holes 331 is integrally formed in hard hat 302.

(45) In another preferred embodiment, the cameras are each mounted securely to the inside surface of the hard hat and are positioned to view the outside world through the holes.

(46) In a preferred embodiment, a BIM is downloaded from a system administrator server into a memory resident in a wearable computer 313. The BIM is transmitted from wearable computer 313 through headset 303 and projector 325 for viewing adjacent eye 323 of user 301 to augment the vision of user 301, as will be further described below. The user can select different layers of the BIM to view via voice control. For example, the BIM includes an electrical layer, which shows the location of electrical conduit, connection points, and equipment. As the user moves, headset 303 and wearable computer 313 tracks the location of user 301 and the position and orientation of the user's head using camera 326 and/or camera matrix 318.

(47) In one embodiment, a set of data is downloaded, selected, and displayed to user 301. In one embodiment, the position and orientation of the user's head is not tracked in a display mode. Rather, the data is displayed without regard to the position of the user or hard hat. Any type of data content may be selected, formatted, scaled and displayed, including images, photos, text messages, videos, emails, graphics, documents, drawings, and sketches.

(48) In a preferred embodiment, processor 310 is a 2.8 GHz octa-core Snapdragon 810 processor available from QUALCOMM® Technologies, Inc. Other suitable processors known in the art may be employed.

(49) In a preferred embodiment, sensors 312 is a 9-axis motion tracking system-in-package sensor, model no. MP11-9150 available from InverSense®, Inc. In this embodiment, the 9-axis sensor combines a 3-axis gyroscope, a 3-axis accelerometer, an on-board digital motion processor, and a 3-axis digital compass. In other embodiments, other suitable sensors and/or suitable combinations of sensors may be employed.

(50) In a preferred embodiment, memory 311 is a 2 GB LPDDR3 RAM. Other suitable memory known in the art may be employed.

(51) In a preferred embodiment, each of base cameras 320 and angled cameras 322 is a 16 megapixel smartphone camera capable of recording video at 30 fps that includes a CMOS image sensor, part no. 5K3M2 available from Samsung Semiconductor. Other suitable cameras and/or image sensors known in the art may be employed.

(52) Referring to FIG. 4A, user device 400 includes display unit 402 which includes camera 404 on the temporal side of eye 406 of user 401 and display light guide 405. Camera 404 has field of view 413. Display light guide 405 is positioned in the field of view and adjacent to eye 406. Display unit 402 is movably connected to headset 403, which is detachably mounted to hard hat 407. Display unit 409 is movably connected to headset 408, which is detachably mounted to hard hat 407. Display unit 409 includes camera 410 on the temporal side of eye 412 of user 401. Camera 410 has field of view 414. Display light guide 411 is in the field of view and adjacent to eye 412. Display units 402 and 409 and headsets 403 and 408 are the same as previously described. Display units 402 and 409 provide a stereoscopic augmented view to user 401.

(53) In a preferred embodiment, each of cameras 404 and 410 is a 16 megapixel smartphone camera capable of recording video at 30 fps that includes a CMOS image sensor, part no. 5K3M2 available from Samsmung Semiconductor. Other suitable cameras and/or image sensors known in the art may be employed.

(54) Referring to FIGS. 4B, 4C, and 4D in one embodiment, hard hat 407 includes pocket 450 integrally formed in hard hat 407 and adjacent to pocket 449. Glasses 451 includes display arm 452 adjustably engaged with connector 453 and display arm 469 adjustably engaged with connector 470. Mounting arm 454 is adjustably engaged with connector 453. Mounting arm 471 is adjustably engaged with connector 470. Data connection 468 connects display unit 409 to headset 408 detachably mounted to pocket 449, as previously described. Display arm 452 includes a set of ridges 462 integrally formed on it. Mounting arm 454 includes flexible mounts 455 to detachably mount glasses 451 to hard hat 407. Mounting arm 454 further includes a set of ridges 463 integrally formed on it. Connector 453 has mount portion 456 and display portion 457. Mount portion 456 includes channel 458 integrally formed in it. Channel 458 has ridge 459 integrally formed on it. Mounting arm 454 slidingly engages with channel 458. Set of ridges 463 engages with ridge 459 to enable adjustable positional movement along directions 464 and 465. Display portion 457 includes channel 460 integrally formed in it. Channel 460 includes ridge 461 integrally formed on it. Display arm 452 slidingly engages with channel 460. Set of ridges 462 engages with ridge 461 to enable adjustable positional movement along directions 466 and 467.

(55) Likewise, display arm 469 includes a set of ridges 472 integrally formed on it. Mounting arm 471 includes flexible mounts 476 to detachably mount glasses 451 to a pocket in hard hat 407. Mounting arm 471 further includes a set of ridges 477 integrally formed on it. Connector 470 has mount portion 478 and display portion 479. Mount portion 478 includes channel 475 integrally formed in it. Channel 475 has ridge 480 integrally formed on it. Mounting arm 471 slidingly engages with channel 475. Set of ridges 477 engages with ridge 480 to enable adjustable positional movement along directions 464 and 465. Display portion 479 includes channel 473 integrally formed in it. Channel 473 includes ridge 474 integrally formed on it. Display arm 469 slidingly engages with channel 473. Set of ridges 472 engages with ridge 474 to enable adjustable positional movement along directions 466 and 467. Glasses 451 includes display light guides 405 and 411 and display units 402 and 409, as previously described. Display unit 402 is connected to headset 403 with a data connection.

(56) In a preferred embodiment, channel 458 is generally perpendicular to channel 460 and vice versa. Other arrangements may be employed.

(57) In a preferred embodiment, channel 475 is generally perpendicular to channel 473 and vice versa. Other arrangements may be employed.

(58) In a preferred embodiment, each of display anus 452 and 469, connectors 453 and 470, and mounting arms 454 and 471 is made of an injection molded plastic. Other suitable materials known in the art may be employed.

(59) In one embodiment, mount portion 456 and display portion 457 are separate pieces attached to each other with a suitable adhesive or epoxy. In another embodiment, mount portion 456 and display portion 457 are integrally formed portions of a single piece adjacent to each other. Other attachment means known in the art may be employed.

(60) In one embodiment, mount portion 478 and display portion 479 are separate pieces attached to each other with a suitable adhesive or epoxy. In another embodiment, mount portion 478 and display portion 479 are integrally formed portions of a single piece adjacent to each other. Other attachment means known in the art may be employed.

(61) Referring to FIG. 5, camera matrix 318 will be further described. Camera matrix 318 includes halo 501 and halo 502 connected to halo 501. Each of cameras 503, 504, 505, and 506 is connected to halo 501. Camera 503 has field of view 507. Camera 504 has field of view 508. Camera 505 has field of view 509. Camera 506 has field of view 510. Each of cameras 511, 512, 513, and 514 is connected to halo 502. Camera 511 has field of view 515. Camera 512 has field of view 516. Camera 513 has field of view 517. Camera 514 has field of view 518.

(62) In a preferred embodiment, each of cameras 503, 504, 505, and 506 is positioned approximately 90° with respect to each other around halo 501. Other angular intervals may be employed.

(63) In a preferred embodiment, each of cameras 511, 512, 513, and 514 is positioned approximately 90° with respect to each other around halo 502. Other angular intervals may be employed.

(64) In a preferred embodiment, each of field of views 507, 508, 509, and 510 is approximately 90°. Other field of view ranges may be employed.

(65) In a preferred embodiment, each of field of views 515, 516, 517, and 518 is approximately 90°. Other field of view ranges may be employed.

(66) In a preferred embodiment, camera matrix 318 provides a 360° view of the surroundings of a user. In other embodiments, other numbers of cameras, angular positions, and field of view ranges may be employed to provide a 360° view.

(67) Referring to FIG. 6, each of display units 402 and 409 will be further described as display unit 600. Display unit 600 will be further described with respect to a right eye of a user. It will be appreciated by those skilled in the art that the arrangement of display unit 600 is simply reversed for implementation on a left eye. Display unit 600 includes light guide 601, projector 602 attached to light guide 601, and camera 603 connected to and adjacent to light guide 601 and projector 602. Camera 603 is connected to headset 615 and includes lens 604. Projector 602 includes light source 605. Light source 605 is connected to headset 615. Collimating lens 606 is positioned adjacent to light source 605. Light guide 601 includes input surface 607 and output surface 608, each of which is attached to the interior of light guide 601. Each of input surface 607 and output surface 608 is positioned at angles ω and γ, respectively from front surface 613 to provide total internal reflection (“TIR”) for light guide 601, thereby projecting an image in field of view 610 of user eye 609.

(68) In a preferred embodiment, angles ω and γ, are 30° and 45°, respectively. Any angles may be employed to provide TIR for light guide 601.

(69) In use, light source 605 displays an image received from headset 615. The image is represented by rays 611 and 612. Rays 611 and 612 are transmitted through collimating lens 606 and reflected off of input surface 607 for TIR. Rays 611 and 612 are further reflected off of front surface 613 and rear surface 614 and output surface 608 in field of view 610 of user eye 609.

(70) In a preferred embodiment, light source 605 is an organic light emitting diode (“OLED”) display such as the WUXGA OLED-XL Microdisplay, part no. EMA-100801-01, available from eMagin Corporation. In another embodiment, light source 605 is a light emitting diode (“LED”) display. Other suitable light sources and displays known in the art may be employed.

(71) In a preferred embodiment, light guide 601 is made of acrylic. In another embodiment, light guide 601 is made of poly (methyl methacrylate) (“PMMA”). Other suitable materials known in the art may be employed.

(72) In a preferred embodiment, input surface 607 is a flat mirror and output surface 608 is a partially-reflective mirror, such as a half-silvered mirror. In other embodiments, other combinations for input surface 607 and output surface 608 may employed and are summarized in Table 1 below.

(73) TABLE-US-00001 TABLE 1 Combinations for Input and Output Surfaces Output Surface Diffraction Single Multiple grating Diffraction Reflective Reflective (varying Grating Holographic Input Surface Surface Surfaces index) (lines) Element Single Reflective x x x x x Surface Multiple Reflective x x x x x Surfaces Combination x x x x x Refractive/Reflective Element Diffraction Grating x x x x x (varying index) Diffraction Grating x x x x x (lines) Holographic x x x x x Element

(74) Referring to FIG. 7A in use, lens 705 of camera 704 is automatically focused on real object 706 at a distance d from camera 704 and sent to headset 710 as image 707. Headset 710 and wearable computer 711 determine distance d and the position of display unit 701 with respect to real object 706. Wearable computer 711 generates virtual image 708 based on distance d. Projector 703 projects virtual image 708 into light guide 701, as previously described. Virtual image 708 is displayed as virtual object 709 to appear at distance d′ in view of user eye 702. Virtual object 709 is magnified to coincide with the size and position of real object 706 to create a perceived depth of focus d. In one embodiment, d′ is less than d. In another embodiment, d′ is equal to d. In one embodiment, d′ is a fixed distance from camera 704 for all real objects.

(75) Referring to FIG. 7B, point of view 712 is the view a user sees while wearing a headset and display unit. Point of view 712 includes floor 713 and adjoining wall 714. Registration marker 715 is attached to floor 713. Registration marker 716 is attached to wall 714. Real object 718 is being lowered into position. According to the BIM, the correct location for real object 718 is outlined by virtual object 717. In this way, a user easily determines if real object 718 is properly positioned and can quickly make adjustments to ensure real object 718 is properly positioned.

(76) In one embodiment, a set of data 719 is displayed. The set of data 719 includes image 720 and text 721. Any type of data including images, photos, text messages, videos, emails, graphics, documents, drawings, schematics, diagrams, and hand-drawn sketches may be employed. For example, image 720 is an installation diagram of real object 718 and text 721 is a set of installation instructions for real object 718.

(77) Each of the positions and sizes of image 720 and text 721 is optionally changed by the user.

(78) In one embodiment, set of data 719 is displayed simultaneously with virtual object 717. In another embodiment, set of data 719 is displayed without virtual object 717 in a display mode, as will be further described below.

(79) Referring to FIG. 8, construction site 800 includes floor 801 and adjoining walls 802 and 803. Registration system 804 includes registration markers 805, 806, and 807 positioned at precise locations on floor 801, wall 802 and wall 803, respectively and serve as a set of reference points for user device 808 worn by user 809.

(80) Each of the positions of registration markers 805, 806, and 807 is associated with a position in a BIM. Survey location 810 is precisely positioned at a known location at construction site 800 and saved in the BIM. Reference marker 811 is a master reference point based on the location of the survey location 810. Each of registration markers 805, 806, and 807 is positioned from reference marker 811 to ensure proper location of floor 801 and walls 802 and 803. At least one of registration markers 805, 806, 807, and 811 will be in view of a camera of user device 808 worn by user 809 and at any given time. The camera captures an image of at least one of registration markers 805, 806, 807, and 811. A wearable computer of user device 808 decodes the captured image to determine a real location of at least one of registration markers 805, 806, 807, and 811. The wearable computer determines a corresponding virtual location in the BIM.

(81) For example, user 809 is standing in construction site 800 wearing user device 808 and looking down at location 812 where object 813 is to be installed. Registration marker 805 is in view of user device 808. The projected BIM shows the correct installation position 814 in view of user 809 as if the user were standing inside the BIM. As user 809 tilts his or her head up to look at wall 802 the movement of the user's head is detected by user device 808 and registration marker 806 is in view of user device 808. Based on the position of registration marker 806, the BIM is moved and rotated in real time to align with the user's field of vision and provide an in-person view of the BIM to user 809. Crane 815 lowers object 813 towards location 812. Based on the projected BIM, object 813 should be installed at installation position 814. User 809 uses the projected BIM to properly lower the object 813 and precisely install object 813 at proper installation position 814, thereby saving time and money in the form of overrun construction costs.

(82) If a mistake is found, user 809 captures still images using the camera for upload to the system administrator or records or streams video back to the system administrator. In this way, the party responsible for the mistake can be easily and quickly identified.

(83) Referring to FIGS. 9A and 9B, registration marker 901 includes shape 903 and code 904. Registration marker 902 includes shape 905 and code 906. Any polygon may be employed for shapes 903 and 905.

(84) In a preferred embodiment, each of codes 904 and 906 is a two-dimensional bar code. In this embodiment, each of codes 904 and 906 includes a set of marker information, including a set of dimensions of shapes 903 and 905, and a set of x, y, z coordinates position at which registration markers 901 and 902 are placed, and a description of each shape and location. Any type of code may be employed.

(85) Shapes 903 and 905 enable detection of codes 904 and 906, respectively, at an offset angle. For example, shape 903 is an equilateral triangle and shape 905 is a rectangle. If a camera capturing an image of shapes 903 and 905 is positioned at an offset angle, shapes 903 and 905 will appear as a scalene triangle and a parallelogram, respectively, in a skewed image.

(86) Referring to FIG. 10, data flow 1000 for augmented reality application 1001 for a user device will be described. BIM 1002 is input into augmented reality application 1001. Application commands 1003 provide input control for the processes of augmented reality application 1001. Images 1004 are received and sent by augmented reality application 1001. For example, a set of cameras captures a set of registration images. The set of marker images is used to determine the position of the user. In another example, images 1004 are still or video images captured by a set of cameras adjacent to the eyes of the user and saved to memory for later upload or streamed to a server. Point of view image 1005 is captured by the set of headset cameras adjacent to the eyes of a user. Set of data 1007 is input into augmented reality application 1001.

(87) In a preferred embodiment, the position of the user is determined from the set of code images 1004 by augmented reality application 1001. Augmented reality application 1001 orients BIM 1002 according to the determined position of the user. Commands 1003 determine which layers of BIM 1002 are displayed. Augmented reality application 1001 overlays the selected layers of BIM 1002 at the determined position to generate stereoscopic image overlay 1006 for display.

(88) In one embodiment, commands 1003 determine a subset of set of data 1007 to display and the size and position of the subset of the set of data. Augmented reality application 1001 overlays the selected subset of data 1007 according to the selected size and position of the set of data 1007 for display.

(89) Referring to FIG. 11, commands menu 1100 includes standby/run toggle 1101, BIM layer selection 1102, reload BIM 1103, save overlaid image 1104, and calibrate 1105. Standby/run toggle 1101 toggles the augmented reality application to a standby mode or a run mode. BIM layer selection 1102 enables the user to select any layer of the BIM to view. For example, the layers include, but are not limited to, structural, electrical, plumbing, data, and HVAC. Reload BIM button 1103 downloads the BIM into memory. Save overlaid image 1104 captures a “screen capture” of the point of view and the overlaid BIM from the perspective of the user. Calibrate 1105 executes a calibration process, as will be further described below. Position and orientation toggle 1106 toggles the position and orientation functions on and off to selectively run in a display mode. Select data 1107 enables the user to select which data to display and the size and the position of the selected data. Selection of 1101, 1102, 1103, 1104, 1105, 1106, and 1107 is accomplished via voice controls.

(90) Referring to FIG. 12, state machine method 1200 for an augmented reality application will now be described. State machine method 1200 begins at step 1201 in a power off mode. Once the system is enabled in step 1202 by initiating power, state machine method 1200 proceeds to a standby mode at step 1203. Once a “run” command is received, state machine method 1200 proceeds to step 1204. At step 1204, a position and an orientation function of the augmented reality application is toggled on or off. If toggled off then the augmented reality application runs in a display mode at step 1205 and optionally displays a set of data selectable by the user. The augmented reality application runs in the display mode until the user toggles the position and the orientation function on at step 1204. If turned on, then state machine method 1200 proceeds to step 1206.

(91) At step 1206, state machine method 1200 turns on a set of cameras and begins to search for a registration marker in a loss of “marker lock” mode. At step 1207, a position and orientation of a user device is determined from the registration marker, as will be further described below. If the position and orientation of the user device cannot be determined, then state machine method 1200 returns to step 1206 to search for a registration marker. If the position and orientation of the user device is determined, then state machine method 1200 proceeds to step 1208. At step 1208, the augmented reality application runs in a “marker lock” mode, that is the position and orientation of the user device can repeatedly be determined within a predetermined time. In this step, a runtime loop for the augmented reality application is initiated and a BIM is displayed, as will be further described below. In a preferred embodiment, the predetermined time is 30 seconds. Other times may be employed.

(92) In one embodiment, the set of data is displayed when the augmented reality application runs in the “marker lock” mode.

(93) At step 1209, a consistency is determined. In this step, if the position and orientation of the user device can be repeatedly determined within the predetermined time, then state machine method 1200 returns to step 1208. In this step, if the BIM is properly displayed, i.e., is rotated and aligned with the user point of view, then stated machine method 1200 returns to step 1208. If the position and orientation of the user device cannot be repeatedly determined within the predetermined time or the BIM is not properly displayed, i.e., is not rotated and aligned with the user point of view, then state machine method 1200 proceeds to step 1210. At step 1210, a message is displayed to the user indicating a position and orientation consistency problem and state machine method 1200 begins a calibration process at step 1211, as will be further described below.

(94) Referring to FIG. 13, method 1300 for registering a registration marker for a BIM will be described. The registration marker includes a shape and a code, as previously described. At step 1301, a position of the registration marker is calibrated. In this step, a surveyor or a user positions the registration marker in a desired location. For example, the registration marker is placed in the middle of a wall or a column or a stud. Any desired location may be employed. Measurements are taken to ensure the registration marker is placed in the desired location. At step 1302, a set of location coordinates of the placed registration marker is stored in the code and in the BIM. At step 1303, a set of dimensions for the shape of the registration marker is stored in the code and in the BIM. At step 1304, a description of the registration marker is stored in the code and the BIM. Method 1300 is repeated for each registration marker.

(95) Referring to FIG. 14A, method 1400 for calibrating a position of a user device will be described. At step 1401, a camera of the user device is pointed at a registration marker so that the registration marker is within a field of view of the camera.

(96) Referring to FIGS. 14B and 14C, an offset position of user device 1409 with respect to registration marker 1411 shown in a top view and a side view, respectively, will now be described. User device 1409 has camera 1410. Camera 1410 has camera axis 1414. Registration marker 1411 is in view 1413 of camera 1410. Registration marker 1411 has marker axis 1412. Camera 1410 and user device 1409 is positioned offset with respect to registration marker 1411. Position angle α is the angle between marker axis 1412 and camera axis 1414 in the x-z plane of coordinates 1415. Position angle β is the angle between marker axis 1412 and camera axis 1414 in the y-z plane of coordinates 1415. In one embodiment, registration marker 1411 is rotated about the z-axis of coordinates 1415.

(97) Because of the offset position of user device 1409 and camera 1410 as defined by position angles α and β, the image of registration marker 1411 is skewed.

(98) Returning to FIG. 14A, at step 1402, an image of the registration marker is captured by the camera.

(99) Referring to FIG. 14D, skewed image 1416 includes skewed registration marker 1442. Skewed registration marker 1442 includes skewed shape 1417 and skewed code 1418. Skewed registration mark 1442 is in the x-y plane defined by x-axis 1419 and y-axis 1420. Z-axis 1421 traverses perpendicularly through skewed registration mark 1442. As can be seen in FIG. 14D, skewed registration mark 1442 appears as a parallelogram. In this example, skewed registration mark 1442 is rotated approximately 30° about each of x-axis 1419, y-axis 1420, and z-axis 1421.

(100) Returning to FIG. 14A, at step 1403, a set of edges in the image of the registration marker is located. In this step, Gaussian smoothing is first applied to the image to reduce noise in the image. In a preferred embodiment, Canny edge detection is then employed to locate the set of edges. In other embodiments, other edge detection means may be employed. In one embodiment, edge thinning is applied to the set of edges to remove any unwanted points. In a preferred embodiment, the set of edges is a boundary of the shape of the registration marker.

(101) At step 1404, the image is deskewed in order to determine a set of position angles with respect to the registration marker, as will be further described below.

(102) At step 1405 the code is read to determine the set of dimensions of the shape of the registration marker, including an actual height and an actual width. At step 1406, a distance from the camera to the registration marker is determined.

(103) At step 1407, an absolute position of the user is calculated based on the position angles and the distance from the registration marker.

(104) Referring to FIG. 14E, step 1404 will be further described as method 1422 for deskewing an image. Method 1422 begins at step 1423. At step 1424, a set of reference lines for the set of edges of a registration marker is determined. In a preferred embodiment, the set of references lines is determined by the Hough transform. Other suitable methods known in the art may be employed.

(105) At step 1425, a pair angle is calculated between each pair of intersecting reference lines to generate a set of pair angles. At step 1426, a skew angle is calculated from set of pair angles by averaging the set of pair angles. At step 1427, the image is rotated about an axis by the skew angle. The skew angle is the position angle with respect to each axis, as previously described. At step 1428, whether or not the image has been deskewed for all axes is determined. If not, method 1422 advances to the next axis at step 1429 and returns to step 1424. If so, method 1422 ends at step 1430.

(106) Referring to FIG. 14F, deskewed image 1443 includes deskewed registration marker 1444 along x-axis 1419, y-axis 1420, and z-axis 1421. Deskewed registration marker 1444 includes deskewed shape 1431 and deskewed code 1432. Deskewed registration marker 1444 is approximately perpendicular to z-axis 1421. Deskewed code 1432 can now be read. Deskewed image 1443 has height 1433 and width 1434. Deskewed registration marker 1444 has height 1435 and width 1436.

(107) In a preferred embodiment, each of heights 1433 and 1435 and widths 1434 and 1436 is measured by counting the number of pixels for deskewed registration marker 1444 and deskewed image 1443.

(108) Referring to FIG. 14G, step 1406 will now be further described. Camera 1437 has field of view 1438 spanning an angle θ, which varies depending on the type of camera employed. Registration marker 1439 is in plane 1440. Plane 1440 is distance 1441 from camera 1437. Height 1442 of registration marker 1439 is retrieved from a code contained in registration marker 1439. Distance 1441 is calculated by:

(109) $\begin{array}{cc}d=\frac{h}{x\tan \theta },& \mathrm{Eq}.1\end{array}$
where d is distance 1441, h is height 1442, θ is angle θ of field of view 1438, and x is a height percentage of the height of the deskewed registration marker in the deskewed image to the height of the deskewed image. For example, if the height of the deskewed registration marker is 60% of the height of the deskewed image, then x=0.6.

(110) Referring to FIG. 15, runtime process 1500 for an augmented reality application will now be described. Runtime process 1500 starts at step 1501. At step 1502, a BIM is retrieved. In this step, the BIM is downloaded from a system administration server and saved into memory of a user device. At step 1503, an image is captured from a set of cameras. At step 1504, a position and an orientation of the user device is determined, as will be further described below. At step 1505, a stereoscopic overlay of the BIM is rendered according to the position and the orientation of the user device, as will be further described below. At step 1506, the rendered stereoscopic overlay is output to a display unit of the user device for display to the user. In a preferred embodiment, the rendered stereoscopic overlay is rendered at least 24 fps.

(111) In one embodiment, a set of data is retrieved at step 1507. In this step, the set of data is downloaded from the system administrator and saved into the memory of the user device. In one embodiment, the position and the orientation function is deactivated. In another embodiment, the position and the orientation function remain activated. At step 1508, a subset of the set of data is selected for display including the size and the position of the selected set of data. At step 1509, the selected subset of data is displayed on the display unit.

(112) At step 1510, a determination is made as to whether an end command has been received. If not, runtime process returns to step 1503. If so, runtime process 1500 ends at step 1511.

(113) Referring to FIG. 16, step 1504 will be further described as method 1600 for determining a position and an orientation of a user device. Method 1600 begins at step 1601. At step 1602, a set of registration markers is identified and decoded to determine the position of the user device. In a preferred embodiment, method 1400 is employed. At step 1603, a set of motion detection data is received from a set of sensors in the user device to determine movement of the user device. At step 1604, the set of motion detection data and the position of the user device are combined to determine an x, y, z position of the user device in reality and in the BIM and a roll, pitch, and yaw or detection of the user device in reality and the BIM. In this step, the user device determines which camera captured the image of the registration marker, i.e., a temporal camera or a camera of the camera matrix. If the camera of the camera matrix captures the images, then a difference angle is calculated between an axis of the camera of the camera matrix and an axis of the temporal camera. The orientation is calculated from the set of position angles and the difference angles. The set of motion detection data received is the roll, pitch, and yaw orientation movement of the head of the user. Method 1600 ends at step 1605.

(114) Referring to FIG. 17, step 1505 will be further described as method 1700 for rendering a stereoscopic overlay according to the position and the orientation of the user device for a user device. Method 1700 begins at step 1701. At step 1702, a BIM is rotated and magnified based on the position and the orientation of the user device. At step 1703, the BIM is “clipped” based on a set of barriers in the BIM, i.e., the nearest set of walls. For example, if the user is standing the middle of a room, the BIM is “clipped” to only show the room of the BIM in which the user is standing. Otherwise, the entire BIM of the entire building would be shown to the user. At step 1704, a layer selection of the BIM is determined from the command menu. At step 1705, the selected layers of the “clipped” BIM is rendered as a stereoscopic image, i.e., the BIM image is rendered as a pair of BIM images, a left BIM image for a left display unit and a right BIM image for a right display unit of the user device. Method 1700 ends at step 1706.

(115) In a preferred embodiment, the left BIM image and the right BIM image are shifted with respect to each other, in a range of approximately 2.5 to 3 inches to compensate for the average distance between the pupils of human eyes.

(116) Referring to FIG. 18, method 1800 for updating a BIM will now be described. Method 1800 begins at step 1801. At step 1802, a virtual location of a virtual object in the BIM is determined by viewing the virtual location on a display unit of a user device. At step 1803, an actual location of a real object associated with the virtual object is determined. At step 1804, a tolerance for the real object location is determined by any measuring means. In a preferred embodiment, the tolerance is determined by a set of building codes. At step 1805, the actual location is compared to the virtual location to determine whether the actual location is within the tolerance. If so, then method 1800 ends at step 1809. If the actual location is not within the tolerance, then method 1800 proceeds to step 1806. At step 1806, an image is captured of the actual location and the virtual location as seen through the display by the user. At step 1807, the captured image is uploaded to a system administrator server. At step 1808, the captured image is saved in the BIM as a “mistakes” layer. The “mistakes” layer is then a selectable layer in the BIM once a user reloads the BIM to the user device from the system administrator server. Method 1800 ends at step 1809.

(117) Referring to FIG. 19 in another embodiment, method 1900 for updating a BIM will now be described. Method 1900 begins at step 1901. At step 1902, a streaming session between a user device and a system administrator server is initiated and a video is captured and streamed in real time to the system administrator server. The video includes the point of view of the user captured by a camera with the overlaid BIM. At step 1903, a virtual location of a virtual object in the BIM is determined by viewing the virtual location on the display. At step 1904, an actual location of a real object associated with the virtual object is determined. At step 1905, a tolerance for the real object location is determined by any measuring means. In a preferred embodiment, the tolerance is determined by a set of building codes. At step 1906, the actual location is compared to the virtual location to determine whether the actual location is within the tolerance. If so, then method 1900 ends at step 1908. If the actual location is not within the tolerance, then method 1900 proceeds to step 1907. At step 1907, the video is saved in the BIM in a “mistakes” layer as a selectable element, such as an icon or link. The “mistakes” layer is then a selectable layer in the BIM once a user reloads the BIM to a wearable computer from the system administrator server. The user selects the selectable element to stream and view the video. Method 1900 ends at step 1908.

(118) It will be appreciated by those skilled in the art that the described embodiments disclose significantly more than an abstract idea including technical advancements in the field of data processing and a transformation of data which is directly related to real world objects and situations in that the disclosed embodiments enable a computer to operate more efficiently and make improvements to construction management technology. Specifically, the disclosed embodiments eliminate the remanufacture of construction components and rescheduling of equipment. Further, the disclosed embodiments eliminate the reliance and use of external positioning systems, such as GPS or laser-based systems.

(119) It will be appreciated by those skilled in the art that changes could be made to the embodiments described above without departing from the broad inventive concept. It is understood, therefore, that this disclosure is not limited to the particular embodiments herein, but it is intended to cover modifications within the spirit and scope of the present disclosure as defined by the appended claims.