Provided are methods for end-to-end perception system training using fused images, which can include fusing different types of images to form a fused image, extracting features from the fused image, calculating a loss, and modifying at least one network parameter of an image semantic network based on the loss. Systems and computer program products are also provided.

Systems and methods are provided for generating textual embeddings by tokenizing text data and generating vectors to be provided to a transformer system, where the textual embeddings are vector representations of semantic meanings of text that is part of the text data. The vectors may be averaged for every token of the generated textual embeddings and concatenating average output activations of two layers of the transformer system. Image embeddings may be generated with a convolutional neural network (CNN) from image data, wherein the image embeddings are vector representations of the images that are part of the image data. The textual embeddings and image embeddings may be combined to form combined embeddings to be provided to the transformer system.

An image processing device includes: a reception unit that receives at least one first image provided from at least one first camera capturing an image of a region in which an object exists and a plurality of second images provided from a plurality of second cameras capturing images of a region including a dead region hidden by the object and invisible from a position of the first camera; and an image processing unit that generates a complementary image, as an image of a mask region in the at least one first image corresponding to the object, from the plurality of second images and generates a synthetic display image by combining the at least one first image and the complementary image.

The present teaching relates to method, system, medium, and implementation of in-situ perception in an autonomous driving vehicle. A plurality of types of sensor data are acquired continuously via a plurality of types of sensors deployed on the vehicle, where the plurality of types of sensor data provide information about surrounding of the vehicle. One or more items surrounding the vehicle are tracked, based on some models, from a first of the plurality of types of sensor data from a first type of the plurality of types of sensors. A second of the plurality of types of sensor data are obtained from a second type of the plurality of sensors and are used to generate validation base data. Some of the one or more items are labeled, automatically, via validation base data to generate labeled at least some item, which is to be used to generate model updated information for updating the at least one model.

Various implementations disclosed herein include devices, systems, and methods that provide measurements of objects based on a location of a surface of the objects. An exemplary process may include obtaining a three-dimensional (3D) representation of a physical environment that was generated based on depth data and light intensity image data, generating a 3D bounding box corresponding to an object in the physical environment based on the 3D representation, determining a class of the object based on the 3D semantic data, determining a location of a surface of the object based on the class of the object, the location determined by identifying a plane within the 3D bounding box having semantics in the 3D semantic data satisfying surface criteria for the object, and providing a measurement of the object, the measurement of the object determined based on the location of the surface of the object.

Various implementations disclosed herein include devices, systems, and methods that obtain a three-dimensional (3D) representation of a physical environment that was generated based on depth data and light intensity image data, generate a 3D bounding box corresponding to an object in the physical environment based on the 3D representation, classify the object based on the 3D bounding box and the 3D semantic data, and display a measurement of the object, where the measurement of the object is determined using one of a plurality of class-specific neural networks selected based on the classifying of the object.

In a mobile object, a camera unit including an optical system that forms an optical image having a high-resolution region and a low-resolution region on a light receiving surface of an image pickup element and is disposed on a side of the mobile object, wherein the camera unit is installed to meet the following conditions: A tan (h/(d1+x))−θv/2<φv<A tan (h/(d2+x))+θv/2, φh_limit=max (A tan ((w1−y)/(d1+x))−θh/2, A tan ((w2−y)/(d2+x))−θh/2), φh limit <φh <−A tan (y/(d1+x))+θh/2, where θv and θh denote a vertical and a horizontal field angle of the high-resolution region, φv and φh denote a vertical and a horizontal direction angle of the optical axis of the optical system, x, y, and h denotes offsets, and w1 and w2 denote predetermined widths on the ground at the distances d1 and d2.

In general, devices, systems, and methods for fiducial identification and tracking are provided.

Disclosed is an apparatus for assisting driving of a vehicle including a camera provided in the vehicle, a lidar sensor provided in the vehicle, and a controller configured to control the vehicle to prevent to deviate from a lane based on lane information obtained by the camera and the lidar sensor, wherein the controller excludes the lane information obtained by the camera and controls the vehicle to prevent to deviate from the lane based on the lane information obtained by the lidar sensor, when the vehicle enters a tunnel.

A vehicle driving control apparatus for controlling driving of a vehicle based on information received from a plurality of sensors, and a calibration method performed thereby is provided. The vehicle driving control apparatus includes at least one processor configured to determine a driving route to include a predetermined area of road, upon determining that calibration of a first sensor is required, and a memory configured to store values measured by the plurality of sensors while the vehicle is being driven along the driving route, wherein the at least one processor is further configured to calibrate the first sensor based on the stored measured values and information about the predetermined area of road.