In various examples, systems and methods for machine learning based hazard detection for autonomous machine applications using stereo disparity are presented. Disparity between a stereo pair of images is used to generate a path disparity model. Using the path disparity model, a machine learning model can recognize when a pixel in the first image corresponds to a pixel in the second image even though the pixel in the two images does not have identical characteristics. Similarities in extracted feature vectors can be computed and represented by a vector similarity metric that is input to a machine learning classifier, along with feature information extracted from the stereo image pair, to differentiate hazard pixels from non-hazard pixels. In some embodiments, a V-space disparity map, where a first axis corresponds to disparity values and the second axis corresponds to pixel rows, may be used to simplify estimation of the path disparity model.

A method for ascertaining a dynamic foreign object-driving corridor association with the aid of an image sensor of an ego vehicle, in which images of the environment of the ego vehicle are generated with the aid of the image sensor. A driving corridor of the ego vehicle is ascertained with the aid of the images in the native measuring space of the image sensor from roadway information and/or with the aid of the odometry of the ego vehicle, foreign objects are detected, and at least one kinematic variable is ascertained for at least one of the foreign objects, and at least one dynamic foreign object-driving corridor association for the lateral movement of the foreign object relative to the driving corridor is ascertained based on the at least one kinematic variable and the driving corridor.

A vehicle control device includes a tracking unit estimating a motion of a moving object, a model selection unit selecting a motion model corresponding to a moving object type, an abnormality determination unit determining a presence or absence of an abnormality of the estimation of the motion of the moving object based on the estimated moving object motion and the motion indicated by the motion model, and a control unit. A control mode in which the control unit controls traveling of a host vehicle when the abnormality determination unit determines that the abnormality is present differs from a control mode in which the control unit controls the traveling of the host vehicle when the abnormality determination unit determines that the abnormality is absent.

A monocular image often does not contain enough information to determine, with certainty, the depth of an object in a scene reflected in the image. Combining image data and LIDAR data may enable determining a depth estimate of the object relative to the camera. Specifically, LIDAR points corresponding to a region of interest (“ROI”) in the image that corresponds to the object may be combined with the image data. These LIDAR points may be scored according to a monocular image model and/or a factor based on a distance between projections of the LIDAR points into the ROI and a center of the region of interest may improve the accuracy of the depth estimate. Using these scores as weights in a weighted median of the LIDAR points may improve the accuracy of the depth estimate, for example, by discerning between a detected object and an occluding object and/or background.

Exemplary embodiments described in this disclosure are generally directed to a collaborative relationship between a vehicle and a UAV. In one exemplary implementation, a computer that is provided in the vehicle uses images captured by an imaging system in the UAV together with images captured by an imaging system in the vehicle, to modify a suspension system of the vehicle based on a nature of the terrain located below, or ahead, of the vehicle. The computer may, for example, modify a suspension system before the vehicle reaches a rock or a pothole on the ground ahead. In another exemplary implementation, the computer may generate an augmented reality image that includes a 3D model of the vehicle rendered on an image of a terrain located below, or ahead of, the vehicle. The augmented reality image may be used by a driver of the vehicle to drive the vehicle over such terrain.

Provided is an information processing device that includes an acquisition unit that acquires an image captured by an imaging unit, a recognition unit that recognizes attributes of an object shown in the image captured by the imaging unit, and a generation unit that generates a bird's-eye view map showing the attributes of the object on the basis of the image captured by the imaging unit and information on the attributes of the object recognized by the recognition unit.

A system includes an image sensor having a plurality of pixels that form a plurality of regions of interest (ROIs), and configured to operate at a frame rate higher than a threshold rate. The system also includes an image processing resource. The system further includes control circuitry configured to perform operations that include obtaining, from the image sensor, a full-resolution image of an environment. The full-resolution image contains each respective ROI of the plurality of ROIs. The operations also include selecting a particular ROI based on the full-resolution image, and detecting an object of interest in the particular ROI. The operations include determining a mode of operation by which subsequent image data generated by the particular ROI is to be processed. The operations further include processing, based on the mode of operation and the frame rate, the image data comprising a plurality of ROI images of the object of interest.

A data acquisition system of a vehicle includes an image capture device, a communication interface, and a controller communicatively coupled to the image capture device and communicatively coupled to the communication interface. Processors of the controller are configured to calibrate an image-distance relationship value of an identified component of a first image captured by the image capture device corresponding to a known feature based on established metrics of the known feature. The processors are also configured to provide control of the vehicle or activation of an alert system of the vehicle via the communication interface based on the image-distance relationship value.

A vehicle safety system and technique are described. In one example, a vehicle safety system communicably coupled to a vehicle, comprises processing circuitry coupled to a camera unit and a LiDAR sensor, the processing circuitry to execute logic operative to analyze an image captured by one or more cameras to identify predicted regions of road damage, correlate LiDAR sensor data with the predicted regions of road damage, analyze the LiDAR sensor data correlated with the predicted regions of road damage to identify regions of road damage in three-dimensional space; and output one or more indications of the identified regions of road damage, wherein the processing circuitry is coupled to an interface to the vehicle, the processing circuitry to output an identification of road damage.

Provided are methods for prediction error scenario mining for machine learning methods, which can include determining a prediction error indicative of a difference between a planned decision of an autonomous vehicle and an ideal decision of the autonomous vehicle. The prediction error is associated with an error-prone scenario for which a machine learning model of an autonomous vehicle is to make planned movements. The method includes searching a scenario database for the error-prone scenario based on the prediction error. The scenario database includes a plurality of datasets representative of data received from an autonomous vehicle sensor system in which the plurality of datasets is marked with at least one attribute of the set of attributes. The method further includes obtaining the error-prone scenario from the scenario database for inputting into the machine learning model for training the machine learning model. Systems and computer program products are also provided.