Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using a surfel map to generate long range localization. One of the methods includes obtaining, for a particular location of a vehicle having a camera and a detection sensor, surfel data including a plurality of surfels. Each surfel in the surfel data has a respective location and corresponds to a different respective detected surface in an environment. Image data captured by the camera is obtained. It is determined that a region of interest for detecting objects for a vehicle planning process is outside a detectable region for the detection sensor. In response, it is determined that the image data for the region of interest matches surfel color data for the surfels corresponding to the region of interest. In response, the vehicle planning process is performed with the region of interest designated as having no unexpected objects.

Techniques for determining predictions on a top-down representation of an environment based on object movement are discussed herein. Sensors of a first vehicle (such as an autonomous vehicle) may capture sensor data of an environment, which may include object(s) separate from the first vehicle (e.g., a vehicle, a pedestrian, a bicycle). A multi-channel image representing a top-down view of the object(s) and the environment may be generated based in part on the sensor data. Environmental data (object extents, velocities, lane positions, crosswalks, etc.) may also be encoded in the image. Multiple images may be generated representing the environment over time and input into a prediction system configured to output a trajectory template (e.g., general intent for future movement) and a predicted trajectory (e.g., more accurate predicted movement) associated with each object. The prediction system may include a machine learned model configured to output the trajectory template(s) and the predicted trajector(ies).

Described herein are systems, devices, and methods for localizing and recognizing an object in an environment. In an example, a mobile cleaning robot comprises a drive system to move the mobile cleaning robot about an environment, an imaging sensor to take images of an object in the environment from different perspectives. The multiple observations include images of the object that is at least partially occluded by an obstacle. A controller circuit of the mobile robot can, for multiple different locations in a map of the environment, calculate respective fractional visibility values using the plurality of images. The fractional visibility values each represent a probability of the object being visible through the corresponding location. The controller circuit can localize and recognize the object based on the fractional visibility values at the multiple locations on the map.

One example system comprises an active sensor that includes a transmitter and a receiver, a first camera that detects external light originating from one or more external light sources to generate first image data, a second camera that detects external light originating from one or more external light sources to generate second image data, and a controller. The controller is configured to perform operations comprising determining a first distance estimate to a first object based on a comparison of the first image data and the second image data, determining a second distance estimate to the first object based on active sensor data, comparing the first distance estimate and the second distance estimate, and determining a third distance estimate to a second object based on the first image data, the second image data, and the comparison of the first and second distance estimates.

There is provided a collision avoidance method of a moving body collision avoidance device including acquiring a driving image of the moving body, recognizing an object in the acquired driving image using a neural network model, calculating a relative distance between the moving body and the object based on the recognized object, calculating a required collision time between the object and the moving body based on the calculated relative distance, and controlling an operation of the moving body based on the calculated required collision time.

An apparatus, system, method, and/or non-transitory computer readable media of a distance estimation apparatus including at least one camera includes obtaining a bounding box corresponding to a target vehicle on the basis of an image obtained through the at least one camera, obtaining a first rectilinear distance to the target vehicle, obtaining a first world width on the basis of the first rectilinear distance and a width of the bounding box, obtaining a second ratio of a region, corresponding to a rear surface of the target vehicle, of a region of the bounding box on the basis of a first ratio, and calculating a second world width of the target vehicle on the basis of the second ratio, wherein the first ratio represents a ratio of the rear surface and a side surface of the target vehicle.

A method for monocular 3D object perception is described. The method includes sampling multiple, stochastic latent variables from a learned latent feature distribution of an RGB image for a 2D object detected in the RGB image. The method also includes lifting a 3D proposal for each stochastic latent variable sampled for the detected 2D object. The method further includes selecting a 3D proposal for the detected 2D object using a proposal selection algorithm to reduce 3D proposal lifting overlap. The method also includes planning a trajectory of an ego vehicle according to a 3D location and pose of the 2D object according to the selected 3D proposal.

Techniques are discussed for determining a location of a vehicle in an environment using a feature corresponding to a portion of an image representing an object in the environment which is associated with a frequently occurring object classification. For example, an image may be received and semantically segmented to associate pixels of the image with a label representing an object of an object type (e.g., extracting only those portions of the image which represent lane boundary markings). Features may then be extracted, or otherwise determined, which are limited to those portions of the image. In some examples, map data indicating a previously mapped location of a corresponding portion of the object may be used to determine a difference. The difference (or sum of differences for multiple observations) are then used to localize the vehicle with respect to the map.

Techniques are disclosed for tracking objects in sensor data, such as multiple images or multiple LIDAR clouds. The techniques may include comparing segmentations of sensor data such as by, for example, determining a similarity of a first segmentation of first sensor data and a second segmentation of second sensor data. Comparing the similarity may comprise determining a first embedding associated with the first segmentation and a second embedding associated with the second segmentation and determining a distance between the first embedding and the second embedding. The techniques may improve the accuracy and/or safety of systems integrating the techniques discussed herein.

Methods for operating a picking devices for medicaments are provided. An image of a movement space detectable by an optical detection device is created with an optical detection device inside the picking device after a predetermined event. Predetermined areas of the image of the movement space are compared with corresponding areas of a reference image, and based on the comparison it is determined whether an obstacle is present in the detected portion of the movement space. Based on the determination of the presence of an obstacle, corresponding signals are provided to prevent a movement of the operating device and/or request a removal of the obstacle by a user. Picking devices for medicaments are also provided.