Latest Trends and Efforts in the Development of Autonomous Vehicles

This paper provides an examination of current technologies used in autonomous driving, categorized into hardware and AI models. It will then cover the leading companies in the field in depth and afterwards give an overview about the safety of autonomous driving.

About this paper

Research Essay for the course Seminar: Current Topics in AI (DLMAISCTAI01), Master of Science - Artificial Intelligence at IU International University. Tutor: Gissel Velarde. Matriculation number: IU14090306.

The original LaTeX source, the bibliography and the compiled PDF are available on GitLab: gitlab.com/iu-msc-ai/seminar_current_topics_in_ai_dlmaisctai

Introduction

This paper will analyze the current trends and efforts in the development of autonomous vehicles (AVs). This analysis is based on the various following layers that comprise an autonomous driving stack¹:

1. Input sensors
2. Perception
3. Assessment
4. Planning and decision-making
5. Vehicle controls

Wayve created a good visualization on what they call the traditional AV stack (compare Wayve). This gives a deeper insight into which individual tasks and challenges each step encompasses.

Therefore, the first part of this paper considers the input sensors, which are used to gather data about the environment. The second part is the perception, which processes the data and detects objects in the environment. The third part is the assessment, which evaluates the situation. The fourth part is the planning and decision-making, where the vehicle calculates the best route through the environment and makes the driving decision. The fifth part is the vehicle control, which is how the decision is implemented as physical control of the vehicle.²

The technologies used in autonomous vehicles can be divided into hardware and software, with software being mainly the AI models and the surrounding processing. The hardware part will be separated into the mentioned sensors and the chips, which process the data for the AI model.

The AI models are responsible for the steps from perception until decision-making process. In this case AI models are usually deep learning algorithms, which are trained on large amounts of data. AI models are involved in processing and correcting the sensor data as well, what is not part of this paper.

To align this to the mentioned AV stack, the input sensors will be discussed in the hardware part (compare Sensors), while everything from perception to vehicle control will be discussed in the parts about AI models and applied autonomous driving (compare AI models), with references to the hardware part for the processing power (compare Chips).

The main part of this paper is about information provided by companies in the AV sector, as they gave valuable insights about the current state-of-the-art in autonomous driving. This is split mainly between Tesla and Waymo, as both companies operate on cars which are equipped very differently and therefore have different challenges and solutions (compare Applied autonomous driving).

Levels of automation

The SAE International has defined six levels of automation for vehicles.³ These levels are:

1. Driver assistance
2. Partial driving automation
3. Conditional driving automation
4. High driving automation
5. Full driving automation

In Germany, the federal motor transport authority called KBA has legislation in place to allow automation until level 4 on the roads. Level 5 automation is still in the research phase and is not yet allowed on public roads.⁴

In the USA, where many advanced AV companies are located, the National Highway Traffic Safety Administration (NHTSA) gives out policies on automated driving, but each state is able to allow or restrict AVs on their own. Currently, most states allow level 2 and level 3 automation, while some states even allow level 4 and level 5 automation as well, without a licensed driver on board. Additionally, the individual states have different liability laws in place.⁵

Current commercial situation

The early stages were mostly driven by academic research and organized competitions like the DARPA Grand Challenge. DARPA, the research and development agency of the United States Department of Defense, has been organizing the Grand Challenge from 2004 to 2007, which was a competition for autonomous vehicles. The competition has helped to advance the technology and has led to many breakthroughs in the field. Since then, more and more private companies started developing their own autonomous vehicles, most notably Tesla and Waymo, with Waymo being partly founded by competitors in the DARPA Grand Challenge and being owned by Alphabet Inc. now.

Tesla pushed the development of autonomous vehicles with their Full Self-Driving (FSD) package, which currently operates on level 2 automation and is available in most of their cars. Other car manufacturers started to close the gap and have the same level of automation currently. Recently Tesla took another heavy investment in their compute power (compare Chips) and has the biggest archive of data to train their models.⁶

This year Waymo announced a jump from 50.000 paid rides per week in May in their level 4 autonomous vehicles, to 100.000 paid rides per week in August.⁷⁸ They increased their area of operation from Los Angeles and Phoenix and included San Francisco. They operate on streets which have been mapped out by the company deliberately. Their newest cars are set up with “13 cameras, 4 LiDAR, 6 radar, and an array of external audio receivers”.⁹ Their parent company Alphabet Inc. recently invested additional 5 billion USD in the company.¹⁰

There are a few other companies already which operate at level 4 automation. This includes Baidu, WeRide, AutoX and Cruise. With Waymo being the furthest in commercial operation. Nuro and Starship operate on level 4 for cargo delivery. Cruise has purpose build level 5 cars available, build in cooperation with their parent company GM, but stopped commercial operation for a while, after a series of incidents.¹¹

A new competitor is Zoox, which is a part of Amazon.com Inc. and is operating robotaxis since 2023. Zoox has not started commercial operation yet, but is active in several states in the USA.¹²

McKinsey expects “autonomous driving could create $300 billion to $400 billion in revenue” by 2035.¹³

Hardware

The hardware used in the field of autonomous driving can be separated into the hardware used to train the AI models and the hardware on board of the vehicle to gather and process the data, which lets the model make a decision to control the vehicle. The following part will give an overview about the current state of both kinds of hardware.

Chips

Deep learning algorithms run historically on Graphics Processing Units (GPUs), as GPUs have been a widely available architecture and are capable of solving matrix multiplication in a parallel way. Which is a main part in building and running AI models efficiently and fast. GPUs are used to train the models in large supercomputers and run them afterwards, usually on small edge devices, which are placed in the car. The decision-making needed in the edge devices comes with the need for real time processing capabilities.

Main provider for these chips is NVIDIA. Other companies like AMD or Google with their Tensor Processing Units (TPUs) and are providing chips for training and inference as well. Some companies are developing their own chips, like Tesla with their Full Self-Driving (FSD) chip.¹⁴

According to Epoch AI¹⁵ “FLOP/s performance in 47 ML hardware accelerators doubled every 2.3 years”. The switch “from FP32 to tensor-FP16 led to a further 10x performance increase” and the “memory capacity and bandwidth doubled every 4 years”.¹⁶ Further the costs for floating-point operations/second halfs every ~2.5 years.¹⁷

In 2019 Tesla needed 70.000 GPU hours to train one version of their self-driving AI model end to end.¹⁸ In an article from 2024 they stated to expect a “40x Increase in Computing Power in 3 Years”.¹⁹ To meet this needs they recently choose TSMC for producing their self-designed D1 chips, which as used to build their newest supercomputer called Dojo. The chips used in Dojo are set up as tiles, which are 5×5 arrays of single chips, which act as one. Tesla plans to spend 1 billion Dollar on it and is expected to have 100 exaFlops available by the end of 2023.²⁰

The Dojo supercomputer is additional to continued investment in NVIDIA GPUs.²¹ According to Elon Musk the company will use 130 MW of power for compute and cooling in 2024, which will increase to more than 500 MW in 2025. Half of it will be used with their own chip design, the other half with NVIDIA chips.²²

Further Tesla decided for a mix of 8-bit and 16-bit floating point operations, as they deemed 32-bit as being more precision and range than needed.²³

Sensors

A variety of different sensors are used in autonomous vehicles, to create the input signals for the models. These sensors can be split into visual sensors, sensors which relate to the car (tire angle, rotational speed, etc.) and systems to infer the position (like Global Navigation Satellite Systems). Sensor information is processed via AI models, cross-checked to increase the reliability of the data and sometimes fused over multiple sensors, before the data is forwarded to the AI model, responsible for processing and inference.

Visual sensors are used to capture the environment. They can be passive sensors, which use the visual spectrum of light (cameras), or can be able to inherently measure distance, like radar or LiDAR, via actively sending out signals and capturing the return signals. In all cases the data from visual sensors is used for detecting stationary and moving objects and to calculate their distance.

With LiDAR using laser pulses and radar using radio waves, LiDAR offers better resolution and accuracy, while radar provides longer range and is more reliable in bad weather conditions.²⁴ In 2022 11 billion USD have been invested in LiDAR startups in 2022, compared to 8 Billion USD Radar, to mitigate the individual downsides.²⁵ The referenced article states that most car manufacturers are using a combination of both technologies and radar is expected to take over. LiDAR technology is currently three to five times the cost of radar technology.

Another option is ultrasonic sensors, which are currently used for very short distances, like parking assistance. They are usually not used for more complex autonomous driving.

Cameras do not offer inherent distance measurement, but distances can be inferred. As AI models currently can not process a large sized picture in real time, the available resolution and quality of cameras is already at the limit of what can be processed in real time. Therefore, the better way to increase the benefit from cameras is to increase the number of cameras. Tesla, which relies purely on cameras as visual sensors, is using eight cameras in their cars.

Car sensors are used to measure the current state of the car. A variety of sensors is involved to define the state, which includes a possible accident:

• Steering Angle Sensor
• Throttle position sensor
• Rotational speed sensor on the tire
• Tyre Pressure Sensor
• Accelerometer
• Gyroscope
• etc.

The input of these sensors is used in the AI models as well. Sensor precision is a non-trivial electrical engineering task, but sufficient precision is achieved via fusion of multiple sensor of the same or different types.²⁶

Autonomous car manufacturers expect sensors, which are responsible for measuring the cars angle, to be installed with 0.1º accuracy, so LiDAR and radar systems can function as expected.²⁷ A main challenge is to reduce the noise each sensor measures.

Global Navigation Satellite Systems (GNSSs) currently have a precision of 1-2 meters, which is not precise enough as the only measure of position for autonomous vehicles.²⁸ This accuracy can be increased by using a differential GNSS, which uses fixed stations on earth to correct the signal. The accuracy can be increased to 1 cm with this method and grows around 1m per 150 km from the individual station.²⁹ To have these stations available comes with additional costs though. It is probably not realistic to have stations in all possible environment in a close enough proximity, but may be a solution for specific areas, like a city center.

Precise-Point-Positioning (PPP) and Real-Time Kinematic (RTK) are other ways to increase the accuracy of GNSSs. RTK utilizes differential GPS and real-time kinematic corrections from the cars sensors to enhance the positional accuracy. PPP uses satellite related, atmospheric, and receiver and operational environment to correct the position.³⁰

AI models

As mentioned, the software used to control autonomous vehicles is mainly based on deep learning algorithms. Currently, the most commonly used types of algorithms are Convolutional Neural Networks (CNNs), Reinforcement Learning (RL) and generative models. With CNNs being leading in visual tasks, but recently challenged in this field by the invention and performance of Visual Transformers. General progress in speed, accuracy and capabilities of these technologies will be discussed in the following parts.

Convolutional Neural Networks and Visual Transformers

Processing the visual input from the visual sensors, was for a long time done by Convolutional Neural Networks, which became the standard for image processing tasks since their breakthrough in the 2000s. They are used to detect and label objects in images and to derive measurements like the distance to and between them. This is done either via supervised learning with predefined labels or via semi-supervised learning without a large amount of pre-processed data. With the advent of the transformer architecture and especially the paper about the visual transformer, called An Image Is Worth 16x16 Words, these tasks started to shift to attention based architectures.

CNNs have been implemented in various ways, while one of the most famous and publicly available models for training CNNs is the YOLO model. YOLO is a real-time object detection model that can detect objects in images and videos.

Real time models like YOLO have to balance between speed and accuracy. The YOLO model is able to detect objects in real time, but it is not as accurate as other models which can take more time to process the images. YOLO is available in different versions, which shifts these two dimensions according to the tasks needs.

Over time new versions of YOLO have been released, which have improved in speed and accuracy. Accuracy is measured in mean average precision (mAP) and speed in frames per second (FPS). The mAP depends on the model size, even in the same model generation. The FPS is also dependent on the model size and the hardware used. A comparison between YOLO v5n, which was released in June 2020, and YOLO v8n, which was released in January 2023, shows mAP increased from 45.7 to 52.5 (measured with a threshold of 0.50) and an increase in FPS at the same time, from 934 to 1163 (measured on a RTX 4070 TI chip). This means a 15% increase in accuracy and a 25% increase in speed over 3 years and generations.³¹

Visual transformers (ViTs) are a newer way for visual processing, which are based on the transformer architecture, first introduced in 2017. Transformers are famously being used in natural language processing (NLP) and have since been adapted for image processing tasks. ViTs, like standard transformers, are able to process images in parallel and have an attention mechanism, which allows them to focus on the important parts of the image.

Overall “convolutional structures excel at extracting short-range dependency information among pixels […], transformer structures are adept at capturing global dependency relationships”.³² ViTs can be used for image classification, when a classification token is added to the sequence and have at least comparable results to CNNs, while “requiring substantially fewer computational resources to train”.³³

A paper from Google researchers looked into the differences between CNNs and ViTs in details and concluded that “ViTs can achieve comparable or even superior performance on image classification tasks” compared to CNNs, while maintaining “spatial location information more faithfully than ResNets”.³⁴

Reinforcement learning

RL is mainly used in the planning and decision-making step and provides the decisions to control the vehicle. Deep reinforcement learning (DRL) “consistently outperforms rule-based methods in safety, efficiency, and generalization”, while Deep Q Learning architecture (DQN) provided a big improvement in RL performance.³⁵ Further, as AVs drive in multi-agent settings, with not just cars but a variety of road users interacting, the “increase in mixed traffic, training DRL models gets tougher, highlighting the need for future research in multi-agent RL.”³⁵

Reinforcement learning is notoriously hard for defining the reward function in long term settings. This was shown, among others, by Google Deepmind in their work on agents playing the Starcraft 2 game, which has a long delay between action and overall outcome and therefore a sparse reward structure. To solve this, the Deepmind team used a mix of different short and long term rewards.³⁶ This setting is comparable to autonomous driving, with short term rewards for single maneuvers and long term rewards for the overall trip.

Therefore, a “proper choice or design of the action space can significantly simplify learning”, as the actions are linked to the rewards. A car moving “erratically when exploring the control space, thereby failing to achieve driving maneuvers such as overtaking a leading car” and therefore not getting the reward. This can be mitigated by using a high-level action space, which is abstracted from the low-level control space.³⁷

This is comparable to how a human learns and acts, making “decisions over abstracted high-level options, which we call motion skills”, like “overtaking a leading car” or “merging into a lane”³⁷ and called temporally hierarchical reinforcement learning, which helps to “address large-scale and sparse reward issues”.³⁸ Zhou and Yu go one step further and propose temporal-adaptive hierarchical policy learning, to learn these high-level actions with non-fixed time steps. This helps to learn high-level driving maneuvers, as they can be executed in different time steps, what can be even dependent on the situation.³⁸

Further autonomous driving companies added prediction of human behavior via RL models, to gain more actionable insights about the other agents in the environment (compare Applied autonomous driving).

Generative models

The usage of generative models splits mainly in two areas for autonomous driving. First they are used for creating new data from existing data, to leverage simulated tests. Second, they have been used very recently in driving, predicting human behavior and augmenting the driving decisions of other models (compare Waymo).

Simulations are used to generate more training data or execute current or newly trained agents in situations virtually. The more realistic the generated simulation is, the more of the autonomous driving stack can be tested, e.g. a photorealistic 3D simulation lets the visual processing part be tested as well. Simulations can be created from the ground up, to increase the amount of normal training data, but are used to create edge cases as well, e.g. via adding an uncommon element to a previously captured situation (compare Waymo). This can be done via diffusion models (compare Wayve).

Large language models are a new approach in autonomous driving, where foundational models are used to interpret what the agents are doing and how the environment changes. Sometimes LLMs are even used to create the next action(s) in human language, via next token generation. This is in the early stages and not yet used in commercial autonomous driving yet.

In general the foundational large language models (LLMs) may provide a different technology stack for autonomously driving a car in the future, as they are able to process natural language input and create fitting output. Overall this makes sense, as LLMs are trained on large datasets of human speech and as cars are driven by human decisions, a LLM may be a good fit for modeling human driving behavior.

LLMs are, as the name implies, very big and therefore struggle with real time scenarios. Further, hallucinating information, which is not there, is a problem. That may be resolved with deliberately created training sets just for the task of autonomous driving. This is still a very open field of research.

Applied autonomous driving

The following section tries to give insight into state-of-the-art applied autonomous driving. While this field was in the beginning dominated by Universities (compare Introduction), it is currently led by private companies. Especially as the need for costly hardware has increased (compare Chips).

Tesla

Andrej Karpathy, at that time Senior Director of AI at Tesla, gave a talk in 2021 about the current AI architecture for self-driving at Tesla, which provided a deep insight into their architecture. He describes a variety of models which are involved mostly in the perception section of the AV stack. The following part mainly references this talk, with amendments when newer information was available and relevant.³⁹

The architecture chart will be referenced in the following parts. Tesla separates the backbone and the heads of the model, which they call HydraNet, as it resembles the many heads of a Hydra.

As mentioned, Tesla uses 8 cameras and sensors for velocity and positioning in their cars, which are used as input for the model.⁴⁰ The rough positioning of the cameras gives information about the picture as well (compare 1). Further the camera input gets virtually rectified (compare 2).

The rectified input from the individual cameras is then fed into residual neural networks (ResNets) (compare 3). With Tesla using specifically the RegNet model. ResNet defines the layers as “residual functions with reference to the layer inputs” and applies “Bottleneck Residual Blocks”, where the “bottleneck reduces the number of parameters and matrix multiplications” needed, what makes the residual blocks thinner, so depth can be increased with fewer parameters.⁴¹⁴² RegNet adds a “regulator module as a memory mechanism”, which is “composed of convolutional RNNs”.⁴³ This addresses the problem in ResNet where information learned in blocks get lost over multiple blocks. This step creates a low resolution image with a high number of channels as output, compared to the high resolution, low channel input.

The next step is a feature pyramid networks (FPNs), which uses the individual features, which got created by the multiple layers of the RegNet (compare 4). Tesla uses a specialized implementation of FPNs, called bi-directional feature pyramid network (BiFPN) on top of their RegNet implementation, which allows for “easy and fast multiscale feature fusion”.⁴⁴

The basic idea of a FPN is to use differently scaled versions of the image to detect objects, while not increasing detection time by a lot. Detecting objects in a lower resolution image is faster, but makes the location of the object less accurate. Therefore, after detecting an object in a lower resolution image, a FPN uses the information to detect the object in a higher resolution image with general information about the positions, which then gives a more precise location. This is done by a top-down pathway, which is used to combine the information from the lower resolution image with the higher resolution image.⁴⁵ The output is a fusion of the features, based on the different scales of the image.

All these steps are done for each visual sensor individually. The next part is creating a 3 dimensional vector space out of this data (compare 5), so the upcoming parts of the model can understand situations like cars which show up in different video sensors at the same time are actually the same car or not. It is done via a transformer architecture. Again this has the advantage of being able to train in parallel and it has the attention concept, so the model can focus on the most important parts of the image, for the specific situation. The output is then more accurate than any single camera prediction.

At this time the information from other sensors which give e.g. the current direction and velocity (called kinematics) are ingested into the model as well (compare 6).

The next important part is the memory of the network. As e.g. signs on the road a few meters ago, which indicate a turn lane, may still provide valid information about the road ahead or rules to follow. Another example of the need for information over time are cars which are temporarily hidden behind other cars, but show up a few moments later again. These two examples are fundamentally different for memory. The first one is a static memory, which is valid for a duration based on movement of the car, while the second one is a dynamic memory, which is in most cases only valid for a short time. The model has to be able to distinguish between these two types of memory. Therefore, the model’s memory is not just updated by time, but also by the car movement. Because if it would just update on time, a red light would clear the cache completely and remove still valid information like the latest road signs. This part of the memory is taken care of by the feature queue (compare 6).

The memory is implemented via a spatial recurrent neural network (compare 7). This lets the model be updated just for points close to the car and not for the whole picture. Further it lets the model decide which spatial parts should be overwritten and which should be kept.⁴⁶ This is done by a gating mechanism, which is a common concept in RNNs.

By now the information inside the model has a dimensionality of 20×80×300×60, which is the amount of pixels in the fused image, the feature channels and the time. The feature channels increased, as the kinematics and positional encoding of the car was added. This is commonly trained in a batch size of 32. Processing this amount of data in a neural net is usually to be too big to hold in a single GPU or even node and is therefore processed via model parallelism and data parallelism.¹⁸

For object detection a YOLO like model is used, which puts a raster over the picture and outputs a probability of specific objects in this area (compare 8). Different kinds of information, like cars, traffic lights or the lane, is predicted by different heads, which makes this a parallel task. Each task is again done by a specialized network, e.g. via classification (cls) or regression (reg). Another advantage of this layout is that each head can be trained and replaced individually, without affecting another one.

During the Tesla AI Day in 2022 Tesla showed additions to the described layout.⁴⁷ One of these additions is predicting human behavior (compare Waymo). Tesla implemented that via a parallelized tree search of possible trajectories of all involved objects, to be able to do this in real time. It uses the state of the own car, all other moving objects, static elements, lanes and traffic controls, to propose possible actions as trajectories, while penalizing undesired outcomes.

Another insight they provided was that they moved to automatically labeling images, which has replaced 5 Million hours of manual labeling for 10k trips with 12 hours on a GPU Cluster. Additionally, they showed a 3D simulator, which can be used to play out the models predictions in a simulated environment (compare Generative models).

Waymo

In a talk from 2024 Drago Anguelov, VP and Head of Research, at Waymo, gave an overview of their current technology stack for autonomous driving for vehicles with a LiDAR system.⁴⁸ The following part references this talk.

The big difference between Waymo and Tesla is that Waymo uses LiDAR and radar additionally to the camera input. Waymo is aiming for a reaction time of 10 Hertz or faster, while having a memory of multiple seconds to decide on the next decision.

The LiDAR is very visual in the mentioned demo. As the 3D output from the model is very detailed and even shows the hand position of pedestrians, as high resolution point clouds. This resolution is needed for e.g. construction workers showing stop signs by hand or a police officer regulation traffic at an intersection.⁴⁹ Waymo is doing a multi-modal pose estimation on this input.⁵⁰⁵¹

In comparison, Zoox labels 30 different attributes for people, to check whether they are distracted or not and changes the behavior of the car accordingly. Further, they are using the pose and movement of persons, to predict their future behavior, including a person waving a car through or waving at a car to stop. This is similar to the example from Waymo with the police officer.⁵²

To efficiently process the sparse output of LiDAR sensors Waymo uses a transformer architecture, called SWFormer, a sparse window transformer for 3D objects.⁵³ The SWFormer is based on Google’s work on the Vision Transformer, which, as mentioned, showed that transformers can be used for image classification, and the Swin Transformer. The Swin Transformer has a hierarchical approach, which gives it the ability to use large scale pictures, by splitting the input into non-overlapping windows, with the “self-attention computation in the new windows cross[ing] the boundaries of the previous windows”, to still be able to detect objects over multiple windows.⁵⁴ The SWFormer adds convolution while creating the windows, which reduces the amount of empty space.

As earlier methods like PointPillar scaled quadratically with the distance and several other methods like sparse submanifold convolutions, did not scale well for sparse data. The SWFormer therefore implemented two adjustments. First it groups windows with the same amount of non-sparse data. Second of all it limited window shifting, which connected windows, to grasp objects which are between windows and instead “relies on multi-scale features […] and multi-scale fusion network to effectively combine these features”.⁵³ Therefore the SWFormer is able to run efficiently on chips which are optimized for matrix multiplications. Waymo then uses a FPN (compare Tesla) to make the output from the SWFormer more precise.

On this output the human behavior is modeled then, as mentioned. This is based on a paper from 2019, which uses “future state-sequence anchors” and predicts a “discrete distribution over the anchors”.⁵⁵ The potential trajectories for known objects are predicted for up to 8 seconds, with increasing likelihood as the timeframe decreases. This was upgraded recently to a transformer architecture via a model called Wayformer, which allows multi-modal input without pre-processing, for input which “exist in temporal and spatial dimensions concurrently”. The output is “a mixture of Gaussians to represent the possible trajectories an agent may take”.⁵⁶

This method is not able to predict interactions between multiple objects. So Waymo developed a transformer model to create joint predictions of K options for N agents, via adding “pairwise potentials”, by putting the interactions in the decoder and output stages. The Wayformer then uses message passing, a way to exchange information among nodes in a graph, to “generate scene-consistent joint futures”.⁵⁷

The final output is an agent which makes decisions for the car it is implemented in. These agents are trained in simulated environments and then deployed on the real vehicle. As mentioned, RL is notoriously hard for defining the reward function in long term settings. Therefore, Waymo augmented their reinforcement learning strategies with imitation learning. This shows an increase in robustness and safety of the model. The paper shows “more than 38% reduction in safety events”.⁵⁸

Another approach Waymo is currently testing is moving to LLMs (compare Generative models). They created a model called MotionLM, which predicts the future state via assigning tokens to connect words to the current trajectories of the cars and then using a transformer, with teacher forcing, to predict the future trajectories as words as well. This can be envisioned as a discussion of cars, about what they do next in dependence of each other. This is trained in a simulated environment then, via ensemble distillation.⁵⁹

LLMs are involved as well in the decision-making process, as they can annotate and describe complicated situations, like parking signs or people jaywalking. This is done via unsupervised training and the output is comparable to traditional annotation with additional semantics. Waymo made progress in automated labeling of input as well,⁶⁰ which lets the model create the bounding boxes and labels automatically on raw input, for static and dynamic objects. Further Waymo started to augment LiDAR data, to increase rare examples in training data.⁶¹

Diffusion is used to increase training data, as it can be used to generate realistic data from a small amount of real data. This is created by real language input.⁶²

Cruise

In a talk from 2023 Cruise showed their so-called continuous learning machine, which implements end-to-end learning, without the need for human intervention. It starts with finding situations where the model did not perform well, finding more examples of such a situation and then doing retraining and testing via simulating the situation. In the end an improved model is then deployed to the car.⁶³ Other companies showed similar approaches.

The technical implementation of this end-to-end pipeline is based on best practices from data engineering and DevOps and a heavy reliance on the physical infrastructure. This structure lets multiple data teams and data scientists work on the same data and model, without interfering with each other, while respecting the dependencies. Cruise calls this their One Platform. This increases their deployment time of a new and better model significantly.⁶³

Cruise has a similar approach for implementing LLMs into the model and developed tooling around that, which lets researchers visualize and explore ROS files.⁶⁴⁶⁵ They developed a pipeline to predict human behavior in traffic fully automated. They are able to evaluate the models prediction and the accuracy, while being trained on a recording, by just letting the recording play out further.

Wayve

Wayve claims to have the first car purely driven by reinforcement learning on a public road in 2018. This was a very simple demo of a car equipped with buttons to give training input and after a few tries is able to navigate a deliberately very easy environment.⁶⁶

Currently, Wayve puts a huge emphasis on simulations, as their captured data has not enough rare examples to train the model on. They are using photorealistic 3D simulations to play out the models predictions in a simulated environment. They can amend the real world data with virtual edge cases, via adding dynamic agents, e.g. adding a person crossing the road unexpectedly or a car switching into the main agents’ lane. The model is then monitored and rated on its behavior. The scene augmentation is done by a model called Prism-1 and the generative AI model is called GAIA.² It is based on a VQ-GAN, an autoregressive transformer and a diffusion model.⁶⁷

Wayve emphasizes that 3D simulations have a huge advantage over simulating the processed data, as this way the whole AI stack is used to create the agent’s decisions, which is exactly how it is happening in the real world.

Further Wayve built a model which annotates the driving of an agent in English language, called LINGO-1.⁶⁸ This is a big step in regard to explainable AI and the public getting more trust in automated vehicles. The recent second version of this model is even able to drive. This is comparable to what Waymo build and Wayve called it the first LLM model to drive on public roads.⁶⁹ This is build on top of the vision stack of the model and creates output as text and as driving action.²

Safety

As mentioned, currently one of the main open challenges in autonomous driving is the reliability of the models decisions in varying conditions and therefore the safety of the passengers and other road users.

Tesla had until April 2024 13 fatal crashes, with one or more persons, where the autopilot feature was involved in the US and “since 2016, the NHTSA has opened more than 40 special Tesla crash investigations in cases where driver systems such as Autopilot were suspected of being used, with 23 crash deaths reported to date”.⁷⁰

Waymo had in 1 million miles “2 crashes and 18 minor contact incidents”, without any human injuries.⁷¹ End of 2023 Waymo reported even better numbers, when they analyzed 7.1 million miles and came to the conclusion that their cars are 6.7 times less likely to be involved in a car crash with injuries, compared to human drivers.⁷² These numbers align with a study they did in 2023 together with the insurance company Swiss Re, which included all claims filled with them.⁷³

This can not be a comparison between Waymo and Tesla though. Waymo operates with different technologies, on a different level of automation and in different (selected) environments, while Tesla being more open with the usage of their self-driving vehicles and allowing interaction by the driver. In a very recent interview Andrej Karpathy stated that Tesla maps out environments via LiDAR during for gathering training data as well though.⁷⁴

Cruise reported 36 crashes in their first 1 million miles. The number is higher than the number Waymo reported for their first 1 million miles. While both operate with Level 4 automation, Cruise operated just in San Francisco, while Waymo operated during that time in Phoenix and Los Angeles.⁷⁵

It is important to note that Waymo was in their first 2 million miles not involved in an accident by other people running a red light, Cruise was involved in 7 accidents of this kind in their first million miles.⁷⁶ This could be an important factor, as it seems Waymos model is able to react to other road users, which do not follow the rules.

A paper from June 2024 looked into the difference between accidents with human drivers and autonomous vehicles with level 2 and level 4 automation. With a dataset of accidents from 2.100 assisted driving vehicles (level 4 with 1.099 samples and level 2 with 1.001 samples) and accidents from 35.113 human-driven vehicles.⁷⁷

The study showed that “accidents of vehicles equipped with Advanced Driving Systems generally have a lower chance of occurring than Human-Driven Vehicles in most of the similar accident scenarios”.⁷⁷ Advanced Driving Systems are in this case level 4 autonomous vehicles. This changes under “dawn/dusk or turning conditions” though, where the autonomous vehicles had a higher chance of being involved in an accident. They attribute the lower performance of autonomous vehicles in these conditions to “sensors and cameras used by AVs may not be able to quickly adapt to changes in lighting conditions”.⁷⁷ Further, the study states that accidents with an AV involved are mostly between cars and just 3% of these accidents have a pedestrian involved, compared to 15% for human driven cars.

A lot of other sources, including the US based NHTSA, which has a standing order to report all accidents with level 2 autonomous vehicles, do not report or normalize on the amount of miles driven of AVs. This is a crucial factor in the comparison of the safety of autonomous vehicles to human drivers though.⁷⁸

Conclusion

Overall the main challenges for autonomous driving are safety, in combination with the flexibility of environments to drive in. Andrej Karpathy mentioned very recently in an interview, Waymo having artificial general intelligence (AGI) level of driving skills already.⁷⁴ Tesla, which is used in a larger variety of environments and is using a different sensor stack, still has safety issues though.

Overall the progress over the last years in the hardware development and AI models is incredible. The transformer architecture gave a huge boost in performance and latency for visual inference (compare AI models). The use of reinforcement learning and simulations to train the models has still room to grow and using simulations is heavily correlated to chip capabilities and availability.

The companies active in the field of autonomous driving are using cutting edge research (compare Waymo) and are investing a large amount of money in hardware (compare Chips). For Waymo, especially the fields of human behavior modeling and spatial reasoning are promising field.⁴⁸ While using LLMs in various parts of the model is an interesting new approach.

Even as the field of autonomous driving got a bit more quiet over the last years and on the Gartner hype cycle in 2022 was hovering in the trough of disillusionment, it rightfully got closer to the slope of enlightenment in 2023, as there was a staggering amount of progress in the last three years.⁷⁹⁸⁰

References