Simulation

The Virtual World Helps Waymo Learn Advanced Real-World Driving Skills

In simulation, engineers identify the most challenging situations Waymo has encountered on public roads, and turn them into virtual scenarios for Waymo’s autonomous driving software to practice in simulation.

The simulator can replay the real-world miles Waymo vehicles have driven with each new software version. They can also build completely new realistic virtual scenarios for Waymo’s software to be tested against.

Each day, as many as 25,000 virtual Waymo autonomous vehicles drive up to 10 million miles in simulation every day, which is like driving to the moon and back 15 times.

Like an athlete visualizing the playing field, Waymo virtual vehicles are constantly refining old skills and testing out new maneuvers that help them navigate the real world safely.

For example: at the corner of South Longmore Street and West Southern Avenue in Mesa, Arizona, there’s a flashing yellow arrow for left turns. This type of intersection can be tricky for humans and autonomous vehicles alike—drivers must move into a five-lane intersection and then find a gap in oncoming traffic. A left turn made too early may pose a hazard for oncoming traffic. A turn made too late may frustrate drivers behind.

Simulation lets Waymo turn a single real-world encounter like this into thousands of opportunities to practice and master a skill.

Today, the vast majority of improvements and refinements Waymo makes to its software are based on things it picks up in simulation, before a single real-world car has encountered it on the road.

So how does an encounter on public roads turn into a scenario Waymo can test hundreds of thousands of times in simulation? Here’s the anatomy of a simulation scenario:

Waymo can recreate a highly-detailed, realistic virtual version of the East Valley

Step 1: Start with a Highly-Detailed Vision of the World

Using a powerful suite of custom-built sensors, engineers build a virtual replica of the intersection, complete with identical dimensions, lanes, curbs, and traffic lights. In simulation, Waymo can focus on the most challenging interactions—flashing yellow signals, wrong-way drivers, or nimble cyclists—rather than on monotonous highway miles.

In simulation, Waymo can practice driving the same intersection, in the same driving conditions thousands of times, with different vehicles from its fleet. In this image, Waymo is simulating driving the intersection with one of its Lexus vehicles.

Step 2: Drive, Drive, and Redrive

With this flashing yellow left turn now digitized in its virtual world, Waymo’s software can practice this scenario thousands of times over in a variety of driving conditions. Waymo can practice this new skill on every flashing yellow arrow it has ever come across, in order to improve its software even faster.

Through a process called fuzzing, Waymo can alter the speed, trajectory, and position of objects on these virtual streets.

Step 3: Create Thousands of Variations

Next, Waymo can multiply this one tricky left turn to explore thousands of variable scenarios and “what if’s?” Through a process called fuzzing, Waymo alters the speed of oncoming vehicles and the timing of traffic lights to make sure its vehicles can still find a safe gap in traffic. The scene can be made more complex by adding simulated pedestrians, motorcycles, or even joggers zig-zagging across the street—all to see how that might change Waymo’s driving.

Waymo vehicle simulation. To make a scene more complex, we can add vehicles, pedestrians, and cyclists that never existed in the original scene.

Step 4: Validate and Iterate

Success: Waymo’s autonomous driving technology has learned how to turn confidently at a flashing yellow arrow. That new skill becomes part of Waymo’s permanent knowledge base, shared with every vehicle across the fleet. Waymo uses real-world driving and its private closed-course testing facility to validate its simulated experience. And then the cycle begins again. Each of these eventful simulator miles is guiding Waymo to what everyone wants: billions of safe and uneventful miles in the real world.