The long way to verify and certify autonomous vehicles

25 April 2018

The main challenge to verify and certify a level 5 vehicle is to show sufficient proof that the system works well in all situations. Actual calculations show that around 240 million kilometers are necessary. This huge number makes it also clear that no single OEM will drive this with real cars. Therefore new, more time and cost efficient and reliable testing and verification procedures and processes are necessary and required. In addition, more simulation and Software-in-the-loop (SIL) would have to be used as today. At the moment the ratio between HiL plus real testing to SiL is 80:20. This has to be turned to 80% testing using simulations and SiL and only 20% of HiL and test drives on the roads.

There are many criteria to be considered when talking about the “complete thing”, such as that the processes and the complete tool-chain for testing and verification. It is also important to consider that these would also have to be certified as well as the infrastructure which will be used, plus that real data would needed. In this post I would like to discuss only the data, necessary to run the simulations and the SiL`s for verification of single ADAS features as well as for the complete ADAS system and the vehicle. From level 3 upwards you cannot consider only one ADAS feature, such as ACC, LKA, BSD, high-way pilot, etc. as sensor data fusion is required and that various ADAS features have to work together to make the right decision. In addition all the interfaces to other vehicle systems and sub-systems.

Let´s go back to the “Data“…

It is vital and very important to acquire as much real sensory data as possible. I have personally been involved in various data acquisition projects where we had operated several test cars worldwide and often came across some very interesting ideas. One good approach seen by a client and partner was to equip the test cars with a full set of all sensors they are considering and to start a project to acquire data with an aim to collect real sensory data and not test it. Each car had produced a large volume of data of around 17-20 TB each day. You can imagine the volume of data that could be collected at the end of such a campaign, considering it could involve over 10 vehicles. It will be easily around 20 PB if not more which constitutes a large number of information.

So far so good. Thinking about such data volumes the next question should that arises is “how these data will be structured so that it is possible to select specific traffic scenes for simulation or a HiL or SiL run?”

To structure the raw sensor data so called meta data have to be added and synchronized. When I am asked how to do this I am explaining this with a phased approach as explained below:

Phase 1: Adding GPS data as well as weather data to the raw sensor data. This is quite easy as the cars are equipped with GPS and mostly also with differential GPS. Adding weather data is also quite simple and can run automatically.

Phase 2: We tend to call this “Pre-Labeling”. You are using a tool which will be operated by the co-driver. This person does have a tablet with various icons (configurable). During test drive the person clicks on the icons such as trucks, exits, traffic signs, pedestrians, rain, fog, etc.. This information will be directly synchronized with the raw sensor data.

Phase 3: Manual labeling. This will be done using the camera/video data and a commercial “off the shelf” software which allows one to mark every object, its position and size. As this task is very time and effort consuming usually only 5% of the acquired data will be manually labelled.

To give you a simple example and numerical information of how much effort is required let´s see some average numbers. Exact numbers will obviously always depend on how many objects have to be labelled, which objects, how many frames, etc but let´s see some average numbers. To label 1 minute of a video, 300 minutes for the annotation of 2D data are necessary and up to 3.720 minutes for 2D/3D. This means one person needs to spend up to 62 hours to add solid meta information to the raw sensor data. You think this is not necessary? How will you verify your functions if you do not have the date with this information? Today you are going to test the feature on the real roads with real cars. Just remember the 240 million kilometers mentioned in the beginning of this article.. So in the future there is no way around this..

Having said all this the next question is “how do I get the manual labeling much more efficient in time and cost so I can increase my data base with the detailed information?”

We do have a possible answer and solution. Let talk about it.

Article by Michael Zimmermann

Think we can help you? Send us an email at, or follow us on Twitter and LinkedIn where we release educational and informative content every day.

Related posts

Get in touch with us

    7 Dale Street, Leamington Spa, England, CV32 5HH
    +44 (0)7545942389


    a monthly summary of the significant changes in the mobility ecosystem

      TO TOP