Dr Siddartha Khastgir, head of verification and validation at WMG, University of Warwick, recently discussed the decision to grant public access to the Safety Pool Scenario Database – the world’s largest scenario database for automated vehicles – with ADAS & Autonomous Vehicle International (first published in the January 2023 issue).
In September 2022, leading experts at the UK’s WMG, University of Warwick, and Deepen AI opted to provide credit-based access to the largest public store of scenarios for testing automated vehicles. The Safety Pool Scenario Database features more than 250,000 scenarios.
To date, more than 500 organizations have enrolled in the database, which provides scenarios in different operational design domains that can be leveraged by governments, industry and academia to test and benchmark automated driving systems. Scenarios covered include urban environments, highways and those representing varied environmental conditions, where vehicles perform different maneuvers such as cut-ins and overtaking.
The scenarios are generated using two novel scenario generation methods – knowledge-based and data-based – where scenarios are focused on uncovering failures in automated vehicles as they capture those edge case scenarios. Use cases supported include automated lane keeping systems, low-speed shuttles, urban Level 4, highway ADAS and more.
Users are rewarded with credits for submitting scenarios to the database. Contributions are scored based on the uniqueness of the scenarios and their validity, and corresponding credits are awarded to the contributing organization. These credits can be redeemed to gain access to more scenarios. This system encourages users to contribute to grow the database and make more scenarios available to the self-driving-vehicle community.
The following is an interview conducted between ADAS & Autonomous Vehicle Technology International and Dr Siddartha Khastgir, head of verification and validation at WMG, University of Warwick:
How do scenario databases support the development of ADAS and automated driving?
If we try to prove that automated driving systems (ADS) are safer than human drivers just by driving real-world miles, we would need to drive over 11 billion miles. To put that into perspective, this would be equivalent to driving to the moon and back 20,000 times. This is a lot of work to demonstrate a 20% improvement in safety, and would not be commercially feasible.
As a result, scenario-based testing has quickly become a key method to achieve and demonstrate the safety of ADAS and ADS. Among the pillars of scenario-based testing are the test scenarios used in the testing process.
Given the number of scenarios that will be required to demonstrate safety, a scenario database is an essential part of the safety process for an ADAS or ADS.
A scenario database is a repository of test scenarios that can be used by ADAS and ADS developers to verify and validate these systems.
Some of the most popular scenario databases are the Safety Pool Scenario Database (developed in the UK but with global relevance), Sakura scenario database (Japan), Pegasus scenario database (Germany), StreetWise database (Netherlands) and ADScene (France).
What are the main criteria for AV developers to consider when selecting a scenario database?
The main criteria are ease of use, relevance to the operational design domain (ODD) of the system under test and diversity in test scenarios with respect to ODD.
Each of these factors is key to the success and uptake of a scenario database. Having a database with lots of content should not be the target for the ADS ecosystem. As a community, we need to agree on quality requirements for scenario databases (for each of the three criteria listed above). Such discussions have begun at various international regulatory levels.
What makes Safety Pool different?
The Safety Pool Scenario Database is underpinned by the philosophy that the automated driving ecosystem should not compete on safety. This is a mission that both WMG and Deepen AI all have in common, and it is what brought the two organizations together to collaborate and create the Safety Pool Scenario Database.
Through the Safety Pool Scenario Database, we have created the world’s largest public store of scenarios, intentionally developed to be used globally, not just at a national level.
To promote the pre-competitive aspect of testing the safety of ADS, one of the key features of the Safety Pool Scenario Database is its credit-based scenario exchange mechanism. We have created an ecosystem where organizations are incentivized to share test scenarios with each other. Initially, organizations have access to the public section of the database. When they contribute a scenario, we score the contribution for quality and quantity of the scenarios, and assign credits to the contributing organizations. They can then use these credits to buy other scenarios from the restricted section of the database. No organization can buy their way into the database financially; they must contribute scenarios.
Safety Pool has scenarios covering a diverse set of ODDs (rural, urban, semi-urban, motorway, weather conditions, different actors, etc). This makes it relevant to varied types of ADS developers. Coupled with an ability to search scenarios based on the ODD of the ADS, the testing process becomes more efficient and relevant.
Safety Pool was conceived to meet the needs of a diverse set of stakeholders – not only ADS developers but also OEMs and Tier 1 suppliers. Positioned as a tool to help regulators and type approval bodies as well as researchers, Safety Pool provides ODD-based searching and human-intuitive scenario definition for the test scenarios, making the database more accessible.
With a wide variety of stakeholders in the ADAS and ADS ecosystem, alignment with international standards is a prerequisite to ensure common understanding. The Safety Pool Scenario Database is unique in the sense that it is compliant with the relevant international standards such as ASAM OpenLABEL, ISO 34503 and ASAM OpenSCENARIO. The team at WMG and Deepen AI have played a key leadership role in the creation of these standards.
What are some of the common misconceptions about scenario databases?
It is erroneously assumed that all types of users (ADS developers, regulators, researchers) will use the scenario database in a similar manner. The reality is that user journeys for different types of users will be very different and have varied sets of requirements. For example, a type approval authority or regulator would like to read/understand scenarios (stored in the scenario database) in a human-readable format. In this regard, WMG has led the British Standards Institution (BSI) activity on creation of the BSI Flex 1889 standard, which provides a language for this purpose.
[On the other hand,] an ADS developer performing simulation-based testing would like to have the scenarios in a machine-readable format that can be executed in a simulation platform for virtual testing.
Thus, we have a competing set of demands placed on a scenario database. For Safety Pool, we created a concept that can cater to both sets of users and requirements by having two abstraction levels for a language to describe scenarios.
Another misconception is that all ADS need to be tested for the same set of test scenarios, but in fact every ADS will require a different set of test scenarios depending on its ODD and design. There will be no ‘golden set’ of scenarios for all ADS. We need to create a scenario database that has diversity in scenarios and represents different ODDs – for example, urban, motorway, rural and weather conditions. With more than 250,000 scenarios, Safety Pool covers a wide variety of ODDs to meet this industry need.
In addition to misconceptions, there is still huge confusion in the ADS industry on the meaning/definition of the term ‘test scenarios’ and what constitutes a test scenario. In simple terms, a test scenario should define the behavior of various actors (vehicles, cyclists, pedestrians, etc). At WMG, we have been championing a drive for the industry to agree on terminology, and we actively contribute to various international standards to this end. However, more efforts are needed to drive industry to a consensus to avoid confusion.
The biggest challenge currently is the need to ensure that the scenarios contained in the database offer a representative set of complete scenarios for an ODD of the ADS. Research at WMG currently focuses on creating metrics on scenario completeness as well as requirements on the quality of scenario databases.
What scenarios are of most interest?
Based on our research at WMG, we have coined a concept of hazard-based testing, which focuses on testing how a system fails rather than how a system works. Our focus is on identifying failures.
As a result, we believe that the scenarios that offer the most value are the ones that reveal failures. We have proposed a hybrid approach to scenario generation: 1) data-based scenario generation; 2) knowledge-based scenario generation.
Data-based scenario generation analyzes accident databases, insurance claim records and real-world collected data to identify trends that lead to incidents or near-misses. Knowledge-based scenario generation analyzes the system to understand how its design could lead to failures. Our scenario generation approach has also been incorporated in the EU regulation on automated driving systems that was adopted on August 5, 2022.
To ensure randomization, we treat scenarios at a logical scenario level: all scenarios have parameters, and each parameter has a value range. Depending on the ODD definition of the ADS, we first identify the relevant scenarios (at the logical level) and have intelligent test case generation algorithms that assign values to the parameters, focusing on revealing failures.
How are you working to enhance Safety Pool?
We are continuously working with stakeholders across the ADS ecosystem in the UK and internationally to capture new requirements and user journeys. Currently, we are working with more than 500 organizations worldwide on various user journeys using the Safety Pool Scenario Database.
Regulators, researchers and ADS developers in different countries have a lot of common requirements but also many bespoke needs. Being a public database funded by the UK government allows us to work closely with these partners to understand and implement the enhancements and therefore grow the reach of Safety Pool. END
This interview was first published in the January 2023 issue of ADAS & Autonomous Vehicle International – register for your free subscription, here. Read more on simulation, here.
Don’t miss Siddartha’s presentation Demystifying safety assurance of automated driving systems: The myths and the reality at the ADAS & Autonomous Vehicle Technology Expo Conference, this June in Stuttgart, Germany.
