Autonomous cars promise to change the face of transportation, offering many more mobility options for individual motorists and companies alike. In moving forward with this new technology, our automotive clients have a very important challenge to overcome: processing the petabytes of data that gets collected during the development and testing of autonomous driving systems.
KPIs have always been important to car makers. They are necessary to attain road approvals and to track key competitive differentiators. With autonomous cars, however, car makers are accumulating – and must find ways to process and manage – 10, 20, sometimes 30 times the data as before.
As a result, they need much more efficient data analysis tools that can help them analyze the data for the specific autonomous car KPIs they are looking for. To make this happen, they need to take the following four steps:
- Make sure the car’s sensors are working. There are typically eight to 12 sensor systems in an autonomous vehicle test car. It’s important to look at the data at the very beginning of the workflow by checking the KPIs to ensure that the system works properly. Some of the KPIs car testers evaluate include the following: vehicle operations, safety, environmental impact and in-car network efficiency.
- Scale the workflow to process the data. Traditional architectures of automotive frameworks are not suited for the large-scale data processing workloads required for testing the algorithms used in autonomous car tests. In using traditional data storage methods, vehicle test data gets stored in NAS-based storage and gets then transferred to workstations, where engineers test algorithms under development. This process has two downsides:
- Large amounts of data must be moved, requiring considerable time and network bandwidth.
- Individual workstations do not offer the massive computing power required to return test results fast enough.
Today, testers are extracting each frame of video data with its associated Radar, Lidar and sensor data by using open source Hadoop. The major benefit of Hadoop is that it scales processing and storage to hundreds of petabytes. This makes it a perfect environment for testing autonomous driving systems.
- Make the most of data analytics. In processing petabytes of automotive data, we have to look at how we present the data to higher level services. New data analysis tools can read different automotive formats to give us proper levels of access to the metadata and data. For example, say we have 700 video recordings, we now have tools that can pinpoint footage from the front-right camera alone to show how the car performed making right-hand turns. We can also use the footage to determine the accuracy of a model depicting the autonomous car’s perception of its ambient physical surroundings .
- Run the data analysis. In the end, we want to use data analysis tools to give R&D engineers a complete view of how the car has performed in the field. We want to generate information on how the systems will react under normal driving conditions.
Overcoming these data analysis challenges is critical. Manufacturers can’t obtain permits for releasing their cars until they can show that the cars performed up to certain standards in road tests. And when autonomous cars do start to hit the roadways in the next few years, auto manufacturers might need the KPIs they generated in testing. A few accidents are inevitable and, when questions arise, car makers can use KPIs to show the authorities, insurance companies and the general public how the cars were tested and that proper due diligence was performed.
Right now, there’s some distrust among the driving public of autonomous cars. It will take a massive public relations effort to convince consumers that autonomous cars are safer than traditional manually-driven cars. But proving that case all starts with the ability to process the data more efficiently
Autonomous cars promise to change the face of transportation, offering many more mobility options for individual motorists and companies alike. In moving forward with this new technology, our automotive clients have a very important challenge to overcome: processing the petabytes of data that gets collected during the development and testing of autonomous driving systems.
KPIs have always been important to car makers. They are necessary to attain road approvals and to track key competitive differentiators. With autonomous cars, however, car makers are accumulating – and must find ways to process and manage – 10, 20, sometimes 30 times the data as before.
As a result, they need much more efficient data analysis tools that can help them analyze the data for the specific autonomous car KPIs they are looking for. To make this happen, they need to take the following four steps:
- Make sure the car’s sensors are working. There are typically eight to 12 sensor systems in an autonomous vehicle test car. It’s important to look at the data at the very beginning of the workflow by checking the KPIs to ensure that the system works properly. Some of the KPIs car testers evaluate include the following: vehicle operations, safety, environmental impact and in-car network efficiency.
- Scale the workflow to process the data. Traditional architectures of automotive frameworks are not suited for the large-scale data processing workloads required for testing the algorithms used in autonomous car tests. In using traditional data storage methods, vehicle test data gets stored in NAS-based storage and gets then transferred to workstations, where engineers test algorithms under development. This process has two downsides:
- Large amounts of data must be moved, requiring considerable time and network bandwidth.
- Individual workstations do not offer the massive computing power required to return test results fast enough.
Today, testers are extracting each frame of video data with its associated Radar, Lidar and sensor data by using open source Hadoop. The major benefit of Hadoop is that it scales processing and storage to hundreds of petabytes. This makes it a perfect environment for testing autonomous driving systems.
- Make the most of data analytics. In processing petabytes of automotive data, we have to look at how we present the data to higher level services. New data analysis tools can read different automotive formats to give us proper levels of access to the metadata and data. For example, say we have 700 video recordings, we now have tools that can pinpoint footage from the front-right camera alone to show how the car performed making right-hand turns. We can also use the footage to determine the accuracy of a model depicting the autonomous car’s perception of its ambient physical surroundings .
- Run the data analysis. In the end, we want to use data analysis tools to give R&D engineers a complete view of how the car has performed in the field. We want to generate information on how the systems will react under normal driving conditions.
Overcoming these data analysis challenges is critical. Manufacturers can’t obtain permits for releasing their cars until they can show that the cars performed up to certain standards in road tests. And when autonomous cars do start to hit the roadways in the next few years, auto manufacturers might need the KPIs they generated in testing. A few accidents are inevitable and, when questions arise, car makers can use KPIs to show the authorities, insurance companies and the general public how the cars were tested and that proper due diligence was performed.
Right now, there’s some distrust among the driving public of autonomous cars. It will take a massive public relations effort to convince consumers that autonomous cars are safer than traditional manually-driven cars. But proving that case all starts with the ability to process the data more efficiently
