PPS: Finally, a Consistent Metric to Gauge Lidar Performance

Introduction

Individual performance metrics, such as range, have traditionally been used to define a Lidar system’s overall performance in the market.  As such, too many customers get an incomplete look at a Lidar system’s true capabilities, because a system designed to maximize performance in one dimension (like range) comes at the expense of other key metrics (like resolution).  Said differently, for the four core performance specs that are critical for Lidar to truly support the needs of ADAS/AV (range, field of view, resolution, and refresh rate), maximizing the performance of any one of these specs compromises the other three, thus reducing the system’s overall performance.

At the end of the day, automotive customers are looking for a Lidar sensor that sees a specific minimum distance, across a large FoV, refreshes the scene often, and achieves high resolution – all at the same time. In order to bring clarity and simplicity to the evaluation of the Lidar system’s output, we advocate industry-wide convergence on a simple, impartial metric that can be applied broadly to compare raw performance of competing systems: Points per Second (PPS).  Once established, PPS becomes an incredibly useful metric for automotive OEMs when evaluating like systems. 

As each component of PPS is critically important, let us first examine each of these performance areas individually. 

 

Range

In the past, a significant amount of marketing has focused on ‘range’ as a proxy for Lidar performance.  This is understandable, because being able to detect objects in front of you is of critical importance — particularly when a vehicle is closing on those objects at high speed.  However, this singular focus on detection range led to a press announcement arms race of 200m, then 300m, 450m and even 1000m detections, without mention of how other measures were compromised.  Note that range should always be specified in full daylight (100 kLux equivalent), without applying selective regions of interest and measured at a very high probability (Pd>90%) of detection with a low false alarm rate.

To ensure passenger and pedestrian safety, autonomous vehicles must be able to detect and identify objects at a distance of 200 to 250 m. A vehicle traveling 100 km/hr covers 200 m in about 7 sec and Federal safety standards require a stopping distance of 70 meters for light vehicles (3,500 kg GVW [gross vehicle weight]) — but cars today commonly stop well short of this requirement. However, a generous amount of additional margin in detection range is needed to account for weather / road / passenger-specific conditions. As a result, vehicles need to be able to detect and identify an object, say a small child wearing a dark sweater (of ~10% reflectivity), and have enough reaction time to safely plan and execute a maneuver. Counting these considerations, 200m of range for a normally oriented 10% reflectivity object under bright daylight conditions is generally considered as sufficient to fulfil automotive L2+/ L3 ADAS long range requirements.  

Although range itself is not a component of PPS, it remains a critical evaluation metric for Lidar, so it is best utilized as the normalizing constant when comparing similar systems (e.g. evaluating the PPS of three systems that operate at 100m). 

 

Field of View (FoV)

Intuitively speaking, a larger FoV leads to more complete coverage, which is highly desired. While there is no such thing as too much coverage, the sky in particular contains very little information pertinent to autonomous navigation, and it does not make much sense to waste power tracking an area with no relevant data. In the past, long-range Lidar customers have typically set the Vertical FoV to match the largest slopes and highest objects that need to be navigated, and set the Horizontal FoV to match the number of lanes that need to be seen (and be able to detect cut-ins from vehicles that wander into the vehicle’s lanes). For a short-range Lidar, FoV requirements are typically tailored to the desired form-factor for the vehicle with as close to 360 degrees of coverage as possible to achieve a cocooning effect, preventing blind spots and ensuring the vehicle does not collide with any objects or obstacles. 

 

Resolution

The practical requirement around angular resolution is a sufficient resolution to resolve objects at a long range (e.g. > 150m). While 0.1 x 0.1 degrees has been the standard to date, the industry is quickly converging around requiring higher vertical resolution (ex: 0.1 x 0.05) at long ranges in order to detect low profile obstacles (such as black tires) as well as potholes in the path of the vehicle.

Angular resolution becomes difficult to compare between systems because resolution is often not reported uniformly across the FoV.  With most scanning technologies (including MEMS-based and macro-mechanical designs), Lidar returns start to diverge at a long distance, so achieving and maintaining resolution at range becomes difficult.  To overcome this, some systems promote ‘regions of interest’ (ROIs) where overlapping beams of light are diverted into a small region and spatial resolution enhancement techniques are used to obtain temporarily higher resolution within the ROI.  The reality is that this increase in resolution within this region comes with a severe tradeoff in performance parameters (range, resolution, refresh rate) in the rest of the FoV, which is why we advocate for high angular resolution that can be maintained uniformly across the required FoV.

 

Refresh Rate

Until 2018, 10 Hz of sampling rate (one data update every 1/10th of a second) had been the standard in the industry.  As Lidar data tends to be ‘lighter’ than camera data, this rate was deemed sufficient and relatively easy on data bandwidth from a processing pipeline perspective.  However, given recent advances in processing — combined with incidents such as the one in 2018 involving Uber, which has increased pressure from NTSB, NHTSA, NCAP and other regulatory bodies — safety has become the #1 concern across both ADAS and AV automotive applications.  The industry, especially for automotive series production programs, is converging on higher refresh rates as standard, based on first-hand experience from supervised driving data collected over the past few years. Higher refresh rates of 20 Hz or more are quickly becoming the standard as ADAS and AV systems make incremental progress to operate at higher speeds. 

 

The Need to Understand Raw Performance / Output

To summarize, the automotive industry needs a Lidar sensor with adequate detection range across a wide FoV while achieving high angular resolution at a high frame rate – all at the same time. In parallel, it needs to simultaneously extract all useful information from the scene (which is achieved in Lidar science using the concept of multiple echoes or returns).

Among Lidars achieving the same equivalent range (for example a set of sensors competing for a Highway Pilot application- typically 200m of range @ 10%), the universal metric of raw performance comparison is calculated as follows:

 

PPS or Points/Second = (FoV * Refresh Rate* nReturns)/(HRes x VRes). 

 

Sabbir Rangwalla, contributing editor for Forbes, discussed price/performance metrics in his recent articles, Money for Somethin? and Money for Everythin’, and arrived at the same conclusion: Points Per Second (PPS) is an effective way to measure system performance for equivalent Lidar systems.   By comparing different products on the market using this standard, it was clear to Sabbir that some systems offer more ‘raw performance’ than others. 

NOTE:  Any sensor that would be selected for an automotive production program is also going to have to meet the customer’s requirements for size, weight, power, cost, cooling and compliance (SwaP-C3), but that is the subject of another article; our first priority is establishing a consistent metric for performance.

Why is Points Per Second the right metric of raw performance?  Data in the form of returned ‘points’ translate to richness of the sensor output, which in turn translates to the level of confidence that the perception systems require in order to maneuver safely.   The advantages of Gen 4 Lidar that utilizes CMOS SPAD and VCSEL technology with a true global shutter flash – such as the system that we are developing at Sense Photonics (referenced in a recent press release) will always deliver superior 3D data over systems using scanning technology – even the ones that refer to themselves as “solid state”.  Whether mechanically scanning (using galvos, MEMS, polygonal mirror) or electronically scanning, these systems have inherent drawbacks when imaging high-velocity road objects resulting in motion blur. Their laser only illuminates a small ‘spot’ instead of the entire scene requiring the laser to scan the entire FoV. In order to try and overcome the motion blur, these systems can operate at higher frame rates but only at the expense of reduced range and/or resolution causing PPS to change unpredictably based on their architectural limitations. However, our Sense Flash LiDAR architecture maintains range and resolution regardless of frame rate resulting in a predictable way to track PPS.

 

A Balloon of Points

It is instructive to think of the PPS bang-for-buck metric as a ‘solid putty/balloon of points’ whose size and heft is proportional to the size of the PPS, overlaid on a 3D grid representing X, Y, Z. The balloon can be squeezed in different ways to get coverage matching the use case across the grid. The heft signifies resolution (wide coverage without adequate resolution is not useful). With more PPS, a beefier balloon is possible with a wider and denser coverage. With fewer PPS, the customer has to pick one or the other, or sacrifice another key metric such as refresh rate.

Figure: PPS Visualized as a solid balloon, this comparison shows that high PPS (left) systems have wider and denser coverage than low PPS (right) systems.  The heftier balloon signifies a Lidar’s bang for buck (assuming a similar cost)

Embedded within the hefty balloon analogy – although somewhat difficult to display visually – the PPS metric also encompasses the time element that signifies how frequently information is sampled through refresh rate:  more frequent = safer (up to a point of diminishing returns, which is around 30 Hz for practical automotive applications). 

With the PPS spec serving as the basis of a holistic performance comparison that can be applied to any Lidar, it becomes hard for a Lidar supplier to hide the flaws of an underpowered system.

 

Looking Ahead

PPS is an impartial metric to measure the raw output of Lidar systems.  If the Lidar industry can coalesce around this metric as a standard, derived utility metrics such as PPS/Dollar and PPS/Watt will represent better ways for customers to holistically compare product performance.

 

Aravind Ratnam

Head of Product
Aravind leads Sense’s product management team with a background steeped in automotive, software, lasers, optics, and product management. Previously, he served as product management head for connected car solutions at Wind River, an embedded systems company that was part of Intel’s IoT group, where he successfully led corporate strategy and execution of the Wind River automotive portfolio.

WANT TO ELEVATE YOUR SENSES?

Reach out to our expert team members. 

Contact Us