 |
|
Fundamentally, the goal of performance testing is to measure accuracy, speed and reliability so that all of these things may be improved. No system is error free and this includes biometric products. Consequently, performance testing standards establish methods and procedures for how performance should be evaluated and reported, sometimes including specific thresholds for how well a product should perform.
In the case of the ILO SID, for instance, the thresholds for false accept rate (i.e., the rate at which false claims of identity are confirmed) and false reject rate (i.e., the rate at which true claims of identity are rejected) are both 1%. For any product to be approved, it must achieve an average false reject rate when interoperating with all other ILO SID approved products of 1% or less at a false accept rate of 1%.
Standard measures used to evaluate a biometric product’s performance include:
- Enrolment: failure to enroll rate (FTE) and failure to acquire rate (FTA)
- Matching: false match rate (FMR) and false non-match rate (FNMR)
- Decision: false accept rate (FAR) and false reject rate (FRR)
However, there are many other measures of how a product performs. Similarly, products can be tested in a number of ways and under a number of different conditions.
Traditionally, performance testing takes one of three forms:
- Technology evaluation: evaluates the performance of a biometric product (e.g., algorithm and/or sensor) using a fixed and reusable data set (i.e., offline testing).
- Scenario evaluation: evaluates the performance of a biometric product in a simulated application (e.g., enrollment), where data is collected in real time from a given population.
- Operational evaluation: evaluates how the biometric product performs when integrated into live operations (i.e., online testing) and how it has altered, negatively and positively, the operational environment and outcomes.
Ideally, performance testing should be a three-step process starting with technology evaluation, continuing with scenario evaluation and finishing with operational evaluation. Clearly, testing for operational performance before evaluating the technology would be impractical and inefficient. Also, during these sequential steps, modifications and further tests and research can be completed to improve the product’s performance. Prior to system deployment, technology and scenario testing are the most important and bring the greatest benefits. After system deployment, operational testing will indicate how well the deployed system is working, although due to the uncontrolled nature of a system in the real world, it is seldom possible to determine all of the standard performance measures. Despite this all three forms of testing have benefit under the appropriate circumstances.
At Bion Biometrics, we have the expertise to design and conduct all forms of performance testing, using standard and customized measures. We also bring our clients the added benefit of having firsthand knowledge of biometric product design and integration. We can identify where your product needs improving, but we can also provide you with professional advice on how to improve the technology (e.g., algorithms, sensors and software) or make changes in integration or operational procedures. See our consulting services for further information.
|
