Message Sent

2022 Biometric Rally Results

The 2022 Biometric Technology Rally (2022 Rally) challenged industry to produce face recognition systems capable of autonomously identifying small groups of people passing through the system.​ These systems were also expected to maintain the privacy of people not explicitly using the system.

Four acquisition systems and ten matching systems were evaluated based on their efficiency, user satisfaction, effectiveness, and privacy. A total of 575 volunteers diverse in race, gender, age, and skin tone passed through each acquisition system in a group of 2 and a group of 4 while 8-10 others passed around the system via adjacent bypass lanes. Acquisition systems were tasked with capturing one best photo per person passing through the system. Matching systems had to identify the individual in each submitted photo​.

Performance was measured separately by group size. Efficiency (transaction times) and user satisfaction (positive ratings) were measured only for acquisition systems. Privacy was measured for each unique combination of acquisition and matching systems as the proportion of images that matched to any volunteers that passed around the system.

Two effectiveness metrics were also calculated for each system combination:

  • Operational Focused True Identification Rate (TIR) - the performance of an acquisition and matching system combination.​​
  • Matching Focused True Identification Rate (matching-TIR) - the performance of an acquisition and matching system combination, discounting any failures of the acquisition system to submit images of suitable quality for matching.​

Effectiveness metrics are also reported disaggregated by race, gender, and skin tone to measure equitability.​​​​

For further information about the 2022 Biometric Technology Rally metrics and test design, see the technical webinar (pdf). Select performance results are available below. To comply with information sharing agreements between DHS and the various system providers, all system names are aliased. If interested in contacting the system providers, please click here to send an email to DHS S&T.

Comprehensive effectiveness results are available for download here and comprehensive privacy results are available for download here.

The 2022 Rally assessed the efficiency, satisfaction, effectiveness, equitability, and privacy of 4 biometric face acquisition systems and 10 biometric face matching systems. The infographics below show insights on efficiency, satisfaction, effectiveness, and privacy. Please see the other portions of the results section of this website for information about the demographic equitability of these systems.

Group Processing Applications

Overview of the 2022 Rally

Key findings from the 2022 Rally

Privacy Performance

Efficiency

Groups of Two

Efficiency was measured as the average transaction time required to process each volunteer in a group.

All acquisition systems met the efficiency goal of a per volunteer average transaction time of under 3 seconds for groups of 2 volunteers.

GROUPS OF
2 4

Satisfaction

Groups of Two

Volunteers rated their satisfaction with each acquisition system immediately after use. The acquisition system’s satisfaction rating is the proportion of test volunteers who provided positive responses after using the acquisition system.

One acquisition system, Longs, met the goal of over 95% positive satisfaction for groups of 2 volunteers.​

GROUPS OF
2 4

Effectiveness - TIR

Groups of 2 Operational Focus Overall

Acquisition System
Bison Longs Wilson Borah
Matching System
Kenai 97.4● 96.5 93.2 74.1
Miami 97.4● 96.5 93.2 74.1
Tioga 97.4● 96.5 93.2 73.9
Mill 97.4● 96.3 93.2 73.4
Bronx 97.0 96.3 93.0 73.6
Grant 97.4● 96.0 93.0 73.0
Hop 96.9 95.8 92.8 73.7
Entiat 96.7 95.5 92.3 73.7
Flag 97.2 93.4 93.0 72.3
Row 83.7 83.8 79.2 62.4
Table legend
Max● >99% >95% >90% ≤90%
Hover over each TIR value to see detailed results.

True Identification Rate (TIR) measures the percentage of volunteers using the system that were correctly identified.

Seventeen (17) system combinations met the >95% Rally TIR threshold over all volunteers in groups of 2. These system combinations all included acquisition systems Bison or Longs.

Groups of 2 Operational Focus Female

Acquisition System
Bison Longs Wilson Borah
Matching System
Kenai 98.6● 96.6 93.5 71.2
Miami 98.6● 96.6 93.5 71.2
Tioga 98.6● 96.6 93.5 70.8
Mill 98.6● 96.6 93.5 69.8
Bronx 98.0 96.3 93.2 70.2
Grant 98.6● 95.9 93.2 69.2
Hop 98.0 95.9 93.5 70.8
Entiat 97.6 95.6 92.2 70.5
Flag 98.3 92.9 93.5 69.2
Row 85.1 85.0 80.6 62.0
Table legend
Max● >99% >95% >90% ≤90%
Hover over each TIR value to see detailed results.

True Identification Rate (TIR) measures the percentage of volunteers using the system that were correctly identified.

Seventeen (17) system combinations met the >95% Rally TIR threshold for volunteers in groups of 2 who self-reported as female. These system combinations all included acquisition systems Bison or Longs.

Groups of 2 Operational Focus Male

Acquisition System
Bison Longs Wilson Borah
Matching System
Kenai 96.0 96.4● 92.8 77.3
Miami 96.0 96.4● 92.8 77.3
Tioga 96.0 96.4● 92.8 77.3
Mill 96.0 96.0 92.8 77.3
Bronx 96.0 96.4● 92.8 77.3
Grant 96.0 96.0 92.8 77.3
Hop 95.7 95.7 92.1 77.0
Entiat 95.7 95.3 92.4 77.3
Flag 96.0 93.9 92.4 75.9
Row 82.0 82.7 77.6 62.9
Table legend
Max● >99% >95% >90% ≤90%
Hover over each TIR value to see detailed results.

True Identification Rate (TIR) measures the percentage of volunteers using the system that were correctly identified.

Seventeen (17) system combinations met the >95% Rally TIR threshold for volunteers in groups of 2 who self-reported as male. These system combinations all included acquisition systems Bison or Longs.

Groups of 2 Operational Focus Asian

Acquisition System
Bison Longs Wilson Borah
Matching System
Kenai 95.4 92.2 96.9● 72.3
Miami 95.4 92.2 96.9● 72.3
Tioga 95.4 92.2 96.9● 72.3
Mill 95.4 92.2 96.9● 70.8
Bronx 95.4 92.2 96.9● 70.8
Grant 95.4 90.6 96.9● 70.8
Hop 95.4 92.2 96.9● 70.8
Entiat 95.4 92.2 96.9● 70.8
Flag 93.8 92.2 96.9● 70.8
Row 86.2 84.4 86.2 64.6
Table legend
Max● >99% >95% >90% ≤90%
Hover over each TIR value to see detailed results.

True Identification Rate (TIR) measures the percentage of volunteers using the system that were correctly identified.

Seventeen (17) system combinations met the >95% Rally TIR threshold for volunteers in groups of 2 who self-reported as Asian. These system combinations all included acquisition systems Bison or Wilson.

Groups of 2 Operational Focus Black

Acquisition System
Bison Longs Wilson Borah
Matching System
Kenai 98.1● 95.5 92.9 70.3
Miami 98.1● 95.5 92.9 70.3
Tioga 98.1● 95.5 92.9 70.3
Mill 98.1● 95.5 92.9 69.9
Bronx 98.1● 95.2 92.9 69.5
Grant 98.1● 94.8 92.5 69.5
Hop 97.4 94.4 92.2 70.3
Entiat 97.4 94.1 91.8 70.3
Flag 98.1● 92.6 92.9 68.4
Row 84.0 82.9 77.6 59.1
Table legend
Max● >99% >95% >90% ≤90%
Hover over each TIR value to see detailed results.

True Identification Rate (TIR) measures the percentage of volunteers using the system that were correctly identified.

Fourteen (14) system combinations met the >95% Rally TIR threshold for volunteers in groups of 2 who self-reported as Black. These system combinations all included acquisition systems Bison or Longs.

Groups of 2 Operational Focus White

Acquisition System
Bison Longs Wilson Borah
Matching System
Kenai 97.2 99.1● 92.6 78.7
Miami 97.2 99.1● 92.6 78.7
Tioga 97.2 99.1● 92.6 78.2
Mill 97.2 98.6 92.6 77.8
Bronx 96.3 99.1● 92.1 78.7
Grant 97.2 99.1● 92.6 77.3
Hop 96.8 98.6 92.6 78.2
Entiat 96.3 98.1 91.6 78.2
Flag 97.2 94.4 92.1 76.9
Row 83.3 85.6 80.0 65.3
Table legend
Max● >99% >95% >90% ≤90%
Hover over each TIR value to see detailed results.

True Identification Rate (TIR) measures the percentage of volunteers using the system that were correctly identified.

Five (5) system combinations met the >99% Rally TIR goal for volunteers in groups of 2 who self reported as White. These system combinations all included acquisition system Longs. Twelve (12) more system combinations met the >95% Rally TIR threshold.

Grups of 2 Operational Focus Darker Skin

Acquisition System
Bison Longs Wilson Borah
Matching System
Kenai 98.4● 94.7 91.5 70.9
Miami 98.4● 94.7 91.5 70.9
Tioga 98.4● 94.7 91.5 70.9
Mill 98.4● 94.7 91.5 70.4
Bronx 98.4● 94.2 91.5 69.8
Grant 98.4● 93.7 91.0 69.8
Hop 97.4 93.1 90.4 70.9
Entiat 97.4 92.6 89.9 70.9
Flag 98.4● 91.5 91.5 67.7
Row 83.6 82.0 75.5 61.4
Table legend
Max● >99% >95% >90% ≤90%
Hover over each TIR value to see detailed results.

True Identification Rate (TIR) measures the percentage of volunteers using the system that were correctly identified.

Nine (9) system combinations met the >95% Rally TIR threshold for volunteers in groups of 2 with darker skin tone. These system combinations all included acquisition system Bison.

Groups of 2 Operational Focus Lighter Skin

Acquisition System
Bison Longs Wilson Borah
Matching System
Kenai 96.3● 96.3 93.1 75.1
Miami 96.3● 96.3 93.1 75.1
Tioga 96.3● 96.3 93.1 74.6
Mill 96.3● 96.3 93.1 74.1
Bronx 95.8 96.3 92.6 74.6
Grant 96.3● 96.3 93.1 73.0
Hop 95.8 95.7 93.1 74.1
Entiat 95.8 95.7 92.6 74.1
Flag 96.3● 93.1 92.6 74.1
Row 83.6 84.6 81.9 64.0
Table legend
Max● >99% >95% >90% ≤90%
Hover over each TIR value to see detailed results.

True Identification Rate (TIR) measures the percentage of volunteers using the system that were correctly identified.

Seventeen (17) system combinations met the >95% Rally TIR threshold for volunteers in groups of 2 with lighter skin tone. These system combinations all included acquisition systems Bison and Longs.

GROUPS OF
2 4
SELECT DEMOGRAPHIC GROUP

Effectiveness - Matching-TIR

Groups of 2 Matching Focus Overall

Acquisition System
Bison Longs Wilson Borah
Matching System
Kenai 99.8● 99.8 99.8 99.5
Miami 99.8● 99.8 99.8 99.5
Tioga 99.8● 99.8 99.8 99.5
Mill 99.8● 99.8 99.8 98.6
Bronx 99.5 99.6 99.6 98.8
Grant 99.8● 99.3 99.8 98.4
Hop 99.3 99.1 99.4 99.3
Entiat 99.1 98.7 99.1 99.1
Flag 99.8 99.8 99.8 98.8
Row 87.1 87.3 86.5 86.5
Table legend
Max● >99% >95% >90% ≤90%
Hover over each TIR value to see detailed results.

Matching TIR measures the percentage of volunteers with successfully acquired images that were correctly identified.

Discounting failures to acquire high quality images, thirty-one (31) system combinations met the >99% Rally matching-TIR goal over all volunteers in groups of 2. An additional five (5) system combinations met the >95% Rally TIR threshold.

Groups of 2 Matching Focus Female

Acquisition System
Bison Longs Wilson Borah
Matching System
Kenai 99.7● 99.6 99.6 99.1
Miami 99.7● 99.6 99.6 99.1
Tioga 99.7● 99.6 99.6 99.1
Mill 99.7● 99.6 99.6 97.2
Bronx 99.0 99.3 99.3 97.6
Grant 99.7● 98.9 99.6 96.7
Hop 99.0 98.9 99.6 99.1
Entiat 98.6 98.6 98.5 98.1
Flag 99.7 99.6 99.6 98.1
Row 88.1 88.0 87.5 88.0
Table legend
Max● >99% >95% >90% ≤90%
Hover over each TIR value to see detailed results.

Matching TIR measures the percentage of volunteers with successfully acquired images that were correctly identified.

Discounting failures to acquire high quality images, twenty-four (24) system combinations met the >99% Rally matching-TIR goal for volunteers in groups of 2 who self-reported as female. An additional twelve (12) system combinations met the >95% Rally matching-TIR threshold. Matching systems Kenai, Miami and Tioga met the goal with all 4 acquisition systems.

Groups of 2 Matching Focus Male

Acquisition System
Bison Longs Wilson Borah
Matching System
Kenai 100.0● 100.0● 100.0● 100.0●
Miami 100.0● 100.0● 100.0● 100.0●
Tioga 100.0● 100.0● 100.0● 100.0●
Mill 100.0● 100.0● 100.0● 100.0●
Bronx 100.0● 100.0● 100.0● 100.0●
Grant 100.0● 99.6 100.0● 100.0●
Hop 99.6 99.3 99.2 99.5
Entiat 99.6 98.9 99.6 100.0●
Flag 100.0● 100.0● 100.0● 99.5
Row 86.0 86.4 85.3 85.0
Table legend
Max● >99% >95% >90% ≤90%
Hover over each TIR value to see detailed results.

Matching TIR measures the percentage of volunteers with successfully acquired images that were correctly identified.

Discounting failures to acquire high quality images, thirty-five (35) system combinations met the >99% Rally matching-TIR goal for volunteers in groups of 2 who self-reported as male.

Groups of 2 Matching Focus Asian

Acquisition System
Bison Longs Wilson Borah
Matching System
Kenai 100.0● 100.0● 100.0● 100.0●
Miami 100.0● 100.0● 100.0● 100.0●
Tioga 100.0● 100.0● 100.0● 100.0●
Mill 100.0● 100.0● 100.0● 97.9
Bronx 100.0● 100.0● 100.0● 97.9
Grant 100.0● 98.3 100.0● 97.9
Hop 100.0● 100.0● 100.0● 97.9
Entiat 100.0● 100.0● 100.0● 97.9
Flag 100.0● 100.0● 100.0● 100.0●
Row 91.8 91.5 91.8 93.3
Table legend
Max● >99% >95% >90% ≤90%
Hover over each TIR value to see detailed results.

Matching TIR measures the percentage of volunteers with successfully acquired images that were correctly identified.

Discounting failures to acquire high quality images, thirty (30) system combinations met the >99% Rally matching-TIR goal for volunteers in groups of 2 who self-reported as Asian. Matching systems Kenai, Miami, Tioga and Flag achieved 100% matching-TIR with all 4 acquisition systems.

Groups of 2 Matching Focus Black

Acquisition System
Bison Longs Wilson Borah
Matching System
Kenai 100.0● 100.0● 100.0● 99.5
Miami 100.0● 100.0● 100.0● 99.5
Tioga 100.0● 100.0● 100.0● 99.5
Mill 100.0● 100.0● 100.0● 98.9
Bronx 100.0● 99.6 100.0● 98.4
Grant 100.0● 99.2 100.0● 98.4
Hop 99.2 98.8 99.2 99.5
Entiat 99.2 98.4 99.2 99.5
Flag 100.0● 100.0● 100.0● 99.5
Row 86.9 87.5 85.6 85.9
Table legend
Max● >99% >95% >90% ≤90%
Hover over each TIR value to see detailed results.

Matching TIR measures the percentage of volunteers with successfully acquired images that were correctly identified.

Discounting failures to acquire high quality images, thirty-one (31) system combinations met the >99% Rally matching-TIR goal for volunteers in groups of 2 who self-reported as Black.

Groups of 2 Matching Focus White

Acquisition System
Bison Longs Wilson Borah
Matching System
Kenai 100.0● 100.0● 100.0● 100.0●
Miami 100.0● 100.0● 100.0● 100.0●
Tioga 100.0● 100.0● 100.0● 100.0●
Mill 100.0● 100.0● 100.0● 98.8
Bronx 99.0 100.0● 99.5 100.0●
Grant 100.0● 100.0● 100.0● 98.8
Hop 99.5 99.5 100.0● 100.0●
Entiat 99.0 99.1 99.0 99.4
Flag 100.0● 100.0● 100.0● 98.2
Row 87.4 87.2 87.3 86.0
Table legend
Max● >99% >95% >90% ≤90%
Hover over each TIR value to see detailed results.

Matching TIR measures the percentage of volunteers with successfully acquired images that were correctly identified.

Discounting failures to acquire high quality images, thirty-two (32) system combinations met the >99% Rally matching-TIR goal for volunteers in groups of 2 who self-reported as White. Matching systems Kenai, Miami, and Tioga achieved 100% matching-TIR with all 4 acquisition systems.

Groups of 2 Matching Focus Darker Skin

Acquisition System
Bison Longs Wilson Borah
Matching System
Kenai 100.0● 100.0● 100.0● 99.3
Miami 100.0● 100.0● 100.0● 99.3
Tioga 100.0● 100.0● 100.0● 99.3
Mill 100.0● 100.0● 100.0● 98.5
Bronx 100.0● 99.4 100.0● 97.8
Grant 100.0● 98.9 100.0● 97.8
Hop 98.9 98.3 98.8 99.3
Entiat 98.9 97.8 98.8 99.3
Flag 100.0● 100.0● 100.0● 99.2
Row 86.3 87.1 85.0 86.6
Table legend
Max● >99% >95% >90% ≤90%
Hover over each TIR value to see detailed results.

Matching TIR measures the percentage of volunteers with successfully acquired images that were correctly identified.

Discounting failures to acquire high quality images, twenty-six (26) system combinations met the >99% Rally matching-TIR goal for volunteers in groups of 2 with darker skin tone. An additional ten (10) system combinations met the >95% Rally matching-TIR threshold.

Groups of 2 Matching Focus Lighter Skin

Acquisition System
Bison Longs Wilson Borah
Matching System
Kenai 99.5● 99.5 99.4 99.3
Miami 99.5● 99.5 99.4 99.3
Tioga 99.5● 99.5 99.4 99.3
Mill 99.5● 99.5 99.4 97.9
Bronx 98.9 99.5 98.9 98.6
Grant 99.5● 99.5 99.4 97.2
Hop 98.9 98.9 99.4 98.6
Entiat 98.9 98.9 98.9 97.9
Flag 99.5● 99.4 99.4 97.9
Row 87.3 88.3 88.5 87.7
Table legend
Max● >99% >95% >90% ≤90%
Hover over each TIR value to see detailed results.

Matching TIR measures the percentage of volunteers with successfully acquired images that were correctly identified.

Discounting failures to acquire high quality images, twenty-three (23) system combinations met the >99% Rally matching-TIR goal for volunteers in groups of 2 with lighter skin tone. An additional thirteen (13) system combinations met the >95% Rally matching-TIR threshold.

GROUPS OF
2 4
SELECT DEMOGRAPHIC GROUP

Privacy

Groups of Two

Acquisition System
Bison Longs Wilson Borah
Matching System
Kenai 0.00● 0.00● 0.00● 0.64
Miami 0.00● 0.00● 0.00● 0.64
Tioga 0.00● 0.00● 0.00● 1.06
Mill 0.00● 0.00● 0.00● 0.85
Bronx 0.00● 0.00● 0.00● 1.27
Grant 0.00● 0.18 0.00● 0.64
Hop 0.00● 0.00● 0.00● 1.06
Entiat 0.00● 0.00● 0.00● 0.85
Flag 0.00● 0.00● 0.00● 0.65
Row 0.00● 0.00● 0.00● 0.88
Table legend
Min● =0% ≤1% ≤5% >5%
Hover over each NU-IR value to see detailed results.

A non-user identification occurred when an acquisition system submitted an image that matched to a volunteer in a bypass lane. Non-user identification rate is the proportion of images for which this occurred.

Twenty-nine (29) system combinations met the 2022 Rally NU-IR goal of 0% for groups of 2 volunteers. Acquisition systems Bison and Wilson met the goal with all 10 matching systems. Eight (8) more system combinations met the Rally NU-IR threshold of 1% or less.

GROUPS OF
2 4
Message Sent!
Error. Please try again.