Results - Maryland Test Facility

2020 Privacy Technology Demonstration Results

The 2020 Privacy Technology Demonstration (2020 Demo) is an event sponsored by the Department of Homeland Security (DHS) Science & Technology (S&T) Directorate to survey the state of privacy technology. Privacy technologies obscure or block face identity information to prevent people in videos from being identified, and are an emerging field of interest to DHS S&T as a potential solution to address public concerns about video surveillance, while still allowing safety organizations to safeguard the homeland.

A total of five systems participated in the 2020 Privacy Demonstration, and their goal was to obfuscate faces that appeared in a challenge video that highlighted use-cases seen in DHS operations:

Travelers interacting with a security officer one-on-one,
Travelers waiting in a crowded queue,
Travelers walking through an open space, and
Individuals wearing face masks that cover the nose and mouth regions of the face.

2020 Demo Results Summary

Below is a summary of how well the privacy systems processed different aspects, or factors, of the challenge video. Each segment of the outer ring is dedicated to one system with the color indicating whether a system did well (teal) or if there is room for improvement (orange). To read a description of each factor and a summary of how all the systems performed on that factor, hover your mouse cursor over each graphic to reveal additional information. A smaller version of each graphic is displayed in the top right corner of each summary.

Barrel Distortion

Barrel distortion occurs when straight lines bow outwards. This effect is commonly seen in body-worn cameras or in some wall-mounted security cameras. Overall, most systems did well with barrel distortion, maintaining obfuscations even when there were slight distortions to the face.

Motion Blur

Motion blur is when streaking appears due to movement from either the volunteer or the camera. This effect is commonly seen in body-worn cameras, particularly ones that are head or shoulder mounted. All five systems experienced disruptions in obfuscation when motion blur was present, leading to times when faces were fully visible.

Pose Variation

Pose variation is when volunteers in the video clips change their position or way they stand. This occurs when subjects walk from one area to another or turn a corner, which can alter their face from a profile view to a three-quarter or straight-on view. Only one system was resilient to pose changes, whereas the remaining four systems only obfuscated faces when they appeared straight on.

Expression Variation

Expression variation is when volunteers in the video clips change the emotion expressed on their faces. From furrowed brows to smiling and laughing, all five privacy systems did well with expression variation. No disruptions due to expression changes were observed.

Occlusions

Occlusions are objects that partially block regions of the face. Occlusions to the face are often from headwear or eyewear, but may also be caused by the environment, such as walking behind objects like trees or poles. Only two privacy systems did well with occlusions by preserving obfuscations, even when subject wore hoods or hats; the remaining three privacy systems had disruptions to obfuscations when occlusions were present.

Clipping

Clipping is when a face appears partially outside of a video frame. This occurs when volunteers in a video clip are moving freely through an open area, entering and exiting the view of the camera. Only one system was resilient to partial face images, with the remaining four privacy systems sometimes dropping obfuscations when faces were partially out of frame.

Crowding

Crowding is when there are multiple volunteers are in close proximity to each other. There were sections of the challenge video where there is considerable crowding, with many volunteers appearing close together, as well as periods where there are low or moderate levels of crowding. Four systems were resilient to considerable and moderate levels of crowding.

Masks

Masks are a specific type of occlusion that covers the mouth and nose regions of the face, which may disrupt face detection in privacy systems. All privacy systems’ performance suffered when masks were present; however, three systems had minimal or extremely brief disruptions to face obfuscations when volunteers wore masks.