The Rally Participant Training webinar was not recorded but the slides from this webinar will be available for your reference.
Please email rally@mdtf.org to receive information regarding the Slack Channel.
Rally Participants may bring up to three individuals for the VIP day. If individuals will only be present for the VIP day they do not need to complete the NIH Human Subject Training.
Rally Participants will not be provided internet access while at the MdTF. In the event that you will need internet access, please bring your own hotspot.
The slides should be available by 1/25 COB.
The Slides are posted here. Please email rally@mdtf.org to obtain the password.
Please use the link provided in the slides to verify the safety of your systems. Additional requirements may be brought up by the IRB and will be handled on a case-by-case basis.
This Rally is primarily a test of unmanned/unattended collection processes. The Rally seeks improved ways to collect biometrics quickly and effectively.
The Rally seeks to evaluate both face and combined face/iris biometric systems for DHS use-cases.
Rally Participants will receive the following biometric data:
DHS plans to include representatives from DHS S&T, NIST, CBP, TSA, and DoD. The panel may also include representatives from critical infrastructure sectors.
Not at this time.
Yes, vendors may submit white papers on more than one system. If a vendor has multiple systems selected to participate in the Rally, each system will be assigned to separate station.
This will be determined by emerging DHS priorities and available resources.
No, the gallery will not be available prior to installation at the MdTF. You will receive the gallery on a USB after you have installed and tested your system.
There will be jpeg and png images in the gallery. Information about the participant will be included in the file name.
Face images for identification operations have been acquired by MdTF staff in prior testing. Rally Participants will be provided a gallery of face images for those systems performing identification operations. The gallery of face images will include one or more photos of individuals. Images may be collected from different camera systems and at varying times.
The face image gallery will include approximately 300-400 unique individuals. However, there may be multiple images (from varying devices) per individual.
Most, but not all, individuals in the gallery will be encountered during the Rally.
It is expected that ~80% of the volunteers will be present in the gallery of images provided to those Rally Participants performing face identifications.
Gallery images will include a variety of image types and styles. The photos within the gallery will be mostly frontal images but will vary in style (e.g., portrait, “selfies”), file type, and file size. A sample gallery will be available to Rally Participants upon request.
Yes. Rally Participants performing face identifications will be provided with data structure/metadata prior to installation.
No. A gallery of iris images will not be provided for identification operations during testing. The MdTF will be collecting same-day iris and face images to calculate iris identification metrics after testing with volunteers.
No. Volunteers will not be provided with a travel token (e.g. document or device).
Rally Systems should be applicable to DHS use-cases where there is no token available. DHS is interested in seeing if a process based on a token today can be made token-less (e.g. UK Iris).
Yes. Only ~80% of the volunteers will be in the provided face gallery. Volunteers that are not in the gallery must be identified as such by the system.
Yes. Biometric accuracy for systems that do not submit identifications will be calculated by matching submitted images on MdTF systems after the Rally.
Rally Systems should be applicable to DHS use-cases where there is no token available. DHS is interested in seeing if a process based on a token today can be made token-less (e.g. UK Iris).
Yes. Only ~80% of the volunteers will be in the provided face gallery. Volunteers that are not in the gallery must be identified as such by the system.
Yes. Biometric accuracy for systems that do not submit identifications will be calculated by matching submitted images on MdTF systems after the Rally.
No. Iris images will not be provided to Rally Participants for matching operations. Following the Rally, submitted face and iris images will be matched on MdTF systems using multiple biometric matching algorithms and galleries.
All iris matching will be performed on MdTF systems after the Rally.
The Rally seeks biometric systems useful for various DHS biometric use-cases such as identifying previously encountered, or trusted individuals.
The general use case of Rally systems will be to verify the identity of an individual. This may apply to travel and other security use cases. Some specific use cases may be security check points, or identifying individuals at restricted access-points.
Yes, these are some potential use cases for this activity. Others include physical access control systems for sensitive buildings or areas.
Yes. Systems must be able to automatically submit at least one face image, through the API, in an un-attended operation mode. No Rally Participant staff will be present on the test floor during the Rally.
The volunteers will make good faith effort to perform tasks as indicated by the Rally Systems. Rally Systems are responsible for all signage/instructions that inform the volunteers of any required actions.
This particular Rally is intended to evaluate the trade-offs between collection time and accuracy for individuals – not large groups. Individual volunteers will interact with test systems for less than 20 seconds each. Interaction/transaction times of less than 5 seconds are desirable. Your test system may also include multiple cameras, however, for the purpose of this test you will only be able to submit a limited amount of still photo and video imagery.
In order to receive the face gallery, the MdTF will provide a USB drive to transfer it to Rally Participants’ systems. After the gallery has been transferred, the USB drive will be given back to the MdTF.
For Rally Participants that are recording video, the MdTF will provide a USB drive to transfer video from Rally Participant’s systems to the MdTF.
To obtain the data your system collected during the Rally, Rally Participants are responsible for bringing a high-speed, encrypted data locker that is FIPS compliant to the MdTF.
The afternoon session does not have a strict end time. Each session will end when all human volunteers have left the MdTF.
Yes, a bitlocker encrypted HD is sufficient. Please ensure that any HD is FIPS compliant.
The CRADA was emailed directly to Rally Participants.
There is no correct sequence of function calls. All calls to the API can be thought of as independent operations, i.e. not related to one another. For example, a call to /v1/face-captures at time t=0 followed by a call to /v1/identifications at time t=1 does not imply the face in the first call was used to calculate the identification result in the second.
Yes, NIR is considered safe depending on intensity. We will measure the intensity of the light off your system during installation, if applicable.
Attesting to attending the webinar on 1/26 constitutes receiving Recipient Training materials. The SOW will be included in the CRADA document.
No, you will not be able to dial into your systems during the Rally. We will provide the capability to monitor your systems output via our backend. Any diagnosis will need to happen locally.
Once Rally Participants believe they have successfully submitted a test transaction with the API, please send an email to rally@mdtf.org with the time of the transaction and the endpoints used.
The green checkmarks on the monitors are for the Test Guides and ensures that the MdTF’s backend systems are functional.
Prior to the test, the human volunteers are provided with the instructions as shown on Slide 21 of the Logistics Webinar Slides. The MdTF will have staff at the front of each station scanning wristbands. The scanners are told to wait until the person prior has cleared the station before scanning the next individual. In this sense these staff members are responsible for the flow, however the pace is dictated by what happens in the station, i.e. the Rally Participant system.
You should arrive approximately 45-minutes before each test session. This will be the time that Rally Participants have to perform starting any hardware or software your system requires. After all human volunteers have exited the afternoon session, there will be a brief period for Rally Participants to power down their systems.
Restrictions associated with the IRB-approved testing protocol and limitations of the MdTF physical infrastructure prevents us from supporting outside observers for Rally activities. There will be a technology demonstration day for US Government and critical infrastructure stakeholders (e.g. airports, airlines), please submit an expression of interest to Peoplescreening@hq.dhs.gov for consideration. A post-rally webinar briefing the results is planned for the April/May timeframe that will be available for third parties participation.
A camera system is in place, however, it will not be accessible by Rally Systems. The MdTF video system is used for diagnostics only. Rally Participants may request a copy of the MdTF video observations for their Rally System. Usage of this video will be limited to activities authorized by the testing protocol and informed consent documents approved by the IRB and signed by the volunteers.
Rally Systems will not have external internet access – all communication will be internal to the MdTF. Prior to the test the MdTF plans to host the MdTF Rally API in the cloud to validate/troubleshoot data submission before Rally System installation at MdTF.
Yes. However, if space permits, we may allow packaging materials to be stored elsewhere.
A physical gate is not required. Participants are encouraged to develop creative approaches to optimize efficiency, effectiveness, and volunteer satisfaction. Please see the Rally Metrics page for more information.
No. Rally Stations will be separated by partitions but will not have roofs or coverings above the partitions. Rally Participants will not be allowed to attach anything to the roof or ceiling of the facility. All equipment must fit within the 7’ x 8’ space.
To reduce delays during the Rally, the MdTF API will be hosted on the local area network. The MdTF API only provides HTTP response status codes indicating success or error. Rally Systems do not need to wait for any reply from the API to complete a transaction. Rally Participants concerned with timing, may implement asynchronous API calls. To reduce any delays associated with image submissions, Rally Participants are encouraged to send the smallest images which they believe will result in high identification performance on MdTF systems.
During the rally no attempt will be made to introduce occlusions; however, volunteers may wear whatever they desire during testing at the MdTF. If you wish for these specific items to be removed, your system should provide appropriate instructions to volunteers. Please be aware that the testing dates will be during winter, so some volunteers may be wearing winter attire (scarfs, hats, etc.). Volunteers will be able to store their personal belongings but they may elect to bring it with them or wear coats, hats, and glasses during their visit.
Occlusions (e.g. due to clothing or hairstyle) may be present during acquisition. These cases will not be marked separately and will be included in the calculated performance of Rally Systems. Any impact of instructions to remove said occlusion will be included in the Rally Station’s Transaction Time, Failure to Acquire Rate, and volunteers’ Satisfaction rating. Rally Participants should be mindful of how the human factors of their system’s process may impact Rally metrics.
Rally Participants may use any kind of camera or cameras during the test as long. However, any all rally systems must meet Rally requirements and only data sent through the MdTF API will be used for evaluation. Further, all Rally Systems must meet standard safety regulations.
Rally Participants should consider the properties of the images acquired from their camera(s) and the time to it takes for systems to send images to the API. It takes approximately 600 milliseconds to send a pair of 512x512 iris images to the API and approximately 300 milliseconds to send a 512x512 face image. Any time required to send images to the API will be included in the Rally metrics
There are no strict usability or signage requirements for Rally Systems; however, volunteers will not be instructed by MdTF staff on how to use each Rally System. MdTF staff will instruct volunteers that the purpose of each Rally System is to gather face/iris images and that they should attempt to use each system in accordance to any provided instructions. Rally Participants are encouraged to implement any usability components so that volunteers comprehend what to do, including when to exit the system.
Volunteers will have IDs to establish ground truth during the Rally. However, this ID will not be available to Rally Systems. Volunteers will not be equipped with any token or ticket by the MdTF and Rally Participants that are completing the optional identification requirements should not rely on a token or ticket being presented.
You may capture as many images per person as you like. However, the API will only accept 3 calls to the face-captures, iris-captures, and identifications endpoint per person. A 4th call will result in an error code being returned. Rally Participants who capture more images per person will need to have a triage capability to decide which ones to send.
Yes, the last image submitted by each system should be of the best quality as this image will be used to evaluate Rally Participants’ system. As a corollary there is no downside to submitting low quality images early.
Definitions of the metrics are provided in the Webinar slide deck.
Yes. Biometric accuracy for all Rally Systems will be independently evaluated on MdTF systems after the Rally using ground-truthed galleries.
Yes. Biometric accuracy for iris systems will be evaluated on MdTF systems after the Rally using ground-truthed iris image gallery.
The first Biometric Technology Rally does not include a financial prize. Rally Participants will receive data and information to improve the performance of their systems. Rally Participants may choose to use specific DHS S&T test results to describe the performance of their systems to third parties. DHS S&T may use test results to identify systems for DHS operational tests or limited pilots and share results as market research with other government agencies. There is no guarantee of an RFQ or RFP after the Rally.
Rally metrics will not combine the results of facial and iris recognition; the computed True Identification Rate (TIR) for each biometric modality will be calculated separately.
We will use multiple commercial and government algorithms to compute TIR for face and iris images against relevant galleries.