Full-stack software engineer building useful tools. Constantly learning, with a habit of connecting dots across different domains.
Authenticating faces that are half-covered — a two-stage detect-then-recognize pipeline. B.Tech final-year project.
A face-recognition system that still works when the face is masked, the kind of occlusion that COVID-era checkpoints, polluted cities, and clean-room labs made an everyday problem. My B.Tech final-year project at NIT Srinagar (2023), with Huzaif Mushtaq Mir, under Dr. Sparsh Sharma.
Conventional recognition models lean on the nose, mouth, and cheeks exactly the region a mask hides, so their accuracy collapses on masked faces. This is the sub-problem of known occlusion: you know which part is covered, so you can design around it.
Removing a mask for every verification is, at best, irritating and at worst dangerous, like in a high-profile lab where someone has to break protective gear just to badge in. After the pandemic mandated masks worldwide, recognizing the masked face stopped being a niche and became an urgent gap in the traditional models.
We surveyed the field first, datasets, algorithms, and the NIST benchmarks of 266 post-2020 recognition algorithms, then built our own.
Two models in sequence. A detector first finds each face and labels it masked or unmasked, transfer-learned from ssd_mobilenet_v2_fpnlite, chosen over heavier ResNet backbones for its low inference time, with focal loss to handle class imbalance and smooth-L1 for the bounding boxes. A recognizer then turns the cropped face into an embedding: a Siamese network built on Xception (pretrained on ImageNet, last three blocks unfrozen) trained with triplet loss, producing a 256-d vector matched against the database by distance with a rejection threshold for unknown faces.
The detector trained on ~1,850 masked/unmasked images filtered from MAFA and MOXA3K; the recognizer on LFW with simulated masks applied via MaskTheFace (~7,150 images → 11,231 triplets).
Our Xception encoder reached 84.57% on Masked-LFW — within a whisker of the 84.60% FaceNet baseline from the literature on the same dataset, and well ahead of our own InceptionResNetV2 (73.89%) attempt.
Coming soon — technical notes breaking down the pipeline: the two-stage detect-then-recognize design, triplet-loss embeddings for occluded faces, and designing around known occlusion.