What is facial recognition? How facial recognition works

Facial recognition defined 

Facial recognition is a way of recognizing a human face through technology. A facial recognition system uses biometrics to map facial features from a photograph or video. It compares the information with a database of known faces to find a match. Facial recognition can help verify a person’s identity, but it also raises privacy issues.

The facial recognition market is expected to grow to $7.7 billion in 2022, an increase from $4 billion in 2017. That’s because facial recognition has many commercial applications. It can be used for everything from surveillance to marketing.

But that’s where it gets complicated. If privacy is important to you, you probably want some control over how your personal information — your data — is used. And here’s the thing: Your “faceprint” is data.

How facial recognition works

You might be good at recognizing faces. You probably find it a cinch to identify the face of a family member, friend, or acquaintance. You’re familiar with their facial features — their eyes, nose, mouth — and how they come together.

That’s how a facial recognition system works, but on a grand, algorithmic scale. Where you see a face, recognition technology sees data. That data can be stored and accessed. For instance, half of all American adults have their images stored in one or more facial-recognition databases that law enforcement agencies can search, according to a Georgetown University study.

So how does facial recognition work? Technologies vary, but here are the basic steps:

Step 1. A picture of your face is captured from a photo or video. Your face might appear alone or in a crowd. Your image may show you looking straight ahead or nearly in profile.

Step 2. Facial recognition software reads the geometry of your face. Key factors include the distance between your eyes and the distance from forehead to chin. The software identifies facial landmarks — one system identifies 68 of them — that are key to distinguishing your face. The result: your facial signature.

Step 3. Your facial signature — a mathematical formula — is compared to a database of known faces. And consider this: At least 117 million Americans have images of their faces in one or more police databases. According to a May 2018 report, the FBI has had access to 412 million facial images for searches.

Step 4. A determination is made. Your faceprint may match that of an image in a facial recognition system database.

A brief history of facial recognition

You can trace the history of facial recognition to the 1960s. That’s when mathematician and computer scientist Woodrow Wilson Bledsoe first developed a system of measurements that could be used to put photos of faces in different classifications. Because of this work, Bledsoe is known as the unofficial father of facial recognition technology. 

Law enforcement agencies soon became interested in Bledsoe’s work. And in the 1970s through the 1990s, agencies developed their own facial recognition systems. These were crude compared to the technology today, but the work on these systems did lead the way to modern facial recognition programs.

Many point to 2001 as a key year for facial recognition technology. That’s when law enforcement officials used facial recognition to help identify people in the crowd at Super Bowl XXXV. That same year, the Pinellas County Sheriff’s Office in Florida created its own facial recognition database.

It wasn’t until the 2010s, though, that computers grew powerful enough to make facial recognition a more standard feature. In 2011, in fact, facial recognition software confirmed the identity of terrorist Osama bin Laden. In 2015, the Baltimore police department used facial recognition to identify those who participated in protests after Freddie Gray was killed by a spinal injury that he suffered while being transported in a police van. 

Consumers now use facial recognition with their smartphones and other personal devices. Windows Hello and Android’s Trusted Face in 2015 allowed people to log into their devices by simply aiming them at their faces. Apple’s iPhone X unveiled its Face ID facial recognition technology in 2017.

There has been controversy over this technology, with critics saying it is an invasion of privacy. Cities such as San Francisco, Oakland, and Boston have banned governments from using facial recognition. And after Black Lives Matter protests against police brutality in the summer of 2020, several tech giants, including Amazon, Microsoft, and IBM, announced that they would no longer sell their facial recognition technology to law enforcement agencies.

How accurate is facial recognition?

Critics worry that facial recognition could lead to false identifications. What if a police department uses facial recognition technology to incorrectly identify someone breaking a store window during a riot as a person who was nowhere near the event? How likely is it that this could happen? 

That depends. Tests by the National Institute of Standards and Technology say that as of April of 2020, the best face identification algorithm boasted an error rate of just 0.08%. That's a big improvement from 2014, when the best algorithm had an error rate of 4.1%.

Accuracy, though, is higher when identification algorithms are used to match people to clear, static images, such as a passport photo or mugshot, according to a story by the Center for Strategic & International Studies (CSI) in 2020. The story said that facial recognition algorithms can hit accuracy scores as high as 99.97% on the National Institute of Standards and Technology's Facial Recognition Vendor Test when used in this way.

In the real world, though, accuracy rates are usually lower. According to the CSI story, the Facial Recognition Vendor Test found that the error rate for one algorithm rose from 0.1% when faces were matched against high-quality mugshots to 9.3% when matched to pictures of individuals captured in public. Error rates rose especially when subjects were not looking directly at the camera or were partially hidden by shadows or objects. 

Aging is another challenge. The Facial Recognition Vendor Test said that middle-tier facial recognition algorithms had error rates that jumped by nearly a factor of 10 when they attempted to match photos of subjects that had been taken 18 years earlier.

Who uses facial recognition 

A lot of people and organizations use facial recognition — and in a lot of different places. Here’s a sampling:

• U.S. government at airports. Facial recognition systems can monitor people coming and going in airports. The Department of Homeland Security has used the technology to identify people who have overstayed their visas or may be under criminal investigation. Customs officials at Washington Dulles International Airport made their first arrest using facial recognition in August of 2018, catching an impostor trying to enter the country.
  
• Mobile phone makers in products. Apple first used facial recognition to unlock its iPhone X, and has continued with the technology with the iPhone XS. Face ID authenticates — it makes sure you’re you when you access your phone. Apple says the chance of a random face unlocking your phone is about one in 1 million.
• Colleges in the classroom. Facial recognition software can, in essence, take roll. If you decide to cut class, your professor could know. Don’t even think of sending your brainy roommate to take your test.
• Social media companies on websites. Facebook uses an algorithm to spot faces when you upload a photo to its platform. The social media company asks if you want to tag people in your photos. If you say yes, it creates a link to their profiles. Facebook can recognize faces with 98 percent accuracy.
• Businesses at entrances and restricted areas. Some companies have traded in security badges for facial recognition systems. Beyond security, it could be one way to get some face time with the boss.
• Religious groups at places of worship. Churches have used facial recognition to scan their congregations to see who’s present. It’s a good way to track regulars and not-so-regulars, as well as to help tailor donation requests.
• Retailers in stores. Retailers can combine surveillance cameras and facial recognition to scan the faces of shoppers. One goal: identifying suspicious characters and potential shoplifters.
• Airlines at departure gates. You might be accustomed to having an agent scan your boarding pass at the gate to board your flight. At least one airline scans your face.
• Marketers and advertisers in campaigns. Marketers often consider things like gender, age, and ethnicity when targeting groups for a product or idea. Facial recognition can be used to define those audiences even at something like a concert.

Facial recognition and its use in law enforcement

Facial recognition databases play a significant role in law enforcement today. According to a report by the Electronic Frontier Foundation, law enforcement agencies routinely collect mugshots from those who have been arrested and compare them to local, state, and federal facial recognition databases.

Law enforcement agencies can sift through these mugshot databases to identify people in photos taken from a variety of sources: closed-circuit television cameras, traffic cameras, social media, or photos that police officers have taken themselves.

Police officers can also use their smartphones, tablets, or other mobile devices to snap photos of drivers or pedestrians and immediately compare their photo against the faces in one or more facial recognition databases, the Electronic Frontier Foundation says. 

And law enforcement has used facial recognition at large events such as concerts, sporting events, or the Olympics to identity people who might be wanted in connection with crimes.

The federal government can use several facial recognition systems. The database it relies on most frequently, though, is the FBIs Next Generation Identification system. This database contains more than 30 million facial records.

Facial recognition examples

Businesses use facial recognition in a variety of ways today, usually to make it easier for consumers to use their products or services. Here are some examples:  

Traveling: British Airways uses facial recognition to make it easier for U.S. passengers to board their flights. Passengers can have their faces scanned by a camera to verify their identity. This way, they can board their flights without having to show a passport or boarding pass.

Apple: Apple could be considered a pioneer in facial recognition. The tech giant has long allowed consumers to unlock their phones, log into apps, and make purchases just by showing their face to their smartphones and other devices.

Driving: Automakers are testing facial recognition technology to help cut down on car theft. Consider Project Mobil: Ford and Intel are testing a project in which a dashboard camera uses facial recognition to identify the primary driver of a car and, perhaps, other authorized drivers. The tech could prevent a car from starting if someone other than a rightful driver is sitting behind the wheel. 

Banking: Banking giants such as HSBC and Chase already use Apple's FaceID to let customers log into their mobile banking apps. Other financial institutions are testing facial recognition to allow customers to use their phone's cameras to approve online purchases. 

Insurance: Cigna allows customers in China to file health insurance claims using their photos instead of a written signature. The insurance company says it’s a way to cut down on insurance fraud.

Even soft drinks: Coca-Cola has been a longtime user of facial recognition. For instance, the company uses the technology to reward customers for recycling at some of its vending machines in China. It also uses facial recognition to send customers in some countries personalized ads when they use vending machines.

Facial recognition pros and cons

As a relatively new technology, we're still understanding the pros and cons of facial recognition. But here is a brief list of both the positives and possible negatives of this technology.

Pros 

Finding missing people: With facial recognition, law enforcement agencies have been able to track down missing children, sometimes even after they've been missing for years. 

Identifying criminals: Law enforcement agencies can also use facial recognition to identify criminals or suspects in crimes. 

Making flying safer: Airports across the globe are using facial recognition to identify criminals and potential threats as they enter airports or try to board flights.

More efficient shopping? Retailers can use facial recognition to make it easier for consumers to check out. Instead of forcing customers to pay with cash or credit, retailers can use facial recognition to immediately charge their purchases to their accounts.

Cons 

A threat to privacy? Do you want your face saved in a database that law enforcement agencies can tap? Do you want retailers to have a saved image of your face? If you don’t, you're not alone. Many critics worry that facial recognition is one more erosion of personal privacy.  

Mistaken identity: Facial recognition isn't perfect. What if a law enforcement agency mistakenly identifies you as a criminal suspect when you're filing into your favorite ballpark?

It can be tricked: Criminals can trick facial recognition by wearing masks or facial disguises. This could lessen the effectiveness of this tech.  

Aging lowers its effectiveness: Studies have found that as people age, and their features change, facial recognition has an increasingly difficult time identifying them. Other studies have shown that facial recognition is less effective in identifying people of color and women.

Reasons to be concerned about your privacy

Privacy matters. Privacy refers to any rights you have to control your personal information and how it’s used — and that can include your faceprint.

So, what are the issues? Here are some:

• Security. Your facial data can be collected and stored, often without your permission. It’s possible hackers could access and steal that data.
• Prevalence. Facial recognition technology is becoming more widespread. That means your facial signature could end up in a lot of places. You probably won’t know who has access to it.
• Ownership. You own your face — the one atop your neck — but your digital images are different. You may have given up your right to ownership when you signed up on a social media network. Or maybe someone tracks down images of you online and sells that data.
• Safety. Facial recognition could lead to online harassment and stalking. How? For example, someone takes your picture on a subway or some other public place and uses facial recognition software to find out exactly who you are.
• Mistaken identity. Say, for instance, law enforcement uses facial recognition to try to identify someone who robbed a corner store. Facial recognition systems may not be 100 percent accurate. What if the police think the suspect is you? 
• Basic freedoms. Government agencies and others could have the ability to track you. What you do and where you go might no longer be private. It could become impossible to remain anonymous.

How you can help protect yourself against facial recognition 

Want to protect your privacy in a world in which facial recognition technology is becoming more common? Here are some reasons for hope.

Tech innovation: Concerns about facial recognition could spur innovation.

Consider this: Two universities have developed anti-facial recognition glasses to make wearers undetectable.

The glasses — the work of researchers at Carnegie Mellon University and the University of North Carolina at Chapel Hill — could be one way to help protect yourself.

Social networks: Changing the way you interact with social media could help protect you from facial recognition-based privacy invasions.

For example, Facebook allows you to opt out of its facial recognition system.

It’s smart in general to be careful about what you share on social networks. Posting too much personal information, including photos, could lead to identity theft. For instance, you might share your dog’s name or your high school mascot. Those details might give an identity thief a clue to the answers to your security questions for your bank or credit card accounts.

The Internet of things: It’s also a good idea to consider the so-called Internet of Things — those devices in your home that connect to the internet. IoT devices that use face recognition include iPads, Xboxes, and video systems.

One possible solution? A secure router can help safeguard your network and your connected devices, which in turn could help protect your facial image.

How can you find more protection against facial recognition systems?

Will hackers really want to steal your face? If your facial data can be used to commit fraud or turn a profit, the answer is “yes.” Add that to the list of cyber safety risks.

A holistic cyber safety package is worth considering for help protecting your online privacy and security.

Still, facial recognition represents a challenge to your privacy. After all, there are few rules governing its use.

In the meantime, maybe those anti-facial-recognition glasses won’t look so bad. 

FAQs — frequently asked questions about facial recognition

What is facial recognition?
Facial recognition is a way of recognizing a human face through technology. 

How does facial recognition work?
A facial recognition system uses biometrics to map facial features from a photograph or video. It compares the information with a database of known faces to find a match.

Can facial recognition work with a mask?
The masks that people are wearing during the COVID-19 pandemic do pose challenges for facial recognition. But companies are working to overcome this by focusing their technology on the facial features visible above these masks. That could mean that a COVID mask won’t thwart facial recognition technology for long. 

What is facial recognition used for? Facial recognition has many uses. Companies can use it for marketing, sending targeted ads to consumers. Law enforcement agencies use it to identify suspects or track down missing persons. And tech companies use it to allow consumers to easily unlock their devices.

Is facial recognition accurate? That depends. Studies have found that facial recognition is highly accurate when comparing faces to static images. This accuracy drops, though, when matching faces to photos taken in public.

What are the benefits of facial recognition? Law enforcement agencies can use facial recognition to locate missing persons and identify the perpetrators of crimes. It can also be used to find criminal suspects in large crowds, such as those attending sporting events or concerts.

What are the disadvantages of facial recognition? Critics says that facial recognition represents a further eroding of people’s privacy. There are also questions of false identification: What if law enforcement agencies incorrectly identify an innocent person as a suspect in a crime?