Consider everything your smartphone has done for you today. Counted your steps? Transcribed notes? Navigated you somewhere new?
Smartphones make for versatile pocket assistants. That’s because they’re equipped with a suite of sensors. And some of those sensors you may never think — or even know — about. They sense light, humidity, pressure, temperature and other factors.
Smartphones have become essential companions. So those sensors probably stayed close by throughout your day. They sat in your backpack or on the dinner table or nightstand. If you’re like most smartphone users, the device was probably on the whole time, even when its screen was blank.
“Sensors are finding their ways into every corner of our lives,” says Maryam Mehrnezhad. She’s a computer scientist at Newcastle University in England. That’s a good thing when phones are using their powers to do our bidding. But the many types of personal information that phones have access to also makes them potentially powerful spies.
Online app store Google Play has already discovered apps that are abusing their access to those sensors. Google recently booted 20 apps from Android phones and its app store. Those apps could record with the microphone, monitor a phone’s location, take photos and then extract the data. And they could do all of this without a user’s knowledge!
Stolen photos and sound bites pose obvious privacy invasions. But even seemingly innocent sensor data might broadcast sensitive information. A smartphone’s motions might reveal what a user is typing. Or it might disclose a someone’s location. Even barometer readings could be misused. These readings subtly shift with increased altitude. That could give away which floor of a building you’re on, suggests Ahmed Al-Haiqi. He’s a security researcher at the National Energy University in Kajang, Malaysia.
Such sneaky intrusions may not be happening in real life — yet. However, concerned researchers are working to head off eventual invasions.
Some scientists have designed invasive apps. Afterward, they tested them on volunteers to highlight what smartphones can reveal about their users. Other researchers are building new phone security systems to help guard users from invasions of their privacy. They could thwart efforts to do everything from stalking a user to stealing the PIN codes needed to access their bank accounts.
Motion detectors are some of the tools within smartphones that are collecting data. These include their accelerometer (Ak-sell-ur-AHM-eh-tur) and the rotation-sensing gyroscope. Such bits of technology could be prime tools for sharing data without your knowing it.
One reason: They’re not permission-protected. That means a phone’s user doesn’t have to give a newly installed app permission to access those sensors. So motion detectors are fair game for any app downloaded onto a device.
In an April 2017 study, Mehrnezhad’s team at Newcastle showed that touching different regions of a screen makes the phone tilt and shift just a tiny bit. You may not notice it. But your phone’s motion sensors will. The data they collect may “look like nonsense” to the human eye, says Al-Haiqi. Yet clever computer programs can tease out patterns in that mess. They can then match segments of motion data to taps on various regions of the screen.
For the most part, these computer programs are algorithms that make up a type of machine learning, Al-Haiqi says. Researchers first train the programs to recognize keystrokes. They do this by feeding the programs lots of motion-sensor data. Those data are then labeled with the key tap that produced a particular movement.
A pair of researchers built TouchLogger. It’s an app that collects sensor data on a phone’s orientation in space. It uses these data to figure out how a user had been tapping on a smartphone’s number keyboard. In a 2011 test on phones made by a company in Taiwan, called HTC, TouchLogger figured out more than 70 percent of key taps correctly.
Since then, more studies have come out showing similar results. Scientists have written code to infer keystrokes on number and letter keyboards for different types of phones. In one 2016 study, Al-Haiqi’s team reviewed how successful these efforts were. And they concluded that only a snoop’s imagination limits the ways motion data could be translated into key taps. Those keystrokes could reveal everything from the password entered on a banking app to the contents of a text message.
Story continues below image.
A more recent application used a whole fleet of smartphone sensors to guess PINs. (A PIN is a sequence of numbers used to access a bank account.) The app analyzed a phone’s movement. It also noted how, during typing, the user’s finger blocked the light sensor. When tested on a pool of 50 PIN numbers, the app could discern keystrokes with 99.5 percent accuracy. The researchers reported this in December 2017 on the Cryptology ePrint Archive.
Other researchers have paired motion data with microphone recordings. A phone’s mic can pick up the soft sound of a fingertip tapping on a screen. One group designed a malicious app. It could masquerade as a simple note-taking tool. When the user tapped on the app’s keyboard, the app covertly recorded the keys’ input. It also recorded the simultaneous microphone and gyroscope readings. That let it learn the sound and feel to correctly diagnose each keystroke.
The app could even listen in the background when the user entered sensitive info on other apps. This phone app was tested on Samsung and HTC phones. It inferred the keystrokes of 100 four-digit PINs with 94 percent accuracy.
Such high success rates come mostly from tests made in controlled settings, notes Al-Haiqi. Those tests assume that users will hold their phones a certain way each time or will sit down while typing. How these info-extracting programs fare in a wider range of real-world conditions remains to be seen. But the answer to whether motion and other sensors would open the door for new privacy invasions is “an obvious yes,” he says.
Motion sensors also can help map someone’s travels, such as on a subway or bus ride. A trip produces motion data that are different from the more brief, jerkier movements of something like a phone being pulled from a pocket.
For a 2017 study, researchers designed an app to extract the data signatures of various subway routes. They used accelerometer readings from Samsung smartphones of people riding the subway in Nanjing, China.
A tracking app picked out which segments of the subway system a user was riding. It did this with an accuracy of 59 to 88 percent. How well it performed depended on how many subway stations the people road through. (The app improved as the rides lengthened from three stations to seven stations long.) Someone who can trace a user’s subway movements might figure out where the traveler lives and works. They might tell where the user shops or map out someone’s entire daily schedule. It might even — if the app is tracking multiple people — figure out who the user meets at various places.
Accelerometer data also can plot driving routes. And other sensors can be used to track people in more confined spaces.
One team, for instance, synced a smartphone mic and portable speaker. That let them create an on-the-fly sonar system to map movements throughout a house. The team reported the work in a September 2017 study.
Selcuk Uluagac is an electrical and computer engineer. He works at Florida International University in Miami. “Fortunately, there is not anything like [these sensor spying techniques] in real life that we’ve seen yet,” he notes. “But this doesn’t mean there isn’t a clear danger out there that we should be protecting ourselves against.”
That’s because the types of algorithms that researchers have used to comb through sensor data are getting more advanced and user-friendly all the time, says Mehrnezhad at Newcastle University. It’s not just people with PhDs who can design these types of privacy invasions, she says. App developers who don’t understand machine-learning algorithms can easily get this kind of code online to build sensor-sniffing programs.
What’s more, smartphone sensors don’t just provide snooping opportunities for cybercrooks who peddle info-stealing software. Legitimate apps often harvest info to compile such things as your search-engine and app-download history. The makers of these apps sell that info to advertising companies and outside parties. They could use the data to learn aspects of a user’s life that this person might want to keep private.
Take a health-insurance company. It may charge more to insure someone who don’t get much exercise. So “you may not like them to know if you are a lazy person or you are an active person,” Mehrnezhad says. Yet with your phone’s motion sensors, “which are reporting the amount of activity you’re doing every day, they could easily identify what type of user you are.”
It’s getting ever easier for an untrustworthy party to figure out private details of your life from data they get from your phone’s sensors. So researchers are devising ways to give people more control over what information apps can siphon data from their devices.
Some safeguard apps could appear as standalone programs. Others are tools that would be built into future updates of the operating system for your phone’s onboard computer.
Uluagac and his colleagues recently proposed a system called 6thSense. It monitors a phone’s sensor activity. Then it alerts an owner when it detects unusual behaviors. Users train this system to recognize their phone’s normal sensor behavior. This might include tasks like calling, Web browsing or driving. Then, 6thSense continually checks the phone’s sensor activity against these learned behaviors.
That program is on the lookout for something odd. This might be the motion sensors reaping data when a user is just sitting and texting. Then, 6thSense alerts the user. Users can check if a recently downloaded app is responsible for a suspicious activity. If so, they can delete the app from their phones.
Uluagac’s team recently tested a prototype of 6thSense on Samsung smartphones. The owners of 50 of these phones trained with 6thSense to identify their typical sensor activity. The researchers then fed the 6thSense system examples of benign data from daily activities mixed with bits of malicious sensor operations. 6thSense correctly picked out the problematic bits more than 96 percent of the time.
Supriyo Chakraborty is a privacy and security researcher at IBM in Yorktown Heights, N.Y. His team devised DEEProtect for people who want more active control over their data. It’s a system that blunts the ability of apps to draw conclusions about user activity from a phone’s sensor data. People could use DEEProtect to specify what their apps would be allowed to do with sensor data. For example, someone may want an app to transcribe speech but not identify the speaker.
DEEProtect intercepts whatever raw sensor data an app tries to access. It then strips those data down to only the features needed to make user-approved inferences.
Consider speech-to-text translation. For this, the phone typically needs sound frequencies and the probabilities of particular words following each other in a sentence. But sound frequencies could also help a spying app deduce a speaker’s identity. So DEEProtect distorts the dataset before releasing it to the app. However, it leaves alone data on word orders. Those data have little or no bearing on a speaker’s identity.
Users get to control how much DEEProtect changes the data. More distortion offers more privacy — but at a price: It degrades app functions.
Giuseppe Petracca is a computer scientist and engineer at Pennsylvania State University in University Park. He and his colleagues took a different approach. They are trying to protect users from accidentally allowing sensor access to deceitful apps. Their security system is called AWare.
When they are first installed, apps have to get a user permission to access certain sensors. This might include the mic and camera. But people can be careless about granting those permissions, Uluagac says. All too often, “people blindly give permission,” he says, to use the phone’s camera or microphone. They may give no thought to why the apps might — or might not — need them.
AWare would instead request permission from a user before an app can access a certain sensor the first time a user provided a certain input. For instance, this might happen when you press a camera’s button the first time after downloading an app. On top of that, the AWare system memorizes the state of the phone when the user grants that first permission. It remembers the exact appearance of the screen, the sensors that were requested and other information. That way, AWare can tell users if and when the app later attempts to trick them into granting unintended permissions.
The Penn State researchers imagined a crafty data-stealing app. It would ask for camera access when the user first pushes a camera button. But it would then also try to access the mic when the user later pushes that same button. The AWare system would realize the mic access wasn’t part of the initial deal. It would then ask the user again if he or she would like to grant this additional permission.
Petracca and his colleagues tested AWare with people using Nexus smartphones. Those using phone equipped with AWare avoided unwanted authorizations about 93 percent of the time. That’s compared with just 9 percent among people using smartphones with typical first-use or install-time permission policies.
The price of privacy
The security team in Google’s Android division is also trying to mitigate the privacy risks posed by app sensor data collection. Rene Mayrhofer is an Android security engineer in Austria at Johannes Kepler University in Linz. He and his colleagues are keeping tabs on the latest security studies coming out of university labs.
But just because someone has a successful prototype of a new smartphone-security system doesn’t mean it will show up in future phone updates. Android hasn’t incorporated any of these proposed sensor safeguards yet. That’s because its security team is still looking for the right balance. The team wants to restrict access for nefarious apps but not slow or degrade the functions of trustworthy programs, Mayrhofer explains.
“The whole [app] ecosystem is so big,” he notes. “And there are so many different apps out there that have a totally legitimate purpose.” Any kind of new security system that curbs an app’s access to the phone’s sensors, he says, could pose “a real risk of breaking” legitimate apps.
Tech companies may also be reluctant to adopt more security measures. Why? These extra protections can come at the cost of user friendliness. (AWare’s additional permissions pop-ups, for instance.)
Mani Srivastava is an engineer at the University of California, Los Angeles. There’s always a trade-off between security and convenience, he says. “You’re never going to have this magical sensor shield [that] gives you this perfect balance of privacy and utility.”
But phones are relying on ever more — and more powerful — sensors. And algorithms for analyzing their data are becoming more wise. Because of this, even smartphone makers may eventually admit that the current sensor protections aren’t cutting it. “It’s like cat and mouse,” Al-Haiqi says. “Attacks will improve. Solutions will improve.” Then more clever attacks will emerge. And security teams will engineer still more clever solutions. And on and on it goes.
The game will continue, Chakraborty agrees. “I don’t think we’ll get to a place where we can declare a winner and go home.”
accelerometer An instrument for measuring vibrations or a change in the rate of movement. These sensors typically can measure movement changes in all three dimensions (front-to-back, side-to-side and up-and-down).
algorithm A group of rules or procedures for solving a problem in a series of steps. Algorithms are used in mathematics and in computer programs for figuring out solutions.
app Short for application, or a computer program designed for a specific task.
application A particular use or function of something.
barometer An instrument that measures atmospheric pressure and is especially helpful in both weather forecasting and determining altitude.
behavior The way something, often a person or other organism, acts towards others, or conducts itself.
benign Not harmful to one’s health. Malignant, in contrast, means harmful and generally refers to cancer.
code (in computing) To use special language to write or revise a program that makes a computer do something.
colleague Someone who works with another; a co-worker or team member.
computer program A set of instructions that a computer uses to perform some analysis or computation. The writing of these instructions is known as computer programming.
degrade To break down into smaller, simpler materials — as when wood rots or as a flag that’s left outdoors in the weather will fray, fade and fall apart. (in chemistry) To break down a compound into smaller components.
distort (n. distortion) To change the shape or image of something in a way that makes it hard to recognize, or to change the perception or characterization of something (as to mislead).
ecosystem A group of interacting living organisms — including microorganisms, plants and animals — and their physical environment within a particular climate. Examples include tropical reefs, rainforests, alpine meadows and polar tundra.
engineer A person who uses science to solve problems. As a verb, to engineer means to design a device, material or process that will solve some problem or unmet need.
environment The sum of all of the things that exist around some organism or the process and the condition those things create. Environment may refer to the weather and ecosystem in which some animal lives, or, perhaps, the temperature and humidity (or even the placement of components in some electronics system or product).
factor Something that plays a role in a particular condition or event; a contributor.
function A relationship between two or more variables in which one variable (the dependent one) is exactly determined by the value of the other variables.
gyroscope A device to measure the 3-dimensional orientation of something in space. Mechanical forms of the device tend to use a spinning wheel or disc that allows one axle inside it to take on any orientation.
humidity A measure of the amount of water vapor in the atmosphere. (Air with a lot of water vapor in it is known as humid.)
infer (n. inference) To conclude or make some deduction based on evidence, data, observations or similar situations.
information (as opposed to data) Facts provided or trends learned about something or someone, often as a result of studying data.
machine learning A technique in computer science that allows computers to learn from examples or experience. Machine learning is the basis of some forms of artificial intelligence (AI). For instance, a machine-learning system might compare X-rays of lung tissue in people with cancer and then compare these to whether and how long a patient survived after being given a particular treatment. In future, that AI system might be able to look at a new patient’s lung scans and predict how well they will respond to a treatment.
monitor (v.) To test, sample or watch something, especially on a regular or ongoing basis.
online (n.) On the internet. (adj.) A term for what can be found or accessed on the internet.
operating system The software that controls a computer to supports its basic activities, such as scheduling tasks; assigning storage space (memory) to programs or data; performing applications, and controlling linked up devices (peripherals such as a printer, keyboard or computer monitor).
prototype A first or early model of some device, system or product that still needs to be perfected.
risk The chance or mathematical likelihood that some bad thing might happen. For instance, exposure to radiation poses a risk of cancer. Or the hazard — or peril — itself. (For instance: Among cancer risks that the people faced were radiation and drinking water tainted with arsenic.)
search engine (in computing) A computer program that allows a computer to search for information on the Internet. Common examples include Google, Yahoo and Bing.
sensor A device that picks up information on physical or chemical conditions — such as temperature, barometric pressure, salinity, humidity, pH, light intensity or radiation — and stores or broadcasts that information. Scientists and engineers often rely on sensors to inform them of conditions that may change over time or that exist far from where a researcher can measure them directly.
smartphone A cell (or mobile) phone that can perform a host of functions, including search for information on the internet.
software The mathematical instructions that direct a computer’s hardware, including its processor, to perform certain operations.
sonar A system for the detection of objects and for measuring the depth of water. It works by emitting sound pulses and measuring how long it takes the echoes to return.
subtly An adverb to describe something that may be important, but can be hard to see or describe. For instance, the first cellular changes that signal the start of a cancer may be only subtly different — as in small and hard to distinguish from nearby healthy tissues.
technology The application of scientific knowledge for practical purposes, especially in industry — or the devices, processes and systems that result from those efforts.
texting The sending of a text message from a mobile (cell) phone.
Web (in computing) An abbreviation of World Wide Web, it is a slang term for the internet.
Journal: D. Berend, B. Jungk and S. Bhasin. There goes you PIN: Exploiting smartphone sensor fusion under single and cross user setting. Cryptology ePrint Archive. Published online December 6, 2017.
Journal: C. Liu, S. Chakraborty, P. Mittal. DEEProtect: Enabling inference-based access control on mobile sensing applications. arXiv:1702.06159. Posted September 20, 2017.
Journal: R. Nandakumar et al. CovertBand: Activity information leakage using music. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. September 2017. doi: 10.1145/3131897.
Journal: M. Mehrnezhad et al. Stealing PINs via mobile sensors: actual risk versus user perception. International Journal of Information Security. Published online April 7, 2017. doi: 10.1007/s10207-017-0369-x.
Meeting: G. Petracca et al. AWare: Preventing abuse of privacy-sensitive sensors via operation bindings. 26th USENIX Security Symposium, Vancouver, Canada, August 16, 2017.
Meeting: A.K. Sikder, H. Aksu, A.S. Uluagac. 6thSense: A Context-aware sensor-based attack detector for smart devices. 26th USENIX Security Symposium, Vancouver, Canada, August 16, 2017.
Journal: J. Hua, Z. Shen, S. Zhong. We can track you if you take the Metro: Tracking Metro riders using accelerometers on smartphones. IEEE Transactions on Information Forensics and Security. Published online September 20, 2016, p. 286. doi: 10.1109/TIFS.2016.2611489.
Journal: M. Hussain et al. The rise of keyloggers on smartphones: A survey and insight into motion-based tap inference attacks. Pervasive and Mobile Computing. Vol. 25, January 2016. doi: 10.1016/j.pmcj.2015.12.001.
Journal: S. Narain, A. Sanatinia, G. Noubir. Single-stroke language-agnostic keylogging using stereo-microphones and domain specific machine learning. Proceedings of the ACM conference on security and privacy in wireless & mobile networks. January 23-25, 2014, p. 201. doi: 10.1145/2627393.2627417.
Meeting: H. Han et al. ACComplice: Location inference using accelerometers on smartphones. 4th International Conference on Communication Systems and Networks, Bangalore, India, January 6, 2012.
Meeting: L. Cai and H. Chen. TouchLogger: Inferring keystrokes on touch screen from smartphone motion. 6th USENIX workshop on hot topics in security, San Francisco, Calif., August 9, 2011.