Vehicles as sensor platforms

4 November 2021

We're watching you...

Almost everybody in the automotive world is familiar with sensors. Lambda, coolant temperature, oil pressure, knock, MAP, MAF, speed, tyre pressure and others are all well known. But these days there’s even more to the story. All sorts of things can be monitored, and everything can be logged. Even more importantly, all the logged data can and will be transferred from your car to other interested parties, with a variety of consequences.

Sensor systems can determine several things besides the conditions they’re designed to monitor directly. The latest tyre pressure sensors provide a good example of how secondary inferences can be derived from directly sensed primary data. Tyre pressure sensors have been mandatory across the EU since 2014 in the name of increased safety, reduced fuel consumption and lower CO2 emissions. While not mandatory in Australia, these will eventually be compulsory here.
 
So far, tyre pressure sensors have been relatively bulky components fitted in conjunction with valve stems. They are powered by small disc-type batteries that have relatively significant mass, despite their diminutive size. Newer types will utilise thin-film technology. This will allow such sensors to be embedded directly in the cases of tyres where they can gather a wider range of data.

A company called IEE is developing this embedded sensor technology and says these new ‘intelligent tyres’ will be able to sense internal and external temperatures, tread depth, road condition, axle load and wheel alignment. As such technology becomes available, every tyre on Earth will be numbered and monitored from manufacture to end-of-life. But why?
While tyres are essential for the modern world, they’re also a big problem. We all know that, but we just don’t think about it much. Tyre manufacturing consumes both resources in enormous measure and energy in copious quantities. End-of-life disposal is also an issue. Maximising service longevity and re-use (by retreading) will be important to minimise environmental impact. Also, thin-film tyre-embedded technology can generate power for thin-film sensors by piezoelectrical means.

Aside from the directly sensed data mentioned above, all sorts of other things will be inferred from the primary data, like driving style and habits, speed, attention to maintenance schedules, etc. These will all be logged and transferred back to a central server somewhere. Now many of these things, like speed, will be directly detected by other dedicated sensors. However, anyone who’s thought about it knows that wildly excessive speed is not the only form of dangerous driving.

If wheel speed sensors show that speed is under the limit, yet heat and stresses in the tyre tread are excessive and coupled with a high slip angle, a quick cross-reference to steering angle input will show that a car is being driven on the edge. So, to be clear, the systems to which automotive sensors are connected will create secondary data. But we won’t mind, we never do. We’re like frogs in water slowly coming to the boil, wondering why things are getting so hot.

Apart from sensors that monitor things like tyres, suspension, drivetrain and engine, cars of the not-too-distant future will have a whole new range of in-cabin biometric sensors that monitor occupants. Of course, many of these systems exist now in nascent form, at least, but there will be more, and they’ll be more advanced.

First, all cars will have driver identification, whether drivers like it or not. This is for vehicle personalisation. Also, when fully autonomous vehicles are on the road and offering Transport as a Service (TaaS), it will be imperative to record who has been in a car. This is because some people do unspeakable things when they can remain anonymous. Certainly, most people are fine and decent, but it only takes one to leave an unpleasant reminder of their presence.

In a TaaS car the sole, or primary, occupant will be known because of booking and payment information. However, cars will also know exactly how many people are in the vehicle, their age, weight, size, and more. This information is needed for tailoring the vehicle accident restraint system to each occupant. In fact, there will likely be 3D maps of each occupant, along with vital sign sensors that monitor health and well-being.

If a biometric sensing system detects that a driver is suffering, or is about to suffer, a health problem, like a heart attack or stroke, the car will act appropriately. A fully autonomous vehicle will take an occupant suffering a medical episode to hospital, or to the closest ambulance (which will already be making its way to the patient). The car of the not-too-distant future will also have aids like gesture recognition and hands-off wheel detection, which will be a factor for activating autonomous mode. No more accidents when wrestling with something that ideally shouldn’t be taking your attention. There will be unattended child detection, drowsiness detectors, distraction detectors (both already available) and emotion detection. IEE describes all of this as The Car as a Wellness Oasis. It’s all brilliant and useful technology.

So, how is all of this, particularly emotion, to be sensed?

Well, it turns out that ordinary CCD cameras can be built to notice all sorts of things, like facial expressions. These, as we all know, indicate emotions. Images of faces produced by such cameras have position indicating nodes overlaid on them and as facial features move, the nodes move with them. Analysing the movements of the nodes in relation to each other allows the system to determine if the subject is smiling, frowning, laughing, showing revulsion, anger, surprise, fear and other emotions based on the movements of the nodes. This type of analysis can also indicate distraction and drowsiness. None of this is particularly new. Research in the field has been going on for years. What will be new is the way the data will be processed and used in fully connected vehicles.

Cameras can also be used to take our pulses by analysing changes to the light-reflective properties of small portions of our skin. These changes to reflectivity occur because of fluctuations in blood flow caused by heartbeat. This will be linked to the entertainment we’re consuming at the time. It’s interesting that online games offer heart rate monitors that are fully intertwined with game play. It’s suggested that this provides a more immersive experience, but an equally important reason is to gather data about how players react to features within the game. This will feed into product development.

Voice is another emotional indicator, but it’s even more than that. Analysis of the way we speak is used as a form of unique identification around the world. However, speech pattern analysis can also reveal much more about us, like level of education, financial position, and social standing. Cars of the future will listen to us and analyse the pronouns, metaphors, tenses, and many other factors we use to come up with a comprehensive view of what we are and what we like. This will also feed into product development and likely spending preferences. Also, keep in mind that data gathered from cars is just one part of the digital classification scheme in which we are enmeshed. Siri, Alexa, our phones, and an increasing number of other listening devices will continue to form the backbone of such data-building systems. Our cars will be another of those devices. Let’s shift gear and talk about China to see where all of this is leading.

China is in love with biometric sensing, particularly facial recognition technology. By now, most people have probably heard of the Chinese mass surveillance system known as Sharp Eyes, often referred to as Black Shield after the dystopian work of fiction with the same name. Sharp Eyes is an indiscriminate mass public surveillance system that spies on all its citizens through a network of over 600 million cameras. The system observes behaviour and adjusts a social score attached to each person observed. Acts judged as positive for society increase scores. Those seen as socially negative deduct points. The system can also identify individuals by the way they walk. It also reads vehicle number plates. 

China has a lot of people, which makes it impossible to analyse the data gathered by manual means. So, the Sharp Eyes camera network is backed by artificial intelligence (AI). People who accrue positive scores receive discounted bills and have their faces put up on public bulletin boards for all to admire. Those who have low scores also have their images posted for all to see. They also have more difficulty getting loans, finding accommodation, and even travelling. The Chinese government says the system is not used for political oppression but there appears to be testimony to the contrary. It is hard to believe but most Chinese seem to approve of the system.

Most Westerners think they would never support such a system. The only problem is that we do, and heartily. Every time we post something on social media, we help to build a data-defined version of ourselves on a server somewhere. All the media we absorb, all the things we buy and all the places we go will eventually feed into our digital profile. Much of it already does. Also, this constantly updated, involuntary digital adjunct to our ‘real’ selves will be aware of our friends and their leanings. Our cars and their sensing abilities will be part of this.

The data that describes us consists of magnetically stored bits. Electromagnetism is a physical system so such data can be considered a physical part of us. True, such data can’t exist without us, but we’re rapidly reaching a point at which we can’t exist without the data. Data from cars of the near future will be a part of the sensing network that defines our lives. It’s also likely that fully connected cars will also control our lives.

Sharp Eyes in China already restricts the mobility of people judged to be unworthy of free movement. When cars are fully connected, the system will probably prevent individuals from using them, even if they own them. This hasn’t happened yet, but we’re betting that it will. What about here? Would that level of control ever be accepted in a free and democratic society like ours? Interesting question.

The recent protests that governments around the country did not want people to attend went ahead anyway. We must wonder if governments would have allowed that to occur if they had the ability to simply turn off cars belonging to protesters. It’s conceivable that the authorities could apply some sort of reverse geo-fencing that denies access to certain areas by people judged likely to be headed for a protest based on their data signatures. We think yes. Really, it wouldn’t be a lot different to setting physical traffic control barriers in place.

None of these concepts are outlandish. All this sensed data is stored on servers around the world. Government agencies routinely get court orders against the companies holding such data. There are court cases in progress now trying to decide all of this. These are real issues.

There’s a great deal more to this subject and an American professor called Shoshana Zuboff covers it in her book, entitled The Age of Surveillance Capitalism. At the very least, her many interviews and lectures online are extremely informative. The upshot of her work is that surveillance is everywhere in the West and it's completely secretive and habituated. Cars of the near-future are going to be an interlinked part of this system and you will not be able to refuse the gathering, processing, and on-selling of data from your car and other sources.

Words: Paul Tuzson. As featured in Australian Automotive August 2020.