From sex to wealth... ...web searches reveal what we're really thinking
From racism to our secret concerns, our search history reveals the dark side of humanity
As sure as night follows day, we all tend to hide our real selves. To our loved ones, on our social media pages, to anonymous pollsters and to ourselves.
Studies have shown we are even dishonest in anonymous surveys - to hide embarrassing thoughts and behaviours so we can look better than we really are.
But a data scientist who specialises in human behaviour has used our Google searches to shine an insightful, fascinating and sometimes disturbing light into the darkest corners of the human mind.
Seth Stephens-Davidowitz, a PhD student in economics at Harvard, releases his book this week, Everybody Lies: Big Data, New Data and What the Internet Can Tell Us About Who We Really Are.
And it turns out we are meaner, more self-absorbed, racist and - most of all - more sexually complex than anyone could have previously imagined.
After four years analysing Google data on issues such as sex, social media, body image, politics and mental health, Stephens-Davidowitz found: "The more impersonal the conditions, the more honest people will be. Everybody lies and yet Google data reveals big truths."
The Ivy League scientist explains how our everyday searches, when multiplied by millions, reveal profound realities.
"People tend to keep their prejudices to themselves, but Google lets us see inside their heads. For example the 'N' word is included in seven million American searches every year. Rap songs can't account for this because the word almost always used is "n*gga", not "n*gger", so the results can't be accounted for in hip-hop lyrics.
"We could also see when searches for 'N' word jokes were most common - basically whenever African-Americans are leading the news. The times when such searches were highest was immediately after Hurricane Katrina struck in 2005, when the media showed images of black people in New Orleans suffering, and they also rose during Barack Obama's first presidential election victory. Even on Martin Luther King Jr Day searches for the N word rose an average of 30pc."
In addition to subconscious racist tendencies, Stephens-Davidowitz was able to uncover an equally disturbing truth about sexism - it often starts at home.
"I was able to use Google searches to find evidence of prejudice against young girls by their own parents," he says. "Parents are two-and-a-half times more likely to ask 'Is my son gifted?' than 'Is my daughter gifted?'. This is despite the fact that young girls have consistently been shown to have larger vocabularies and are more likely to use more complex sentences.
"Where parents do seem to take more interest in their daughters is in their appearance. Parents search 'Is my daughter overweight?' twice as frequently as they enter the the same question about their sons. Parents are also one-and-a-half times more likely to ask whether their daughter is beautiful than whether their son is handsome."
Asked if the research has changed his views on humanity, Stephens-Davidowitz says he is still inclined towards optimism.
"Yes, some of the research is definitely disturbing, but I do think there is a positive in all of this - while it's not always the most cheery data, instead of just pretending these problems don't exist, we can now acknowledge the reality and work on improving it."
On the area that presented Stephens-Davidowitz with his most surprising revelations, he quotes Steven Pinker who said 'I like to think that nothing can shock me but I was shocked aplenty by what the internet reveals about human sexuality': "For example Google showed that women are more concerned with finding tell-tale signs about their partner's sexuality than the prospect he is cheating. The question: 'Is my husband gay?' is 10pc more likely to complete searches that begin 'Is my husband...' than the second-place word, 'cheating'."
Stephens also explains: "I found there is twice as many complaints that 'a boyfriend won't have sex with me' than a 'girlfriend won't'. Which goes against the conventional wisdom that says men want sex all the time while women are the ones who withhold it."
In terms of men's questions about their body, the data shows what many already knew: "Men are obsessed with their penis size. It is very striking in the data," he says.
"Men ask more questions about their penis than any other body part - they also Google more questions about it than about their ears, nose, throat, lungs, liver, feet and brain combined
"They also ask more about how to make their penis bigger than how to tune a guitar, change a tyre or make an omelette. When it comes to men and the ageing process, more men will search 'Is my penis getting smaller?' than other health concerns such as 'Is my blood pressure getting higher?' or 'Is my memory getting worse?'
"Another surprising finding was the fact that while men are more interested in finding out how to last longer in bed, women are more frequently searching how to make a man climax quicker, so they are kind of on two different planes."
But the most shocking revelations concern our secret sexual desires. Looking at one particular subset, he says: "Among women's top PornHub searches is a genre of pornography that will disturb many readers - sex featuring violence against women. In fact 25pc of female searches for straight porn focus on the pain and/or the humiliation of the woman - through searched words like 'painful' and 'crying', for example.
"Searches for non-consensual sex through words such as 'rape' or 'forced' - show that, even though these videos are banned on PornHub, they are twice as common among women as men. If there is a genre of porn in which violence is perpetrated against a woman, my analysis of the data shows that it almost always appeals disproportionately to women."
But he warns: "To stress the obvious, this does not mean women want to be raped in real life. It also doesn't make rape any less horrific a crime. But what the data does tell us is that sometimes people have fantasies they wish they didn't have and which they may never share with others."
In the same way, he says: "Among the most surprising findings I discovered during my data investigations is that a shocking number of people visiting mainstream porn sites are looking for portrayals of incest - including mens' fantasies about their own mother."
He goes on: "The other thing that was very striking was just how much variation there is in what we are attracted to. We usually divide people sexually into two groups -you're either gay or straight. But you see there is so much variation within these groups that I think there are so many more dimensions to sexuality than I think we originally thought. People just search for 'overweight women' or 'skinny women' or 'elderly women' or 'older men' - like that's all they are attracted to really. They have primary types - similar to the way that some people are just attracted to one gender, people are just attracted to one body type or one type of person."
Elsewhere, when it comes to our physical well-being, searches for health conditions are most common in the middle of the night - and at 2am people turn to Google for information and reassurance. It is here that the area of anxiety showed other trends: "Urban areas are places with higher education and people tend to be more open in discussing anxiety there, but actually anxiety tends to be higher in rural areas with lower levels of education. I also think some of the things that cause anxiety are a lot different to what people say."
The data scientist found people often pretend to care about others and the world far more than they actually do - and in fact we are quite selfish animals at heart.
"People try to say they are very anxious about world events, political developments, global warming and legal changes, but I haven't seen that in the data," he says.
"They tend to be more anxious about their own personal situations - their jobs, health and relationships. Very rarely do you see evidence of world events playing a role there. Of all the things people worry about, money is the biggest concern."
When it comes to people feeling isolated, he says: "That is one where people just type 'I am lonely' into Google where it is not really clear what they are hoping to find. That's one of the stranger things about Google - people don't just type questions, they type statements. "Sometimes it's harder to know why things are happening - just the fact that they are actually happening."
Seth Stephens-Davidowitz is author of Everybody Lies: What the Internet Can Tell Us About Who We Really Are, published by Bloomsbury, price €19.18