Human beings are naturally good at deciding when somebody is trying to cheat them. Evolution has
created these skills over eons of time. Often we are so good at not being cheated that we don’t even
recognise how we successfully avoided a trap. Much of our skill lies in the interpretation of minute
visual facial cues and nuances that helped build the trust for early humans to live in communities
and work together. Yet today a growing number of our interactions take place online and we have
yet to evolve the skills we need to not only survive but prosper in this new environment.
People worry about artificial intelligence (AI) as though this was a problem for the future.
Unfortunately the future is already with us. Surprisingly, it only takes an extremely tiny piece of
artificial intelligence to fool us, just a few relatively simple lines of scripting code. For example,
frequently those Facebook likes, or Twitter tweets, or trending stories on Reddit that you receive
haven’t originated from a human being but come from an automated “bot” and you probably didn’t
even notice. Today there is more traffic from automated bots on the Internet than there is human
traffic and as the chart above shows – unfortunately there are more bad bots than good bots. That
means a lot of people are being duped a lot of the time when they use the Internet.
Unless you are somebody who follows Internet trends closely you probably know very little bots but
its important that you start to learn about them - what they can do and how they fool people into
believing they are human. And their numbers are rising. Don’t get me wrong I’m not against bots -
some good bots can be very useful, its the bad bots you have to be concerned about. Bots are a
made up of a series of automated instructions (scripts) that can be run over the Internet. They are
becoming increasingly sophisticated and therefore very much harder to detect. In the early days of
the Internet a Web server could control the behaviour of a bot by having a “robot.txt” file that
instructed what a bot could or could not do with the information on that Web server. Web servers
identify humans apart from bots because humans use browsers to view Internet content and bots
don’t. Now many bots disguises themselves as humans by including a few extra lines of code. Today
Web servers are fooled by bots who appear to be using a Web browser and can also appear to
emulate human mouse click behaviour. Humans are just as easily fooled sometimes to devastating
For example: a few rudimentary lines of code instructed a small army of bots masquerading as
humans to help Russia support Donald Trump’s election campaign. Maybe Trump would have won in
any event, we will never know the effect that thousands of non-human Twitter accounts firing off
their messages in the alphabetical order of their false identities, a few seconds apart, on the day of
the election had on the U.S. result. Each message was tailored to appeal different groups of people
based on their previous Twitter behaviour so the message would chime with their own personal
biases. Most people thought these messages were coming from real people, few people realised
these messages were from automated bots with fake human identities.
Gartner, the information technology consultancy, has amongst their 2018 predictions forecast that
“By 2022, the majority of individuals in mature economies will consume more false information than
true information.” To which I would add that the bulk of that false information will be coming from
bots disguised as humans. Of course some pundits think this prediction is absurd, to quote Steve
Andriole writing in Forbes magazine “While I agree that fake news will increase, it’s impossible to
predict what regulations might appear, what real-time fact-checking might occur, and how the
technology giants will address the fake news problem. Bad prediction.” I disagree and think that this
view greatly underestimates the rapid progress being made in bot technology. Gartner being a
technology consultancy has understood that we are about to experience an invasion of bots.
Fake news is a new name for old style propaganda with a new social media twist. These days the
quantity of fake news is being greatly amplified by fake human messenger bots. Part of the reason
for this is that it is getting very, very easy to make a bot. Back in 2016 at Microsoft Build 2016
conference, CEO Satya Nadella said "Bots are the new apps," as he announced Microsoft’s Bot
Framework. This is a set of tools for software developers that does a lot of the lower level work in
building a sophisticated bot. This Bot Framework has now reached the marketplace as has
Facebook’s Workplace Bots. Facebook even has instructional videos like this one on how to build a
great bot. And you can even build a bot without coding using tool like Converse AI.
The chart above uses data from Incapsula’s 2016 Bot Traffic Report, so let’s look at some of the good
bots and the proportion of Web traffic they take up. Firstly, there are the “monitoring” bots that
check that websites and all their functions are up and running correctly. Monitoring bots make up
1.2% of Web traffic and they help maintain the sites we visit. The second smallest proportion of bot
Web traffic is generated by “commercial” bots that crawl the Web automatically extracting data from
the authorised websites they visit. Think price comparison websites or advertising or marketing
trackers. To my mind some of these bots border onto the dark side of the Web although compared
to the bad bots these commercial bots behave in a much less malign way. These represent 2.9% of
Web traffic. Thirdly, there are search engine bots (sometimes called spiders and crawlers) like the
Googlebot or Bingbot that captures and indexes Web pages so they can be retrieved by search
engines. Most people want their website content to be discoverable by people using search engines
so these are considered good bots and they can be controlled using a robots.txt file mentioned
earlier. Search engine bots make-up 6.6% of Web traffic. The last and most frequent good bots are
the “feed fetchers” that capture content from websites and shuttle it to mobile and Web applications
e.g. news feeds. Feed fetchers make-up 12.2% of Web traffic.
Turning to the bad bots, these can range from being just a nuisance to the criminal or downright
diabolical. You will probably noticed the results of “spammer” bots, although they only make-up
0.3% of Web traffic. Using millions of captured email addresses spammer bots automatically
proliferate messages about Ukrainian ladies seeking love and sex or Nigerians seeking investment
partners. Spammer bots are the polluters of the Web although good spam filters can block the
majority of their output. 1.7% of Web traffic comes from “scraper” bots who lift unauthorised data
for websites, including content to repackage for free or the prices used by commercial websites.
2.6% of Web traffic originates from “hacker” bots that are permanently scouring the Web for
vulnerabilities to be exploited by data theft and malware injection. The creators of these bots can be
criminals after money or foreign states searching for secret information. A good example of the
latter is the constant attacks by both Western powers and the Russian FSB to use Kaspersky anti-virus
software as a search engine to detect top secret information.
By far the largest amount of Web traffic from any type of bot at the moment comes from
“impersonator” bots at 24.3% of Web traffic. Using false identities to bypass log-in details these bots
break most types of security. Identified as human these impersonator bots can be bought by the
thousand for Facebook Likes, Twitter or Instagram Followers. Hard to detect on social media
because their identities have usually been created manually, they are difficult to distinguish from
real people. They even come complete with matching mobile numbers for two step identification. If
you want to understand how this is done read this article, alternatively select a country and build
your own profile using an online Random Identity Generator. Some Facebook advertisers reckon
that 90% of their clicks can come from bots and not human beings. So far their only solution to the
problem is to try and claim a refund. When combined into automated armies, impersonator bots
can bring down government websites and freeze a postal tracking service, or extort money from
commercial websites. These attacks are called Distributed Denial of Service (DDoS) and are more
common than you think. Check out this website to view real-time attacks that are happening today,
or view previous attacks and the traffic they generated.
It is ironic, that Alan Turing, the mathematical genius who conceived the modern computer in the
1930s, created his famous “imitation test” to show how it was possible for a machine to appear to
convince a human that it was capable of thought. Now commonly called the Turing Test, examples of
this are played out by the millions everyday as humans interact with fake human bots on Facebook,
Instagram and Twitter in complete acceptance they are human. Without noticing it, we are
undoubtedly going into a future where an increasing amounts of our time will be spent interacting
with machine (artificial) intelligence at the expense of interaction with our fellow humans. What the
current data above indicates is that unfortunately more time is being spent interacting with bad bots
that good bots. You have been warned!
...with analysis & insight...
Online nobody knows you’re a bot...