9th January, 2022
8 Min read
Book a Demo
Introduction
Bots today outnumber human users in eCommerce sites: From 15% in 2017, to 30% in 2019, to 64% in 2021. Some extreme cases we’ve witnessed peaked in 90-99.8% bot traffic. But perhaps the more concerning bit is the traffic share of bad bots: an approximate 39% of all internet traffic in 2021.
Hackers are bad enough, but the bad bot is the most prolific of their tools of trade: The bad bot never has a timeout because the design enables it to be an ongoing, ever-evolving, over-encompassing entity of destruction and mayhem. A bad bot is like the terminator but a lot more stealthy and compact.
Reduce your AWS costs by over 50%
Discover your Cloud Saving Potential – Answer just 5 simple questions. AppsFlyer, Playtika, Lufthansa, IBM, top leading companies are already using our FinOps services.
Despite their size, the destruction of these bad bots can speak volumes. In February of 2020, Amazon’s AWS fell victim to the largest DDoS attack in history. Thankfully, AWS was relatively well-prepared for the attack so the outage was minimal. However, the attack requested roughly 2.3Tbps of data from AWS.
This example is far from the only instance. Still, it is a sobering case, reminding business owners that if AWS was brought down, it is unlikely any other business can completely elude bot attacks. Still, Anti-Bot innovation can take you a great way in defending your business from their adversary effects.
Check out this case study for some real-life examples:
The Dangers of Bad Bots
Bad bot attacks can cause all sorts of damage, from messing with your performance data through to bringing down your website. Here are some main forms of attacks utilizing bad bots:
- Skewing analytics: When your website is targeted by massive bot traffic distributed upon the globe, it prevents website admins from knowing where real user traffic is coming from and optimizing accordingly. It can lead to your organization making poor decisions due to low-quality data.
- Application DDoS: This type of attacks is meant to make your website unavailable to human users, by flooding your backend servers with bot requests. Downtime results in reputational and financial damage, as every minute of it might cost thousands of dollars to the average online business.
- Account takeovers or card skimming: Here, bad bots repeatedly attempt to login to an account with compromised credentials, or checkout with stolen credit or debit card information – even information leaked from other sources. These repeated attempts occur in rapid succession until an active account or card is found. This allows the bot operator to steal any funds or store credit currently held in the account, or use any valid payment details to carry out fraudulent transactions.
Why are Bad Bots so Hard to Defeat?
There are many reasons why bad bots are continuously threatening the safety and security of businesses throughout all industries. Here are the two most dynamic reasons why bots have become bad news:
Bad Bots have become sophisticated: (And cheap!) The rise of using bots in everything from eCommerce to customer service has made the market for bad bots sophisticated and widely accessible. You can get a botnet (an army of a few hundred bots) for between $200 and $700. This price averages at about $0.50 per bot, and for that price, you receive an SLA (Service Level Agreement) that offers bots that can imitate human behavior so well that common security challenges are rendered useless.
Good & Bad bots aren’t so easy to tell apart: You shouldn’t ward off bots completely, as you can have bots that are “good” to your site, even if they originate somewhere else. (From search engines or business partners.) However, bots have become so advanced, the proper flagging of good and bad bots can become skewed if you do not have the appropriate fraud detection systems in place.
Watch on-demand webinar: Defending eCommerce from Next-Gen Bad Bots.
Solution Evaluation
Sadly, the most widely-known fraud detection technique is also the least efficient one: CAPTCHA. This annoying attempt at bot detection is a UX disaster. It would be one thing if CAPTCHA worked, but the evolution of bots has made CAPTCHA more ensnaring to humans than bots.
So, is there an option that helps navigate the tricky cyber collection of human and bot attention?
Yes, there is. And it stems from a new, mindful approach to categorize each visitor.
Friend, Foe, or Human
Simply put, the way today’s latest bot mitigation techniques decipher between a good bot, a bad bot, and a human is the javascript. The inherent Javascripts extract details about how bots/humans interact with the web page throughout these fraud detection tactics. Some of the easiest examples include mouse movement and keystrokes. Yet, there are plenty of other methods javascript can employ to make this distinction:
- Analyzing HTTP headers
- Check browser type (bots typically use the same browser to hide under)
- Detecting non-human mouse movement and keystroke speed across the page
- Via telemetry information:
- Does the phone battery go down over time?
- Is the accelerometer in the phone moving (and in a non-robotic way)?
- Is the time correct?
Unfortunately, even if the javascript suspects the user to be a bot, the platform cannot be 100% sure. So, rather than blocking the user outright, the system sends a new challenge to the bots. This challenge is not easy to get the first time; it uses repetition and evolving understanding to single out the bots.
How Bots Get Their Covers Blown
As you can see, there are many different methods to root out a bot, but due to the volume of users, it is important to know the indications your fraud detection has for bots.
Token Missing
Any bot mitigation solution will send some sort of challenge to incoming requests to distinguish between bots and real users. If the result concludes there is a ‘missing token” or missing fingerprint, it means that the bot simply didn’t execute that challenge and didn’t send the response to that same challenge.
There are many reasons the system may initiate this response:
- The instances/bots used by the bot operator can’t execute the challenge
- Commercially, it doesn’t compensate for upgrading my instances and my bots’ capabilities to bypass this challenge, but I’ll keep trying to break through in hopes that the “gates open someday.”
- The action can be to continue generating noise. The more noise the bad bot makes, the more work the target has to do.
Explicitly Non-Human Behavior
One main component of human behavior, that I addressed in some of my research papers during my PhD as well as other researchers, is human randomness. Humans are always random in the way they interact with a page and bots cannot imitate this randomness so easily. For instance, please look at the 2 pictures below that easily depict how bot management solutions use “AI” to distinguish between humans and bots:
Bots are becoming ever more sophisticated at mimicking human activity, and from an analytical perspective, humans aren’t that hard to copy. Therefore, the chances of tracking only behavior and blocking traffic based on such a basic algorithm are minimal. Therefore, the algorithm needs to track multiple behaviors and ingest large amounts of data on users & bots alike, in order to characterize the behavior of each.
How 2022 Bot Mitigation Treats Bots Once Detected
When a user trips a warning in a fraud detection system, it sends challenges (like CAPTCHA) to suspicious users. Of course, these fail-safes don’t always work, but when they do, bot mitigation has specified reactions: Anti-bot solutions separate the UX for bots and humans once it recognizes them, by two main techniques:
Option 1: Error Redirection
This is the easier option to develop, and it immediately blocks the bad bot traffic. Error redirections can be:
- Sending an HTTP 403 ‘forbidden’ message
- Send a custom HTTP status code
- Leave the connection to time out
The latter two allow your mitigation to remain undiscovered by the bots. Either way, the bot will uncover little information before it’s thrown out from your website.
Option 2: Fake Information
A common strategy involves serving fake content to bots. For instance, an airline can let bots through but add an extra booking fee. Another tactic is serving bots with old prices, so the airline’s booking backend infrastructure is relieved from dealing with those requests. This increases the “look-to-book” ratio which is a critical metric in airlines, or conversion rate in other forms of eCommerce.
Is There a Cost-Effective Way to Root Out Bots from My Website?
First of all, be advised that being targeted by bots is actually good news: It means your business is successful and being noticed worldwide.
But honestly, there is no such thing as cheap bot mitigation solutions: The effective solutions are managed ones, with a team of security analysts constantly monitoring and updating mitigation rules. Also, there’s lots of R&D invested to make these solutions capable of ingesting huge amounts of data.
However, bot mitigation usually has a positive ROI: It greatly reduces your eCommerce operational costs, saving both compute and fraud costs inflicted by bot traffic.
The goal of bot mitigation is to make bot activity economically counterproductive, so the bot operators won’t come back and won’t retool. Some bot operators, like Online Travel Agencies (OTAs, e.g. Kayak) or ill-intentioned governments, have dedicated full-time teams constantly writing new bots to bypass mitigation solutions. These ones can never be abolished for good, and good bot mitigation is your weapon in this cat-and-mouse game.
Good Bot Mitigation = Revenue Increase
Bot protection will always result in revenue increase – basically you keep sending bad bots to the end of the queue and prevent them from making transactions. This means that all of your inventory is available for human users. It also means that human users will have a faster shopping experience, with less security friction sent their way. Both increase the chances to complete the transaction (AKA look-to-book ratio or conversion rate).
The key is to be able to tell automation from humans and then add friction to bots while reducing it on humans.
Do Buyers Have a Responsibility for Keeping Themselves Safe?
Even with all of the precautions and bot mitigation implemented into a system, safety is not the sole responsibility of the merchants. Merchants need to be vigilant in spotting abnormal behavior, blocking known-bad IPs, and identifying malicious traffic.
However, buyers have a responsibility for their safety, too.
All the outward safety parameters in the world cannot completely help a person who uses their email and password multiple times. Making simple mistakes in username and password creation makes individuals an easy mark, which negates the assurance of most outside security influences.
Therefore, people need to take responsibility for their safety, both as merchants and buyers.
So, Which is the Best Bot Mitigation Come 2022?
While it is essential to ensure your bot mitigation solution is up-to-date, there isn’t a single fraud detection system that fits all. Industry, activity volume and IT department size have a lot to do with the perfect fit for your business.
GlobalDots has access and integration expertise of today’s latest, most innovative options. This allows our solution architects to perfect your posture against bad bots with absolutely no effort.
Contact GlobalDots Today for commitment-free Web Security consultation.