How to detect (modified, headless) Chrome instrumented with Selenium (2024 edition)
In this article, we present 4 efficient techniques to detect bots that leverage Selenium with headless and non-headless Chrome. These techniques have been tested in June 2024.
TL;DR:
If you just want the code of the detection techniques, you can only have a look at the code snippet below. The remainder of this article goes into the details of these techniques and explains how some of them can be bypassed by attackers. The 4 techniques work as follows:
- Using the user agent HTTP headers or with
navigator.userAgent
in JS to detect user agents linked to Headless Chrome: isMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/126.0.6478.114 Safari/537.36
- Similarly, by detecting the presence of the
HeadlessChrome
substring in thesec-ch-ua
header
- By detecting if
navigator.webdriver = true
in JavaScript
- By detecting the side effects of CDP (Chrome DevTools Protocol)
JavaScript code to detect (headless) Chrome instrumented with Selenium:
let isBot = false;
if (navigator.userAgent.includes("HeadlessChrome")) {
isBot = true;
}
// Test the sec-ch-ua header
const hints = ['fullVersionList'];
const res = await navigator.userAgentData.getHighEntropyValues(hints);
if (res['brands'].some((obj) => { return obj.brand.indexOf('HeadlessChrome') > -1})) {
isBot = true;
}
if (navigator.webdriver) {
isBot = true;
}
var cdpDetected = false;
var e = new Error();
Object.defineProperty(e, 'stack', {
get() {
cdpDetected = true;
}
});
// This is part of the detection, the console.log shouldn't be removed!
console.log(e);
if (cdpDetected) {
isBot = true;
}
if (isBot) {
console.log("Your bot has been detected!")
}
Technique 1: How to detect an unmodified Headless Chrome automated with Selenium
To illustrate the first detection technique, we create a simple bot based on Headless Chrome and Selenium. The bot visits https://deviceandbrowserinfo.com/http_headers and we take a screenshot of the page to observe the HTTP headers sent by our bot:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(options=chrome_options)
try:
driver.get('https://deviceandbrowserinfo.com/http_headers')
screenshot_path = './vanilla-hc-selenium.png'
driver.save_screenshot(screenshot_path)
finally:
driver.quit()
We obtain the following HTTP headers:
![](/assets/img/headers_selenium_chrome_vanila.png)
Thus, we notice that the user agent sent by an unmodified
headless chrome instrumented with Selenium indicates the presence of Headless Chrome. Note that the user
agent can also be obtained from the client side, e.g. if you want to exclude headless Chrome traffic from
your google analytics. You can access it using navigator.userAgent
. In the case of our bot, it
returns
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/126.0.6478.114 Safari/537.36
.
Disclaimer: this detection technique
works only for Headless Chrome. It doesn’t work for normal Chrome instrumented with Selenium since the user
agent won’t contain the HeadlessChrome
substring.
Technique 2: Using the sec-ch-ua to detect headless Chrome instrumented with Selenium
Similarly to the previous technique, this detection
technique works only for Headless Chrome. In the previous screenshot, we also notice that the
sec-ch-ua
header indicates the presence of Headless Chrome:
"Not/A)Brand";v="8", "Chromium";v="126", "HeadlessChrome";v="126"
.
Thus, we can use it to detect the presence of Headless
Chrome instrumented with Selenium, either on the server side by looking at the value of the
sec-ch-ua
HTTP header, or on the client side by collecting information about the client hints
using the navigator.userAgentData.getHighEntropyValues
function:
const hints = ['fullVersionList'];
const res = await navigator.userAgentData.getHighEntropyValues(hints);
if (res['brands'].some((obj) => { return obj.brand.indexOf('HeadlessChrome') > -1})) {
console.log("It's a bot!");
}
Technique 3: How to detect a modified (headless) Chrome instrumented with Selenium
The third technique works both for headless and non-headless Chrome and can also be used to detect bots that changed their user agent.
To lie about the user agent, we can use
chrome_options.add_argument(f"user-agent={user_agent}")
with the user agent we want to
forge.
chrome_options = Options()
chrome_options.add_argument("--headless")
user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
chrome_options.add_argument(f"user-agent={user_agent}")
driver = webdriver.Chrome(options=chrome_options)
try:
driver.get('https://deviceandbrowserinfo.com/http_headers')
user_agent_in_browser_js = driver.execute_script("return navigator.userAgent")
print(f"The user agent in the browser is {user_agent_in_browser_js}")
# The user agent in the browser is Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
webdriver_value = driver.execute_script("return navigator.webdriver")
print(f"Value of webdriver is {webdriver_value}")
# Value of webdriver is True
finally:
driver.quit()
Once we change the user, it doesn’t contain the
HeadlessChrome
substring anymore, even when using Headless Chrome.
However, from JavaScript, you can still detect bots that
forged their user agent by verifying if the navigator.webdriver
property is equal to
true
.
Disclaimer: note that if you change the
user agent this way, the sec-ch-ua
header still contains HeadlessChrome
.
Technique 4: How to detect a modified (headless) Chrome instrumented with Selenium that removed navigator.webdriver = true
The navigator.webdriver = true
property
presented in the previous section can easily be removed by using the
chrome_options.add_argument("--disable-blink-features=AutomationControlled")
argument
when creating the Selenium Chrome instance:
chrome_options = Options()
# Note that we removed the Headless flag since this detection also work on non-headless Chrome
chrome_options.add_argument("--disable-blink-features=AutomationControlled")
user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
chrome_options.add_argument(f"user-agent={user_agent}")
When a browser is created this way,
navigator.webdriver
returns false
and can’t be used anymore for detection.
Thus, to leverage attackers that actively lie about their
nature by forging their user agent and by getting rid of navigator.webdriver
, we need to find
another detection technique. We can use CDP detection, a technique I presented in a recent DataDome
blog post.
Under the hood, Selenium leverages the Chrome DevTools Protocol (CDP) to instrument (headless) Chrome. By using a specially crafted challenge shown below, we can detect the use of CDP, and therefore the fact that a browser is automated:
var detected = false;
var e = new Error();
Object.defineProperty(e, 'stack', {
get() {
detected = true;
}
});
console.log(e);
If the value of detected
is equal to
true
then it means that the browser is automated. One of the side effects of this detection
technique is that it will flag human users with dev tools open as bots. Note that the
console.log(e)
is part of the challenge since it is what triggers the serialisation in CDP. You
can find more details about this challenge in my DataDome
blog post.
Bot detection test on the modified Selenium Chrome
We execute the bot detection test of https://deviceandbrowserinfo.com/are_you_a_bot
on our modified version of Selenium Chrome. As shown on the screenshot below, even the version that 1) is
not based on Headless Chrome and 2) removes navigator.webdriver
is detected thanks to the CDP
detection.
![](/assets/img/results-bot-test-selenium-modified.png)
Limits of current detection techniques
In this article, we presented 4 different detection techniques that can be used to detect bots based on (headless) Chrome instrumented with Selenium:
- Using the user agent HTTP headers or with
navigator.userAgent
in JS to detect user agents linked to Headless Chrome: isMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/126.0.6478.114 Safari/537.36
- Similarly, by detecting the presence of the
HeadlessChrome
substring in thesec-ch-ua
header
- By detecting if
navigator.webdriver = true
in JavaScript
- By detecting the side effects of CDP (Chrome DevTools Protocol)
Even though these detection techniques are quite effective
— they have no false positives, besides the CDP detection technique that will flag people with the dev tools
open — sophisticated attackers are aware of it and started to develop countermeasures to avoid being
detected. In particular, certain frameworks, such as nodriver enable bot developers to bypass CDP
detection by avoiding the use of the Runtime.enable
CDP command.