In this article, we present 4 efficient techniques to detect bots that leverage Selenium with headless and non-headless Chrome. These techniques have been tested in June 2024.

TL;DR:

If you just want the code of the detection techniques, you can only have a look at the code snippet below. The remainder of this article goes into the details of these techniques and explains how some of them can be bypassed by attackers. The 4 techniques work as follows:

1. Using the user agent HTTP headers or with navigator.userAgent in JS to detect user agents linked to Headless Chrome: is Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/126.0.6478.114 Safari/537.36

2. Similarly, by detecting the presence of the HeadlessChrome substring in the sec-ch-ua header

3. By detecting if navigator.webdriver = true in JavaScript

4. By detecting the side effects of CDP (Chrome DevTools Protocol)

JavaScript code to detect (headless) Chrome instrumented with Selenium:

let isBot = false;
if (navigator.userAgent.includes("HeadlessChrome")) {
    isBot = true;
}

// Test the sec-ch-ua header
const hints = ['fullVersionList'];
const res = await navigator.userAgentData.getHighEntropyValues(hints);
if (res['brands'].some((obj) => { return obj.brand.indexOf('HeadlessChrome') > -1})) {
	isBot = true;
}

if (navigator.webdriver) {
    isBot = true;
}

var cdpDetected = false;
var e = new Error();
Object.defineProperty(e, 'stack', {
   get() {
    cdpDetected = true;
   }
});

// This is part of the detection, the console.log shouldn't be removed!
console.log(e);

if (cdpDetected) {
    isBot = true;
}

if (isBot) {
		console.log("Your bot has been detected!")
}

Technique 1: How to detect an unmodified Headless Chrome automated with Selenium

To illustrate the first detection technique, we create a simple bot based on Headless Chrome and Selenium. The bot visits https://deviceandbrowserinfo.com/http_headers and we take a screenshot of the page to observe the HTTP headers sent by our bot:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_argument("--headless")

driver = webdriver.Chrome(options=chrome_options)

try:
    driver.get('https://deviceandbrowserinfo.com/http_headers')
    screenshot_path = './vanilla-hc-selenium.png'
    driver.save_screenshot(screenshot_path)

finally:
    driver.quit()

We obtain the following HTTP headers:

Thus, we notice that the user agent sent by an unmodified headless chrome instrumented with Selenium indicates the presence of Headless Chrome. Note that the user agent can also be obtained from the client side, e.g. if you want to exclude headless Chrome traffic from your google analytics. You can access it using navigator.userAgent. In the case of our bot, it returns Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/126.0.6478.114 Safari/537.36.

Disclaimer: this detection technique works only for Headless Chrome. It doesn’t work for normal Chrome instrumented with Selenium since the user agent won’t contain the HeadlessChrome substring.

Technique 2: Using the sec-ch-ua to detect headless Chrome instrumented with Selenium

Similarly to the previous technique, this detection technique works only for Headless Chrome. In the previous screenshot, we also notice that the sec-ch-ua header indicates the presence of Headless Chrome: "Not/A)Brand";v="8", "Chromium";v="126", "HeadlessChrome";v="126" .

Thus, we can use it to detect the presence of Headless Chrome instrumented with Selenium, either on the server side by looking at the value of the sec-ch-ua HTTP header, or on the client side by collecting information about the client hints using the navigator.userAgentData.getHighEntropyValues function:

const hints = ['fullVersionList'];
const res = await navigator.userAgentData.getHighEntropyValues(hints);
if (res['brands'].some((obj) => { return obj.brand.indexOf('HeadlessChrome') > -1})) {
    console.log("It's a bot!");
}

Technique 3: How to detect a modified (headless) Chrome instrumented with Selenium

The third technique works both for headless and non-headless Chrome and can also be used to detect bots that changed their user agent.

To lie about the user agent, we can use chrome_options.add_argument(f"user-agent={user_agent}")with the user agent we want to forge.

chrome_options = Options()
chrome_options.add_argument("--headless")
user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
chrome_options.add_argument(f"user-agent={user_agent}")

driver = webdriver.Chrome(options=chrome_options)

try:
    driver.get('https://deviceandbrowserinfo.com/http_headers')

    user_agent_in_browser_js = driver.execute_script("return navigator.userAgent")
    print(f"The user agent in the browser is {user_agent_in_browser_js}")
    # The user agent in the browser is Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36

    webdriver_value = driver.execute_script("return navigator.webdriver")
    print(f"Value of webdriver is {webdriver_value}")
    # Value of webdriver is True

finally:
    driver.quit()

Once we change the user, it doesn’t contain the HeadlessChrome substring anymore, even when using Headless Chrome.

However, from JavaScript, you can still detect bots that forged their user agent by verifying if the navigator.webdriver property is equal to true.

Disclaimer: note that if you change the user agent this way, the sec-ch-ua header still contains HeadlessChrome.

Technique 4: How to detect a modified (headless) Chrome instrumented with Selenium that removed navigator.webdriver = true

The navigator.webdriver = true property presented in the previous section can easily be removed by using the chrome_options.add_argument("--disable-blink-features=AutomationControlled") argument when creating the Selenium Chrome instance:

chrome_options = Options()
# Note that we removed the Headless flag since this detection also work on non-headless Chrome
chrome_options.add_argument("--disable-blink-features=AutomationControlled")
user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
chrome_options.add_argument(f"user-agent={user_agent}")

When a browser is created this way, navigator.webdriver returns false and can’t be used anymore for detection.

Thus, to leverage attackers that actively lie about their nature by forging their user agent and by getting rid of navigator.webdriver , we need to find another detection technique. We can use CDP detection, a technique I presented in a recent DataDome blog post.

Under the hood, Selenium leverages the Chrome DevTools Protocol (CDP) to instrument (headless) Chrome. By using a specially crafted challenge shown below, we can detect the use of CDP, and therefore the fact that a browser is automated:

var detected = false;
var e = new Error();
Object.defineProperty(e, 'stack', {
   get() {
       detected = true;
   }
});
console.log(e);

If the value of detected is equal to true then it means that the browser is automated. One of the side effects of this detection technique is that it will flag human users with dev tools open as bots. Note that the console.log(e) is part of the challenge since it is what triggers the serialisation in CDP. You can find more details about this challenge in my DataDome blog post.

Bot detection test on the modified Selenium Chrome

We execute the bot detection test of https://deviceandbrowserinfo.com/are_you_a_bot on our modified version of Selenium Chrome. As shown on the screenshot below, even the version that 1) is not based on Headless Chrome and 2) removes navigator.webdriver is detected thanks to the CDP detection.

Limits of current detection techniques

In this article, we presented 4 different detection techniques that can be used to detect bots based on (headless) Chrome instrumented with Selenium:

1. Using the user agent HTTP headers or with navigator.userAgent in JS to detect user agents linked to Headless Chrome: is Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/126.0.6478.114 Safari/537.36

2. Similarly, by detecting the presence of the HeadlessChrome substring in the sec-ch-ua header

3. By detecting if navigator.webdriver = true in JavaScript

4. By detecting the side effects of CDP (Chrome DevTools Protocol)

Even though these detection techniques are quite effective — they have no false positives, besides the CDP detection technique that will flag people with the dev tools open — sophisticated attackers are aware of it and started to develop countermeasures to avoid being detected. In particular, certain frameworks, such as nodriver enable bot developers to bypass CDP detection by avoiding the use of the Runtime.enable CDP command.