In your bot detection systems I do not see open source system like anubis, safeline or sibuna (I am the author of sibuna). Sibuna is built as self-hosted alternative to cloudflare with console UI providing OWASP CRS, JA4 TLS signature from ingress, these are still necessary besides bot detection and cloudflare offer it as enterprise offering.

I have a smaller benchmark for them in my book at https://insanai.github.io/sibuna/. If you can provide your benchmarking code as open source will incorporate this report into sibuna book.

Based on trying to buy anything retail that is scarce, bot detection systems are definitely not working. I wonder how their system would do against https://refractbot.com/ or other popular retail bots.

When they run these benchmarks, they should measure false positives - because it's annoying for a real human to solve a captcha.

If blocking 1 bot request is +1 then making a human solve a captcha should be something like -100. That's how we should grade.

I don't want to say "that's an ad", but I can see an obvious conflict of interest there.

Also, the methodology just compare the true-negative detection rate for known negative, it seems?

> "Proof of Human identified 99% of automated sessions"

> "We measured detection rates for automated sessions but did not evaluate false-positive rates among human respondents."

Trivially easy measure to game if so. "isHuman(x) = return false" gets you a 100%.

Did they use the browsers against sites they owned or 3rd party ones? If the latter, then lots of sites have Cloudflare Bot Management on, without actual strict restrictions on bots (a lot of sites using CLoudflare as a CDN for instance)

While building a project, I needed a crawler to read robots.txt. Out of 10,000 sites, just over 1,200 refused access outright.