Given the ever-increasing number of malicious bots scouring the web, many websites are turning to specialized services that advertise their ability to detect bots and block them. In this paper, we investigate the design and implementation details of commercial anti-bot services in an effort to understand how they operate and whether they can effectively identify and block malicious bots in practice. We analyze the JavaScript code which their clients need to include in their websites and perform a set of gray box and black box analyses of their proprietary back-end logic, by simulating bots utilizing well-known automation tools and popular browsers. On the positive side, our results show that by relying on browser fingerprinting, more than 75% of protected websites in our dataset, successfully defend against attacks by basic bots built with Python scripts or PhantomJS. At the same time, by using less popular browsers in terms of automation (e.g., Safari on Mac and Chrome on Android) attackers can successfully bypass the protection of up to 82% of protected websites. Our findings show that the majority of protected websites are prone to bot attacks and the existing anti-bot solutions cannot substantially limit the ability of determined attackers. We have responsibly disclosed our findings with the anti-bot service providers.