Anti-PBN Crawler
Constantine Kalivas
It is well known that black hats and PBN are blocking all formal seo trackers (especially ahrefs)
Some may also present ahrefs with different content altogether. In such cases it is frequent to have sites ranking in 1st page of google with practically zero back-links reported by ahrefs.
It would be nice when the ahrefs bot gets blocked while crawling from a specific domain multiple times, to have an alternated IP/hostname that will by more dynamic (and not publicly declared) thus not blacklisted, so it would go on and retry the carwl and potentially flag the site as PBN as well.
This would provide tremendous insight when going after for competitive keywords & against blackhats.
A
André Deiß
In my opinion it would be even better if the ahrefs bot does not respect the robots.txt at all. How a backlink tool can do that and still work holistic, like your wrote Constantine, its strictly not possible. Therefore it would be much better when the bot does not respect this at all. Or you think Google will do that? For sure not, even when they declare that they know every single website on the net and if they dont know about nobody else will as well.