Ahrefs
Create
Log in
Sign up
Roadmap
Feedback
Site Audit
288
Voters
Jaume Bosch
Powered by Canny
1
Claify how many pages can't be scrapped due robots.txt directive
Jaume Bosch
Just alert the number of pages that can be scrapped or, if it can't be reached (no internal links pointing to it) clarify it's because robots.txt rule
October 28, 2024
Powered by Canny