Add support to start the crawl both on domain and sitemap (including sitemap index) - flag any differences, e.g any links that is found within sitemap that is not found in crawl, vice versa.
The new field “Is in sitemap” has been added to the Data Explorer. Before the crawl we analyse the sitemap from domain.tld/sitemap.xml and then show if the page is present in the sitemap. You can also start crawling your website from sitemap by choosing sitemap as a seed. Please let me know if this is something you were looking for.
@Andrey Kirillov: Yes, workaround. But check the settings again after saving. Go back and open the settings again. The saves sitemap url is gone (or only not visible?)
I would add this feature in the core, that the crawler check the sitemap if it exist, crawl it, if not then not -> and then in the project settings
"crawl the sitemap if available?" Yes/No. done.
> But check the settings again after saving. Go back and open the settings again. The saves sitemap url is gone (or only not visible?)
This is a bug, we are already working to fix it. Sorry for the inconveniences.
And thanks for the suggestion. This feature is planned to be improved.
@Andrey Kirillov: Check ;-)
http://take.ms/0uGkV i told you check the competitors :D
@André Deiß: great
@Tim Soulo is there only an ETA?
Also we want to know is there a 404 or 500 pages in sitemap files. So we want to crawl our sitemap files too.