Crawl both site and sitemap
complete
Andrey Kirillov
complete
This has been implemented a while ago. It's possible to start a crawls from both homepage and sitemap. There is a chart called "Discoverable URLs by crawl source" that whether all URLs are discoverable from different crawl sources.
Andrey Kirillov
The new field “Is in sitemap” has been added to the Data Explorer. Before the crawl we analyse the sitemap from domain.tld/sitemap.xml and then show if the page is present in the sitemap. You can also start crawling your website from sitemap by choosing sitemap as a seed. Please let me know if this is something you were looking for.
A
André Deiß
Andrey Kirillov: Yes, workaround. But check the settings again after saving. Go back and open the settings again. The saves sitemap url is gone (or only not visible?)
I would add this feature in the core, that the crawler check the sitemap if it exist, crawl it, if not then not -> and then in the project settings
"crawl the sitemap if available?" Yes/No. done.
Andrey Kirillov
André Deiß:
> But check the settings again after saving. Go back and open the settings again. The saves sitemap url is gone (or only not visible?)
This is a bug, we are already working to fix it. Sorry for the inconveniences.
And thanks for the suggestion. This feature is planned to be improved.
A
André Deiß
Andrey Kirillov: Check ;-)
http://take.ms/0uGkV i told you check the competitors :D
Ankur Kanasagara
André Deiß: great
Ankur Kanasagara
like
A
André Deiß
Tim Soulo is there only an ETA?
Kerem Süha Mete
Also we want to know is there a 404 or 500 pages in sitemap files. So we want to crawl our sitemap files too.
Tim Soulo
in progress