Crawl both site and sitemap
complete
Andrey Kirillov
marked this post as
complete
This has been implemented a while ago. It's possible to start a crawls from both homepage and sitemap. There is a chart called "Discoverable URLs by crawl source" that whether all URLs are discoverable from different crawl sources.
Photo Viewer
View photos in a modal
Andrey Kirillov
The new field “Is in sitemap” has been added to the Data Explorer. Before the crawl we analyse the sitemap from domain.tld/sitemap.xml and then show if the page is present in the sitemap. You can also start crawling your website from sitemap by choosing sitemap as a seed. Please let me know if this is something you were looking for.
Photo Viewer
View photos in a modal
A
André Deiß
Andrey Kirillov: Yes, workaround. But check the settings again after saving. Go back and open the settings again. The saves sitemap url is gone (or only not visible?)
I would add this feature in the core, that the crawler check the sitemap if it exist, crawl it, if not then not -> and then in the project settings
"crawl the sitemap if available?" Yes/No. done.
Andrey Kirillov
André Deiß:
> But check the settings again after saving. Go back and open the settings again. The saves sitemap url is gone (or only not visible?)
This is a bug, we are already working to fix it. Sorry for the inconveniences.
And thanks for the suggestion. This feature is planned to be improved.
A
André Deiß
Andrey Kirillov: Check ;-)
http://take.ms/0uGkV i told you check the competitors :D
Ankur Kanasagara
André Deiß: great
Ankur Kanasagara
like
A
André Deiß
Tim Soulo is there only an ETA?
Kerem Süha Mete
Also we want to know is there a 404 or 500 pages in sitemap files. So we want to crawl our sitemap files too.
Tim Soulo
marked this post as
in progress