A user agent setting is a must for me.

1. To be able to Crawl mobile site (many sites have separate site URLs or Dynamic serving of mobile theme)

2. Crawl as Google-bot. Some sites do different things depending on user agent, such as pre-rendering if the site is based on JS