Late to this thread, and strictly amateur myself, but on the basis of advice from others smarter our system picks up a lot. FYI:



We have a decent long list of denys in .htaccess
Robots.txt...