# /robots.txt file for http://webcrawler.com/ # mail webmaster@webcrawler.com for constructive criticism User-agent: * Disallow: /logs Disallow: /wsb3972182501 Disallow: /moodle disallow: /moodledata disallow: /Software disallow: /TrainingVideos disallow: /Update disallow: /phpBB2 disallow: /data disallow: /download disallow: /33modfurnonline disallow: /33sequences disallow: /33twostage disallow: /aconline disallow: /acsequences disallow: /Airside and Psychrometrics disallow: /download disallow: /pt disallow: /software disallow: /online disallow: /links disallow: /yorkinstructor