ensl.org/public/robots.txt

# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
User-Agent: *
Disallow: /data_files
Disallow: /directories
Disallow: /movies
Purged git history and removed sensitive information. 2014-03-23 00:22:25 +00:00			`# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file`
			`#`
			`# To ban all spiders from the entire site uncomment the next two lines:`
Prevent crawls of large data files 2015-08-20 12:56:07 +00:00			`User-Agent: *`
			`Disallow: /data_files`
			`Disallow: /directories`
			`Disallow: /movies`