ensl/ensl.org
0
0
Fork 0
mirror of https://github.com/ENSL/ensl.org.git synced 2024-12-27 04:51:14 +00:00
Code Issues Projects Releases Packages Wiki Activity

Merge pull request #44 from cblanc/robots

Browse source 
Prevent crawls of large data files
This commit is contained in:
simplefl 2015-08-23 21:34:23 +02:00
commit 2e0bf01460
1 changed files with 4 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
public/robots.txt
View file

@ -1,5 +1,7 @@
# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-Agent: *
# Disallow: /
User-Agent: *
Disallow: /data_files
Disallow: /directories
Disallow: /movies