ensl/ensl.org
0
0
Fork 0
mirror of https://github.com/ENSL/ensl.org.git synced 2024-12-26 04:21:36 +00:00
Code Issues Projects Releases Packages Wiki Activity

Prevent crawls of large data files

Browse source
This commit is contained in:
Chris Blanchard 2015-08-20 13:56:07 +01:00
parent 8285f15587
commit 6312e177f5
1 changed files with 4 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
public/robots.txt
View file

@ -1,5 +1,7 @@
# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file # See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
# #
# To ban all spiders from the entire site uncomment the next two lines: # To ban all spiders from the entire site uncomment the next two lines:
# User-Agent: * User-Agent: *
# Disallow: / Disallow: /data_files
Disallow: /directories
Disallow: /movies