| Robots Robots files informs spiders/crawlers
of the time of day to crawl the site, which directories are restricted, the number of
simultaneous fingers, frequency of the hits, and so on.
Standards
None
Guidelines
- A "robots.txt" file should be established on each web server’s site root
directory.
- Subdirectories within the web can also have their own "robots.txt" file.
- Procedures should be established for how to deal with spiders that don’t follow the
rules.
- Reference the HHS World Wide Web Applications and the Internet Best Practices and
Guidelines, http://www.hhs.gov/progorg/oirm/bestguid.html
for additional information.
| Date |
Change
History |
| |
No changes have been made to this page. |
|