User Guide

Robots file generation
A robots.text file enables you to block specific automated bots and web
spiders, which crawl the web in order to find more webpages to add to
their company's servers. Some bots you want to have full access to your
website, such as Google's or Yahoo's, and some others you'd rather not,
such as spam bots, content stealers, e-mail collectors, etc. Note that the
worst of bots do not respect this files directive, as it is purely advisory.
PrestaShop's robots.txt generation tools simply creates a file with
exclusion directives for files and directories that are not meant to be
public, and should not be indexed.
These rules apply to all bots (note the "User-agent: *" string).