User Guide

Path and URL options 115
Path and URL options
The following sections describe the Verity Spider path and URL options.
-auth
Syntax:
-auth path_and_filename
Specifies an authorization file to support authentication for secure paths.
Use the
-auth option to specify the authorization file. The file contains one record per line. Each
line consists of server, realm, username, and password, separated by whitespace.
The following is a sample authorization file:
# This is the Authorization file for HTTP's Basic Authentication
#server realm username password
doleary MACR my_username my_password
-cgiok
Ty pe: Web crawling only
Lets you index URLs containing the question mark (?). This typically means that the URL leads
to a CGI or other processing program.
The return document produced by the web server is indexed and parsed for document links,
which are followed and in turn indexed and parsed. However, if the web server does not return a
page, perhaps because the URL is missing parameters that are required for processing in order to
produce a page, nothing happens. There is no page to index and parse.
Example
The following is a URL without parameters:
http://server.com/cgi-bin/program?
If you include parameters in the URL to be indexed, as specified with the -start option, those
parameters are processed and any resulting pages are indexed and parsed.
By default, a URL with a question mark (?) is skipped.
-domain
Ty pe: Web crawling only
Syntax:
-domain name_1 [name_n] ...
Limits indexing to the specified domain(s). You must use only complete text strings for domains.
You cannot use wildcard expressions. URLs not in the specified domain(s) are not downloaded or
parsed.
You can list multiple domains by separating each one with a single space.
Note: You must have the appropriate Verity Spider licensing capability to use this option. The Verity
Spider that is included with ColdFusion MX is licensed for websites that are defined and reside on the
same machine on which ColdFusion MX is installed. Contact Verity Sales for licensing options
regarding the use of Verity Spider for external websites.