Cloudmark Cartridge Installation and Administration Guide
© 2001-2007 Cloudmark, Inc. All rights reserved. Cloudmark, the Cloudmark logo, and other trademarks, service marks, and designs are registered or unregistered trademarks of Cloudmark Inc. and its subsidiaries in the United States and in foreign countries. Other brands and products are trademarks of their respective holders. All product information is subject to change without notice.
Contents CHAPTER 1 Introduction . . . . . . . . . . . . . . . .1 What’s new in Cartridge 3048 . . . . . . . . . . . . . . . . . .1 Cloudmark fingerprinting algorithms . . . . . . . . . . . . . . . .2 Cloudmark Global Threat Network. . . . . . . . . . . . . . . . .2 Micro-updates . . . . . . . . . . . . . . . . . . . . . . . .3 Message scoring . . . . . . . . . . . . . . . . . . . . . . .3 Message categorization . . . . . . . . . . . . . . . . . . . . .4 Cartridge statistics . . . . . . . . . . . . . .
Contents Network interaction. . . . . . . . . . . . . . . . . . . . . . 19 Using HTTP proxies . . . . . . . . . . . . . . . . . . . . . 19 Connection timeout logic . . . . . . . . . . . . . . . . . . . 19 Data files. . . . . . . . . . . . . . . . . . . . . . . . . . 20 Offline files . . . . . . . . . . . . . . . . . . . . . . . . . 20 Data file integrity and security. . . . . . . . . . . . . . . . . . 21 Advanced micro-update configurations . . . . . . . . . . . . . . 21 Using an HTTP proxy . . .
Contents Index . . . . . . . . . . . . . . . . . .
Contents vi
CHAPTER 1 Introduction Cloudmark’s gateway solutions use the Cloudmark Cartridge to deliver the latest Cloudmark anti-abuse technology for your email platform. This guide explains how to install, configure, and administer the cartridge. You can find out what’s new in this version of the cartridge in “What’s new in Cartridge 3048” below.
Cloudmark Cartridge Installation and Administration Guide Chapter 1 • A new fingerprinting scheme provides faster processing. • A new statistics field reports your unique installation ID. See “What statistics are collected” on page 31. Cloudmark fingerprinting algorithms The Cloudmark Cartridge includes Cloudmark’s fingerprinting algorithms, designed to target the most current spamming techniques. Using these algorithms, the Cloudmark Cartridge generates a set of fingerprints for each incoming message.
Chapter 1 Introduction Micro-updates Cloudmark stores message fingerprints generated though the Global Threat Network in near-real-time. Micro-updates are the mechanism that allows Cloudmark customers to download the latest fingerprint data at regular intervals.
Cloudmark Cartridge Installation and Administration Guide Chapter 1 Message categorization When scoring a message with the Cloudmark Authority Engine SDK’s CMAE_Score() function, an application can request that the cartridge return a category and a subcategory for the message. Categories and subcategories are expressed as integers, which are mapped to categories in the .cats file. See the .cats file for the list of categories.
Chapter 1 Introduction Whitelisting A whitelist is a list of trusted senders from whom you always accept email, or email characteristics which indicate a trusted message. This feature of the Cloudmark Cartridge minimizes the filtering of legitimate messages and allows system administrators to conveniently manage the receipt of messages from known safe senders. For complete information, see Chapter 5, “Whitelisting”.
Cloudmark Cartridge Installation and Administration Guide 6 Whitelisting Chapter 1
CHAPTER 2 Cloudmark Cartridge Installation This chapter provides the Cartridge installation instructions: • “The Cartridge installation package” below. • “Installing or updating the Cartridge” on page 8 ! Be sure to refer to the release notes of each Cartridge version for special installation instructions. The Cartridge installation package The Cartridge installation package is provided in either a TAR or a ZIP file, depending on your platform.
Cloudmark Cartridge Installation and Administration Guide Chapter 2 • etc/micro_updates/.implv1 • etc/micro_updates/states/srl_set.package • etc/whitelist.cfg.sample • etc/cartridge.cfg.sample • lib/cartridge.so Additional files are downloaded as micro-updates. For more information about these files, see “Data files” on page 20.
Chapter 2 Cloudmark Cartridge Installation gzip –d –c < SpamDNA-3048.x.x.x-.tar.gz | tar xvf – • For Windows installation, double-click the .zip file, then click Extract. 5 Create the etc/license.cfg file. This file must contain the two-line license text that you received from Cloudmark. 6 If you are updating an existing Cartridge installation, update your cartridge.cfg to the latest defaults listed in the file etc/cartridge.cfg.sample.
Cloudmark Cartridge Installation and Administration Guide Chapter 2 5 If you are upgrading from Cartridge 3046 or earlier, create the etc/license.cfg file. This file must contain the two-line license text that you received from Cloudmark. 6 Update your cartridge.cfg with the latest defaults listed in the file etc/cartridge.cfg.sample.
Chapter 2 Cloudmark Cartridge Installation • (Immunity 2.0.1) /setup/sql/sqlite/cm_egm.db ( is typically /srv/immunity) 9 The default cartridge.cfg and whitelist.cfg files will not be installed in etc/. To create the default configuration files, copy etc/cartridge.cfg.sample to etc/cartridge.cfg and etc/whitelist.cfg.sample to etc/whitelist.cfg.
Cloudmark Cartridge Installation and Administration Guide Chapter 2 See also the Cloudmark Authority Engine Extensions Service for Openwave Email Mx Administration Guide. Installation for Openwave Edge Gx Before starting this procedure, Cloudmark recommends backing up entire $AUTH_HOME directory.
CHAPTER 3 Cloudmark Cartridge Configuration This chapter discusses the configuration settings in the cartridge.cfg file. Please also refer to the Cartridge release notes for updates to these settings. You can configure Cartridge parameters in the cartridge.cfg file – located in the etc/ directory in the root directory of your Cloudmark installation – using any text editor. Table 1 below lists the configuration parameters, as specified in cartridge.cfg. All configuration parameters are case-insensitive.
Cloudmark Cartridge Installation and Administration Guide Table 1 Chapter 3 Micro-updates configuration settings Parameter Value(s) Default Description exclude from stats reports whitelist or proxy auth or whitelist, proxy auth none If present, the corresponding values in statistics reports will be replaced by the string “”. favor analysis over speed yes or no no When set to “yes”, the Cartridge calculates all fingerprints before returning a result for the message.
Chapter 3 Cloudmark Cartridge Configuration Table 1 Micro-updates configuration settings Parameter Value(s) Default Description image processing depth none or low or medium or high high When analyzing images, the Cartridge can apply a variety of fingerprinting algorithms, some more resource-intensive than others. The default value of “high” applies all image-specific fingerprinting algorithms, achieving the highest possible accuracy.
Cloudmark Cartridge Installation and Administration Guide Table 1 Chapter 3 Micro-updates configuration settings Parameter Value(s) Default Description micro-update port Positive integer 80 Connect to an alternate port on the host specified by micro-update hostname. Note that Cloudmark’s micro-update servers only accept connections on port 80. micro-update timeout Positive integer 60 Specifies the timeout period (in seconds) for HTTP requests used when checking for micro-updates.
CHAPTER 4 Micro-Updates Micro-updates is the mechanism that allows the Cloudmark Cartridge to regularly download the latest fingerprint data used to identify abusive messages. The fingerprint data is provided by highly trusted reporters and analysis by the Trust Evaluation System in the Cloudmark Global Threat Network in near realtime. Therefore, proper use of the micro-update mechanism is critical in maintaining Cartridge accuracy as new types of spam, phishing, and virus messages are reported.
Cloudmark Cartridge Installation and Administration Guide Chapter 4 These options are configured using the ‘micro-update interval’ configuration setting in the cartridge.cfg file. See also Chapter 3, “Cloudmark Cartridge Configuration”. Automatic micro-updates By setting the “micro-update interval” configuration setting to ‘auto’, the Cloudmark Cartridge will automatically download the latest micro-update information at the interval defined by Cloudmark in the .acf file.
Chapter 4 Micro-Updates Network interaction Micro-updates are downloaded using standard HTTP requests. If an HTTP proxy is not enabled, then at the specified interval, a download will be attempted over port 80 by default, or the port configured by “micro-update port” on page 16.
Cloudmark Cartridge Installation and Administration Guide Chapter 4 Data files The Cloudmark service currently generates new versions of the full microupdates data files at intervals designed to balance bandwidth usage with the required amount of updates. Delta updates are generated at more frequent intervals to complement the micro-updates files. ! New files will only be generated if new data is available.
Chapter 4 Micro-Updates When a valid set of data files is downloaded from the micro-updates service, they are saved to the /micro_updates/ directory with version numbers in the filenames. New files are loaded from disk at startup. ! Delta micro-updates files are read into memory and are not kept on disk. Data file integrity and security The micro-update files containing data are compressed and encrypted. Data that is not encrypted with the correct key will be ignored.
Cloudmark Cartridge Installation and Administration Guide 22 Advanced micro-update configurations Chapter 4
CHAPTER 5 Whitelisting The Cloudmark product provides system-level whitelisting support, allowing you to pass messages automatically based on domains, IP ranges, envelope, header or body features. Whitelisting configuration settings are stored in the file whitelist.cfg located in the etc/ directory.
Cloudmark Cartridge Installation and Administration Guide Chapter 5 Following are example configurations that allow mail from hosts with the specified IP addresses to bypass spam-filtering: type=host; address=[1.2.3.4] type=host; address=[1.2.3] type=host; address = [192.168.32.0/24]; In the first example, only mail from the host at the exact IP address bypasses spam filtering. The second example allows mail to bypass spam-filtering if only the first three octets of the IP address match.
Chapter 5 Whitelisting Following are example configurations which use header whitelisting to bypass spam filtering for all addresses from the .gov and doj.org domains, respectively: type=envelope; command=[mail from]; value=[@.*\.gov\b]; type=envelope; command=[rcpt to]; value=[@.*\.doj\b]; To match an explicit email address: type=header; header=[From]; value=[\buser@domain\.com\b]; Body whitelisting Body whitelisting checks the body of the email to see if it matches the given regular expression.
Cloudmark Cartridge Installation and Administration Guide Chapter 5 Note that email address strings contain a leading less than sign (<) and trailing greater than sign (>). Either of the following examples can be used when searching for an explicit email address (for both header and envelope whitelisting): type=envelope; command=[rcpt to]; value=[$]; type=envelope; command=[rcpt to]; value=[\buser@domain\.
Chapter 5 Whitelisting Sample whitelist configuration file ## Whitelist Configuration File # # # # # This configuration file defines whitelisted domains, IPs and headers. When an item is matched, the message is guaranteed to be let through, unmodified. # Empty lines or lines where the first non-whitespace character is a ``#'' are ignored. # # # # # # # Type: Host This type of whitelist entry applies to any kind of ip or domain name.
Cloudmark Cartridge Installation and Administration Guide # affect performances. # type = body; regex = [CLOUD.
CHAPTER 6 Cartridge Statistics Reporting The Cloudmark Cartridge collects statistics about message classification and reports these statistics to Cloudmark. These statistics are used in conjunction with feedback data collected from the Cloudmark Network Feedback System (CNFS) to provide customers and Cloudmark with visibility into filtering accuracy at customer sites. ! As of Cartridge 3047, statistics reporting is mandatory. If your organization has special privacy concerns, contact Cloudmark.
Cloudmark Cartridge Installation and Administration Guide Chapter 6 The POST body will be of content type text/plain and contain a collection of keyvalue pairs. Below is a list of the key-value pairs in the POST body: Table 2 POST body key-value pairs Key-value pair Description report = spamdna stats This identifies that this report consists of Cartridge statistics. It may be different for future possible communications from the Cartridge to Cloudmark.
Chapter 6 Cartridge Statistics Reporting What statistics are collected Information regarding the specific Cartridge instance is reported to Cloudmark upon each Cartridge installation. The following is a list of statistics that are collected by the Cartridge if reporting is enabled. Table 3 Statistics key-value pairs Key-value pair Description customer id = The exact value of “customer id” in cartridge.cfg.
Cloudmark Cartridge Installation and Administration Guide Table 3 32 Chapter 6 Statistics key-value pairs Key-value pair Description application name = This identifies the application using this Cartridge. It may be "unknown", meaning that the Cartridge is being used by a non-CMAE application or CMAE < 2.0. It may also be "unspecified", meaning that a CMAE 2.0 application chose to not provide an application name.
Chapter 6 Cartridge Statistics Reporting Table 3 Statistics key-value pairs Key-value pair Description identified e4 = identified e7 = identified e8 = identified e9 = identified e10 = identified e14 = identified e15 = identified e16 = identified e17 = identified e18 = identified as empty = Count of messages identified by specific fingerprint algorithms in the last data period.
Cloudmark Cartridge Installation and Administration Guide Table 3 Chapter 6 Statistics key-value pairs Key-value pair Description category = subcategory . = is the category number, is the subcategory number. Key-value pairs are only sent when > 0. Since subcategories are not always used, there may be more messages classified as a particular category than messages classified in that category's subcategories.
Chapter 6 Cartridge Statistics Reporting Table 3 Statistics key-value pairs Key-value pair Description license id = 0> The license number as found in license.cfg. If the license is not found, the value of this key is zero. clean shutdown = [01] This key indicates whether the cartridge shut down properly during its previous run; 1 if it did, 0 if it did not. installation id = This is a base64-encoded string that uniquely identifies a cartridge installation.
Cloudmark Cartridge Installation and Administration Guide 36 Cartridge reporting configuration Chapter 6
APPENDIX A Logging Log messages are passed programmatically from the cartridge to your application. See your application’s documentation for information about how these messages are exposed.
Cloudmark Cartridge Installation and Administration Guide Table 4 Appendix A Log variables Variable Sample Values Description “Permission Denied”, “Not a Directory”, “No Such File or Directory” Any of the standard error strings returned on Unix or Windows systems related to disk I/O N/A An error code related to encryption or decryption N/A Directory containing authority.cfg, immunity.
Appendix A Logging Copied package microupdate files to "safe" set Displayed during first cartridge initialization after installation when the package files are considered safe. Committed key "report statistics" from "License Cfg" value=yes A configuration parameter was set from the LDAC. Licensing OK (session #)" A new session was retrieved from Cloudmark’s licensing host. () Backend whitelist enabled with (entry/entries) This message is logged when a .
Cloudmark Cartridge Installation and Administration Guide Appendix A extension .part in the micro-updates directory. These files may exist if the cartridge is shut down while a micro-update download is in progress Successful download from network (new serial ). This message is logged when the cartridge successfully downloads and decodes a micro-update file from the network Removed old file .
Appendix A Logging Statistics initialized. Current settings... This message is logged when the statistics module is initialized, and gives a summary of statistics related configuration Connecting to licensing host without authentication Cloudmark’s licensing host has served a checksum file unencrypted. WARN log messages Can not create directory (). Expect errors saving micro-update files.
Cloudmark Cartridge Installation and Administration Guide Appendix A high accuracy, it is important to wait until files are successfully downloaded from the network before passing messages through the cartridge. Can not read information about micro-update directory (). This warning is logged when the cartridge is unable to read any information about the configured micro-updates directory. This message will not be logged if the directory does not exist.
Appendix A Logging instances, but a large number of warnings indicate a problem with the inmemory set of signatures. Could not update signature from local set ([add/remove] portion) Incremental micro-update files may contain updates to meta-data associated with a signature, and this log message indicates a problem updating signature meta-data. Unable to parse mime parts of message, skipping This message is logged when the cartridge is unable to parse the mime parts present in a message.
Cloudmark Cartridge Installation and Administration Guide Appendix A ERROR log messages Failed to read current serials from disk srl_set.current file is unexpectedly missing. The etc/micro_updates/states/ Could not find source %s in current serial state file srl_set.current file is corrupted. Failed to write the current state file (%s) may be a permission error. The The disk may be full or there Failed to remove %s There was a permission error in the etc/ micro_updates/states/ directory.
Appendix A Logging Could not update signatures from network (authorization denied). The license is not authorized to download micro-updates. Could not update signatures from network (license verification failed tid:38977 (No sessions available)). The number of sessions authorized for this license has been reached. Could not update signatures from network (microupdate file XXXX failed the integrity check). The computed checksum is incorrect.
Cloudmark Cartridge Installation and Administration Guide Appendix A Value for “enable micro-updates” must be yes/no/1/0-keeping downloads enabled. This message is logged at startup when the value for “enable micro-updates” in cartridge.cfg is not a yes/no-style value. Can not start networking subsystem (), expect network operations to fail This error is logged on Windows platforms when the cartridge could not initialize networking via WSAStartup().
Appendix A Logging Could not construct HTTP POST for statistics report, will not send report This message is logged when there is a problem constructing the HTTP POST command used to submit a statistics report. If this message occurs, the statistics report will not be sent to Cloudmark. Could not connect to micro-updates host (), will not send report This message is logged when there is a problem connecting to the configured micro-updates host (potentially via a proxy if a proxy is configured).
Cloudmark Cartridge Installation and Administration Guide Appendix A Could not initialize encryption, expect problems reporting statistics This message is logged when there is a problem initializing the module used to encrypt statistics reports. You can expect subsequent failures with sending statistics reports. CRITICAL log messages () Could not initialize micro-update module This message is logged when there is a problem initializing the module used to interact with micro-updates servers.
Index A H authentication 14, 19, 35 NTLM 14, 19 header whitelisting 24 host whitelisting 23 HTTP proxy 14, 19, 21, 46 B body whitelisting 25 C cartridge.cfg file 10, 11, 13, 18, 19 cartridge.
Index signature files 20 SpamAssassin 9 spoofing 21 state files 20 statistics 4, 14, 16, 29, 43, 47 configuration 35 T timeout 19 Trust Evaluation System 2, 17 50 U updating 8 W whitelist.cfg 34 whitelist.
