UPGRADING

Follow the steps sequentially from the base version you are running up to
the top.

UPGRADING FROM 3.9.0
--------------------

1. Ensure MySQL is using latin1 character set. This should reduce the needed
   database space. The following clauses should be executed for upgrading 3.9.0
   DSPAM MySQL schema:
     ALTER TABLE `dspam_signature_data`
       CHANGE `signature` `signature` VARCHAR(32) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL;
     ALTER TABLE `dspam_signature_data`
       DEFAULT CHARACTER SET latin1 COLLATE latin1_general_ci;
     ALTER TABLE `dspam_stats`
       DEFAULT CHARACTER SET latin1 COLLATE latin1_general_ci;
     ALTER TABLE `dspam_token_data`
       DEFAULT CHARACTER SET latin1 COLLATE latin1_general_ci;

   If you are using preference extension with DSPAM, then you should execute
   the following clause for upgrading DSPAM preference MySQL schema to use
   latin1 character set:
     ALTER TABLE `dspam_preferences`
       CHANGE `preference` `preference` VARCHAR(32) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL;
     ALTER TABLE `dspam_preferences`
       CHANGE `value` `value` VARCHAR(64) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL;
     ALTER TABLE `dspam_preferences`
       DEFAULT CHARACTER SET latin1 COLLATE latin1_general_ci;

   If you are using the DSPAM virtual user feature then execute the following
   clauses to update the DSPAM virtual user table to use latin1 character set:
     ALTER TABLE `dspam_virtual_uids`
       CHANGE `username` `username` VARCHAR(128) CHARACTER SET latin1 COLLATE latin1_general_ci NOT NULL;
     ALTER TABLE `dspam_virtual_uids`
       DEFAULT CHARACTER SET latin1 COLLATE latin1_general_ci;


UPGRADING FROM 3.8
------------------

1. Ensure MySQL is using the new database schema. The following clauses should
   be executed for upgrading pre-3.9.0 DSPAM MySQL schema to the 3.9.0 schema:
     ALTER TABLE `dspam_signature_data`
       CHANGE `uid` `uid` INT UNSIGNED NOT NULL,
       CHANGE `data` `data` LONGBLOB NOT NULL,
       CHANGE `length` `length` INT UNSIGNED NOT NULL;
     ALTER TABLE `dspam_stats`
       CHANGE `uid` `uid` INT UNSIGNED NOT NULL,
       CHANGE `spam_learned` `spam_learned` BIGINT UNSIGNED NOT NULL,
       CHANGE `innocent_learned` `innocent_learned` BIGINT UNSIGNED NOT NULL,
       CHANGE `spam_misclassified` `spam_misclassified` BIGINT UNSIGNED NOT NULL,
       CHANGE `innocent_misclassified` `innocent_misclassified` BIGINT UNSIGNED NOT NULL,
       CHANGE `spam_corpusfed` `spam_corpusfed` BIGINT UNSIGNED NOT NULL,
       CHANGE `innocent_corpusfed` `innocent_corpusfed` BIGINT UNSIGNED NOT NULL,
       CHANGE `spam_classified` `spam_classified` BIGINT UNSIGNED NOT NULL,
       CHANGE `innocent_classified` `innocent_classified` BIGINT UNSIGNED NOT NULL;
     ALTER TABLE `dspam_token_data`
       CHANGE `uid` `uid` INT UNSIGNED NOT NULL,
       CHANGE `spam_hits` `spam_hits` BIGINT UNSIGNED NOT NULL,
       CHANGE `innocent_hits` `innocent_hits` BIGINT UNSIGNED NOT NULL;

   If you are using preference extension with DSPAM, then you should execute
   the following clause for upgrading pre-3.9.0 DSPAM preference MySQL schema
   to the 3.9.0 schema:
     ALTER TABLE `dspam_preferences` 
       CHANGE `uid` `uid` INT UNSIGNED NOT NULL;

   If you are using virtual users (with AUTO_INCREMENT) in DSPAM, then you
   should execute the following clause for upgrading pre-3.9.0 DSPAM virtual
   uids MySQL schema to the 3.9.0 schema:
     ALTER TABLE `dspam_virtual_uids`
       CHANGE `uid` `uid` INT UNSIGNED NOT NULL AUTO_INCREMENT;

   If you are using virtual user aliases (aka: DSPAM in relay mode) in DSPAM,
   then you should execute the following clause for upgrading pre-3.9.0 DSPAM
   virtual uids MySQL schema to the 3.9.0 schema:
     ALTER TABLE `dspam_virtual_uids`
       CHANGE `uid` `uid` INT UNSIGNED NOT NULL;

   If you need to speed up the MySQL purging script and can afford to use more
   disk space for the DSPAM MySQL data, then consider executing the following
   clause for adding additional indices to the dspam_token_data table:
     ALTER TABLE `dspam_token_data`
       ADD INDEX(`spam_hits`,`innocent_hits`),
       ADD INDEX(`last_hit`);

   You can of course combine all three fields in one index (and save some space):
     ALTER TABLE `dspam_token_data`
       ADD INDEX(`spam_hits`,`innocent_hits`,`last_hit`);

   Those additional indices/index can speed up the purging for MyISAM tables. If you
   are using InnoDB tables then the difference with/without them is not that amazing
   and in most cases not worth the needed additional space/processing power.

2. Ensure PosgreSQL is using the new database schema. The following clauses should
   be executed for upgrading pre-3.9.0 DSPAM PosgreSQL schema to the 3.9.0 schema:
     ALTER TABLE dspam_preferences ALTER COLUMN uid TYPE integer;
     ALTER TABLE dspam_signature_data ALTER COLUMN uid TYPE integer;
     ALTER TABLE dspam_stats ALTER COLUMN uid TYPE integer;
     ALTER TABLE dspam_token_data ALTER COLUMN uid TYPE integer;
     DROP INDEX IF EXISTS id_token_data_sumhits;

   If you are using virtual users in DSPAM, then you should execute the following
   clause for upgrading pre-3.9.0 DSPAM virtual uids to the 3.9.0 schema:
     ALTER TABLE dspam_virtual_uids ALTER COLUMN uid TYPE integer;

   You need to add an second lookup_tokens function to PostgreSQL. The additional
   lookup_tokens function can be found in pgsql_objects.sql and it's a slightly
   different and enhanced then the function originally introduced in 3.6.8.

3. Follow steps in "UPGRADING FROM 3.9.0".


UPGRADING FROM 3.6
------------------

1. Add 'Tokenizer' setting to dspam.conf
   The 'Tokenizer' setting in 3.8.0 replaces tokenizer definitions in the 
   "Feature" clause of previous version configurations. See src/dspam.conf
   (after make) for more information about this seting.
 
2. Check calls to dspam_logrotate
   Earlier versions of 3.6 did not prepend a leading "-l" flag to specifying
   log file selection. This is now required.

3. Ensure 3.6.0 malaligned hash databases are converted
   Version 3.6.0 failed to align hash databases to 8-byte boundaries. If you
   are upgrading from v3.6.0 and are using the hash_drv storage driver, you
   should run cssconvert to upgrade your .css files to a fully aligned format.

4. Invert "SupressWebStats" setting in dspam.conf
   SupressWebStats has been changed to simply WebStats, and the setting is
   inverted. Be sure to update this in dspam.conf.

5. Add "ProcessorURLContext" setting in dspam.conf
   ProcessorURLContext has been added to toggle whether URL specific tokens
   are created in the tokenizer process. The "on" value is default for previous
   versions of DSPAM.

6. Follow steps in "UPGRADING FROM 3.8".


UPGRADING FROM 3.4
------------------

Follow all of the steps above, and the following steps:

1. Add "ProcessorBias" setting to dspam.conf
   ProcessorBias has been added to dspam.conf and must be specified.
   Since ProcessorBias is the default behavior for previous versions of DSPAM,
   you will need to add "ProcessorBias on" to dspam.conf. If you have
   specifically disabled bias, or are using a technique such as Markovian
   discrimination, you may leave this feature off.

2. Ensure references to SBLQueue are changed to RABLQueue.
   Older versions of DSPAM used the SBLQueue setting to write files for a 
   DSPAM SBL setup. This has been renamed to RABLQueue. Please change this in 
   dspam.conf if you are writing to a SBL/RABL installation.

3. Add "TestConditionalTraining" setting to dspam.conf
   TestConditionalTraining has been added to dspam.conf and must be specified
   to be enabled. Since TestConditionalTraining is the default behavior
   in DSPAM, it is strongly recommended that you add 
   "TestConditionalTraining on" to dspam.conf

4. Ensure PostgreSQL installation have a lookup_tokens function
   PostgreSQL systems running v8.0+ must create the function lookup_tokens
   added to pgsql_objects.sql. The driver now checks your version and uses this
   function to improve performance on 8.0+.

5. Ensure you are specifying the correct storage driver.
   hash_drv is now the new default storage driver. hash_drv has no dependencies
   and is extremely fast/efficient. If you're not familiar with it, you should
   check out the readme. If you were previously using SQLite, you will now need
   to specify it as the storage driver: --with-storage-driver=sqlite_drv

   NOTE: Berkeley DB drivers (libdb3_drv, libdb4_drv) are deprecated and have
         been removed from the build. You will need to select an alternative
         storage driver in order to upgrade.

6. Follow steps in "UPGRADING FROM 3.6".