June 27, 2008

 Optimizing @Mail Performance using APC

This article will detail the procedures required to install and configure the APC opcode cache on your server. This article does not cover setting up APC on Windows servers.

What is APC?

From php.net:

"The Alternative PHP Cache (APC) is a free and open opcode cache for PHP. It was conceived of to provide a free, open, and robust framework for caching and optimizing PHP intermediate code."
So basically we can use APC to squeeze even better performance out of @Mail.

Installing APC

APC is a PHP PECL extension so I will cover using the pecl command to insall APC. If you find that pecl is not available on your system performing the following command for your system should install it:

  • Ubuntu/Debian: apt-get install pear
  • FreeBSD: pkg_add -r php5-pear
  • RedHat Linux (inc Centos, Fedora etc): yum install php5-pear
  • You will also need the apache apxs tool. If apxs is not on your system you can install it as such:

  • Ubuntu/Debian: apt-get install apache2-threaded-dev
  • FreeBSD:
  • RedHat Linux (inc Centos, Fedora etc):
  • Once you have pecl and apxs installed perform the following command:

    # pecl install apc

    Once the compile and installation is complete you will need to enable the extension by adding the following line to your php.ini:

    extension=apc.so

    You can test that the APC module is installed by using the 'php -m' command. This will list the installed PHP modules and APC should be near the top:

    # php -m
    [PHP Modules]
    apc
    bcmath
    bz2
    calendar
    ctype
    date
    dba
    ...etc

    Once you have enabled APC in php.ini, in order to load APC for the Apache PHP module simply restart Apache, usually

    apachectl restart

    or

    apache2ctl restart

    Now APC is ready and waiting to spice up performance.

    Configuring APC

    APC has several settings you can tweak according to your setup and desired functionality. The first two settings you should consider are

    apc.shm_size

    and

    apc.stat

    These two settings adjust how much shared memory you want to set aside for APC and whether you want APC to check for file modification on every request, respectively.

    First lets consider

    apc.shm_size

    How large a memory segment you can assign to APC depends on amount of available RAM and your OS type and configuration. The amount is specified in MB and the default is 30. In my testing with APC and @Mail I maxed out at 8.895MB of memory usage for the APC cache so I'd suggest the recommended minimum setting for a server running @Mail is about 10MB. If the server is running other php scripts via apache then you will need to either increase this value or use the technique described in the "Conditional Caching" section.

    This setting determines whether APC checks for file modification on every request. The default is 'On', and APC will check each script for modification upon each request. If the script has been modified then it will not use the cached version, but recompile and re-cache the new version. Also with apc.stat

    on, with every request made APC will have to find absolute paths for all files included or required with a relative path. This obviously adds some overhead.

    Changing apc.stat to 'Off' can produce a significant performance gain and simply means if you modify any @Mail files you will need to restart Apache before the changes will come into effect. Not a big deal on a production server where files stay static for long periods.

    Conditional Caching

    If the server running @Mail is also serving other PHP scripts then you may want to limit the caching to @Mail scripts only in order to reduce the chance of running out of cache slots or allocated RAM. You can do this by using the following settings: apc.cache_by_default

    apc.cache_by_default is On by default and means all php scripts served by Apache are cached. If you want to cache only @Mail scripts then you will need to set apc.cache_by_default = Off

    in your php.ini or apc.ini file.

    Once you have turned apc.cache_by_default off then you can use a .htaccess file in @Mail's web-root to turn it back on for @Mail only. If you have other scripts/apps you would like to enable caching for then simply add the .htaccess file to their web-root also. The .htaccess file should contain just this line:

    php_value apc.cache_by_default On

    Once you have the .htaccess file in place all @Mail scripts will be cached. You will also need to restart Apache for the main apc.cache_by_default = Off setting to take.

    How Can I Tell What is Currently Cached?

    A simple solution is to create a short php script with this in it:

    print_r(apc_cache_info());

    Save it to a file that is web readable and load it within your web browser. This will give you some information on the current state of your APC cache, including what files are cached. Here is an excerpt of the output given after having used @Mail:

    Array
    (
    [num_slots] => 2000
    [ttl] => 0
    [num_hits] => 11
    [num_misses] => 35
    [start_time] => 1214447228
    [expunges] => 0
    [mem_size] => 4645714
    [num_entries] => 35
    [num_inserts] => 35
    [file_upload_progress] => 1
    [memory_type] => mmap
    [locking_type] => pthread mutex
    [cache_list] => Array
    (
    [0] => Array
    (
    [filename] => /usr/local/atmailphp/webmail/libs/PEAR/Mail/mime.php
    [device] => 2051
    [inode] => 261912
    [type] => file
    [num_hits] => 0
    [mtime] => 1203282990
    [creation_time] => 1214447477
    [deletion_time] => 0
    [access_time] => 1214447477
    [ref_count] => 0
    [mem_size] => 125640
    )
    
    [1] => Array
    (
    [filename] => /usr/local/atmailphp/webmail/libs/PEAR/Mail/RFC822.php
    [device] => 2051
    [inode] => 261911
    [type] => file
    [num_hits] => 0
    [mtime] => 1207549212
    [creation_time] => 1214447477
    [deletion_time] => 0
    [access_time] => 1214447477
    [ref_count] => 0
    [mem_size] => 120242
    )
    
    [2] => Array
    (
    [filename] => /usr/local/atmailphp/webmail/libs/PEAR/Mail/mimePart.php
    [device] => 2051
    [inode] => 261908
    [type] => file
    [num_hits] => 0
    [mtime] => 1203282990
    [creation_time] => 1214447477
    [deletion_time] => 0
    [access_time] => 1214447477
    [ref_count] => 0
    [mem_size] => 58013
    )

    Final Words

    There are several other APC settings that you may wish to tweak for your system. More information on these settings can be found here: http://php.net/manual/en/apc.configuration.php

    You should place all these setting in your php.ini, or alternatively if you have a directory on your system that php will read .ini files from then create a file in there called apc.ini and place the APC settings in it.

    You should find that @Mail and APC work just fine together and you should see improved performance, especially on busier servers - of course if you have any feedback or experience problems let us know.


    Filed under: Linux version, PHP version, Optimization — Brad Kowalczyk @ 12:02 am

     

    June 25, 2008

     Migrating Imail to @Mail 5.0+ (PHP versions)

    Migrating from IMail to @Mail 5.0+ (PHP versions) running under Linux is now possible with new PHP migration scripts (Note, we use the term 'Linux' but this article applies to any UNIX or UNIX like OS that can run @Mail Server 5.0+). The process requires various steps in order to migrate user-authentication data, system accounts, settings, messages and addressbook information into @Mail.To proceed follow the steps below:

    1: First install @Mail on the new server or existing machine.

    2: Download the Imail migration utilities from: http://atmail.com/attach/migrate-imail-php.tgz

    3: Login to the server running Imail - Download ActivePerl from http://www.activestate.com/ and install Win32 Perl on your Imail server. This is required to run the migration script that pulls Imail data from the registry.

    4: During the ActivePerl installation select to install under C:\perl\

    5: Next, run the migrate-imail.pl script to dump the Imail database into a format @Mail can read. You can extract the file from your base @Mail installation webmail/modules/migrate-imail.pl or the archive given above, into directory C:/perl/

    Usage: migrate-imail.pl [options]
    
    Options:
    -help            brief help message
    -man             full documentation
    -users           dump users
    -useraliases     dump user aliases
    -lists           dump lists
    -domains         dump domains
    -hostaliases     dump host aliases
    -orphans         dump domains orphaned with no users

    The script is designed to dump user account ( and decrypted passwords from the registry ) , aliases, lists, domains and
    host-aliases

    In order to migrate the data into @Mail run the migration script via the command-line as follows

    cmd> cd C:/perl
    cmd> C:/perl/bin/perl.exe migrate-imail.pl -users
    cmd> C:/perl/bin/perl.exe migrate-imail.pl -useraliases
    cmd> C:/perl/bin/perl.exe migrate-imail.pl -hostaliases
    cmd> C:/perl/bin/perl.exe migrate-imail.pl -domains

    These commands will create a text-file of the Imail database into a format @Mail can parse.

    6: The next step is to migrate the exported data into @Mail. Copy the following files onto the new server running @Mail:

    C:/perl/users.txt
    C:/perl/useraliases.txt
    C:/perl/hostaliases.txt
    C:/perl/domains.txt

    Copy the data into the pathname:

    /usr/local/atmail/migrate/

    7: Next, on the server running @Mail migrate the user data into @Mail by using the following commands:

    cd /usr/local/atmail/webmail/modules/
    php migrate-imail-import.php /usr/local/atmail/migrate/

    This command will loop through each of the data dumps from Imail and import domains, users, aliases and host-aliases into the new @Mail system.

    8: The next step is to migrate existing messages from Imail into @Mail.

    Usage: php migrate-imail-msgs.php [directory of messages] [skip msgs older then X days]

    Note you can optionally specify the number of days to ignore messages if they are older the X days

    When migrating from Windows to Linux, you need to copy the entire Imail message store to the new server.

    e.g C:\imail\domain.com -> Linuxserver:/usr/local/atmail/imail-data/

    Once you have copied the Imail message store to the @Mail server perform the following commands:

    cd /usr/local/atmail/webmail/modules/
    php migrate-imail-msgs.php /usr/local/atmail/imail-data/

    Note: This process picks up domain name and username from the directory structure of the Imail message store. The message store directory you copy to the @Mail server should have the following structure:

    /usr/local/atmail/imail-data/domain_name/Users/username

    Where domain_name is replaced by each domain the system hosts and username is replaced by each user under that domain. For example if Imail was hosting the accounts brad@mydomain.com, ben@mydomain.com, jason@anotherdom.com and john@anohterdom.com then these directories would exist:

    /usr/local/atmail/imail-data/mydomain.com/Users/brad
    /usr/local/atmail/imail-data/mydomain.com/Users/ben
    /usr/local/atmail/imail-data/anotherdom.com/Users/jason
    /usr/local/atmail/imail-data/anotherdom.com/Users/john
    The command will recurse through these directories and migrate all existing folders, sub-folders and
    messages for those users into @Mail. The @Mail account that the data is migrated to is decided by adding together the name of the username directory with the domain_name directory. So when reading

    /usr/local/atmail/imail-data/mydomain.com/Users/brad

    all messages and folders there will be migrated into the brad@mydomain.com account in @Mail.
    9: Optionally you can migrate existing user addressbooks created via WebMail in Imail to @Mail.

    Usage: php migrate-imail-abook.php [directory of users] [domain name]

    Perform the following commands:

    cd /usr/local/atmail/webmail/modules/
    php migrate-imail-abook.php /usr/local/atmail/imail-data/ domain.com

    10: Once you have completed the steps above the new system is live and ready to use! Login to the @Mail Webadmin -> Users -> List Users and you can login to existing accounts and select messages.

    11: The next step is to disable Imail and activate @Mail

    If migrating to a new machine move the DNS/MX records for all domains to point to the new @Mail server.
    Existing users can login to POP3/IMAP without any changes to their desktop client. Existing users will need to be made aware of the new @Mail Webmail interface.

    If you have any questions regarding the migration steps for Imail, please contact us. We also provide remote migration assistance for existing clients.


    Filed under: Migration — Brad Kowalczyk @ 7:45 pm

     

    June 11, 2008

     Virus Oversized.Zip detected

    If you receive the SMTP error:

    "550 Virus Oversized.Zip detected. Mail delivery avoided"

    This is due to the ClamAV engine of Atmail scanning a message attachment and the zip file compression ratio is too high. For example, zipping a large number of BMP files with high compression.

    This feature is implemented in Clam to avoid a special crafted Zip file to loop on extraction causing a denial-of-service attempt.

    To change the compression ratio simply edit: /usr/local/atmail/av/etc/clamd.conf

    # If a file in an archive is compressed more than ArchiveMaxCompressionRatio
    # times it will be marked as a virus (Oversized.ArchiveType, e.g. Oversized.Zip)
    # Value of 0 disables the limit.
    # Default: 250
    ArchiveMaxCompressionRatio 0

    Specify 0 to disable, or increase the ratio above the default 250, then restart the Atmail services.


    Filed under: Uncategorized, Anti-Virus — info @ 9:23 pm