procmail
[Top] [All Lists]

Re: mail corruption with dotlock/nfs

2008-03-05 15:26:14
On Wednesday, 5 בMarch 2008 23:56, Bart Schaefer wrote:

Hello,


     We're using postfix+procmail with NFS on netapp for
many years, on what I believe to be a largish setup
for a NFS environment, about 60,000 users. We never
saw mail corruption except on one case, that is, when
one of the mounts was using UDP and the filer hit
100% cpu for more than a few minutes. 
    I never used the `sync' mount option, due to it's
enormous hit on performance. Also, I believe that the examples
and data quoted about "data corruption when using async"
was related to exporting filesystems over NFS from a Linux
NFS server(!) with the sync/async option enabled in the
export.

Are you using NFS over UDP ?  Are you using soft mounts ?


Our setup:

2 postfix mail servers
2 imap servers (used to be UW, now using dovecot)
2 pop servers (using UW ipop3d).

All 6 servers mount the same /var/spool/mail, and
the same home directory trees, all via NFS from the
same filer (FAS3050, not clustered). All servers are
active (load balanced via an external load balancer),
so a mailbox may be open from some least 3 locations
(2 mails delivered to the mailbox at the same time
by mailserver1 or 2, and also an active pop or imap
session for the user).

We use NFSv3 only, and on the netapp we're not allowing
mounts with rsize/wsize greater than 8k (due to switches
having buffering problems with larger packets).

Our mount options:

rw,rsize=8192,wsize=8192,lock,tcp,nfsvers=3,hard,intr,bg,nosuid,nodev

On the netapp, our options for NFS (and other relevant stuff) are:

ip.tcp.newreno.enable        on         
ip.tcp.sack.enable           on         
nfs.acache.persistence.enabled on         
nfs.assist.queue.limit       40         
nfs.export.allow_provisional_access on         
nfs.export.auto-update       on         
nfs.export.exportfs_comment_on_delete on         
nfs.export.harvest.timeout   1800       
nfs.export.neg.timeout       60         
nfs.export.pos.timeout       3600       
nfs.export.resolve.timeout   6          
nfs.hide_snapshot            off        
nfs.ifc.xmt.high             16         
nfs.ifc.xmt.low              8          
nfs.kerberos.enable          off        
nfs.kerberos.file_keytab.enable off        
nfs.locking.check_domain     on         
nfs.mount_rootonly           on         
nfs.mountd.trace             off        
nfs.netgroup.strict          off        
nfs.notify.carryover         on         
nfs.ntacl_display_permissive_perms off        
nfs.per_client_stats.enable  on         
nfs.require_valid_mapped_uid off        
nfs.response.trace           off        
nfs.response.trigger         60         
nfs.rpcsec.ctx.high          0          
nfs.rpcsec.ctx.idle          360        
nfs.tcp.enable               on         
nfs.tcp.xfersize             8192       
nfs.udp.enable               on         
nfs.udp.xfersize             8192       
nfs.v2.df_2gb_lim            off        
nfs.v3.enable                on         
nfs.v4.acl.enable            off        
nfs.v4.enable                off        
nfs.v4.id.domain             taucc.tau.ac.il 
nfs.v4.read_delegation       off        
nfs.v4.write_delegation      off        

Some optimization of /etc/sysctl.conf for the imap/pop servers:

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 30
# Decrease the time default value for tcp_keepalive_time connection
net.ipv4.tcp_keepalive_time = 600
# Turn off the tcp_window_scaling
net.ipv4.tcp_window_scaling = 0
# Turn on the tcp_sack
net.ipv4.tcp_sack = 0
# Turn off the tcp_timestamps
net.ipv4.tcp_timestamps = 0
# Increase max files in the system
fs.file-max = 65536
kernel.threads-max = 65536
#
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 87380 8388608
net.ipv4.ipfrag_low_thresh=262144
net.ipv4.ipfrag_high_thresh=39321
#
# cd into /proc/sys/net/core
net.core.rmem_max = 1048576
net.core.wmem_max = 1048576
net.core.rmem_default = 65536
net.core.wmem_default = 65536
#
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle=1


YMMV

--Ariel 
 --
 Ariel Biener, CISO
 Tel-Aviv University CIT div.
 e-mail: ariel(_at_)aristo(_dot_)tau(_dot_)ac(_dot_)il phone: 03-6406086
 PGP key:    http://www.tau.ac.il/~ariel/pgp.html

____________________________________________________________
procmail mailing list   Procmail homepage: http://www.procmail.org/
procmail(_at_)lists(_dot_)RWTH-Aachen(_dot_)DE
http://MailMan.RWTH-Aachen.DE/mailman/listinfo/procmail