Kerberos, Exim, Dovecot and Initial-Response

Someday I will sit down and put to paper the exact steps to get Dovecot and Exim to use GSSAPI authentication. This is not that day.

One thing I did find out is that the base64 encoding of the tickets that mail clients like Mac Mail and Thunderbird attempt to authenticate with are quite long. In fact, they are so long that certain client/server combinations don't work without changing the server source and recompiling. Here are the combinations I found that don't work:

• SMTP using Thunderbird 6 and Exim 4.72 from Enterprise Linux 6 EPEL
• IMAP using Apple Mail (10.7 Lion) and Dovecot 2.0.13 from AT RPMs

In both non-working combinations, the client attempts to use the Initial-Response feature of the respective protocol. If you haven't heard of this before (I sure hadn't) then you can check out the relevant RFCs for IMAP and SMTP. TL;DR: Initial-Response allows clients to save round trips by sending everything the server needs to authenticate them in the initial auth (SMTP) or login (IMAP) call. The problem with this is that the server software doesn't know that a huge base64 encode of the client's Kerberos ticket is coming, has a sane limit of the number of bytes it will accept from the client in a single command and will truncate the ticket that the client sends.

Unfortunately, the fix for this seems to be recompiling the server software with bigger input buffers. This post was my first clue about the problem and contains what I believe to be a safe value for an input buffer that could possibly see a base64 Kerberos ticket along with a handy patch that applies the fix to Exim. For dovecot (2.0.13) you'll want to change MASTER_AUTH_MAX_DATA_SIZE in src/lib-master/master-auth.h and LOGIN_MAX_INBUF_SIZE in src/login-common/client-common.h.

Dovecot and SSO Kerberos With Active Directory

If you have an Active Directory environment, you have multiple authentication options for most open-source software, including the excellent Dovecot IMAP/POP server. You could use:

• Dovecot's PAM method
• ...to connect to LDAP on AD
• ...to connect to Kerberos on AD
• Dovecot's LDAP method (if you need more granular control than PAM)
• Dovecot's Kerberos method (if you want SSO from AD-bound clients)

The last method, I believe is the hardest to achieve. First, Dovecot's Kerberos authn page tells you that you need service keys on your server, but offers no indication of how to do that. There are some very helpful posts to the Dovecot mailing list, but none that fit what I was looking for. My problem was:

• I'm using nslcd on RHEL6 to look at LDAP on AD for POSIX user information like uid, uidNumber, homeDirectory, so I don't really need winbind. (<- I could be dead wrong about winbind but it didn't seem to work for me and I didn't spend too much time investigating why.)
• The hostname that users will connect to is load-balanced across multiple hosts. Because SPN (Service Principal Names) in AD must be unique, I could not bind each mail host to AD using Samba and add IMAP/mailname.domain.edu to each machine account.

Some Google sleuthing revealed that the way most people get around this is to create a user in Active Directory, and add SPNs that will be used on multiple hosts to that account. The best documentation I found for doing this was this post by Achim Grolms. I used those instructions to create the SPN, which got mapped onto a user I created, and generate a keytab for Dovecot to use to validate client requests to that service.

Some pitfalls to watch for when configuring Dovecot for this method:

1. The user that runs Dovecot's auth process needs to be able to read the keytab file you get from ktpass, but you probably don't want anyone else to read it.
2. Set auth_gssapi_hostname to "$ALL" since I'm not sure if these hosts would consider themselves all the hostname mailname.domain.edu.

PSA: NFS Locking in RHEL6 Needs Reverse Lookup of NFS Server

The RHEL6 version of rpc.statd needs to be able to reverse lookup a hostname for the IP address of your NFS server if you want to lock files mounted from that server. If you don't have a reverse lookup for the IP of your NFS server, just throw its IP and some kind of name for it in /etc/hosts.

Otherwise you get errors in syslog to the effect of:

Feb 23 13:14:57 wardentest3 rpc.statd[1259]: No canonical hostname found for 10.238.3.20
Feb 23 13:14:57 wardentest3 rpc.statd[1259]: STAT_FAIL to wardentest3.geneseo.edu for SM_MON of 10.238.3.20
Feb 23 13:14:57 wardentest3 kernel: lockd: cannot monitor 10.238.3.20

Oracle Linux 5.6 Anaconda, HTTP RPM Repo, device-mapper-multipath

Including the "base" RPM channel of Oracle Linux 5.6 (a copy of the Server directory) that is accessed over HTTP causes the last RPM fail to install during a cobbler-assisted pxeboot install. For me it's device-mapper-multipath-0.4.9-23.0.8.el5.x86_64.rpm. Anaconda says it cannot download/install that package and asks me if I want to try to download/install it again or not.

The line in the Apache logs each time I pick "retry":

137.238.2.94 - - [10/Feb/2011:21:02:19 +0000] "GET /oracle/OracleLinux/OL5/6/base/x86_64/device-mapper-multipath-0.4.9-23.0.8.el5.x86_64.rpm HTTP/1.1" 416 394

A packet capture reveals that anaconda is issuing a GET request for the RPM in question but only asking for bytes 440-25514. This corresponds to that RPM's information in the repo:

rpm:header-range start="440" end="25515"


It looks like when Anaconda says "beginning installation" it's querying the header-range of each RPM it's interested in, then when it has resolved dependencies/order of install it downloads the full contents of each RPM.

In the packet capture the GET request has the Range header set to "bytes=86392-" which I assume means it wants from bytes 86392 to the end of the file. The only problem is that the file is only 86392 bytes, so it's asking for a byte range past the end of the file and thus Apache returns the 416 error.

The strange thing is this is the second time during the "fetch and install each RPM" phase where this file was requested. The first time, anaconda was smart enough to fetch the whole file and presumably install it.

As I mentioned earlier, this problem is only apparent when making the 5.6 "base" repo available to Anaconda over HTTP. I disabled that repo in cobbler and Anaconda was able to use the NFS path to the folder that has the extracted contents of the Oracle Linux 5.6 DVD to install without a problem. This is strange because I created our 5.6 "base" repo by copying all the .rpm files from the 5.6 DVD and running "createrepo ." in the new directory. I even ran diff to make sure the device-mapper-multipath-0.4.9-23.0.8.el5.x86_64.rpm is the same in my repo and on the 5.6 DVD.

I even tried using Oracle Linux 5.5's installer with the 5.6 "base" repo which should effectively install Oracle Linux 5.6 packages with 5.5's version of Anaconda. That had the same problem. It really looks like there's something about the 5.6 "base" repo that Anaconda doesn't like accessing over HTTP.

Android Z-Push Issues

I've been testing Z-Push (r502/1.5) some more and ran into a frustrating bug when sending from an Android 2.2 emulator.

The problem was I was sending mail through the Android emulator and the body of all of these messages were getting garbled in Z-Push clients. I noticed this in an iPhone running 3.1.2 and the Android 2.2 emulator. The debug.txt file for Z-Push was saying things like "Detected an illegal character in input string" when attempting to read the copy of the message in my Sent folder or Inbox.


Basically, Android sends mail bodies as UTF-8, encoded in base64. When configured to send mail through Z-Push, the messages were being decoded from base64 and the decoded version of the body was being sent to the local sendmail binary. This was Very Bad because Z-Push was leaving the Content-Transfer-Encoding header as base64, meaning any attempt to MIME parse the body will treat the already-decoded base64 text as base64-encoded text. And when that happens, things look garbled.

It turns out Z-Push already had logic for dealing with base64-encoded bodies, but was not re-encoding the body before sending it to the sendmail binary. The problem was the following line in backend/imap.php:

255         // encode the body to base64 if it was sent originally in base64 by the pda
256         if ($body_base64 && !isset($forward)) $body = base64_encode($body);

For messages with base64 bodies, $body_base64 was being properly set to true. The problem was $forward. Forward will always be set, because it is a function parameter that defaults to false:

 123     function SendMail($rfc822, $forward = false, $reply = false, $parent = false) {

The result is the base64_encode will never run, regardless of the value of $body_base64. I changed the line to look like:

 255         // encode the body to base64 if it was sent originally in base64 by the pda
 256         if ($body_base64 && isset($forward) && !$forward) $body = base64_encode($body);

Which maybe wasn't the best fix since if $forward ever is not set as a default function parameter then this will fail again. But it did fix the problem and now I can send mail from an Android 2.2 emulator without a problem.

I should note that this flaw is present in r502/Z-Push 1.5 AND trunk. I should figure out how to let them know...

Hopefully this saves someone half the frustration I just experienced today tracking this down.