CAS-ifying Roundcube

Quick background/summary:

• CAS is a web-based Single Sign-On (SSO) system. http://www.jasig.org/cas
• Roundcube is a web-based email client, often called a "webmail" system. http://roundcube.net/
• Alex Li wrote a plugin for Roundcube to use CAS as an authentication source. It doesn't work with Roundcube 0.6 or higher without modifying the Roundcube source. http://code.google.com/p/rc-cas-plugin/
• I forked Alex's plugin on Github and made it work with new, stock versions of Roundcube. https://github.com/dfwarden/Roundcube-CAS-Authn

We have a CAS server for our campus, and we use Roundcube for our webmail system. Some time ago, I was asked to extend our Roundcube system to authenticate against CAS. Doing so would net us some nice benefits:

• Graceful handling of locked accounts/expired passwords.
• SSO to Webmail if already signed in to another SSO service.
• Easy access for custom webapps wanting to access IMAP.

One of the first things I had to do was configure our IMAP server to accept CAS tickets as passwords. Our IMAP server (Dovecot) uses PAM for authentication, so all I had to do was follow the instructions on the Google translation of this page (French):

• Create CAS services for HTTPS and IMAP URIs that ignore attributes and can proxy. On our older version of CAS I also had to restart the servlet container for the new services to be active.
• Download, compile, install and configure pam_cas from https://sourcesup.cru.fr/frs/?group_id=213.
• Add pam_cas to Dovecot's PAM stack with the IMAP service I created in CAS and the pam_cas.conf file I created in the options.
• Test that Dovecot validates CAS tickets. You could use the CASTest utility that comes with pam_cas, but I chose to just open https://<CAS server>/login?service=<IMAP service> in a web browser and copy/paste the service ticket I got from CAS into a telnet session that was authenticating to our IMAP server. Be careful when testing! The default CAS Service Ticket life is 10 seconds, so you may have to move fast if you follow my testing method.

If you run in to problems in any of those steps, the best thing you can do is turn debug on in /etc/pam_cas.conf and send *.debug in your syslog config to a file that you can read. You should get good info from that as to why pam_cas is failing.

Once I had IMAP authenticating users with CAS tickets, Alex's plugin got me most of the way, but I found that I had to make 2 modifications to Roundcube's index.php file to get the plugin to work properly. I also wanted a few additional features that his plugin lacked:

• The option of not using CAS for authentication. Alex's plugin forces it for authentication to Roundcube.
• SMTP authentication using a CAS proxy ticket.

I was able to remove the need to modify Roundcube's source and add those features to my fork of his plugin. It is currently in production in our 8-node Roundcube cluster. There were a few "gotchas" along the way that I don't want to forget, and other people may find useful when working with my plugin or CAS' proxy feature in general:

• If you run multiple Roundcube servers behind a load balancer, you need to be sure of 2 things:
• You have the same version of phpCAS on all of your Roundcube servers.
• You are using shared storage for PGTIOUs coming back from the CAS server. This page talks about this process but basically part of proxy CAS authentication is the CAS server will make an HTTP call to a URL that is supposed to store a necessary piece of information. Unless you're absolutely sure the CAS server will be sent to the same Roundcube server that the user is currently hitting, you need to make sure that all Roundcube servers can share the directory where PGTIOUs are saved.
• If you are using CAS' proxy method for authenticating to IMAP, you all but need an IMAP proxy. Without it, you will need to generate and validate a new ticket from CAS (multiple HTTP trips) for each page load. With an IMAP proxy, the user's first IMAP ticket is validated by pam_cas by your actual IMAP server and saved in your IMAP proxy. Roundcube will continue to send the same ticket and the IMAP proxy will continue to believe the user is logged in, even though that ticket has been used and cannot be validated again by the IMAP server.

Increasing VMWare SCSI Timeout on RHEL6 and ESX5

VMWare recommends increasing the disk timeouts for linux guests to 180 seconds to handle things like a storage path failover. There are a few ways to accomplish this. If you are a Netapp customer you could run their script, or you could follow the instructions in the VMWare article.

Unfortunately, both of these solutions were written for ESX4 and RHEL5. If you are on ESX5 with a RHEL6 guest, do the following to increase your timeout for VMWare SCSI disks to 180 seconds:

1. Put the following in a file in /etc/udev/rules.d. I chose /etc/udev/rules.d/99-vmware-scsi-timeout.rules.
   SUBSYSTEM=="block", ACTION=="add", BUS=="scsi", SYSFS{vendor}=="VMware", SYSFS{model}=="Virtual disk",   RUN+="/bin/sh -c 'echo 180 >/sys$DEVPATH/device/timeout'"
2. Run udevadm test /block/sda and make sure you see a line that indicates the echo 180 would run. If it doesn't then your SYSFS values are somehow different than mine. The output you just got should help you figure out the correct values. Look for the entries that start with ID_.
3. Once you have udevadm telling you it would run the echo 180 command you can restart your machine or restart udev during your maintenance window, if this is a production VM. If you don't want to reboot the machine, you can restart udev with:
1. udevadm control --reload-rules
2. start_udev
4. Verify it worked by looking at the timeout file in the /sys tree for your devname. For me this was accomplished with: cat /sys/block/sda/device/timeout

Kerberos, Exim, Dovecot and Initial-Response

Someday I will sit down and put to paper the exact steps to get Dovecot and Exim to use GSSAPI authentication. This is not that day.

One thing I did find out is that the base64 encoding of the tickets that mail clients like Mac Mail and Thunderbird attempt to authenticate with are quite long. In fact, they are so long that certain client/server combinations don't work without changing the server source and recompiling. Here are the combinations I found that don't work:

• SMTP using Thunderbird 6 and Exim 4.72 from Enterprise Linux 6 EPEL
• IMAP using Apple Mail (10.7 Lion) and Dovecot 2.0.13 from AT RPMs

In both non-working combinations, the client attempts to use the Initial-Response feature of the respective protocol. If you haven't heard of this before (I sure hadn't) then you can check out the relevant RFCs for IMAP and SMTP. TL;DR: Initial-Response allows clients to save round trips by sending everything the server needs to authenticate them in the initial auth (SMTP) or login (IMAP) call. The problem with this is that the server software doesn't know that a huge base64 encode of the client's Kerberos ticket is coming, has a sane limit of the number of bytes it will accept from the client in a single command and will truncate the ticket that the client sends.

Unfortunately, the fix for this seems to be recompiling the server software with bigger input buffers. This post was my first clue about the problem and contains what I believe to be a safe value for an input buffer that could possibly see a base64 Kerberos ticket along with a handy patch that applies the fix to Exim. For dovecot (2.0.13) you'll want to change MASTER_AUTH_MAX_DATA_SIZE in src/lib-master/master-auth.h and LOGIN_MAX_INBUF_SIZE in src/login-common/client-common.h.

Dovecot and SSO Kerberos With Active Directory

If you have an Active Directory environment, you have multiple authentication options for most open-source software, including the excellent Dovecot IMAP/POP server. You could use:

• Dovecot's PAM method
• ...to connect to LDAP on AD
• ...to connect to Kerberos on AD
• Dovecot's LDAP method (if you need more granular control than PAM)
• Dovecot's Kerberos method (if you want SSO from AD-bound clients)

The last method, I believe is the hardest to achieve. First, Dovecot's Kerberos authn page tells you that you need service keys on your server, but offers no indication of how to do that. There are some very helpful posts to the Dovecot mailing list, but none that fit what I was looking for. My problem was:

• I'm using nslcd on RHEL6 to look at LDAP on AD for POSIX user information like uid, uidNumber, homeDirectory, so I don't really need winbind. (<- I could be dead wrong about winbind but it didn't seem to work for me and I didn't spend too much time investigating why.)
• The hostname that users will connect to is load-balanced across multiple hosts. Because SPN (Service Principal Names) in AD must be unique, I could not bind each mail host to AD using Samba and add IMAP/mailname.domain.edu to each machine account.

Some Google sleuthing revealed that the way most people get around this is to create a user in Active Directory, and add SPNs that will be used on multiple hosts to that account. The best documentation I found for doing this was this post by Achim Grolms. I used those instructions to create the SPN, which got mapped onto a user I created, and generate a keytab for Dovecot to use to validate client requests to that service.

Some pitfalls to watch for when configuring Dovecot for this method:

1. The user that runs Dovecot's auth process needs to be able to read the keytab file you get from ktpass, but you probably don't want anyone else to read it.
2. Set auth_gssapi_hostname to "$ALL" since I'm not sure if these hosts would consider themselves all the hostname mailname.domain.edu.

PSA: NFS Locking in RHEL6 Needs Reverse Lookup of NFS Server

The RHEL6 version of rpc.statd needs to be able to reverse lookup a hostname for the IP address of your NFS server if you want to lock files mounted from that server. If you don't have a reverse lookup for the IP of your NFS server, just throw its IP and some kind of name for it in /etc/hosts.

Otherwise you get errors in syslog to the effect of:

Feb 23 13:14:57 wardentest3 rpc.statd[1259]: No canonical hostname found for 10.238.3.20
Feb 23 13:14:57 wardentest3 rpc.statd[1259]: STAT_FAIL to wardentest3.geneseo.edu for SM_MON of 10.238.3.20
Feb 23 13:14:57 wardentest3 kernel: lockd: cannot monitor 10.238.3.20

Oracle Linux 5.6 Anaconda, HTTP RPM Repo, device-mapper-multipath

Including the "base" RPM channel of Oracle Linux 5.6 (a copy of the Server directory) that is accessed over HTTP causes the last RPM fail to install during a cobbler-assisted pxeboot install. For me it's device-mapper-multipath-0.4.9-23.0.8.el5.x86_64.rpm. Anaconda says it cannot download/install that package and asks me if I want to try to download/install it again or not.

The line in the Apache logs each time I pick "retry":

137.238.2.94 - - [10/Feb/2011:21:02:19 +0000] "GET /oracle/OracleLinux/OL5/6/base/x86_64/device-mapper-multipath-0.4.9-23.0.8.el5.x86_64.rpm HTTP/1.1" 416 394

A packet capture reveals that anaconda is issuing a GET request for the RPM in question but only asking for bytes 440-25514. This corresponds to that RPM's information in the repo:

rpm:header-range start="440" end="25515"


It looks like when Anaconda says "beginning installation" it's querying the header-range of each RPM it's interested in, then when it has resolved dependencies/order of install it downloads the full contents of each RPM.

In the packet capture the GET request has the Range header set to "bytes=86392-" which I assume means it wants from bytes 86392 to the end of the file. The only problem is that the file is only 86392 bytes, so it's asking for a byte range past the end of the file and thus Apache returns the 416 error.

The strange thing is this is the second time during the "fetch and install each RPM" phase where this file was requested. The first time, anaconda was smart enough to fetch the whole file and presumably install it.

As I mentioned earlier, this problem is only apparent when making the 5.6 "base" repo available to Anaconda over HTTP. I disabled that repo in cobbler and Anaconda was able to use the NFS path to the folder that has the extracted contents of the Oracle Linux 5.6 DVD to install without a problem. This is strange because I created our 5.6 "base" repo by copying all the .rpm files from the 5.6 DVD and running "createrepo ." in the new directory. I even ran diff to make sure the device-mapper-multipath-0.4.9-23.0.8.el5.x86_64.rpm is the same in my repo and on the 5.6 DVD.

I even tried using Oracle Linux 5.5's installer with the 5.6 "base" repo which should effectively install Oracle Linux 5.6 packages with 5.5's version of Anaconda. That had the same problem. It really looks like there's something about the 5.6 "base" repo that Anaconda doesn't like accessing over HTTP.

Android Z-Push Issues

I've been testing Z-Push (r502/1.5) some more and ran into a frustrating bug when sending from an Android 2.2 emulator.

The problem was I was sending mail through the Android emulator and the body of all of these messages were getting garbled in Z-Push clients. I noticed this in an iPhone running 3.1.2 and the Android 2.2 emulator. The debug.txt file for Z-Push was saying things like "Detected an illegal character in input string" when attempting to read the copy of the message in my Sent folder or Inbox.


Basically, Android sends mail bodies as UTF-8, encoded in base64. When configured to send mail through Z-Push, the messages were being decoded from base64 and the decoded version of the body was being sent to the local sendmail binary. This was Very Bad because Z-Push was leaving the Content-Transfer-Encoding header as base64, meaning any attempt to MIME parse the body will treat the already-decoded base64 text as base64-encoded text. And when that happens, things look garbled.

It turns out Z-Push already had logic for dealing with base64-encoded bodies, but was not re-encoding the body before sending it to the sendmail binary. The problem was the following line in backend/imap.php:

255         // encode the body to base64 if it was sent originally in base64 by the pda
256         if ($body_base64 && !isset($forward)) $body = base64_encode($body);

For messages with base64 bodies, $body_base64 was being properly set to true. The problem was $forward. Forward will always be set, because it is a function parameter that defaults to false:

 123     function SendMail($rfc822, $forward = false, $reply = false, $parent = false) {

The result is the base64_encode will never run, regardless of the value of $body_base64. I changed the line to look like:

 255         // encode the body to base64 if it was sent originally in base64 by the pda
 256         if ($body_base64 && isset($forward) && !$forward) $body = base64_encode($body);

Which maybe wasn't the best fix since if $forward ever is not set as a default function parameter then this will fail again. But it did fix the problem and now I can send mail from an Android 2.2 emulator without a problem.

I should note that this flaw is present in r502/Z-Push 1.5 AND trunk. I should figure out how to let them know...

Hopefully this saves someone half the frustration I just experienced today tracking this down.