The Long Road to Logstash

I'm a Splunk addict. I use it almost every day, primarily for problem investigation. So when we started going over our daily indexing limit every day at the start of this semester, I knew I was in trouble. After being locked out from searches for the second time I started to look for alternatives. I found 3 serious candidates:

• logstash+kibana - http://logstash.net/ and http://kibana.org/
• graylog2 - http://graylog2.org/
• ELSA - http://code.google.com/p/enterprise-log-search-and-archive/

I found things I didn't like about all 3 of them but Logstash was by far the most flexible. After a lot of confusion and frustration I finally have it at a point where it is useful. What follows are the things I wish I had known before undertaking this project. It more or less assumes you have had some introduction to Logstash. The most recent presentation from PuppetConf 2012 is quite good: http://www.youtube.com/watch?v=RuUFnog29M4

Logstash

• "Use the grep filter to determine if something exists. Use the grok filter only if you know it will be successful." - This is a big problem with complex log format, like dovecot. I spent many hours trying to write a grok filter that would match every possible Dovecot log line. It's futile. Use the grep filter to grep for something like 'interestingfield=".*"' and add a tag to indicate the field's presence, then grok for 'interestingfield=%{QUOTEDSTRING}' on just that tag. Grok failures are bad. They add the _grokparsefailure tag, and they seemed to contribute to the next problem I ran into, watchdog thread timeouts.
• Logstash, by default, has a 2 second timeout to all filter operations. If it hits that timeout, it will kill itself. I was probably getting this because I was trying to run logstash, elasticsearch and kibana all on the same underpowered development VM, but I think part of the problem was I was doing lots of grok filters that were failing. The recommended solution to the "watchdog timeout" is to run logstash under some system that automatically restarts it. On RHEL6-based distros (and debian-based systems) you probably have upstart. On RHEL5-based distros you can use inittab. There's a good upstart entry in the new logstash cookbook (http://cookbook.logstash.net/recipes/using-upstart/). For inittab you should have a wrapper script that waits a few seconds before attempting to launch logstash just so the old logstash process can give up the TCP ports it was listening on.

Elasticsearch

If you want to be able to search your logs with Kibana, logstash needs to output to Elasticsearch. Unless you already have experience deploying Elasticsearch, you will probably spend more time learning about Elasticsearch than Logstash. I was an Elasticsearch newbie, so some of these may seem like common sense to people familiar with Elasticsearch.

• There are 2 plugins I would consider essential for running Elasticsearch: head (management), and bigdesk (monitoring). Paramedic is also extremely useful for monitoring multiple cluster nodes. If you decide to use the "elasticsearch_river" output, you will need a river plugin. Unlike other plugins, you *must* restart Elasticsearch after installing a river plugin (at least with the rabbitmq one). A good list of plugins: http://www.elasticsearch.org/guide/reference/modules/plugins.html
• Elasticsearch is a java program, so you will need to tune the JVM to your machine. bin/elasticsearch.in.sh is a good place to start. I'd personally recommend setting ES_HEAP_SIZE and telling Elasticsearch to lock that memory. You will almost definitely need to increase the limit of open files. In RHEL it seems to default to 1024. Most recommendations for Elasticsearch are around 300k or 600k.
• There are 3 "outputs" in logstash for elasticsearch. The "elasticsearch" output will run embedded elasticsearch and can either store data in itself or connect to an Elasticsearch cluster and send the data to that. I couldn't get "elasticsearch_http" to work with bulk indexing. I'm currently using "elasticsearch_river", which sends events from logstash to an AMQP (rabbitmq) queue which Elasticsearch indexes data from.
• You need to configure the query cache. I went with setting the query cache to "soft" which lets the JVM garbage collection expire the query cache. For more info, see: http://blog.sematext.com/2012/05/17/elasticsearch-cache-usage/
• Compress/optimize old indices.
• Use mappings and the options of the special _source field to limit the number of fields Elasticsearch saves and their type. (http://www.elasticsearch.org/guide/reference/mapping/ and http://www.elasticsearch.org/guide/reference/mapping/source-field.html)
• Best practice: Make a template that will be applied to all logstash indexes. I started with http://untergeek.com/2012/09/20/using-templates-to-improve-elasticsearch-caching-with-logstash/   

Compiling posix-winsync Plugin for 389 Directory Server on Linux

https://github.com/cgrzemba/Posix-Winsync-Plugin-for-389-directory-server is a plugin for the 389 directory server that enables the syncing of posix attributes between 389 and Active Directory. It was written for Solaris and I was unable to produce a working linux binary of it using the supplied files. I was able to compile and link it by hand in RHEL 6.2. To do this you will need the binary and devel packages for 389 and nspr. The pkgconfig files (.pc) for both of those should help you if my gcc flags or ld flags don't work on your system. Once you have those in place, the following commands in the directory for the project should produce a shared object file that can be copied to where your 389 directory server plugins are (for me, /usr/lib64/dirsrv/plugins).

• gcc -fPIC -I/usr/include/nspr4 -DUSE_OPENLDAP -I/usr/include/dirsrv -I /usr/include/ -c posix-winsync.c
• gcc -fPIC -I/usr/include/nspr4 -DUSE_OPENLDAP -I/usr/include/dirsrv -I /usr/include/ -c posix-winsync-config.c
• gcc -fPIC -I/usr/include/nspr4 -DUSE_OPENLDAP -I/usr/include/dirsrv -I /usr/include/ -c posix-group-func.c
• ld -shared -L/usr/lib64 -lplds4 -lplc4 -lnspr4 -lpthread -ldl -L/usr/lib64/dirsrv -lslapd posix-group-func.o posix-winsync-config.o posix-winsync.o -o libposix-winsync.so

At this point you should run

ldd libposix-winsync.so

to make sure all the libraries required by that file can be found. I had to create a new entry in /etc/ld.so.conf.d to point to /usr/lib64/dirsrv and run ldconfig for it to find libslapd.so.0. I'm not sure how the other 389 plugins worked without setting that.

Then you need to import import the ldif file that comes with the plugin in to your 389 server. The way the plugin seems to work is when you set up a windows sync agreement it will also sync posix attributes. If it cannot find a required attribute, it will not sync that user/group.

It would be nice to create an RPM of this and extend the plugin so the list of attributes it syncs can be dynamic/optional, but for now it gets the job done.

CAS-ifying Roundcube

Quick background/summary:

• CAS is a web-based Single Sign-On (SSO) system. http://www.jasig.org/cas
• Roundcube is a web-based email client, often called a "webmail" system. http://roundcube.net/
• Alex Li wrote a plugin for Roundcube to use CAS as an authentication source. It doesn't work with Roundcube 0.6 or higher without modifying the Roundcube source. http://code.google.com/p/rc-cas-plugin/
• I forked Alex's plugin on Github and made it work with new, stock versions of Roundcube. https://github.com/dfwarden/Roundcube-CAS-Authn

We have a CAS server for our campus, and we use Roundcube for our webmail system. Some time ago, I was asked to extend our Roundcube system to authenticate against CAS. Doing so would net us some nice benefits:

• Graceful handling of locked accounts/expired passwords.
• SSO to Webmail if already signed in to another SSO service.
• Easy access for custom webapps wanting to access IMAP.

One of the first things I had to do was configure our IMAP server to accept CAS tickets as passwords. Our IMAP server (Dovecot) uses PAM for authentication, so all I had to do was follow the instructions on the Google translation of this page (French):

• Create CAS services for HTTPS and IMAP URIs that ignore attributes and can proxy. On our older version of CAS I also had to restart the servlet container for the new services to be active.
• Download, compile, install and configure pam_cas from https://sourcesup.cru.fr/frs/?group_id=213.
• Add pam_cas to Dovecot's PAM stack with the IMAP service I created in CAS and the pam_cas.conf file I created in the options.
• Test that Dovecot validates CAS tickets. You could use the CASTest utility that comes with pam_cas, but I chose to just open https://<CAS server>/login?service=<IMAP service> in a web browser and copy/paste the service ticket I got from CAS into a telnet session that was authenticating to our IMAP server. Be careful when testing! The default CAS Service Ticket life is 10 seconds, so you may have to move fast if you follow my testing method.

If you run in to problems in any of those steps, the best thing you can do is turn debug on in /etc/pam_cas.conf and send *.debug in your syslog config to a file that you can read. You should get good info from that as to why pam_cas is failing.

Once I had IMAP authenticating users with CAS tickets, Alex's plugin got me most of the way, but I found that I had to make 2 modifications to Roundcube's index.php file to get the plugin to work properly. I also wanted a few additional features that his plugin lacked:

• The option of not using CAS for authentication. Alex's plugin forces it for authentication to Roundcube.
• SMTP authentication using a CAS proxy ticket.

I was able to remove the need to modify Roundcube's source and add those features to my fork of his plugin. It is currently in production in our 8-node Roundcube cluster. There were a few "gotchas" along the way that I don't want to forget, and other people may find useful when working with my plugin or CAS' proxy feature in general:

• If you run multiple Roundcube servers behind a load balancer, you need to be sure of 2 things:
• You have the same version of phpCAS on all of your Roundcube servers.
• You are using shared storage for PGTIOUs coming back from the CAS server. This page talks about this process but basically part of proxy CAS authentication is the CAS server will make an HTTP call to a URL that is supposed to store a necessary piece of information. Unless you're absolutely sure the CAS server will be sent to the same Roundcube server that the user is currently hitting, you need to make sure that all Roundcube servers can share the directory where PGTIOUs are saved.
• If you are using CAS' proxy method for authenticating to IMAP, you all but need an IMAP proxy. Without it, you will need to generate and validate a new ticket from CAS (multiple HTTP trips) for each page load. With an IMAP proxy, the user's first IMAP ticket is validated by pam_cas by your actual IMAP server and saved in your IMAP proxy. Roundcube will continue to send the same ticket and the IMAP proxy will continue to believe the user is logged in, even though that ticket has been used and cannot be validated again by the IMAP server.

Increasing VMWare SCSI Timeout on RHEL6 and ESX5

VMWare recommends increasing the disk timeouts for linux guests to 180 seconds to handle things like a storage path failover. There are a few ways to accomplish this. If you are a Netapp customer you could run their script, or you could follow the instructions in the VMWare article.

Unfortunately, both of these solutions were written for ESX4 and RHEL5. If you are on ESX5 with a RHEL6 guest, do the following to increase your timeout for VMWare SCSI disks to 180 seconds:

1. Put the following in a file in /etc/udev/rules.d. I chose /etc/udev/rules.d/99-vmware-scsi-timeout.rules.
   SUBSYSTEM=="block", ACTION=="add", BUS=="scsi", SYSFS{vendor}=="VMware", SYSFS{model}=="Virtual disk",   RUN+="/bin/sh -c 'echo 180 >/sys$DEVPATH/device/timeout'"
2. Run udevadm test /block/sda and make sure you see a line that indicates the echo 180 would run. If it doesn't then your SYSFS values are somehow different than mine. The output you just got should help you figure out the correct values. Look for the entries that start with ID_.
3. Once you have udevadm telling you it would run the echo 180 command you can restart your machine or restart udev during your maintenance window, if this is a production VM. If you don't want to reboot the machine, you can restart udev with:
1. udevadm control --reload-rules
2. start_udev
4. Verify it worked by looking at the timeout file in the /sys tree for your devname. For me this was accomplished with: cat /sys/block/sda/device/timeout