Rock Band Ion Drums Replacement Nut Size

If you have an Ion Drum Rocker that has seen serious use, the nuts in your plastic wingnuts that tighten the clamps for pads and cymbals are probably worn. On my 2008 PS3 Ion Drum Rocker some of the nuts were worn to the point that they wouldn't tighten all the way. The correct size for replacement is:

6mm x 1.00, or M6 x 1.00.

If you live near a Lowes, this is the item: https://www.lowes.com/pd/The-Hillman-Group-5-Count-6mm-Zinc-Plated-Metric-Hex-Nuts/3012716.

Now go forth and rock.

All My Yak Shaves #1:One SSSD Config for RHEL 5-7

TL;DR

Even though Kerberos in CentOS/RHEL 5 (1.6.x) supports KEYRING credential cache, SSSD requires a function from Kerberos 1.10.x to use it.

Why SSSD?

I had been meaning to refactor our Puppet authconfig management for a while. (Authconfig is the recommended utility to configure how user information lookups and authentications are performed on RedHat-based Linux distributions.) Until recently we had been using a mix of nss_ldap, nss-pam-ldapd (nslcd) and sssd packages on our (RedHat Enterprise Linux clone) Oracle Enterprise Linux  5-7 hosts. We're lucky in that we generally only have to support RHEL-based Linux, but simultaneously supporting 3 versions can be tricky. My goal was to standardize our authconfig settings as much as possible while still supporting:

1. nested group membership
2. Kerberos authentication and password changes

for our POSIX users and groups in Active Directory. RedHat's documentation recommends SSSD for this task, and since it exists in versions 5, 6, and 7 I hoped the goals could be achieved.

Version Issues Discovered

RHEL 5 ships with SSSD 1.5.1. This version technically supports nested groups, but querying recursive group membership for a user forces SSSD to also get all members of those groups, which was frustrating  sudo users who would have to wait about 5 seconds for the system to check if they were in a sudo group. A coworker discovered this issue was addressed in the 1.9.x version of SSSD with the addition of the "ignore_group_members" configuration parameter.

Another issue was support for Kerberos KEYRING credential caches in SSSD. I discovered the hard way that pam_krb5 will block logins if your disk is full or it cannot create the default Kerberos credential cache (/tmp/krb5cc_%{uidNumber}). I wanted to avoid that with SSSD, and using the Linux kernel's keyring functionality seemed like a good way to avoid the full disk issue. Unfortunately, while the the version of Kerberos that ships with RHEL 5 (1.6.1) supports KEYRING credential caches (even if the default credential cache is not configurable in /etc/krb5.conf), SSSD support for KEYRING was not added until SSSD 1.10.x. That last bit was the hard part - so much of SSSD's Kerberos implementation seems to simply rely on the system Kerberos libraries, but it turns out that SSSD's KEYRING implementation requires krb5_cc_get_full_name(), which was first introduced in Kerberos 1.10.x, which is only available in RHEL 6 and newer.

Version Issues Resolved-ish

I discovered a RHEL 5 backport of SSSD 1.9.x that ended up working out well for us. It gives us "ignore_group_members" which makes sudo fast. It doesn't give us KEYRING on RHEL 5, but given the prospect of trying to compile and statically link newer Kerberos libraries into a newer, hand-compiled version of SSSD, I chose to accept that older servers might not allow remote users' authentications if the disk was full.

Why Did I Just Buy A 16-Year Old Car?

Nostalgia.

Like many adults, I yearn for a car from my childhood. That car is my father's GC Mazda 626 GT sedan. The thing I loved most about the car was its sleeper status; its subdued styling and straight lines easily masked the 2.0L turbocharged beast underneath. It is not fast by today's standards, cranking out only 120 horsepower and 150 pound-feet of torque. But especially on days we were late for church, my father made it fast.

Until a catastrophic timing belt failure hit my daily driver, I was generally uneducated and uninterested in cars. I bought a Honda Civic on reputation and fuel efficiency alone and always paid for oil changes and whatever service my arbitrarily chosen mechanic told me I needed. When I (perhaps foolishly) decided to repair the timing belt failure damage, my sticker shock at the repair bill motivated me to begin educating myself about auto systems and repair. I have always been an iterative learner: try, fail, try again with gained knowledge. I decided I wanted a manual transmission car on which I would teach myself auto repair.

As you can imagine, it's almost impossible to find one of those Mazdas for sale here in the salt belt as they are about 30 years old at this point. They do not have much of an enthusiast following keeping them alive, either. Discouraged, I started looking for cars with a similar "sleeper" quality. Driving a friend's late-model Acura TSX was a revelation that some luxury cars were more than just coddling technology and actually were high performance machines. Perhaps because of that, most of my early searches focused Hondas, Acuras, and Mazdas. Then a friend suggested I look at older BMWs.

I was not sure if I could maintain any BMW, even an older model. All I knew about BMW at that point was the German performance car stereotype of high maintenance costs. Then I discovered the community of seemingly normal people that managed to maintain their own older BMWs and talk about them on internet forums and the myriad of vendors still selling parts for those old models. The community seemed like the same kind of normal people that run Mac OS on PC hardware: small but dedicated group of people willing to go to great lengths to figure things out and share that knowledge because they are passionate about flexibility and craftsmanship.

I would've preferred a BMW from the late 80's or early 90's, but they have become quite collectible and expensive at this point. Many people suggested looking somewhere around the late 90's E36 3-series or E39 5-series. For sleeper status, a 5-series probably would've made more sense. I knew that if I was going to spend thousands of dollars to buy one then I wanted a powerful one. The M5 was very attractive, but had terrible gas mileage. I also wasn't a huge fan of all the now-outdated technology it provided; I wanted something as simple as my no-frills Honda Civic. The E36 M3 got about 27 mpg on the highway and kept the luxury comforts to as much of a minimum as they could be in the late 90s, so I started looking for one of those.

Of course I didn't want just any E36 M3. I wanted a sedan with manual transmission and cruise control. This was a difficult combination to find. The sedan M3 was only available in 1997 and 1998, and not many remain with both stick shift and the optional cruise control. Eventually I found one that looked good in Florida. It wasn't perfect, but it was a southern car for a good price. Looking back, I realize I would've had an easier time with a slightly more expensive and better maintained car. I don't regret having to learn so much so early in my ownership as I did primarily get it to teach myself new skills, but there are days when I wish the previous owner had replaced more wear items before I bought it.

Between when I bought it in February and now I have:

• Learned how to actually drive manual transmission
• Replaced the valve cover gasket
• Replaced the rear trailing arm bushings
• Replaced rear exhaust hangers
• Replaced the windshield cowl cover
• Replaced the rubber gaskets around the door handles
• Replaced the rubber gasket around the rear windscreen
• Reverted to stock shift knob and e-brake handle
• Installed aftermarket shift and e-brake boot

I've learned there's never a point where you are "done." There's always something else to fix, replace, tweak. It's not a cheap hobby in terms of money or time, but I believe it was worth it for the satisfaction I feel fixing and driving it.

Query Google Admin SDK Reporting API For Google Apps Domain, Output to Graphite

The "Reports" section in the Google Apps (gapps) Admin Dashboard has some interesting stats, but only retains data for the past 6 months. If you want to have that data for longer, Google now recommends the Admin SDK's Reports API instead of the deprecated Google Apps Reporting API. I did have a python repo on Github to query the old Reporting API and send the output to graphite, but I have I have since rewritten it to use the new API. It can be found here.

As with the Groups Settings API, you can use these Google Drive SDK instructions to set up a service account and get sample code to access the Admin SDK Reports API for your gapps domain using OAuth 2.0. It seems like most of Google's APIs are moving towards requiring OAuth 2.0 so this two-legged OAuth 2.0 (2LO) method of access for domain-wide delegation seems like it will soon be required for scripts that need gapps domain-wide authority. I just wish Google did a better job telling gapps admins how to set that up.

Google Groups Settings API for GApps Admins Using OAuth 2.0

TL;DR: Need to administratively access Groups Settings API (or other Google APIs that aren't in gdata)? Follow these instructions and try this example script.

As a relatively new Google Apps for Education (GApps) admin it's somewhat surprising to me how confusing Google's API ecosystem is. For python you have gdata and apiclient. Gdata is fairly well documented and only requires a GApps admin username/password for administrative access. You can use gdata to do useful administrative things for your domain like enable create and populate Google groups. But if you want to configure moderation or other Google group settings, you'll need to use apiclient.

My primary complaint with apiclient (the groups settings API in particular) is that the majority of the documentation expects you to be writing for 3-legged OAuth flow where some user must authorize your script in a web browser before it can access Google resources as that user. If you want to use GApps admin credentials to administratively manage group settings without user interaction (from authorizing requests):

If your application has certain unusual authorization requirements, such as logging in at the same time as requesting data access (hybrid) or domain-wide delegation of authority (2LO), then you cannot currently use OAuth 2.0 tokens. In such cases, you must instead use OAuth 1.0 tokens and an API key. You can find your application's API key in the Google APIs Console, in the Simple API Access section of the API Access pane. 

The problem? The apiclient python package has no documentation or examples of this. In fact, it seems to have removed all OAuth 1.0 functionality.

The solution? After many Google searches I finally stumbled across the following Google Drive (!!!) doc page: https://developers.google.com/drive/delegation. I created a service account, granted it access to the Groups Settings API in the API console, enabled the groups settings scope in our GApps admin dashboard, and downloaded the .p12 file for the service account. The only stumbling block was "user_email" (that gets passed as "sub" to SignedJwtAssertionCredentials() should be a GApps domain admin account.

Hopefully this helps someone. And hopefully someone at Google cleans up their documentation.

The Long Road to Logstash

I'm a Splunk addict. I use it almost every day, primarily for problem investigation. So when we started going over our daily indexing limit every day at the start of this semester, I knew I was in trouble. After being locked out from searches for the second time I started to look for alternatives. I found 3 serious candidates:

• logstash+kibana - http://logstash.net/ and http://kibana.org/
• graylog2 - http://graylog2.org/
• ELSA - http://code.google.com/p/enterprise-log-search-and-archive/

I found things I didn't like about all 3 of them but Logstash was by far the most flexible. After a lot of confusion and frustration I finally have it at a point where it is useful. What follows are the things I wish I had known before undertaking this project. It more or less assumes you have had some introduction to Logstash. The most recent presentation from PuppetConf 2012 is quite good: http://www.youtube.com/watch?v=RuUFnog29M4

Logstash

• "Use the grep filter to determine if something exists. Use the grok filter only if you know it will be successful." - This is a big problem with complex log format, like dovecot. I spent many hours trying to write a grok filter that would match every possible Dovecot log line. It's futile. Use the grep filter to grep for something like 'interestingfield=".*"' and add a tag to indicate the field's presence, then grok for 'interestingfield=%{QUOTEDSTRING}' on just that tag. Grok failures are bad. They add the _grokparsefailure tag, and they seemed to contribute to the next problem I ran into, watchdog thread timeouts.
• Logstash, by default, has a 2 second timeout to all filter operations. If it hits that timeout, it will kill itself. I was probably getting this because I was trying to run logstash, elasticsearch and kibana all on the same underpowered development VM, but I think part of the problem was I was doing lots of grok filters that were failing. The recommended solution to the "watchdog timeout" is to run logstash under some system that automatically restarts it. On RHEL6-based distros (and debian-based systems) you probably have upstart. On RHEL5-based distros you can use inittab. There's a good upstart entry in the new logstash cookbook (http://cookbook.logstash.net/recipes/using-upstart/). For inittab you should have a wrapper script that waits a few seconds before attempting to launch logstash just so the old logstash process can give up the TCP ports it was listening on.

Elasticsearch

If you want to be able to search your logs with Kibana, logstash needs to output to Elasticsearch. Unless you already have experience deploying Elasticsearch, you will probably spend more time learning about Elasticsearch than Logstash. I was an Elasticsearch newbie, so some of these may seem like common sense to people familiar with Elasticsearch.

• There are 2 plugins I would consider essential for running Elasticsearch: head (management), and bigdesk (monitoring). Paramedic is also extremely useful for monitoring multiple cluster nodes. If you decide to use the "elasticsearch_river" output, you will need a river plugin. Unlike other plugins, you *must* restart Elasticsearch after installing a river plugin (at least with the rabbitmq one). A good list of plugins: http://www.elasticsearch.org/guide/reference/modules/plugins.html
• Elasticsearch is a java program, so you will need to tune the JVM to your machine. bin/elasticsearch.in.sh is a good place to start. I'd personally recommend setting ES_HEAP_SIZE and telling Elasticsearch to lock that memory. You will almost definitely need to increase the limit of open files. In RHEL it seems to default to 1024. Most recommendations for Elasticsearch are around 300k or 600k.
• There are 3 "outputs" in logstash for elasticsearch. The "elasticsearch" output will run embedded elasticsearch and can either store data in itself or connect to an Elasticsearch cluster and send the data to that. I couldn't get "elasticsearch_http" to work with bulk indexing. I'm currently using "elasticsearch_river", which sends events from logstash to an AMQP (rabbitmq) queue which Elasticsearch indexes data from.
• You need to configure the query cache. I went with setting the query cache to "soft" which lets the JVM garbage collection expire the query cache. For more info, see: http://blog.sematext.com/2012/05/17/elasticsearch-cache-usage/
• Compress/optimize old indices.
• Use mappings and the options of the special _source field to limit the number of fields Elasticsearch saves and their type. (http://www.elasticsearch.org/guide/reference/mapping/ and http://www.elasticsearch.org/guide/reference/mapping/source-field.html)
• Best practice: Make a template that will be applied to all logstash indexes. I started with http://untergeek.com/2012/09/20/using-templates-to-improve-elasticsearch-caching-with-logstash/   

Compiling posix-winsync Plugin for 389 Directory Server on Linux

https://github.com/cgrzemba/Posix-Winsync-Plugin-for-389-directory-server is a plugin for the 389 directory server that enables the syncing of posix attributes between 389 and Active Directory. It was written for Solaris and I was unable to produce a working linux binary of it using the supplied files. I was able to compile and link it by hand in RHEL 6.2. To do this you will need the binary and devel packages for 389 and nspr. The pkgconfig files (.pc) for both of those should help you if my gcc flags or ld flags don't work on your system. Once you have those in place, the following commands in the directory for the project should produce a shared object file that can be copied to where your 389 directory server plugins are (for me, /usr/lib64/dirsrv/plugins).

• gcc -fPIC -I/usr/include/nspr4 -DUSE_OPENLDAP -I/usr/include/dirsrv -I /usr/include/ -c posix-winsync.c
• gcc -fPIC -I/usr/include/nspr4 -DUSE_OPENLDAP -I/usr/include/dirsrv -I /usr/include/ -c posix-winsync-config.c
• gcc -fPIC -I/usr/include/nspr4 -DUSE_OPENLDAP -I/usr/include/dirsrv -I /usr/include/ -c posix-group-func.c
• ld -shared -L/usr/lib64 -lplds4 -lplc4 -lnspr4 -lpthread -ldl -L/usr/lib64/dirsrv -lslapd posix-group-func.o posix-winsync-config.o posix-winsync.o -o libposix-winsync.so

At this point you should run

ldd libposix-winsync.so

to make sure all the libraries required by that file can be found. I had to create a new entry in /etc/ld.so.conf.d to point to /usr/lib64/dirsrv and run ldconfig for it to find libslapd.so.0. I'm not sure how the other 389 plugins worked without setting that.

Then you need to import import the ldif file that comes with the plugin in to your 389 server. The way the plugin seems to work is when you set up a windows sync agreement it will also sync posix attributes. If it cannot find a required attribute, it will not sync that user/group.

It would be nice to create an RPM of this and extend the plugin so the list of attributes it syncs can be dynamic/optional, but for now it gets the job done.

CAS-ifying Roundcube

Quick background/summary:

• CAS is a web-based Single Sign-On (SSO) system. http://www.jasig.org/cas
• Roundcube is a web-based email client, often called a "webmail" system. http://roundcube.net/
• Alex Li wrote a plugin for Roundcube to use CAS as an authentication source. It doesn't work with Roundcube 0.6 or higher without modifying the Roundcube source. http://code.google.com/p/rc-cas-plugin/
• I forked Alex's plugin on Github and made it work with new, stock versions of Roundcube. https://github.com/dfwarden/Roundcube-CAS-Authn

We have a CAS server for our campus, and we use Roundcube for our webmail system. Some time ago, I was asked to extend our Roundcube system to authenticate against CAS. Doing so would net us some nice benefits:

• Graceful handling of locked accounts/expired passwords.
• SSO to Webmail if already signed in to another SSO service.
• Easy access for custom webapps wanting to access IMAP.

One of the first things I had to do was configure our IMAP server to accept CAS tickets as passwords. Our IMAP server (Dovecot) uses PAM for authentication, so all I had to do was follow the instructions on the Google translation of this page (French):

• Create CAS services for HTTPS and IMAP URIs that ignore attributes and can proxy. On our older version of CAS I also had to restart the servlet container for the new services to be active.
• Download, compile, install and configure pam_cas from https://sourcesup.cru.fr/frs/?group_id=213.
• Add pam_cas to Dovecot's PAM stack with the IMAP service I created in CAS and the pam_cas.conf file I created in the options.
• Test that Dovecot validates CAS tickets. You could use the CASTest utility that comes with pam_cas, but I chose to just open https://<CAS server>/login?service=<IMAP service> in a web browser and copy/paste the service ticket I got from CAS into a telnet session that was authenticating to our IMAP server. Be careful when testing! The default CAS Service Ticket life is 10 seconds, so you may have to move fast if you follow my testing method.

If you run in to problems in any of those steps, the best thing you can do is turn debug on in /etc/pam_cas.conf and send *.debug in your syslog config to a file that you can read. You should get good info from that as to why pam_cas is failing.

Once I had IMAP authenticating users with CAS tickets, Alex's plugin got me most of the way, but I found that I had to make 2 modifications to Roundcube's index.php file to get the plugin to work properly. I also wanted a few additional features that his plugin lacked:

• The option of not using CAS for authentication. Alex's plugin forces it for authentication to Roundcube.
• SMTP authentication using a CAS proxy ticket.

I was able to remove the need to modify Roundcube's source and add those features to my fork of his plugin. It is currently in production in our 8-node Roundcube cluster. There were a few "gotchas" along the way that I don't want to forget, and other people may find useful when working with my plugin or CAS' proxy feature in general:

• If you run multiple Roundcube servers behind a load balancer, you need to be sure of 2 things:
• You have the same version of phpCAS on all of your Roundcube servers.
• You are using shared storage for PGTIOUs coming back from the CAS server. This page talks about this process but basically part of proxy CAS authentication is the CAS server will make an HTTP call to a URL that is supposed to store a necessary piece of information. Unless you're absolutely sure the CAS server will be sent to the same Roundcube server that the user is currently hitting, you need to make sure that all Roundcube servers can share the directory where PGTIOUs are saved.
• If you are using CAS' proxy method for authenticating to IMAP, you all but need an IMAP proxy. Without it, you will need to generate and validate a new ticket from CAS (multiple HTTP trips) for each page load. With an IMAP proxy, the user's first IMAP ticket is validated by pam_cas by your actual IMAP server and saved in your IMAP proxy. Roundcube will continue to send the same ticket and the IMAP proxy will continue to believe the user is logged in, even though that ticket has been used and cannot be validated again by the IMAP server.

Increasing VMWare SCSI Timeout on RHEL6 and ESX5

VMWare recommends increasing the disk timeouts for linux guests to 180 seconds to handle things like a storage path failover. There are a few ways to accomplish this. If you are a Netapp customer you could run their script, or you could follow the instructions in the VMWare article.

Unfortunately, both of these solutions were written for ESX4 and RHEL5. If you are on ESX5 with a RHEL6 guest, do the following to increase your timeout for VMWare SCSI disks to 180 seconds:

1. Put the following in a file in /etc/udev/rules.d. I chose /etc/udev/rules.d/99-vmware-scsi-timeout.rules.
   SUBSYSTEM=="block", ACTION=="add", BUS=="scsi", SYSFS{vendor}=="VMware", SYSFS{model}=="Virtual disk",   RUN+="/bin/sh -c 'echo 180 >/sys$DEVPATH/device/timeout'"
2. Run udevadm test /block/sda and make sure you see a line that indicates the echo 180 would run. If it doesn't then your SYSFS values are somehow different than mine. The output you just got should help you figure out the correct values. Look for the entries that start with ID_.
3. Once you have udevadm telling you it would run the echo 180 command you can restart your machine or restart udev during your maintenance window, if this is a production VM. If you don't want to reboot the machine, you can restart udev with:
1. udevadm control --reload-rules
2. start_udev
4. Verify it worked by looking at the timeout file in the /sys tree for your devname. For me this was accomplished with: cat /sys/block/sda/device/timeout

Kerberos, Exim, Dovecot and Initial-Response

Someday I will sit down and put to paper the exact steps to get Dovecot and Exim to use GSSAPI authentication. This is not that day.

One thing I did find out is that the base64 encoding of the tickets that mail clients like Mac Mail and Thunderbird attempt to authenticate with are quite long. In fact, they are so long that certain client/server combinations don't work without changing the server source and recompiling. Here are the combinations I found that don't work:

• SMTP using Thunderbird 6 and Exim 4.72 from Enterprise Linux 6 EPEL
• IMAP using Apple Mail (10.7 Lion) and Dovecot 2.0.13 from AT RPMs

In both non-working combinations, the client attempts to use the Initial-Response feature of the respective protocol. If you haven't heard of this before (I sure hadn't) then you can check out the relevant RFCs for IMAP and SMTP. TL;DR: Initial-Response allows clients to save round trips by sending everything the server needs to authenticate them in the initial auth (SMTP) or login (IMAP) call. The problem with this is that the server software doesn't know that a huge base64 encode of the client's Kerberos ticket is coming, has a sane limit of the number of bytes it will accept from the client in a single command and will truncate the ticket that the client sends.

Unfortunately, the fix for this seems to be recompiling the server software with bigger input buffers. This post was my first clue about the problem and contains what I believe to be a safe value for an input buffer that could possibly see a base64 Kerberos ticket along with a handy patch that applies the fix to Exim. For dovecot (2.0.13) you'll want to change MASTER_AUTH_MAX_DATA_SIZE in src/lib-master/master-auth.h and LOGIN_MAX_INBUF_SIZE in src/login-common/client-common.h.

Dovecot and SSO Kerberos With Active Directory

If you have an Active Directory environment, you have multiple authentication options for most open-source software, including the excellent Dovecot IMAP/POP server. You could use:

• Dovecot's PAM method
• ...to connect to LDAP on AD
• ...to connect to Kerberos on AD
• Dovecot's LDAP method (if you need more granular control than PAM)
• Dovecot's Kerberos method (if you want SSO from AD-bound clients)

The last method, I believe is the hardest to achieve. First, Dovecot's Kerberos authn page tells you that you need service keys on your server, but offers no indication of how to do that. There are some very helpful posts to the Dovecot mailing list, but none that fit what I was looking for. My problem was:

• I'm using nslcd on RHEL6 to look at LDAP on AD for POSIX user information like uid, uidNumber, homeDirectory, so I don't really need winbind. (<- I could be dead wrong about winbind but it didn't seem to work for me and I didn't spend too much time investigating why.)
• The hostname that users will connect to is load-balanced across multiple hosts. Because SPN (Service Principal Names) in AD must be unique, I could not bind each mail host to AD using Samba and add IMAP/mailname.domain.edu to each machine account.

Some Google sleuthing revealed that the way most people get around this is to create a user in Active Directory, and add SPNs that will be used on multiple hosts to that account. The best documentation I found for doing this was this post by Achim Grolms. I used those instructions to create the SPN, which got mapped onto a user I created, and generate a keytab for Dovecot to use to validate client requests to that service.

Some pitfalls to watch for when configuring Dovecot for this method:

1. The user that runs Dovecot's auth process needs to be able to read the keytab file you get from ktpass, but you probably don't want anyone else to read it.
2. Set auth_gssapi_hostname to "$ALL" since I'm not sure if these hosts would consider themselves all the hostname mailname.domain.edu.

PSA: NFS Locking in RHEL6 Needs Reverse Lookup of NFS Server

The RHEL6 version of rpc.statd needs to be able to reverse lookup a hostname for the IP address of your NFS server if you want to lock files mounted from that server. If you don't have a reverse lookup for the IP of your NFS server, just throw its IP and some kind of name for it in /etc/hosts.

Otherwise you get errors in syslog to the effect of:

Feb 23 13:14:57 wardentest3 rpc.statd[1259]: No canonical hostname found for 10.238.3.20
Feb 23 13:14:57 wardentest3 rpc.statd[1259]: STAT_FAIL to wardentest3.geneseo.edu for SM_MON of 10.238.3.20
Feb 23 13:14:57 wardentest3 kernel: lockd: cannot monitor 10.238.3.20

Oracle Linux 5.6 Anaconda, HTTP RPM Repo, device-mapper-multipath

Including the "base" RPM channel of Oracle Linux 5.6 (a copy of the Server directory) that is accessed over HTTP causes the last RPM fail to install during a cobbler-assisted pxeboot install. For me it's device-mapper-multipath-0.4.9-23.0.8.el5.x86_64.rpm. Anaconda says it cannot download/install that package and asks me if I want to try to download/install it again or not.

The line in the Apache logs each time I pick "retry":

137.238.2.94 - - [10/Feb/2011:21:02:19 +0000] "GET /oracle/OracleLinux/OL5/6/base/x86_64/device-mapper-multipath-0.4.9-23.0.8.el5.x86_64.rpm HTTP/1.1" 416 394

A packet capture reveals that anaconda is issuing a GET request for the RPM in question but only asking for bytes 440-25514. This corresponds to that RPM's information in the repo:

rpm:header-range start="440" end="25515"


It looks like when Anaconda says "beginning installation" it's querying the header-range of each RPM it's interested in, then when it has resolved dependencies/order of install it downloads the full contents of each RPM.

In the packet capture the GET request has the Range header set to "bytes=86392-" which I assume means it wants from bytes 86392 to the end of the file. The only problem is that the file is only 86392 bytes, so it's asking for a byte range past the end of the file and thus Apache returns the 416 error.

The strange thing is this is the second time during the "fetch and install each RPM" phase where this file was requested. The first time, anaconda was smart enough to fetch the whole file and presumably install it.

As I mentioned earlier, this problem is only apparent when making the 5.6 "base" repo available to Anaconda over HTTP. I disabled that repo in cobbler and Anaconda was able to use the NFS path to the folder that has the extracted contents of the Oracle Linux 5.6 DVD to install without a problem. This is strange because I created our 5.6 "base" repo by copying all the .rpm files from the 5.6 DVD and running "createrepo ." in the new directory. I even ran diff to make sure the device-mapper-multipath-0.4.9-23.0.8.el5.x86_64.rpm is the same in my repo and on the 5.6 DVD.

I even tried using Oracle Linux 5.5's installer with the 5.6 "base" repo which should effectively install Oracle Linux 5.6 packages with 5.5's version of Anaconda. That had the same problem. It really looks like there's something about the 5.6 "base" repo that Anaconda doesn't like accessing over HTTP.

Android Z-Push Issues

I've been testing Z-Push (r502/1.5) some more and ran into a frustrating bug when sending from an Android 2.2 emulator.

The problem was I was sending mail through the Android emulator and the body of all of these messages were getting garbled in Z-Push clients. I noticed this in an iPhone running 3.1.2 and the Android 2.2 emulator. The debug.txt file for Z-Push was saying things like "Detected an illegal character in input string" when attempting to read the copy of the message in my Sent folder or Inbox.


Basically, Android sends mail bodies as UTF-8, encoded in base64. When configured to send mail through Z-Push, the messages were being decoded from base64 and the decoded version of the body was being sent to the local sendmail binary. This was Very Bad because Z-Push was leaving the Content-Transfer-Encoding header as base64, meaning any attempt to MIME parse the body will treat the already-decoded base64 text as base64-encoded text. And when that happens, things look garbled.

It turns out Z-Push already had logic for dealing with base64-encoded bodies, but was not re-encoding the body before sending it to the sendmail binary. The problem was the following line in backend/imap.php:

255         // encode the body to base64 if it was sent originally in base64 by the pda
256         if ($body_base64 && !isset($forward)) $body = base64_encode($body);

For messages with base64 bodies, $body_base64 was being properly set to true. The problem was $forward. Forward will always be set, because it is a function parameter that defaults to false:

 123     function SendMail($rfc822, $forward = false, $reply = false, $parent = false) {

The result is the base64_encode will never run, regardless of the value of $body_base64. I changed the line to look like:

 255         // encode the body to base64 if it was sent originally in base64 by the pda
 256         if ($body_base64 && isset($forward) && !$forward) $body = base64_encode($body);

Which maybe wasn't the best fix since if $forward ever is not set as a default function parameter then this will fail again. But it did fix the problem and now I can send mail from an Android 2.2 emulator without a problem.

I should note that this flaw is present in r502/Z-Push 1.5 AND trunk. I should figure out how to let them know...

Hopefully this saves someone half the frustration I just experienced today tracking this down.

CAS-ifying Roundcube 0.4.2

I've been testing Roundcube 0.4.2 as a replacement for our now-decrepit Squirrelmail 1.4.17 Webmail. I've been extremely impressed so far. It does natively or there is a plugin for almost everything I can think of. It acts so much like an application that I found myself angry that I couldn't dynamically resize the Subject/From/Date columns in the folder view. It had tricked me so much into thinking it was a native app that when the illusion broke I was actually confused.

Only one plugin has given me trouble so far, and that's the CAS plugin. For those who aren't familiar with CAS, it is an open-source Central Authentication System that many campuses are using these days for their portals. The benefits to us for using CAS for Roundcube in addition to our portal would be that users only have to sign in once and they can access either Roundcube or the portal without needing to authn again (as long as their cookie/session on the CAS server is still good). We've been trying to convert as many open-source login pages as we can to use CAS.

The problem is perhaps best described here:

http://code.google.com/p/rc-cas-plugin/issues/detail?id=1

Basically even if you do everything right, you will get a redirect loop because somewhere between the version of Roundcube this plugin was written for and 0.4.2, the code to kill/regenerate the PHP session changed. I made some comments in that ticket that include what I was able to do in order to actually log in (comment out a call to kill_session()) but that is not a very clean solution, and it doesn't solve the problem of users logging in from a different browser or IP. Really, this plugin needs work.

Hopefully this helps someone or helps remind me what the problem was should I find the time to work on fixing this plugin.

Z-Push Woes

Z-Push (http://z-push.sourceforge.net/) is an interesting project. It is a PHP web app that lets you serve different backends for mail/contacts to clients that support ActiveSync. The most common use for it seems to be to give devices with zero or minimal IMAP support (but decent ActiveSync support) access to their mail by letting them connect to that IMAP through ActiveSync.

The first thing that bothered me was it REALLY wants to run under mod_php in Apache. People have been able to get it to run in fastCGI by adding the following to compat.php:

if (!function_exists("apache_request_headers")) {
    function apache_request_headers() {
        $headers = array();
        foreach ($_SERVER as $key => $value) {
            if (substr($key, 0, 5) != 'HTTP_') {
                continue;
            }
            $headername = strtr(ucwords(strtolower(strtr(substr($key, 5), '_', ' '))), ' ', '-');
            $headers[$headername] = $value;
        }

        return $headers;
    }

(Taken from http://z-push.sourceforge.net/phpbb/viewtopic.php?f=2&t=36#p1349)
Sadly, the z-push developers didn't accept that patch.

The second problem I ran in to was z-push was segfaulting in PHP's imap_search function. I looked at the raw logs and saw that the client was getting a good response from the search request, but PHP still segfaulted. Some investigation led me to suspect the SE_FREE flag, per this PHP bug report: http://bugs.php.net/48619

Basically I changed the following line in backend/imap.php:

$search = @imap_search($this->_mbox, "SINCE ". date("d-M-Y", $cutoffdate));

to look like:

$search = @imap_search($this->_mbox, "SINCE ". date("d-M-Y", $cutoffdate), ! SE_FREE);

and things started working. So if your PHP is newer than 5.2.10 you hopefully don't have to deal with this. But if not, hopefully this helps you.

Update Fink For 10.6

Lots of people may have figured this out, but if you're getting the following error when trying to install a package with Fink in Mac OS 10.6:

Use of uninitialized value $darwin_osx in numeric eq (==) at /sw/lib/perl5/Fink/Services.pm line 1404, line 1.

Use of uninitialized value $darwin_osx in concatenation (.) or string at /sw/lib/perl5/Fink/Services.pm line 1404, line 1.

Argument "10.6 does not match the expected value of . Please run `..." isn't numeric in exit at /sw/lib/perl5/Fink/Services.pm line 1404, line 1.

 You can change line 1448 of /sw/lib/perl5/Fink/Services.pm look like this:

my %darwin_osx = ('1' => '10.0', '5' => '10.1', '6' => '10.2', '7' => '     10.3', '8' => '10.4', '9' => '10.5', '10' => '10.6');

Basically I added '10' => '10.6' after '9' => '10.5'. And I was able to install things again.

Windows Media Player 12 UPNP Client Sucks.

I was impressed when Windows Media Player 12 saw my Mediatomb UPNP server I use to stream video to a Playstation 3. "Now I don't have to use iTunes/mt-daapd to listen to my music library," I thought.

Then I browsed my music. I'm not exactly sure who is at fault, but either way Media Player is pretty stupid in that it has no way I could find to clear it's cache of what it thinks the UPNP server is offering. The only way to start fresh I've found was to go to C:\Users\%you\AppData\Local\Microsoft\Media Player and delete the files that start with "_HME_home" with Media Player not running. Next time you start Media Player, it should re-index all files.

What I can't figure out it why it presents so many duplicate entries unless I remove the artist/album containers that Mediatomb creates.

Netapp + Rquotad + NFS + RHEL5

Those mounting volumes via NFS on a Netapp filer wanting to use rquotad on a volume with qtrees may notice that, under some conditions, things do not work.

If you are using qtrees, you can still get rquota to work by mounting each qtree individually, each with the "nosharecache" option. This forces each mounted qtree to have different st_dev values so that remote quota calls are enabled for all mounted qtrees, instead of just the first one. Without "nosharecache", all mounts from a volume on a Netapp filer will have the same st_dev value and any rquota calls will only query the first mounted qtree.

See http://linux.die.net/man/5/nfs for info on the "nosharecache" . . . there are some caveats to using it.

Create Maildir Folders

mkdir -m 700 $MAILBOX/{,tmp,new,cur}

Substitute $MAILBOX for what you want to call your Maildir folder.

(Taken from http://plasmasturm.org/log/81/)