Tuesday, August 5, 2014

Why Did I Just Buy A 16-Year Old Car?

Nostalgia.


Like many adults, I yearn for a car from my childhood. That car is my father's GC Mazda 626 GT sedan. The thing I loved most about the car was its sleeper status; its subdued styling and straight lines easily masked the 2.0L turbocharged beast underneath. It is not fast by today's standards, cranking out only 120 horsepower and 150 pound-feet of torque. But especially on days we were late for church, my father made it fast.

Until a catastrophic timing belt failure hit my daily driver, I was generally uneducated and uninterested in cars. I bought a Honda Civic on reputation and fuel efficiency alone and always paid for oil changes and whatever service my arbitrarily chosen mechanic told me I needed. When I (perhaps foolishly) decided to repair the timing belt failure damage, my sticker shock at the repair bill motivated me to begin educating myself about auto systems and repair. I have always been an iterative learner: try, fail, try again with gained knowledge. I decided I wanted a manual transmission car on which I would teach myself auto repair.

As you can imagine, it's almost impossible to find one of those Mazdas for sale here in the salt belt as they are about 30 years old at this point. They do not have much of an enthusiast following keeping them alive, either. Discouraged, I started looking for cars with a similar "sleeper" quality. Driving a friend's late-model Acura TSX was a revelation that some luxury cars were more than just coddling technology and actually were high performance machines. Perhaps because of that, most of my early searches focused Hondas, Acuras, and Mazdas. Then a friend suggested I look at older BMWs.

I was not sure if I could maintain any BMW, even an older model. All I knew about BMW at that point was the German performance car stereotype of high maintenance costs. Then I discovered the community of seemingly normal people that managed to maintain their own older BMWs and talk about them on internet forums and the myriad of vendors still selling parts for those old models. The community seemed like the same kind of normal people that run Mac OS on PC hardware: small but dedicated group of people willing to go to great lengths to figure things out and share that knowledge because they are passionate about flexibility and craftsmanship.

I would've preferred a BMW from the late 80's or early 90's, but they have become quite collectible and expensive at this point. Many people suggested looking somewhere around the late 90's E36 3-series or E39 5-series. For sleeper status, a 5-series probably would've made more sense. I knew that if I was going to spend thousands of dollars to buy one then I wanted a powerful one. The M5 was very attractive, but had terrible gas mileage. I also wasn't a huge fan of all the now-outdated technology it provided; I wanted something as simple as my no-frills Honda Civic. The E36 M3 got about 27 mpg on the highway and kept the luxury comforts to as much of a minimum as they could be in the late 90s, so I started looking for one of those.

Of course I didn't want just any E36 M3. I wanted a sedan with manual transmission and cruise control. This was a difficult combination to find. The sedan M3 was only available in 1997 and 1998, and not many remain with both stick shift and the optional cruise control. Eventually I found one that looked good in Florida. It wasn't perfect, but it was a southern car for a good price. Looking back, I realize I would've had an easier time with a slightly more expensive and better maintained car. I don't regret having to learn so much so early in my ownership as I did primarily get it to teach myself new skills, but there are days when I wish the previous owner had replaced more wear items before I bought it.


Between when I bought it in February and now I have:

  • Learned how to actually drive manual transmission
  • Replaced the valve cover gasket
  • Replaced the rear trailing arm bushings
  • Replaced rear exhaust hangers
  • Replaced the windshield cowl cover
  • Replaced the rubber gaskets around the door handles
  • Replaced the rubber gasket around the rear windscreen
  • Reverted to stock shift knob and e-brake handle
  • Installed aftermarket shift and e-brake boot
I've learned there's never a point where you are "done." There's always something else to fix, replace, tweak. It's not a cheap hobby in terms of money or time, but I believe it was worth it for the satisfaction I feel fixing and driving it.

Tuesday, July 2, 2013

Query Google Admin SDK Reporting API For Google Apps Domain, Output to Graphite

The "Reports" section in the Google Apps (gapps) Admin Dashboard has some interesting stats, but only retains data for the past 6 months. If you want to have that data for longer, Google now recommends the Admin SDK's Reports API instead of the deprecated Google Apps Reporting API. I did have a python repo on Github to query the old Reporting API and send the output to graphite, but I have I have since rewritten it to use the new API. It can be found here.

As with the Groups Settings API, you can use these Google Drive SDK instructions to set up a service account and get sample code to access the Admin SDK Reports API for your gapps domain using OAuth 2.0. It seems like most of Google's APIs are moving towards requiring OAuth 2.0 so this two-legged OAuth 2.0 (2LO) method of access for domain-wide delegation seems like it will soon be required for scripts that need gapps domain-wide authority. I just wish Google did a better job telling gapps admins how to set that up.

Tuesday, April 30, 2013

Google Groups Settings API for GApps Admins Using OAuth 2.0

TL;DR: Need to administratively access Groups Settings API (or other Google APIs that aren't in gdata)? Follow these instructions and try this example script.

As a relatively new Google Apps for Education (GApps) admin it's somewhat surprising to me how confusing Google's API ecosystem is. For python you have gdata and apiclient. Gdata is fairly well documented and only requires a GApps admin username/password for administrative access. You can use gdata to do useful administrative things for your domain like enable create and populate Google groups. But if you want to configure moderation or other Google group settings, you'll need to use apiclient.

My primary complaint with apiclient (the groups settings API in particular) is that the majority of the documentation expects you to be writing for 3-legged OAuth flow where some user must authorize your script in a web browser before it can access Google resources as that user. If you want to use GApps admin credentials to administratively manage group settings without user interaction (from authorizing requests):
If your application has certain unusual authorization requirements, such as logging in at the same time as requesting data access (hybrid) or domain-wide delegation of authority (2LO), then you cannot currently use OAuth 2.0 tokens. In such cases, you must instead use OAuth 1.0 tokens and an API key. You can find your application's API key in the Google APIs Console, in the Simple API Access section of the API Access pane. 
The problem? The apiclient python package has no documentation or examples of this. In fact, it seems to have removed all OAuth 1.0 functionality.

The solution? After many Google searches I finally stumbled across the following Google Drive (!!!) doc page: https://developers.google.com/drive/delegation. I created a service account, granted it access to the Groups Settings API in the API console, enabled the groups settings scope in our GApps admin dashboard, and downloaded the .p12 file for the service account. The only stumbling block was "user_email" (that gets passed as "sub" to SignedJwtAssertionCredentials() should be a GApps domain admin account.

Hopefully this helps someone. And hopefully someone at Google cleans up their documentation.



Sunday, October 21, 2012

The Long Road to Logstash

I'm a Splunk addict. I use it almost every day, primarily for problem investigation. So when we started going over our daily indexing limit every day at the start of this semester, I knew I was in trouble. After being locked out from searches for the second time I started to look for alternatives. I found 3 serious candidates:
I found things I didn't like about all 3 of them but Logstash was by far the most flexible. After a lot of confusion and frustration I finally have it at a point where it is useful. What follows are the things I wish I had known before undertaking this project. It more or less assumes you have had some introduction to Logstash. The most recent presentation from PuppetConf 2012 is quite good: http://www.youtube.com/watch?v=RuUFnog29M4

Logstash

  • "Use the grep filter to determine if something exists. Use the grok filter only if you know it will be successful." - This is a big problem with complex log format, like dovecot. I spent many hours trying to write a grok filter that would match every possible Dovecot log line. It's futile. Use the grep filter to grep for something like 'interestingfield=".*"' and add a tag to indicate the field's presence, then grok for 'interestingfield=%{QUOTEDSTRING}' on just that tag. Grok failures are bad. They add the _grokparsefailure tag, and they seemed to contribute to the next problem I ran into, watchdog thread timeouts.
  • Logstash, by default, has a 2 second timeout to all filter operations. If it hits that timeout, it will kill itself. I was probably getting this because I was trying to run logstash, elasticsearch and kibana all on the same underpowered development VM, but I think part of the problem was I was doing lots of grok filters that were failing. The recommended solution to the "watchdog timeout" is to run logstash under some system that automatically restarts it. On RHEL6-based distros (and debian-based systems) you probably have upstart. On RHEL5-based distros you can use inittab. There's a good upstart entry in the new logstash cookbook (http://cookbook.logstash.net/recipes/using-upstart/). For inittab you should have a wrapper script that waits a few seconds before attempting to launch logstash just so the old logstash process can give up the TCP ports it was listening on.


Elasticsearch

If you want to be able to search your logs with Kibana, logstash needs to output to Elasticsearch. Unless you already have experience deploying Elasticsearch, you will probably spend more time learning about Elasticsearch than Logstash. I was an Elasticsearch newbie, so some of these may seem like common sense to people familiar with Elasticsearch.
  • There are 2 plugins I would consider essential for running Elasticsearch: head (management), and bigdesk (monitoring). Paramedic is also extremely useful for monitoring multiple cluster nodes. If you decide to use the "elasticsearch_river" output, you will need a river plugin. Unlike other plugins, you *must* restart Elasticsearch after installing a river plugin (at least with the rabbitmq one). A good list of plugins: http://www.elasticsearch.org/guide/reference/modules/plugins.html
  • Elasticsearch is a java program, so you will need to tune the JVM to your machine. bin/elasticsearch.in.sh is a good place to start. I'd personally recommend setting ES_HEAP_SIZE and telling Elasticsearch to lock that memory. You will almost definitely need to increase the limit of open files. In RHEL it seems to default to 1024. Most recommendations for Elasticsearch are around 300k or 600k.
  • There are 3 "outputs" in logstash for elasticsearch. The "elasticsearch" output will run embedded elasticsearch and can either store data in itself or connect to an Elasticsearch cluster and send the data to that. I couldn't get "elasticsearch_http" to work with bulk indexing. I'm currently using "elasticsearch_river", which sends events from logstash to an AMQP (rabbitmq) queue which Elasticsearch indexes data from.
  • You need to configure the query cache. I went with setting the query cache to "soft" which lets the JVM garbage collection expire the query cache. For more info, see: http://blog.sematext.com/2012/05/17/elasticsearch-cache-usage/
  • Compress/optimize old indices.
  • Use mappings and the options of the special _source field to limit the number of fields Elasticsearch saves and their type. (http://www.elasticsearch.org/guide/reference/mapping/ and http://www.elasticsearch.org/guide/reference/mapping/source-field.html)
  • Best practice: Make a template that will be applied to all logstash indexes. I started with http://untergeek.com/2012/09/20/using-templates-to-improve-elasticsearch-caching-with-logstash/   

Monday, May 7, 2012

Compiling posix-winsync Plugin for 389 Directory Server on Linux

https://github.com/cgrzemba/Posix-Winsync-Plugin-for-389-directory-server is a plugin for the 389 directory server that enables the syncing of posix attributes between 389 and Active Directory. It was written for Solaris and I was unable to produce a working linux binary of it using the supplied files. I was able to compile and link it by hand in RHEL 6.2. To do this you will need the binary and devel packages for 389 and nspr. The pkgconfig files (.pc) for both of those should help you if my gcc flags or ld flags don't work on your system. Once you have those in place, the following commands in the directory for the project should produce a shared object file that can be copied to where your 389 directory server plugins are (for me, /usr/lib64/dirsrv/plugins).
  • gcc -fPIC -I/usr/include/nspr4 -DUSE_OPENLDAP -I/usr/include/dirsrv -I /usr/include/ -c posix-winsync.c
  • gcc -fPIC -I/usr/include/nspr4 -DUSE_OPENLDAP -I/usr/include/dirsrv -I /usr/include/ -c posix-winsync-config.c
  • gcc -fPIC -I/usr/include/nspr4 -DUSE_OPENLDAP -I/usr/include/dirsrv -I /usr/include/ -c posix-group-func.c
  • ld -shared -L/usr/lib64 -lplds4 -lplc4 -lnspr4 -lpthread -ldl -L/usr/lib64/dirsrv -lslapd posix-group-func.o posix-winsync-config.o posix-winsync.o -o libposix-winsync.so
At this point you should run
ldd libposix-winsync.so
to make sure all the libraries required by that file can be found. I had to create a new entry in /etc/ld.so.conf.d to point to /usr/lib64/dirsrv and run ldconfig for it to find libslapd.so.0. I'm not sure how the other 389 plugins worked without setting that.

Then you need to import import the ldif file that comes with the plugin in to your 389 server. The way the plugin seems to work is when you set up a windows sync agreement it will also sync posix attributes. If it cannot find a required attribute, it will not sync that user/group.

It would be nice to create an RPM of this and extend the plugin so the list of attributes it syncs can be dynamic/optional, but for now it gets the job done.

Thursday, January 26, 2012

CAS-ifying Roundcube

Quick background/summary:
We have a CAS server for our campus, and we use Roundcube for our webmail system. Some time ago, I was asked to extend our Roundcube system to authenticate against CAS. Doing so would net us some nice benefits:
  • Graceful handling of locked accounts/expired passwords.
  • SSO to Webmail if already signed in to another SSO service.
  • Easy access for custom webapps wanting to access IMAP.
One of the first things I had to do was configure our IMAP server to accept CAS tickets as passwords. Our IMAP server (Dovecot) uses PAM for authentication, so all I had to do was follow the instructions on the Google translation of this page (French):
  • Create CAS services for HTTPS and IMAP URIs that ignore attributes and can proxy. On our older version of CAS I also had to restart the servlet container for the new services to be active.
  • Download, compile, install and configure pam_cas from https://sourcesup.cru.fr/frs/?group_id=213.
  • Add pam_cas to Dovecot's PAM stack with the IMAP service I created in CAS and the pam_cas.conf file I created in the options.
  • Test that Dovecot validates CAS tickets. You could use the CASTest utility that comes with pam_cas, but I chose to just open https://<CAS server>/login?service=<IMAP service> in a web browser and copy/paste the service ticket I got from CAS into a telnet session that was authenticating to our IMAP server. Be careful when testing! The default CAS Service Ticket life is 10 seconds, so you may have to move fast if you follow my testing method.
If you run in to problems in any of those steps, the best thing you can do is turn debug on in /etc/pam_cas.conf and send *.debug in your syslog config to a file that you can read. You should get good info from that as to why pam_cas is failing.

Once I had IMAP authenticating users with CAS tickets, Alex's plugin got me most of the way, but I found that I had to make 2 modifications to Roundcube's index.php file to get the plugin to work properly. I also wanted a few additional features that his plugin lacked:
  • The option of not using CAS for authentication. Alex's plugin forces it for authentication to Roundcube.
  • SMTP authentication using a CAS proxy ticket.
I was able to remove the need to modify Roundcube's source and add those features to my fork of his plugin. It is currently in production in our 8-node Roundcube cluster. There were a few "gotchas" along the way that I don't want to forget, and other people may find useful when working with my plugin or CAS' proxy feature in general:
  • If you run multiple Roundcube servers behind a load balancer, you need to be sure of 2 things:
    • You have the same version of phpCAS on all of your Roundcube servers.
    • You are using shared storage for PGTIOUs coming back from the CAS server. This page talks about this process but basically part of proxy CAS authentication is the CAS server will make an HTTP call to a URL that is supposed to store a necessary piece of information. Unless you're absolutely sure the CAS server will be sent to the same Roundcube server that the user is currently hitting, you need to make sure that all Roundcube servers can share the directory where PGTIOUs are saved.
  • If you are using CAS' proxy method for authenticating to IMAP, you all but need an IMAP proxy. Without it, you will need to generate and validate a new ticket from CAS (multiple HTTP trips) for each page load. With an IMAP proxy, the user's first IMAP ticket is validated by pam_cas by your actual IMAP server and saved in your IMAP proxy. Roundcube will continue to send the same ticket and the IMAP proxy will continue to believe the user is logged in, even though that ticket has been used and cannot be validated again by the IMAP server.

Wednesday, January 25, 2012

Increasing VMWare SCSI Timeout on RHEL6 and ESX5

VMWare recommends increasing the disk timeouts for linux guests to 180 seconds to handle things like a storage path failover. There are a few ways to accomplish this. If you are a Netapp customer you could run their script, or you could follow the instructions in the VMWare article.

Unfortunately, both of these solutions were written for ESX4 and RHEL5. If you are on ESX5 with a RHEL6 guest, do the following to increase your timeout for VMWare SCSI disks to 180 seconds:

  1. Put the following in a file in /etc/udev/rules.d. I chose /etc/udev/rules.d/99-vmware-scsi-timeout.rules.
    SUBSYSTEM=="block", ACTION=="add", BUS=="scsi", SYSFS{vendor}=="VMware", SYSFS{model}=="Virtual disk",   RUN+="/bin/sh -c 'echo 180 >/sys$DEVPATH/device/timeout'"
  2. Run udevadm test /block/sda and make sure you see a line that indicates the echo 180 would run. If it doesn't then your SYSFS values are somehow different than mine. The output you just got should help you figure out the correct values. Look for the entries that start with ID_.
  3. Once you have udevadm telling you it would run the echo 180 command you can restart your machine or restart udev during your maintenance window, if this is a production VM. If you don't want to reboot the machine, you can restart udev with:
    1. udevadm control --reload-rules
    2. start_udev
  4. Verify it worked by looking at the timeout file in the /sys tree for your devname. For me this was accomplished with: cat /sys/block/sda/device/timeout