How to Get Credentials on Eucalyptus 4.2

10 Mar 2016

If you run euca_conf --get-credentials on eucalyptus 4.2 you will see the following warning:

warning: euca_conf is deprecated; use ``eval `clcadmin-assume-system-credentials`''
or ``euare-useraddkey''`

There are numerous reasons for that command’s deprecation, but what causes confusion is the fact that it has two replacements. Setting up a new cloud now involves more than just one set of credentials, and if you’re used to having fully-functional credentials immediately this is likely to trip you up.

Why Change?

One of the most common complaints about euca_conf is that it tries to be everything to everybody. It combines multiple types of functionality that need to run in different places, adding excess dependencies and requiring one to log into systems that one normally shouldn’t have to. Eucalyptus 4.2 introduces new administration tools that break euca_conf‘s functionality down into three groups with more specific purposes:

  1. Whole-cloud administration tools
  2. Cloud controller (CLC) support scripts
  3. Cluster controller (CC) support scripts

Cloud controller and cluster controller support scripts can run only on those specific systems, and thus are only installed alongside them. The rest of the administration tools are web service clients, similar to euca2ools, that can run from anywhere. All they need are access keys.

But where do those access keys come from?

Out with the Old

In the old regime, access keys and other credentials come in the form of a zip file containing a bunch of certificates as well as eucarc, a shell script that sets a bunch of environment variables that include service URLs and the access keys themselves. The first zip file it creates is missing several service URLs because those services have yet to be set up, and it doesn’t use DNS either because that has yet to be set up as well.

Once DNS and all of the services are ready, we then have the cloud generate a new zip file. Everything seems fine until something changes for whatever reason and we need to obtain a third one. Since we can only have two certificates at a time, though, this third zip file will not include one. This causes countless problems for automation that relies on them, including eucalyptus’s own QA scripts.

That said, the zip file still has some particularly useful properties:

  • It’s a single file for the administrator to e-mail to new users
  • It contains both access keys and service URLs
  • It (usually) contains all of the certificates needed to bundle images

A euca2ools.ini file also has the first two of those properties, while also managing to be more flexible. Any euca2ools commands that can create access keys, such as euare-useraddkey and euare-usercreate, can generate euca2ools.ini files automatically. That leaves just certificates, which we dealt with by making them all optional or possible to obtain automatically.

In with the New

In isolation, euca2ools commands alone have a chicken-and-egg problem: they require access keys to run, but a new cloud doesn’t have any access keys. We break this loop by splitting eucalyptus installation into two phases, each with different credentials.

Setup Credentials

A cloud controller support script, clcadmin-assume-system-credentials, provides temporary setup credentials. This script works similarly to euare-assumerole, but it is much more limited and it only works on a cloud controller. Setup credentials cannot be used for normal system operation; they provide access only to service registration, service configuration, and IAM services — the minimum necessary to get up and running with euca2ools.

# eval `clcadmin-assume-system-credentials`
# euserv-register-service -t user-api -h ufs-1
# euctl

Admin Credentials

Once DNS and an IAM service are set up, you can use euca2ools to create long-lived admin credentials that let you access the cloud’s full functionality. It is these credentials that are the replacements for the zip file. Once you create them, you are unlikely to ever need setup credentials again.

# euare-usercreate -wld gholms > ~gholms/.euca/mycloud.ini

Here is an explanation of the various parts of that command:

  • gholms: Create a user named gholms
  • -w: Write out a euca2ools.ini file
  • -l: In that file, make that user the default for this cloud
  • -d Use the domain as the cloud’s DNS domain

Normally, when this command writes a configuration file it will pull the DNS domain from the IAM service’s URL, but since this is the very first user we have to supply it by hand because it has not yet been set.

What now?

Once you have a set of admin credentials you can use this for day-to-day cloud administration the same way you would with a classic eucarc file.

% export
% euare-accountcreate -wl alice > alice.ini
% mail -s "Try out this shiny, new cloud" -a alice.ini ...

Useful Git Commands: a Shortcut for GitHub Pull Requests

3 Nov 2015

Normally, when someone asks me to merge something in git I need to add his or her repository using git remote add, fetch the branch I need, and then merge it. When someone submits a pull request to a project hosted on GitHub, however, GitHub additionally publishes it as something I can fetch from my own repository:

% git ls-remote origin | grep pull/6
03d7fb7af91a74bb7658a0742fa68bfeb5d50a3f	refs/pull/6/head
f8ebbc62555143019947f1255b064cda38fd239f	refs/pull/6/merge
8a42021dbabcfc222b4d47c5ada47e344154d934	refs/pull/60/head
f4d8052000601e59e4e7d4dec4aa4094df4e39a0	refs/pull/61/head
8a519986f5b59721692ec75608edf0f404f88e87	refs/pull/62/head
9c03e723e8b50ca56a1257659d686a68a69e6e40	refs/pull/62/merge
f3a983c6fc3ff236c2bc678cfec3885da609f79a	refs/pull/63/head
e7953c21a77fb37fb7158dd87fc0c156dd8f97ae	refs/pull/63/merge

With this I can use one line of configuration to create a convenient shortcut that lets me immediately check out any pull request:

% git config --add remote.origin.fetch "+refs/pull/*/head:refs/remotes/origin/pull/*"
% git checkout pull/63
Branch pull/63 set up to track remote branch pull/63 from upstream.
Switched to a new branch 'pull/63'

Useful Git Commands: URL Rewriting

29 Aug 2014

People have to use SSH or HTTPS to push to GitHub, but when fetching one can use git’s own network protocol because it is generally faster. You can make a specific repository on your machine use SSH only for pushing by cloning it with the faster git:// URL and running something like this:

% git config remote.upstream.pushurl

That works nicely, but you have to do it once for every single repository you want to interact with. That quickly becomes annoying. Thankfully, you can leverage git’s URL-rewriting mechanism to make this easier:

% git config --global url.git:// github:
% git config --global github:

This adds two new rules to your git configuration:

  1. If a URL starts with github: then replace that with git://
  2. If a URL starts with github: and you are pushing then replace that with

After you do that you can simply use URLs like github:gholms/boto when cloning. They will get rewritten to when pushing, and git:// the rest of the time, speeding things up without creating additional work in the future.

This should work if you prefer HTTPS for pushing to GitHub, or if you use other servers, too. Just tweak the commands.

Useful Git Commands: Summarizing Lots of Log Entries

2 Apr 2014

When looking for a summary of a git repository’s history, the output of git log isn’t always as informative as one might like. It displays every commit in chronological order, which effectively hides the changes that merges bring in. It is also quite verbose, showing complete log messages, author info, commit hashes, and so on, drowning us with so much info that only a few commits will fit on the screen at once. After supplying the command with the right cocktail of options, though, its output becomes a significantly better summary:

Output of ``git graph''

The output above came from a command that is long enough that I made an alias, for it, git graph, in my ~/.gitconfig file:

    graph = log --graph --abbrev-commit --date=relative --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(blue)<%an>%Creset'

Don’t forget that git log accepts a list of things to show logs for as well, so if you want to look at the logs for branch-1 and branch-2 you can simply use git graph branch-1 branch-2 to make them both show up in the graph.

Making FileVault Use a Disk Password

31 Mar 2014

To unlock a disk that is encrypted with OS X’s FileVault feature one needs to type in the password that belongs to any user on the machine who is allowed to unlock the disk. The system then boots and helpfully logs you in as that user. In general that is probably a convenient little feature, but for me it just makes things awkward — I want to use different passwords for unlocking the disk and logging into my user account. To make that work I have to create a second account dedicated to unlocking the disk, get logged into that one when the system boots, then immediately log back out so I can log in as the user I actually want to use.

Or do I?

The system that powers FileVault, Core Storage, combines full disk encryption and some logical volume management features in a manner similar to LVM and LUKS on Linux. As a dedicated user of those features on my Linux-based machines, I jumped at the chance to read more about OS X’s version only to discover next to no official documentation whatsoever. On the bright side, after searching some more and then playing around with it for a while I finally figured out how to make it do everything I wanted it to do, including using a disk password. You just have to boot into recovery (or, in my case, the OS X 10.9 install DVD that I threw together) and use the command line.

Create a partition

Since this is a laptop I want to put everything on one huge partition. The utility that handles nearly everything related to disk management on OS X, diskutil, requires us to choose a type of filesystem for every partition we tell it to create, so for now we’ll just tell it to mark the partition as HFS+ and not bother to format it.

# diskutil partitiondisk disk0 1 gpt jhfs+ %noformat% 100%
Started partitioning on disk0
Unmounting disk
Creating the partition map
Waiting for the disks to reappear
Finished partitioning on disk0
   #:                       TYPE NAME    SIZE       IDENTIFIER
   0:      GUID_partition_scheme        *256.1 GB   disk0
   1:                        EFI EFI     209.7 MB   disk0s1
   2:                  Apple_HFS         255.7 GB   disk0s2

The system actually created two partitions for us. The first is a small EFI system partition, which helps the system start, and the second is the big partition for our data that spans the rest of the disk.

Or does it?

# gpt show disk0
      start        end  index  contents
          0          1         PMBR
          1          1         Pri GPT header
          2         32         Pri GPT header
         34          6
         40     409600      1  GPT part - C12A7328-F81F-11D2-BA4B-00A0C93EC93B
     409640  499446368      2  GPT part - 48465300-0000-11AA-AA11-00306543ECAC
  499856008     262151
  500118159         32         Sec GPT table
  500118191          1         Sec GPT header

It seems to have left a little empty space in between the last partition and the boilerplate bits at the end of the disk. More on that later.

Create a volume group

The next step is quite familiar to LVM users. Before we can add encryption we need to create a logical volume group that combines one or more partitions (physical volumes) into a single virtual “disk” that we can slice up.

Why more than one disk? That’s how Fusion Drive works.

# diskutil cs create vg_sodium disk0s2
Started CoreStorage operation
Touching partition type on disk0s2
Adding disk0s2 to Logical Volume Group
Creating Core Storage Logical Volume Group
Switching disk0s2 to Core Storage
Waiting for Logical Volume Group to appear
Discovered new Logical Volume Group "A642DBC3-644C-4C23-8337-ADBCDD9C85F2"
Core Storage LVG UUID: A642DBC3-644C-4C23-8337-ADBCDD9C85F2
Finished CoreStorage operation

# diskutil cs list
CoreStorage logical volume groups (1 found)
+-- Logical Volume Group A642DBC3-644C-4C23-8337-ADBCDD9C85F2
    Name:         vg_sodium
    Status:       Online
    Size:         255716540416 B (255.7 GB)
    Free Space:   255380987904 B (255.4 GB)
    +-< Physical Volume A39FA7E9-F52F-4FFA-9A70-F07304111115
        Index:    0
        Disk:     disk0s2
        Status:   Online
        Size:     255716540416 B (255.7 GB)

# diskutil list disk0
   #:                       TYPE NAME       SIZE       IDENTIFIER
   0:      GUID_partition_scheme           *256.1 GB   disk0
   1:                        EFI EFI        209.7 MB   disk0s1
   2:          Apple_CoreStorage            255.7 GB   disk0s2
   3:                 Apple_Boot Boot OS X  134.2 MB   disk0s3

This changed the type of the second partition and constructed a volume group out of it, but it also created a third partition on which it can put whatever minimal data are needed to allow the system to boot from a Core Storage volume. With a quick look at the partition table we can see that it carved this out of the space it reserved at the end of the disk earlier:

# gpt show disk0
      start        end  index  contents
          0          1         PMBR
          1          1         Pri GPT header
          2         32         Pri GPT header
         34          6
         40     409600      1  GPT part - C12A7328-F81F-11D2-BA4B-00A0C93EC93B
     409640  499446368      2  GPT part - 48465300-0000-11AA-AA11-00306543ECAC
  499856008     262144      3  GPT part - 426F6F74-0000-11AA-AA11-00306543ECAC
  500118152          7
  500118159         32         Sec GPT table
  500118191          1         Sec GPT header

Create a logical volume

LVM has three layers: physical volumes, volume groups, and logical volumes. Since Core Storage includes encryption it adds a fourth layer, the logical volume family, between the volume group and logical volume levels. When you boot a FileVault-enabled system it’s the volume family that your password unlocks. The final step before we can install OS X is to create an encrypted volume family and the logical volume that we will actually install onto.

# diskutil cs createvolume vg_sodium jhfs+ 'Macintosh HD' 100% -passphrase
Passphrase for new volume:
Confirm new passphrase:
Started CoreStorage operation
Waiting for Logical Volume to appear
Formatting file system for Logical Volume
Initialized /dev/rdisk13 as a 238 GB case-insensitive HFS Plus volume with a 24576k journal
Mounting disk
Core Storage LV UUID: 46A6CA83-9CDC-4978-ADF2-E4DC3F203DBD
Core Storage disk: disk13
Finished CoreStorage operation

# diskutil cs list
CoreStorage logical volume groups (1 found)
+-- Logical Volume Group A642DBC3-644C-4C23-8337-ADBCDD9C85F2
    Name:         vg_sodium
    Status:       Online
    Size:         255716540416 B (255.7 GB)
    Free Space:   0 B (0 B)
    +-< Physical Volume A39FA7E9-F52F-4FFA-9A70-F07304111115
    |   ----------------------------------------------------
    |   Index:    0
    |   Disk:     disk0s2
    |   Status:   Online
    |   Size:     255716540416 B (255.7 GB)
    +-> Logical Volume Family AC7B483C-0524-4ACF-8083-9EFD963F81A5
        Encryption Status:       Unlocked
        Encryption Type:         AES-XTS
        Conversion Status:       Complete
        Conversion Direction:    -none-
        Has Encrypted Extents:   Yes
        Fully Secure:            Yes
        Passphrase Required:     Yes
        +-> Logical Volume 46A6CA83-9CDC-4978-ADF2-E4DC3F203DBD
            Disk:                  disk13
            Status:                Online
            Size (Total):          255380987904 B (255.4 GB)
            Conversion Progress:   -none-
            Revertable:            No
            LV Name:               Macintosh HD
            Volume Name:           Macintosh HD
            Content Hint:          Apple_HFS

Now we have a disk called “Macintosh HD” that is encrypted with its own non-user-specific password, formatted, and ready to go. Just switch back to the OS X installer and let it run.

Create a recovery key

After the installer finishes and the system reboots we see the familiar FileVault login screen with just one option: “Disk Password.” Success! Now we just enter that, run through the usual first-time boot stuff, and finally open up a terminal to create a recovery key:

$ sudo fdesetup changerecovery -personal
Enter a password for '/':
New recovery key = 'XPO6-E4OL-XQG6-TGV9-GFOZ-GB8M'

Add more users

Since I generally don’t log in with an administrative account I created a second, regular user, but then the next time I rebooted I discovered a problem: the FileVault password screen asked for the disk password or the new user’s password. Thankfully, removing that user from the list of choices is simple:

$ sudo fdesetup remove -user gholms

Open questions

Now that I have gone through all this my machine finally seems to be working exactly the way I want it to, but a few unanswered questions still remain in the back of my mind:

  • How am I supposed to do this on a newer Mac that ships with neither an install DVD nor an upgrade application that I can turn into one?
  • How do I make OS X stop automatically adding new users to FileVault’s password list?
  • Where is the official Core Storage documentation?

Do know the answers to any of these? Leave a comment!

A Useful rtadvd Configuration

20 Jan 2014

When most IPv6-capable computers join a network they attempt to automatically find a router on the network so they can figure out what addresses to use, how to set up routing, and so forth. On BSD systems like my router, the rtadvd(8) program manages the router’s side of this exchange. While rtadvd is rather flexible, its configuration file is frustratingly terse and its documentation assumes the reader has a fair amount of knowledge already.

For IPv4, my network uses DHCP to hand out addresses and DNS information. When rolling out IPv6, I wanted to set things up similarly, but without managing addresses centrally with DHCP since machines can configure themselves correctly without one. Configuration like this is the sort of thing that I tend to forget and have to re-learn periodically, so for reference, the rtadvd.conf file I used for that setup looked something like this:


This makes machines configure themselves for the network 2001:db8:1221::/64, DNS domain, and three DNS servers: a local one at 2001:db8:1221::1, and both of Google’s public servers.

The source of most of my confusion was figuring out the raflags option. raflags is a bit mask with two flags: M, which means a DHCP server manages addresses, and O, which means that non-address-related information (in this case, DNS information) is available that way even if addresses are not. This network setup requires the O flag to be set and the M flag to be unset, which means raflags has to be 64.

Now that I’ve rolled this out I get all the niceness of auto-configuration without the need for a central DHCP server to keep track of addresses. Well, for IPv6, at least.

Mr. TV

16 Dec 2013

I previously wrote about the big, Raspberry Pi-powered TV set at Eucalyptus HQ that displays the #eucalyptus-devel IRC channel so developers can always see what is going on and jump in if they need to. That setup has worked quite well for some time now, but I recently came up with a way to make it even better:

Mr. TV

Googly eyes have yet to fail me at improving a machine’s appearance.

Making a Less-Limited USB Stick

9 Dec 2013

The FAT32 filesystem is the closest thing we have to a universal standard for passing data around, but with the capacity of modern USB flash drives its 4 GB file size limitation has become problematic. exFAT is a popular contender for dealing with that, but the patent issues that surround it make true portability a pipe dream at best.

Enter UDF. As the filesystem of choice for DVDs and Blu-Ray disks, UDF support is ubiquitous. Appropriately-formatted disks are readable by operating systems dating back to the early 2000s. All that remains is figuring out how to format it. In general, there seem to be three important things to keep in mind:

  1. Remove all traces of previous filesystems. Different operating systems use different methods to detect what filesystems a disk contains, so ensure maximum reliability by eliminating potential sources of confusion.
  2. Format the entire disk, not just a partition. OS X seems to only look for UDF when the filesystem takes up the whole disk, as it does on DVDs, so overwrite the partition table before formatting the disk.
  3. Use the correct UDF version. UDF has a number of versions that add different features, but as you might expect, newer versions require newer operating systems. Windows XP will read up to version 2.01 out of the box, though some Blu-Ray drive manufacturers have released drivers for newer versions.

I formatted my most recently-purchased USB stick on Fedora. First off, I blew away the partition table and all traces of the FAT32 filesystem it came with:

# dd if=/dev/zero of=/dev/sdb bs=1M count=1
# wipefs -a /dev/sdb

Fedora uses the mkudffs command for creating UDF filesystems, which is part of the udftools package:

# yum install udftools
# mkudffs --revision=0x0201 --media-type=hd /dev/sdb

That’s it! Now the disk seems to be usable on Fedora, OS X, and Windows, which covers just about all of my computers. I will test OpenBSD one of these days.

There are undoubtedly ways to do this on other operating systems. Feel free to comment with instructions for your favorite operating system if you know them.

Every DNS Server Should Support Aliases

22 Jul 2013

Amazon’s Route53 DNS service, along with several content delivery networks and other DNS providers let one create an “alias” pseudo-record that causes the server to respond to requests for one name with results for another name. While the ways current implementations of this function vary a bit, the biggest difference between all of them and a CNAME is that while a CNAME gets applied to every query regardless of the type of record something is looking for, an alias is specific to just one type of record.

While this sounds like a trivial difference, the benefits are surprisingly enormous. The most obvious effect is that it lets you point a bare domain name (e.g. at something else (e.g. The reason you can’t normally do this is because the CNAME record you would normally use to do this would conflict with the SOA record at the top of your domain, but since the alias you would use for this only applies to A address records, this is no longer a problem.

Another property aliases have is that they don’t actually go over the wire. While a CNAME record returns to the machine looking up a DNS name, causing it to restart its search with a different name, the answer for an alias comes right out of the DNS server’s own database. This means that aliases can only be used for records for which the server is authoritative or at least has some means of reliably learning the answer it should return, but that’s good enough for a great deal of use cases, notably including those of most content delivery networks. The fact that servers look up what an alias points to before they send anything over the wire means that they can include this functionality without violating standards — no one else needs to change their servers or their clients to support it. If DNS standards evolve to support it in the future, this makes transitioning even easier as that change rolls out.

In short, aliases would solve one of the most commonly-encountered shortcomings of DNS, namely its inability to use a CNAME to point a bare domain at its www equivalent. Given that there are multiple proprietary systems out there that do this already, it’s about time we standardized on an approach.

Running a Text-based Kiosk with Systemd

9 Jul 2013

Eucalyptus HQ has a big TV on the wall that displays the #eucalyptus-devel IRC channel so developers can always see what is going on and jump in if they need to. Until recently, a laptop drove that display, but that seemed like overkill to me, so I went to employ my Raspberry Pi running the Raspberry Pi Fedora Remix to do that instead. Since the IRC program it’s using, irssi, is text-based I don’t need to use any of the Pi’s precious little memory to run anything graphical, so I just needed to figure out how to make systemd spawn irssi instead of a login prompt on tty1.

I would normally do this by copying /lib/systemd/system/getty@.service to /etc/systemd/system/getty@tty1.service and then editing that, but F18’s version of systemd let me do this in an even simpler manner. By creating a directory with the same name as that file, plus .d, I can add a config file to that directory that overrides only the parts of the original unit file that I need to change:


Now I can just plug the system in and have it automatically up and running irssi in less than a minute.

Unexpected lessons

I didn’t expect to have to run nm-online here because is supposed to wait for a service that runs that itself, but for some reason systemd didn’t order things that way and irssi came up before the network connection did. Running that command as part of this unit worked around that problem.

Use the consoleblank=0 parameter to prevent Linux from blanking the screen after the usual ten minutes of inactivity.

I’m using the TV’s USB “service” port to power the raspberry pi. That usually works just fine, but when the TV turns off it cuts the power to that port as well, abruptly shutting the raspberry pi off. I don’t have any data loss in particular to worry about, but turning the system back on causes some annoyance: when the TV turns on the raspberry pi also powers on and attempts to detect what kind of screen it is plugged into. At that point the TV hasn’t figured out what it wants to display yet, so the detection fails and I’m left with a blank screen until I reboot the computer.