A change in philosophy: declarative programming

The developer is very much in the driver’s seat in version 1 of euca2ools. To use a car analogy, the developer directly controls the code’s direction, speed, and gearbox manually. Version 2 adds a cruise control by centralizing a lot of boilerplate code in the form of boto’s roboto module. Version 3 opts to let the developer give the requestbuilder framework a destination, step aside completely, and let it do the driving for the boring parts of the trip.

Requestbuilder offers a set of base classes and a domain-specific language based on python’s standard argparse library that allows the developer to say exactly how something should look at the command line in addition to how it should look when given to the server all in the same place.

What makes this so powerful is that it lets anybody with a service’s documentation and knowledge of how to use argparse write a command line tool quickly and painlessly. For instance, it took me around a day to write highly-customized command line tools for every operation Amazon’s Elastic Load Balancing service supports. Here’s the code from one of them:

class CreateLBCookieStickinessPolicy(ELBRequest):
    DESCRIPTION = ('Create a new stickiness policy for a load balancer, '
                   'whereby the load balancer automatically generates cookies '
                   'that it uses to route requests from each user to the same '
                   'back end instance. This type of policy can only be '
                   'associated with HTTP or HTTPS listeners.')
    ARGS = [Arg('LoadBalancerName', metavar='ELB',
                help='name of the load balancer to modify (required)'),
            Arg('-e', '--expiration-period', dest='CookieExpirationPeriod',
                metavar='SECONDS', type=int, required=True,
                help='''time period after which cookies should be considered
                stale (default: user's session length) (required)'''),
            Arg('-p', '--policy-name', dest='PolicyName', metavar='POLICY',
                required=True, help='name of the new policy (required)')]

The framework hands everything inside each Arg in this code to argparse to gather input from the command line and then send the results directly to the web server using whatever name argparse gives the input it gets. For instance, whatever a user supplies using the -e option ends up getting sent to the server as a CookieExpirationPeriod parameter. With a small amount of practice it becomes quite easy to write a bunch of commands this way very quickly.

One request, one command

Euca2ools are built around a “one request, one command” tenet. This means that, in general, there is a dedicated command for each thing a web service can do. This philosophy naturally lends itself to the tight coupling between command line options and what gets sent to the server discussed earlier, but it also lends itself to reversing the usual relationship between web services and web service requests. Whereas one typically writes an object that represents the service and uses methods on it to send requests, in euca2ools it is the commands, and thus the requests, which are the first-class citizens. Each command that represents a request instead points to a service, rather than the other way around.

The way this works in practice is by defining a base class for each service and a base class that all methods which use that service share:

class CloudWatch(requestbuilder.service.BaseService):
    NAME = 'monitoring'
    DESCRIPTION = 'Instance monitoring service'
    API_VERSION = '2010-08-01'
    AUTH_CLASS = requestbuilder.auth.QuerySigV2Auth
    URL_ENVVAR = 'AWS_CLOUDWATCH_URL'

    ARGS = [MutuallyExclusiveArgList(
                Arg('--region', dest='userregion', metavar='USER@REGION',
                    route_to=SERVICE, help='''name of the region and/or user
                    in config files to use to connect to the service'''),
                Arg('-U', '--url', metavar='URL', route_to=SERVICE,
                    help='instance monitoring service endpoint URL'))]

class CloudWatchRequest(requestbuilder.request.AWSQueryRequest):
    SERVICE_CLASS = CloudWatch

Services can supply their own command line options in the same way as requests. After it gathers options from the command line, requestbuilder uses route_to to choose where to send it. This also provides a convenient way to tell the framework not to send an option to the server at all when a command needs to process it specially: just use route_to=None.

Convention over configuration

The oft-quoted programming paradigm for frameworks is just as true for euca2ools 3 as it is elsewhere. Want to make a command print something? Just write a print_result method. The result from the server gets passed in as a dictionary.

class TerminateInstances(EucalyptusRequest):
    DESCRIPTION = 'Terminate one or more instances'
    ARGS = [Arg('InstanceId', metavar='INSTANCE', nargs='+',
                help='ID(s) of the instance(s) to terminate')]
    LIST_TAGS = ['instancesSet']

    def print_result(self, result):
        for instance in result.get('instancesSet', []):
            print self.tabify(('INSTANCE', instance.get('instanceId'),
                               instance.get('previousState', {}).get('name'),
                               instance.get('currentState', {}).get('name')))

Want to make a request do fancier preparations than argparse can do on its own? Just write a preprocess method that takes things from self.args and adds things to self.params to be sent to the server.

class DescribeSecurityGroups(EucalyptusRequest):
    DESCRIPTION = ('Show information about security groups\n\nNote that '
                   'filters are matched on literal strings only, so '
                   '"--filter ip-permission.from-port=22" will *not* match a '
                   'group with a port range of 20 to 30.')
    ARGS = [Arg('group', metavar='GROUP', nargs='*', route_to=None,
                default=[], help='limit results to specific security groups')]
    ...
    def preprocess(self):
        for group in self.args['group']:
            if group.startswith('sg-'):
                self.params.setdefault('GroupId', [])
                self.params['GroupId'].append(group)
            else:
                self.params.setdefault('GroupName', [])
                self.params['GroupName'].append(group)

There are also a few other methods one can plug in, such as postprocess, and, for especially early-running code, configure. Expect documentation for requestbuilder that covers this in detail in the future.

Scratching the surface

The examples above cover only a fraction of what is possible with euca2ools 3′s new infrastructure. While you can look forward to some more advanced uses of it in later blog posts, you can also take a look at the current euca2ools code in development to see some of the interesting things one can do with it. Today’s pre-release of that code carries with it commands for all three of AWS’s “triangle” services: Auto Scaling, CloudWatch, and Elastic Load Balancing. Continuing what seems to have become a euca2ools tradition, just look for the commands that start with euscale (pronounced “you scale”) euwatch (“you watch”), and eulb (“you’ll be”).

Packages for Fedora and RHEL 6 are available here. If you’re using another OS or want to build the code yourself you can simply clone euca2ools’s git repository‘s requestbuilder branch. Requestbuilder itself is available on PyPI and GitHub. As always, I encourage you to test this code against AWS and Eucalyptus 3.3 and let me know what you think on the euca-users mailing list. If you encounter bugs, please file them in the project’s bug tracker.

A configuration file

Yes, you read that right: a configuration file. Both euca2ools and the command line tools provided by AWS themselves have astonishingly limited support for configuration, forcing people to resort to writing a separate shell script for each combination of users and clouds one might possibly want to access and then use them in place of one.

Your cries of anguish have been heard, so now we have this:

[user gholms]
key-id = AKIA93F29V0AEXAMPLE
secret-key = vcasd93cm1458un4vj84039vda78mDEXAMPLE

[user ecc-admin]
key-id = EVDB93F29V0AEXAMPLE
secret-key = 38fva93cm1458un4vj84039vda78mDEXAMPLE

[region us-east-1]
ec2-url = https://ec2.amazonaws.com/
iam-url = https://iam.amazonaws.com/
s3-url  = https://s3.amazonaws.com/
user = gholms

[region ecc]
ec2-url = https://communitycloud.eucalyptus.com:8773/services/Eucalyptus/
iam-url = https://communitycloud.eucalyptus.com:8773/services/Euare/
s3-url  = https://communitycloud.eucalyptus.com:8773/services/Walrus/
user = ecc-admin

[global]
default-region = us-east-1

A file like this, combined with the --region option that all tools share, mean you can mix and match users and clouds to you heart’s content. Just throw a file like this inside of ~/.euca, end it with .ini, and away you go! You can add as many files to ~/.euca as you want — they all get combined together.

Friendly error feedback

Another common complaint that people had with euca2ools 2 was its behavior in the face of input that didn’t match what it expected. Some of the worst offenders had error messages ranging from confusing to irrelevant to nonexistent. Euca2ools 3 overhauls the code that does this, replacing it with standard python tools and friendlier code that makes its behavior in the face of errors much better.

Here’s how it behaves in the face of the most common case of this:

% euca-describe-availability-zones
error: missing access key ID; please supply one with -I

Also included is special treatment for “pick one from multiple alternatives” options:

% euare-useraddcert
usage: euare-useraddcert (-c CERT | -f FILE) [-u USER]
                         [--as-account ACCOUNT] [--region REGION | -U URL]
                         [-I KEY_ID] [-S KEY]
euare-useraddcert: error: one of the arguments -c/--certificate-body -f/--certificate file is required

A lot of attention to detail went into dealing with some of the most common mistakes people make:

% euca-register -n myimage -b /dev/sda1=snap-12345678:false
euca-register: error: argument -b/--block-device-mapping: second element of EBS block device mapping "/dev/sda1=snap-00000000:false" must be an integer
% euca-authorize mygroup -p 8773:8777
euca-authorize: error: argument -p/--port-range: multi-port range must be separated by "-", not ":"

Tagging and filtering support

Euca2ools 3 at last offers full support for EC2′s massive sets of resource tags and filters:

% euca-describe-instances -h
usage: euca-describe-instances [-h] [--show-empty-fields]
...
  --filter NAME=VALUE   restrict results to those that meet criteria
...

allowed filter names:
  architecture          CPU architecture
  availability-zone
  block-device-mapping.attach-time
                        volume attachment time
  block-device-mapping.delete-on-termination
                        whether a volume is deleted upon instance
                        termination
  block-device-mapping.device-name
                        volume device name (e.g.  /dev/sdf)
  block-device-mapping.status
                        volume status
  block-device-mapping.volume-id
                        volume ID
  client-token          idempotency token provided at instance run
                        time
  dns-name              public DNS name
  group-id              security group membership
  hypervisor            hypervisor type
  image-id              machine image ID
  instance-id
  instance-lifecycle    whether this is a spot instance
  instance-state-code   numeric code identifying instance state
  instance-state-name   instance state
  instance-type
  ip-address            public IP address
  kernel-id             kernel image ID
  key-name              key pair name provided at instance launch time
  launch-index          launch index within a reservation
  launch-time           instance launch time
  monitoring-state      whether monitoring is enabled
  owner-id              instance owner's account ID
  placement-group-name
  platform              whether this is a Windows instance
  private-dns-name
  private-ip-address
  product-code
  ramdisk-id            ramdisk image ID
  reason                reason for the more recent state change
  requestor-id          ID of the entity that launched an instance
  reservation-id
  root-device-name      root device name (e.g.  /dev/sda1)
  root-device-type      root device type (ebs or instance-store)
  spot-instance-request-id
  state-reason-code     reason code for the most recent state change
  state-reason-message  
                        message for the most recent state change
  subnet-id             ID of the VPC subnet the instance is in
  tag-key               name of any tag assigned to the instance
  tag-value             value of any tag assigned to the instance
  tag:KEY               specific tag key/value combination
  virtualization-type
  vpc-id                ID of the VPC the instance is in

The new foundation this code is based upon makes it incredibly simple to extend support for these features as things change in the future.

What else?

Some other minor, but nonetheless noteworthy, changes include:

• euca-* tools gained a --show-empty-fields option that tweaks their output to make it friendlier for running through the column command.
• All tools that access web services use the same options (-I and -S) for access keys.
• euare-* tools’ --delegate option for cloud administrators is now --as-account.
• Multiple --filter options are handled correctly.
• Machine image device mappings are now handled correctly.

A few tools have yet to be ported to the new framework, but will be in the near future. eustore-installimage is known to be broken. The bundle management tools should work correctly, though their testing to date has been minimal. Finally, do not install them on a system that runs a Eucalyptus node controller.

Isn’t aws-cli the future? Why continue developing euca2ools?

Aws-cli is a great project. Both it and euca2ools tie what the server sees very closely to what the user sees under the hood, but the euca2ools suite does so in a way that makes it trivial to customize tools to do more complicated things behind the hood or to make them easier to use. For instance, consider changing a security group’s permissions in EC2 with aws-cli:

% aws ec2 authorize-security-group-ingress --group-name MySecurityGroup --ip-permissions '{"from_port":22,"to_port":22,"ip_protocol":"tcp","ip_ranges":["0.0.0.0/0"]}'

The exact format we need to use to supply the info the tool needs requires relatively detailed knowledge of what EC2-the-server expects. Compared to that, the euca2ools version of that is easier to remember and much easier to type:

% euca-authorize MySecurityGroup --port 22 --source-subnet 0.0.0.0/0

Aws-cli is a very young project, so people haven’t yet had the chance to iron it out completely. Perhaps some day it will become as user-friendly as euca2ools and finally eclipse it. But we aren’t there yet.

How can I try it out?

If you’re interested in a preview of the next major version of euca2ools, an alpha release is available on GitHub. In addition to the dependencies required to run euca2ools 2, you will also need to install requests and the new requestbuilder framework that drives the new tools. It is still alpha-quality software, so be prepared to find bugs. If you encounter any, feel free to file them in the euca2ools project’s bug tracker.

If you’re interested in helping with development, we are happy to accept pull requests on GitHub. Please also consider joining the euca-users mailing list or stopping by in the #eucalyptus-devel IRC channel on Freenode. I look forward to hearing your feedback. 8^)

1. It binds to a class rather than an instance (hence its name). Thus, its first argument is a class, often called cls rather than the usual self.
2. It can be called on both an instance of a class and the class itself.

In general, they behave similarly, but one area in which they can differ is when we go to override the class method:

class Spam(object):
    @classmethod
    def parrot(cls, message):
        print cls.__name__, "says:", message

class Eggs(Spam):
    @classmethod
    def parrot(cls, message):
        Spam.parrot(cls, message)

This code is broken because Spam.parrot is already bound to the Spam class. This means Spam.parrot‘s cls argument will be the Spam class rather than the Eggs class that we wanted, so Spam.parrot will end up being called in the wrong context. Even worse, everything Eggs.parrot passed to it, including cls, ends up getting passed as regular arguments, resulting in disaster.

>>>> Spam.parrot("Hello, world!")
Spam says: Hello, world!
>>>> Eggs.parrot("Hello, world!")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "<console>", line 4, in parrot
TypeError: parrot() takes exactly 2 arguments (3 given)

To chain up to the Spam class’s implementation we need to use a super object, which will delegate things the way we want.

class Eggs(Spam):
    @classmethod
    def parrot(cls, message):
        super(Eggs, cls).parrot(message)

There’s a shortcut for the last line if you’re using python 3:

        super().parrot(message)

The super object functions as a proxy that delegates method calls to a class higher up in the Eggs class’s hierarchy, and in this case it is critical in ensuring that it gets called in the right context.

>>>> Spam.parrot("Hello, world!")
Spam says: Hello, world!
>>>> Eggs.parrot("Hello, world!")
Eggs says: Hello, world!

If you haven’t used super() before, here’s what it’s doing:

1. The python interpreter looks for Eggs in cls.__mro__, a list that represents the order in which the interpreter tries to match method and attribute names to classes.
2. The interpreter checks the class dictionary for the next class that follows Eggs in that list that contains "parrot".
3. The super object returns that version of "parrot", bound to cls, using the attribute-fetching __get__(cls) method.
4. When Eggs.parrot calls this bound method, cls gets passed to Spam.parrot in place of the Spam class.

In general I tend to stick with the older-style syntax for chaining method calls, but this is one case where super() is simply indispensable.

The thing that makes this feature really nice, however, looks like this:

Just a few hours ago, the new web console that is slated to land alongside Eucalyptus 3.2 later this year gained support for importing key pairs as well. (See the change on GitHub.) I’m sure I am not alone in wanting to thank the contributors who added this.

Inconsolata

Inconsolata is my favorite monospaced font. On top of that, it’s free. It has its own unique sort of style that is easy on the eyes and yet instantly recognizable. It also scales beautifully to sizes both large and small. I use it on everything from terminal windows to code editors to the huge LCD on the wall at work.

Its main downside is that it doesn’t seem to look particularly great on Windows. It looks amazing everywhere else as long as you remember to turn anti-aliasing on. But even at sizes that are too tiny to use anti-aliasing, it still manages to remain legible.

If you use Fedora you can even install Inconsolata using yum.

yum install levien-inconsolata-fonts

Consolas

On Windows I use Consolas, a commercial typeface that Microsoft bundles with Windows and Office. It has a sort of roundish appeal to it but still manages to keep just enough of the boxy look that works well for code. It looks cleaner than the Courier New typeface that Windows programs tend to use by default because it drops a lot of serifs that just take up space. Unsurprisingly, it also blows the ancient, raster-based one that the command prompt uses out of the water. Just don’t forget to leave anti-aliasing on.

" Search for selected text, forwards or backwards
vnoremap <silent> * :<C-U>
  \let old_reg=getreg('"')<Bar>let old_regtype=getregtype('"')<CR>
  \gvy/<C-R><C-R>=substitute(
  \escape(@", '/\.*$^~['), '\_s\+', '\\_s\\+', 'g')<CR><CR>
  \gV:call setreg('"', old_reg, old_regtype)<CR>
vnoremap <silent> # :<C-U>
  \let old_reg=getreg('"')<Bar>let old_regtype=getregtype('"')<CR>
  \gvy?<C-R><C-R>=substitute(
  \escape(@", '?\.*$^~['), '\_s\+', '\\_s\\+', 'g')<CR><CR>
  \gV:call setreg('"', old_reg, old_regtype)<CR>

After adding this, select some text in visual mode (press v and move the cursor around) and search for what you have selected with * or #.

git rev-list --reverse ${last_good_commit}.. | xargs git revert

Why Eucalyptus 2? Most of us don’t get to use Eucalyptus 3.0, so comparing against that wouldn’t exactly be fair, would it? ;-)

Centralized documentation

The documentation for Eucalyptus 2 was strewn about the Eucalyptus website on a number of wiki pages. You had to read all of them to have any hope of ending up with a working cloud.

Eucalyptus 3′s documentation comes in the form of front-to-back PDFs. HTML documentation is forthcoming. You still need to read it all, but it is now in one place so you don’t have to go digging to find it.

The documentation’s source (in DITA format, if you find that sort of thing interesting) is also up on github, so there is now a way to fix errors: just send a pull request.

A new database

Eucalyptus 3.1 switches from HSQLDB to PostgreSQL. Given the number of Eucalyptus users I have seen over time who have experienced problems with HSQLDB’s behavior in the face of faults, I suspect this will make a lot of people happy.

Correct packaging

The RPM and DEB packages for Eucalyptus 2 fail to list a number of important things they depend upon, making the software needlessly complicated to install. In fact, this was so complicated that the popular FastStart distribution became the method of choice for getting started with a new Eucalyptus cloud.

This is no longer the case. Installation now consists of adding package repositories and telling one’s package manager to install a component. No more “install these dependencies first.” No more “download these packages separately and install them.” In fact, short of a script that writes Eucalyptus’s configuration for you, this completely obviates the need for FastStart.

RHEL 6 support

Eucalyptus 2 is supported only on CentOS 5. It also works on RHEL 5, but users of RHEL 6 and friends couldn’t even compile the stuff. That is now fixed; those operating systems now have full support.

A more usable configuration file

Eucalyptus 2′s configuration file jams everything into one huge list. Nothing gives any indication what Eucalyptus components actually care about each option. There is also no indication how options were affected by one’s choice of networking mode.

Eucalyptus 3′s configuration file and documentation break options down by component. For networking-related options, they also list the networking modes in which they apply.

Bugfixes

Eucalyptus 3 fixes bugs. Lots and lots of bugs. So many bugs that release notes cannot possibly list them all. In the future this will be easier, as every future bug report will now go through a new JIRA tracker.

What else?

Those are some of my favorites, but there are lots of other little improvements all over the place. Try Eucalyptus 3 out and see what you think. You may be pleasantly surprised.

% git merge -X ours theirbranch

If “their” changes that we are pulling in should always take precedence instead then things are equally easy:

% git merge -X theirs theirbranch

If “ours” or “theirs” style merges will work, but only for certain files, things get a bit more complicated. But with a bit of background knowledge of how git’s index works we can save ourselves a bunch of work.

The index stores what stage a file is in. Normally, a file’s stage is 0.

% git ls-files -s myfile
100644 4b48deed3a433909bfd6b6ab3d4b91348b6af464 0       myfile

A file with a merge conflict is different because the index actually has three different versions of it: the version that the two branches we are merging most recently had in common, the version on “our” side of the merge, and the version on “their” side of the merge. In git these correspond to stages 1, 2, and 3. This is one reason one has to run git add on every conflicting file before completing a merge.

% git ls-files -s myfile
100644 4b48deed3a433909bfd6b6ab3d4b91348b6af464 1       myfile
100644 5be4a414b32cf4204f889469942986d3d783da84 2       myfile
100644 39c5733494077ae8bc45c45c15a708ffe9871966 3       myfile

Rather than opening up the conflicting file and clearing out the parts that clash by hand, we can instead simply copy the version we want from the index into our working tree:

% git checkout-index -f --stage 3 myfile
% git add myfile
% git commit

In the source repository, add the changes we want to move to a bundle that we can copy to a USB stick:

% git bundle create changes.bundle master..mybranch
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 313 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
% cp changes.bundle /media/usbstick

In the destination repository, ensure that we have the commits necessary to use the bundle and then tell git fetch to grab the changes from it:

% git bundle verify /media/usbstick/changes.bundle
The bundle contains 1 ref
7a1d2087f10e6db33e6b4a28e2c427b65238a62c refs/heads/mybranch
The bundle requires these 1 ref
6f5fced94ef76f1b46e259db72ad6fc39c49ba72 
/media/usbstick/changes.bundle is okay
% git fetch /media/usbstick/changes.bundle mybranch
Receiving objects: 100% (3/3), done.
Resolving deltas: 100% (2/2), completed with 2 local objects.
From /media/usbstick/changes.bundle
 * branch            mybranch   -> FETCH_HEAD
% git merge FETCH_HEAD
Updating 6f5fced..7a1d208
Fast-forward
 README |    2 ++
 1 file changed, 2 insertions(+)