What’s New in Euca2ools 3, Part 2: A Developer’s Perspective

3 Apr 2013

The upcoming version of euca2ools, version 3, completely reworks the command line suite to make it both easier to write and easier to use. Part 1 of this series discussed the user-facing changes version 3 has to offer, and today we’re going to take a look at how things improve on the developer’s side of the fence.

A change in philosophy: declarative programming

The developer is very much in the driver’s seat in version 1 of euca2ools. To use a car analogy, the developer directly controls the code’s direction, speed, and gearbox manually. Version 2 adds a cruise control by centralizing a lot of boilerplate code in the form of boto’s roboto module. Version 3 opts to let the developer give the requestbuilder framework a destination, step aside completely, and let it do the driving for the boring parts of the trip.

Requestbuilder offers a set of base classes and a domain-specific language based on python’s standard argparse library that allows the developer to say exactly how something should look at the command line in addition to how it should look when given to the server all in the same place.

What makes this so powerful is that it lets anybody with a service’s documentation and knowledge of how to use argparse write a command line tool quickly and painlessly. For instance, it took me around a day to write highly-customized command line tools for every operation Amazon’s Elastic Load Balancing service supports. Here’s the code from one of them:

class CreateLBCookieStickinessPolicy(ELBRequest):
    DESCRIPTION = ('Create a new stickiness policy for a load balancer, '
                   'whereby the load balancer automatically generates cookies '
                   'that it uses to route requests from each user to the same '
                   'back end instance. This type of policy can only be '
                   'associated with HTTP or HTTPS listeners.')
    ARGS = [Arg('LoadBalancerName', metavar='ELB',
                help='name of the load balancer to modify (required)'),
            Arg('-e', '--expiration-period', dest='CookieExpirationPeriod',
                metavar='SECONDS', type=int, required=True,
                help='''time period after which cookies should be considered
                stale (default: user's session length) (required)'''),
            Arg('-p', '--policy-name', dest='PolicyName', metavar='POLICY',
                required=True, help='name of the new policy (required)')]

The framework hands everything inside each Arg in this code to argparse to gather input from the command line and then send the results directly to the web server using whatever name argparse gives the input it gets. For instance, whatever a user supplies using the -e option ends up getting sent to the server as a CookieExpirationPeriod parameter. With a small amount of practice it becomes quite easy to write a bunch of commands this way very quickly.

One request, one command

Euca2ools are built around a “one request, one command” tenet. This means that, in general, there is a dedicated command for each thing a web service can do. This philosophy naturally lends itself to the tight coupling between command line options and what gets sent to the server discussed earlier, but it also lends itself to reversing the usual relationship between web services and web service requests. Whereas one typically writes an object that represents the service and uses methods on it to send requests, in euca2ools it is the commands, and thus the requests, which are the first-class citizens. Each command that represents a request instead points to a service, rather than the other way around.

The way this works in practice is by defining a base class for each service and a base class that all methods which use that service share:

class CloudWatch(requestbuilder.service.BaseService):
    NAME = 'monitoring'
    DESCRIPTION = 'Instance monitoring service'
    API_VERSION = '2010-08-01'
    AUTH_CLASS = requestbuilder.auth.QuerySigV2Auth
    URL_ENVVAR = 'AWS_CLOUDWATCH_URL'

    ARGS = [MutuallyExclusiveArgList(
                Arg('--region', dest='userregion', metavar='USER@REGION',
                    route_to=SERVICE, help='''name of the region and/or user
                    in config files to use to connect to the service'''),
                Arg('-U', '--url', metavar='URL', route_to=SERVICE,
                    help='instance monitoring service endpoint URL'))]

class CloudWatchRequest(requestbuilder.request.AWSQueryRequest):
    SERVICE_CLASS = CloudWatch

Services can supply their own command line options in the same way as requests. After it gathers options from the command line, requestbuilder uses route_to to choose where to send it. This also provides a convenient way to tell the framework not to send an option to the server at all when a command needs to process it specially: just use route_to=None.

Convention over configuration

The oft-quoted programming paradigm for frameworks is just as true for euca2ools 3 as it is elsewhere. Want to make a command print something? Just write a print_result method. The result from the server gets passed in as a dictionary.

class TerminateInstances(EucalyptusRequest):
    DESCRIPTION = 'Terminate one or more instances'
    ARGS = [Arg('InstanceId', metavar='INSTANCE', nargs='+',
                help='ID(s) of the instance(s) to terminate')]
    LIST_TAGS = ['instancesSet']

    def print_result(self, result):
        for instance in result.get('instancesSet', []):
            print self.tabify(('INSTANCE', instance.get('instanceId'),
                               instance.get('previousState', {}).get('name'),
                               instance.get('currentState', {}).get('name')))

Want to make a request do fancier preparations than argparse can do on its own? Just write a preprocess method that takes things from self.args and adds things to self.params to be sent to the server.

class DescribeSecurityGroups(EucalyptusRequest):
    DESCRIPTION = ('Show information about security groups\n\nNote that '
                   'filters are matched on literal strings only, so '
                   '"--filter ip-permission.from-port=22" will *not* match a '
                   'group with a port range of 20 to 30.')
    ARGS = [Arg('group', metavar='GROUP', nargs='*', route_to=None,
                default=[], help='limit results to specific security groups')]
    ...
    def preprocess(self):
        for group in self.args['group']:
            if group.startswith('sg-'):
                self.params.setdefault('GroupId', [])
                self.params['GroupId'].append(group)
            else:
                self.params.setdefault('GroupName', [])
                self.params['GroupName'].append(group)

There are also a few other methods one can plug in, such as postprocess, and, for especially early-running code, configure. Expect documentation for requestbuilder that covers this in detail in the future.

Scratching the surface

The examples above cover only a fraction of what is possible with euca2ools 3’s new infrastructure. While you can look forward to some more advanced uses of it in later blog posts, you can also take a look at the current euca2ools code in development to see some of the interesting things one can do with it. Today’s pre-release of that code carries with it commands for all three of AWS’s “triangle” services: Auto Scaling, CloudWatch, and Elastic Load Balancing. Continuing what seems to have become a euca2ools tradition, just look for the commands that start with euscale (pronounced “you scale”) euwatch (“you watch”), and eulb (“you’ll be”).

Packages for Fedora and RHEL 6 are available here. If you’re using another OS or want to build the code yourself you can simply clone euca2ools’s git repository‘s requestbuilder branch. Requestbuilder itself is available on PyPI and GitHub. As always, I encourage you to test this code against AWS and Eucalyptus 3.3 and let me know what you think on the euca-users mailing list. If you encounter bugs, please file them in the project’s bug tracker.

One Response to “What’s New in Euca2ools 3, Part 2: A Developer’s Perspective”

  1. Dmitrii Says:

    Nice write-up, Garrett! I like your analogies.


Comments are closed.