Some Atlas Thoughts 28 Dec 2011

Back in June (that long ago, really? wow!) I first talked about Atlas and my, time has flown. While originally created to help address a very specific problem we were facing at work, the general utility of the approach in Atlas is pretty intriguing.

I want to be able to describe a system, such as:

system "blog"
  server "load-balancer:blog"

  server "wordpress", { 
    base: "linux-server",
    cardinality: 2,
    install: ["wordpress?db=blog-db&caches=blog-cache", 
              "lb-add:blog"]
  }

  server "caches", {
    base: "linux-server",
    cardinality: 2,
    install: ["memcached:blog-cache"]
  }

  server "database", base: "mysql:blog-db"
end

Which basically describes how to wire up the abstract servers in a system. This description is paired with an environment definition that explains what things like linux-server are and what exactly is done to install wordpress in a particular environment.

Atlas will need some kind of virtual-installer (which it doesn’t have right now) to get the nice clean descriptor above, as the system described there actually looks like:

wp_url = "http://wordpress.org/wordpress-3.2.1.tar.gz"
mcp_url = "http://downloads.wordpress.org/plugin/memcached.2.0.1.zip"

system "blog" do
  
  server "load-balancer", {
    base: "load-balancer:blog?from=80&to=80"
  }

  server "wordpress", {
    cardinality: 2,
    base: "apache-server",
    install: ["tgz:#{wp_url}?to=/var/www/&skiproot=wordpress",
              "zip:#{mcp_url}?to=/var/www/wp-content/&skiproot=memcached",
              "exec: yes | sudo pecl install memcache",
              "exec: sudo echo 'extension=memcache.so' >> /etc/php5/apache2/php.ini",
              "wait-for:wordpress-db",
              "erb: wp-config.php.erb > /var/www/wp-config.php",            
              "exec: sudo service apache2 restart",
              "elb-add:blog"]
  }

  server "memcached", {
    cardinality: 2,
    base: "server",
    install: ["scratch:memcached=@", 
              "apt:memcached",
              "file:memcached.conf > /etc/memcached.conf",
              "exec: sudo service memcached restart"]
  }

  server "database", {
    base: "mysql:blog",
    install: "scratch:wordpress-db=@"
  }
end

To get to the first system descriptor it will need some concept of a virtual installer, that would be defined in the environment descriptor, and might look like:

environment "ec2" do
  installer "wordpress", {
    virtual: ["tgz:#{wp_url}?to=/var/www/&skiproot=wordpress",
              "zip:#{mcp_url}?to=/var/www/wp-content/&skiproot=memcached",
              "exec: yes | sudo pecl install memcache",
              "exec: sudo echo 'extension=memcache.so' >> /etc/php5/apache2/php.ini",
              "wait-for:{install[db]}",
              "erb: wp-config.php.erb > /var/www/wp-config.php",            
              "exec: sudo service apache2 restart"]
  }
  
  # ... rest of the env descriptor
end

In a non-trivial system I would use Galaxy for deploying services, probably Dain’s fork for something new. But for this example I think it would just confuse matters. I would also be using Chef or Puppet to bring up all the non-application packages and configuration. See Fundamental Components in a Distributed System for more of a look into this reasoning.

Atlas probably also needs explicit stages for installing things. The wordpress servers, for example, query a scratch space for all the memcached servers which happens to work right now because of timing, but you really want to be able to say “all the memcached servers note themselves in the scratch space in phase 1, and in phase 50 the wordpress config template looks them up” where everything in a phase must complete for the phase to be complete.

While cataloging work that needs to be done, Atlas really needs to switch to a better plugin system as right now it isn’t so much a plugin system as a reminder left to use one. OSGI seems like it would work really well for this. Maybe I should pester Bertrand for some help.