Private Apt Repos in S3 12 Oct 2012

Setting up a private apt repository in S3 is actually not too bad. This HOWTO sets up two minimal repositories, one public and one private. You need both. All work is to be done on a Debian or Ubuntu machine with an architecture matching what you ar edeploying to (ie, amd64).

Kyle Shank did the heavy lifting for us by making a s3:// transport scheme for apt. Sadly, that package isn’t in any reliable public repos I know of, so to be safe this HOWTO will have you host it in a repo you control.

The process is therefore two steps, setting up the public repo to hold the s3 apt handler, installing that handler, then setting up a private repo which uses authenticated s3 connections to access your debs.

The first repo is a public repo which exists to hold the apt-transport-s3 package. Check out a fork of apt-s3 and build it using make deb. You probably will want to nest the checkout in a dedicated directory as the build drops the deb one directory up from the one you build from. Go figure. It requires libapt-pkg-dev libcurl4-openssl-dev be installed, see the README for details, it is pretty good.

Once you have that built, you’ll need to put it into our public repo. Doing this looks like:

$ mkdir repos
$ cd repos
$ mkdir -p public-repo/binary
$ cp ~/src/borkage/apt-transport-s3_1.1.1ubuntu2_amd64.deb pubic-repo/binary
$ cd public-repo
$ dpkg-scanpackages binary /dev/null | gzip -9c > binary/Packages.gz
$ dpkg-scansources binary /dev/null | gzip -9c > binary/Sources.gz

We made a repos directory which will hold both of our repos, then made a public-repo/binary directory for our binary artifacts. We copy in our apt-transport-s3 deb and build both package and source indexes. Be sure not to add a trailing slash to the binary bit in the dpkg-* incantations, it will throw off locations in the index. We build a Sources.gz, which will be empty, so that using add-repository doesn’t freak out.

We now have a local copy of our repo, yea! We want to push this up to s3, so make yourself a bucket for the public repo, I’ll call the demo one demo-public-repo. We’re going to sync this up using s3cmd. You should install it and configure it:

$ sudo apt-get install s3cmd
$ s3cmd --configure

Follow the instructions when configuring.

Now we’ll use s3cmd to sync our repo up:

$ cd repos
$ s3cmd sync -P public-repo/ s3://demo-public-repo

Note the -P – we need the artifact herein to be public so that we can install it.

Okay, the private repo will be just like the public repo, except we’ll use a different bucket and not make it world readable:

$ cd repos
$ mkdir -p private-repo/binary
$ cp ~/src/secret-stuff/target/secret_0.0.1.deb private-repo/binary
$ cd private-repo
$ dpkg-scanpackages binary /dev/null | gzip -9c > binary/Packages.gz
$ dpkg-scansources binary /dev/null | gzip -9c > binary/Sources.gz
$ cd ..
$ s3cmd sync -P private-repo/ s3://demo-private-repo

Now log into the host which needs to use the private repo and add the following line to /etc/apt/sources.list

deb http://s3-us-west-2.amazonaws.com/demo-public-repo binary/

Of course, you’re URL will vary – find the HTTP URL for your bucket root and use that. This one happens to be in us-west-2 (Oregon), yours will most likely not be.

Once it is added, install apt-transport-s3 via:

$ sudo apt-get update
$ sudo apt-get install -y --force-yes apt-transport-s3

We need the --force-yes as we didn’t sign the deb.

Now, the magic, this allows us to add a repo url of the form:

deb s3://<access-key>:[<secret-key>]@s3-us-west-2.amazonaws.com/demo-private-repo binary/

to /etc/apt/sources.list where you, again, get the right region and bucket information, and replace <access-key> and <secret-key> with your actual aws access and secret keys. The brackets above are not indicating that the secrrt key is optional, you need to include the brackets. They are there to disambiguate the secret key if any characters show up that confuse things.

You can now update your apt indices again and then go install things from your private repo!