Java Daemonization with posix_spawn(2) 24 Jan 2012
The traditional way of daemonizing a process involves forking the process and daemonizing it from the current running state. This doesn’t work so well in Java because the JVM relies on several additional worker threads, and fork only keeps the thread calling fork. So, basically, you need to start a new JVM from scratch to create a child process.
The traditional way of launching a new program is to fork and exec the program you wish to start. Sadly, this also fails on Java because the calls to fork and exec are seperate, non-atomic (in the platonic sense, not the JMM sense) operations. There is no guarantee that the exec will be reached, or that the memory state of the JVM will even be sound when the exec is reached, because you could be in the middle of a garbage collection and pointers could be all over. In practice, this would happen exceptionally rarely, at least. Charles Nutter has talked about this problem in JRuby as well.
The best method I know of to daemonize a Java process, then, is to use posix_spawn. This function launches a new process based on the program image passed to it – it is like fork and exec in one system call. Over the weekend I took some time to put together a small library to do this, Gressil using jnr-ffi. It turned out that the hardest part, by far, was reconstructing the full ARGV array.
On linux, with ProcFS it is pretty straightforward. On OS X it is supposed to be possible but all the code samples I have found seem to be based on that snippet, and they look forward into the “String area” which usually has ARGV laid out in the correct order, but sometimes doesn’t. Trying to look backward from the pointer returned by the sysctl would be straightforward in C (decrement the pointer) but jnr-ffi seems to copy the memory space into a Java byte array, so there is no going backward. Attempting to guess how far back to jump the pointer to get a new block of memory usually segfaults for me :-) Anyone with thoughts, a version based on the above reference is in Gressil, MacARGVFinder.
Anyway, it turns out you can get most of this information from the Sun JVM. The only dodgy part of that is the program arguments (the stuff passed into Java’s public static void main(String[] args) method. Luckily, those are passed to main, so we can just ask for them in the library.
Given that, you can use Gressil to daemonize Java processes via re-launching like so:
package org.skife.gressil.examples;
import org.skife.gressil.Daemon;
import java.io.File;
import java.io.IOException;
import java.util.Arrays;
import java.util.Date;
import static org.skife.gressil.Daemon.remoteDebugOnPort;
public class ChattyDaemon
{
public static void main(String[] args) throws IOException
{
new Daemon().withMainArgs(args)
.withPidFile(new File("/tmp/chatty.pid"))
.withStdout(new File("/tmp/chatty.out"))
.withExtraMainArgs("hello", "world,", "how are you?")
.withExtraJvmArgs(remoteDebugOnPort(5005))
.daemonize();
while (!Thread.currentThread().isInterrupted()) {
System.out.println(new Date() + " " + Arrays.toString(args));
try {
Thread.sleep(1000);
}
catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
}
In the parent process the call to Daemon#daemonize() will call System.exit(), in the child process it will return normally.
The child process, in this case, will also listen for a Java debugger to attach on port 5005. It will attach stdout to /tmp/chatty.out, and stdin and stderr will default to /dev/null (which stdout would also attach to by default if it were not specified).
I cut a 0.0.1 release which should be synced in maven central by now as org.skife.gressil:gressil:0.0.1 by now. It should work pretty much anywhere with a posix-compliant libc, but I have only extensively tested on OS X.
Configuration Models 20 Jan 2012
I have been noodling on the best general purpose application configuration model, and mechanism, I can find lately – that is, trying to find something general purpose that I don’t think is miserably bad. Specifically, configuration of heterogeneous applications in a hosted/SaaS/distributed/blah type system.
I’ve worked with all kinds of stuff – not everything under the sun, for sure, but quite a few models. For purposes of this discussion, I’ll start with the one I have been working with most closely for the last six years – Ning’s Galaxy Repository.
Galaxy Classic: The Discussion Baseline
Galaxy uses a hierarchical configuration repository. See the “Gepo” and “Config Path” parts of the readme for a description of how it works. The mechanism does work pretty well, it’s gotten us a long way, but it has some horrible warts that I would like to avoid. (Really, go read that page if you are not familiar with Galaxy and want any of the rest of this to make sense. It’s worth it, Galaxy has been a Force for Good).
The first is that the galaxy agent relies on the configuration repository to determine the binary to install. It takes the configuration path for the deployment (I told you, just go read that docu and come back) and constructs a URL for the binary.
The second is that after finding the binary in the config repo, it then passes information about this config repo to the deployment bundle when it deploys it. The agent has a hard dependency on the config repo, and it leaks the dependency into the deployment bundle, which then uses its knowledge of the configuration repository to pull down the application configuration.
Proofpoint’s Galaxy: Another Take
Some very good folks at Proofpoint, who all worked with (and in Martin’s case, helped build) Ning’s Galaxy reimplemented it to fix a number of behaviors that were suitable at Ning, but not Proofpoint. One of the changes was to the configuration mechanism. The deployment command receives a set of coordinates for the binary and a configuration. These coordinates are converted to two URLS by the time they reach the agent.
The configuration URL, in the Proofpoint system, references a resource which is a set of (path, URL) pairs. When the agent deploys the binary bundle, it then pulls down each resource specified in the configuration resource, and puts it at the path specified for it in the deployment.
In this model, the agent just receives URLs, though it understands the resources those URLs point to. The binary one because it is a galaxy package, and deploying them is what it does, the configuration one as it needs to pull down the configuration files.
The Proofpointers are considering reworking that configuation mechanism to be a URL to a configuration tarball which will contain all of the config files needed, which is expanded to a known location (probably /env/) inside the deployment. This would remove one layer of indirection, and makes the configuration a write-once artifact.
Sculptor: Yet Another Take
In parallel, I have been experimenting with another galaxy implementation, in particular to play nicely with Atlas, named Sculptor.
Sculptor also uses (path, URL) pairs, but these pairs are specified as either part of the environment (so properties of the agent) and/or part of the deployment. You can watch a screencast which uses the environment configuration, but the deployment side is just in the noodling stages. In sculptor a deployment resource (submitted to the agent) would look something like:
{
"url":"http://static.skife.org/echo-0.0.2.tar.gz",
"name":"Echo Server",
"configuration": {
"/env/instance.conf":"http://waffles/DEP-123/echo/echo.conf",
"/env/something.else":"http://pancakes/master/global.config"
}
}
This is remarkably similar to the Proofpoint model, but was arrived at bottom up to play nicely with how Atlas works, and be able to support Ning’s applications. I like that they evolved in similar directions.
Using Configuration
In all of these cases, the configuration tends to be very simple key value pairs. Application specific configuration files, such as httpd.conf, usually have values interpolated from the key value pairs coming from configuration. In Ning’s case, this is at deployment time, but in Proofpoint and Sculptor, this is when the service is started.
Under any of these models you can put application-formatted configuration in the repository (and referenced by URL) but the application specific format tends to be stringly tied to the implementation details of the service, and spreading that knowledge around to different artifacts generally makes things more difficult.
Help!
To return to the beginning I want to pick one general purpose model. Sculptor embraces where my mind is right now, but it is an untested model – no one uses Sculptor (yet) so its mechanism and details are unproven. I’d really appreciate feedback, thoughts, and war stories on this stuff. Thank you!
POSIX from Java 10 Jan 2012
I have been doing more traditionally unix-y stuff from Java lately, and one of the things I have needed is proper access to POSIX and libc system calls. Luckily, there are now a couple fabulous libraries to make this easy – no more need to do your own JNI muckery.
I’ve been using jnr-posix with great success. Using it for something like execv(3) looks like:
POSIX posix = POSIXFactory.getPOSIX(new POSIXHandler()
{
@Override
public void error(Errno errno, String s)
{
}
@Override
public void unimplementedError(String s)
{
}
@Override
public void warn(WARNING_ID warning_id, String s, Object... objects)
{
}
@Override
public boolean isVerbose()
{
return false;
}
@Override
public File getCurrentWorkingDirectory()
{
return new File(".");
}
@Override
public String[] getEnv()
{
return new String[0];
}
@Override
public InputStream getInputStream()
{
return System.in;
}
@Override
public PrintStream getOutputStream()
{
return System.out;
}
@Override
public int getPID()
{
return 0;
}
@Override
public PrintStream getErrorStream()
{
return System.err;
}
}, true);
String[] args = new String[] {
"/usr/bin/ssh", "lasker.skife.org"
};
posix.execv("/usr/bin/ssh", args);
The bulk of that snippet is setting up the POSIXHandler which provides nice callbacks for the things that are not obvious how to handle, or might want to be overidden in a specific environment. The boolean flag at the end says to use the native POSIX implementation rather than emulating it in Java. The library will sniff your system and dynamically link the right things – is very nice.
The library doesn’t properly declare its dependencies in its pom, so if you want to use it you need to depend on each of:
<dependency>
<groupId>com.github.jnr</groupId>
<artifactId>jnr-posix</artifactId>
<version>2.0</version>
</dependency>
<dependency>
<groupId>com.github.jnr</groupId>
<artifactId>jnr-ffi</artifactId>
<version>0.6.0</version>
</dependency>
<dependency>
<groupId>com.github.jnr</groupId>
<artifactId>jnr-constants</artifactId>
<version>0.8.2</version>
</dependency>
The jnr-posix pom lists jnr-constants and jnr-ffi as provided for some reason. Hopefully that will be remedied in $version.next().
Some Atlas Thoughts 28 Dec 2011
Back in June (that long ago, really? wow!) I first talked about Atlas and my, time has flown. While originally created to help address a very specific problem we were facing at work, the general utility of the approach in Atlas is pretty intriguing.
I want to be able to describe a system, such as:
system "blog"
server "load-balancer:blog"
server "wordpress", {
base: "linux-server",
cardinality: 2,
install: ["wordpress?db=blog-db&caches=blog-cache",
"lb-add:blog"]
}
server "caches", {
base: "linux-server",
cardinality: 2,
install: ["memcached:blog-cache"]
}
server "database", base: "mysql:blog-db"
end
Which basically describes how to wire up the abstract servers in a system. This description is paired with an environment definition that explains what things like linux-server are and what exactly is done to install wordpress in a particular environment.
Atlas will need some kind of virtual-installer (which it doesn’t have right now) to get the nice clean descriptor above, as the system described there actually looks like:
wp_url = "http://wordpress.org/wordpress-3.2.1.tar.gz"
mcp_url = "http://downloads.wordpress.org/plugin/memcached.2.0.1.zip"
system "blog" do
server "load-balancer", {
base: "load-balancer:blog?from=80&to=80"
}
server "wordpress", {
cardinality: 2,
base: "apache-server",
install: ["tgz:#{wp_url}?to=/var/www/&skiproot=wordpress",
"zip:#{mcp_url}?to=/var/www/wp-content/&skiproot=memcached",
"exec: yes | sudo pecl install memcache",
"exec: sudo echo 'extension=memcache.so' >> /etc/php5/apache2/php.ini",
"wait-for:wordpress-db",
"erb: wp-config.php.erb > /var/www/wp-config.php",
"exec: sudo service apache2 restart",
"elb-add:blog"]
}
server "memcached", {
cardinality: 2,
base: "server",
install: ["scratch:memcached=@",
"apt:memcached",
"file:memcached.conf > /etc/memcached.conf",
"exec: sudo service memcached restart"]
}
server "database", {
base: "mysql:blog",
install: "scratch:wordpress-db=@"
}
end
To get to the first system descriptor it will need some concept of a virtual installer, that would be defined in the environment descriptor, and might look like:
environment "ec2" do
installer "wordpress", {
virtual: ["tgz:#{wp_url}?to=/var/www/&skiproot=wordpress",
"zip:#{mcp_url}?to=/var/www/wp-content/&skiproot=memcached",
"exec: yes | sudo pecl install memcache",
"exec: sudo echo 'extension=memcache.so' >> /etc/php5/apache2/php.ini",
"wait-for:{install[db]}",
"erb: wp-config.php.erb > /var/www/wp-config.php",
"exec: sudo service apache2 restart"]
}
# ... rest of the env descriptor
end
In a non-trivial system I would use Galaxy for deploying services, probably Dain’s fork for something new. But for this example I think it would just confuse matters. I would also be using Chef or Puppet to bring up all the non-application packages and configuration. See Fundamental Components in a Distributed System for more of a look into this reasoning.
Atlas probably also needs explicit stages for installing things. The wordpress servers, for example, query a scratch space for all the memcached servers which happens to work right now because of timing, but you really want to be able to say “all the memcached servers note themselves in the scratch space in phase 1, and in phase 50 the wordpress config template looks them up” where everything in a phase must complete for the phase to be complete.
While cataloging work that needs to be done, Atlas really needs to switch to a better plugin system as right now it isn’t so much a plugin system as a reminder left to use one. OSGI seems like it would work really well for this. Maybe I should pester Bertrand for some help.
In Clauses 21 Dec 2011
The most common feature request I get for jDBI, going back at least seven years now, is for automagic in-clause expansion, or the equivalent. Sadly, there is not correct general case solution for this. There are lots of solutions, but what is the right thing to do is very context dependent. Let’s look at some options.
Database Specific Functionality
The first option is to use database specific functionality to achieve this. The easiest of these I know of is with PostgreSQL, which has very nice sql array support.
We’ll start with how we want to use the feature – ideally we could just bind a collection, but in this case we need to know the type of things in the collection, and we should handle empty collections, so we’ll do something like:
ImmutableSet<String> rs = h.createQuery("select name from something where id = any (:ids)")
.map(StringMapper.FIRST)
.bind("ids", arrayOf(Integer.class, 2, 3))
.list(ImmutableSet.class);
The bind is using a helper function to create an instance of SqlArray which just captures the information being bound:
public class SqlArray<T>
{
private final Object[] elements;
private final Class<T> type;
public SqlArray(Class<T> type, Collection<T> elements) {
this.elements = Iterables.toArray(elements, Object.class);
this.type = type;
}
public static <T> SqlArray<T> arrayOf(Class<T> type, T... elements) {
return new SqlArray<T>(type, asList(elements));
}
public static <T> SqlArray<T> arrayOf(Class<T> type, Iterable<T> elements) {
return new SqlArray<T>(type, elements);
}
public Object[] getElements()
{
return elements;
}
public Class<T> getType()
{
return type;
}
}
When binding a SqlArray it will wind up invoking the Handle#bind(String, Object) method. This is fine as we will intercept the actual binding with an ArgumentFactory. The one here is a toy, specialized solely for binding arrays of integers, but it should be straightforward to generalize:
public class PostgresIntegerArrayArgumentFactory implements ArgumentFactory<SqlArray<Integer>>
{
public boolean accepts(Class<?> expectedType, Object value, StatementContext ctx)
{
return value instanceof SqlArray
&& ((SqlArray)value).getType().isAssignableFrom(Integer.class);
}
public Argument build(Class<?> expectedType,
final SqlArray<Integer> value,
StatementContext ctx)
{
return new Argument()
{
public void apply(int position,
PreparedStatement statement,
StatementContext ctx) throws SQLException
{
// in postgres no need to (and in fact cannot) free arrays
Array ary = ctx.getConnection()
.createArrayOf("integer", value.getElements());
statement.setArray(position, ary);
}
};
}
}
We need to register our argument factory on the DBI, Handle, or SqlStatement for it to be used, we’ll just put it on the handle:
Handle h = dbi.open();
h.registerArgumentFactory(new PostgresIntegerArrayArgumentFactory());
h.registerContainerFactory(new ImmutableSetContainerFactory());
While we were at it we registered a ContainerFactory that knows how to build Guava’s ImmutableSet as well. That is pretty irrelevant to this example, but hey, why not ;-)
The argument factory produces an actual Argument instance used to bind arguments to the the prepared statement. PostgreSQL is the only database I know of which supports the Connection#createArrayOf call, so this is very non-portable, but it works great in PostgreSQL!
Oracle supports something similar, but you need to use Oracle specific APIs and oracle.sql.ARRAY instances. In the Oracle case you have to pre-declare the array type in the database first, and as it stores the array in the database, free it after the call.
I don’t know of anything in MySQL to provide this behavior.
SQL Templating
Much more portable, but requiring more care is rewriting the SQL to provide the right bind slots in the sql statement. If we want to use the same SQL as above, but with an in clause, it would look like select name from something where id in (:ids), to which we want to bind two values, 2 and 3. We need to somehow expand the the statement to select name from
something where id in (:id_1, :id_2). We don’t usually know how many elements will be in the in clause, so… blah.
In jDBI I like to handle this by using StringTemplate statement groups to externalize my SQL. Aside from providing nice grouping, the templating makes the problem of rewriting the sql much more tractable!
We’ll start, again, with what we want the API to look like:
@ExternalizedSqlViaStringTemplate3
@RegisterContainerMapper(ImmutableSetContainerFactory.class)
public interface DAO
{
@SqlQuery
ImmutableSet<String> findNamesForIds(@BindIn("names") List<Integer> names);
}
Note that the @ExternalizedSqlViaStringTemplate3 annotation on the Dao type. This is included with jDBI and tells the sql object to look for sql statements in a StringTemplate Group File loaded from the classpath such that the name is <ClassName>.sql.stg in a directory matching the package of the sql object type. If the above is org.skife.v2.docs.DAO templates for the actual SQL will be looked for on the classpath at /org/skife/v2/docs/DAO.sql.stg.
The group template file, in this case, looks like:
group DAO;
findNamesForIds(names) ::= <<
select name from something where id in (<names>)
>>
Now, looking at this template, select name from something where
id in (<names>) we see something new, the <names> bit. This is stringtemplate, not argument binding. We see the parameter to the template in the template signature as well, findNamesForIds(names).
Looking at the sql object definition, we also see @BindIn("names") instead of the more familiar @Bind("names"). The @BindIn is a custom binding annotation which knows how to work with StringTemplate 3 (specifically version 3, as there is now a version 4 available as well). Let’s look at @BindIn to see how it works:
/*
Please excuse the use of # in place of @, pygments +
maruku is totally confused by the code otherwise -brianm
*/
#Retention(RetentionPolicy.RUNTIME)
#SqlStatementCustomizingAnnotation(BindIn.CustomizerFactory.class)
#BindingAnnotation(BindIn.BindingFactory.class)
public #interface BindIn
{
String value();
public static final class CustomizerFactory implements SqlStatementCustomizerFactory
{
public SqlStatementCustomizer createForMethod(Annotation annotation, Class sqlObjectType, Method method)
{
throw new UnsupportedOperationException("Not supported on method!");
}
public SqlStatementCustomizer createForType(Annotation annotation, Class sqlObjectType)
{
throw new UnsupportedOperationException("Not supported on type");
}
public SqlStatementCustomizer createForParameter(Annotation annotation,
Class sqlObjectType,
Method method,
Object arg)
{
Collection<?> coll = (Collection<?>) arg;
BindIn in = (BindIn) annotation;
final String key = in.value();
final List<String> ids = new ArrayList<String>();
for (int idx = 0; idx < coll.size(); idx++) {
ids.add("__" + key + "_" + idx);
}
StringBuilder names = new StringBuilder();
for (Iterator<String> i = ids.iterator(); i.hasNext();) {
names.append(":").append(i.next());
if (i.hasNext()) {
names.append(",");
}
}
final String ns = names.toString();
return new SqlStatementCustomizer()
{
public void apply(SQLStatement q) throws SQLException
{
q.define(key, ns);
}
};
}
}
public static class BindingFactory implements BinderFactory
{
public Binder build(Annotation annotation)
{
final BindIn in = (BindIn) annotation;
final String key = in.value();
return new Binder()
{
public void bind(SQLStatement q, Annotation bind, Object arg)
{
Iterable<?> coll = (Iterable<?>) arg;
int idx = 0;
for (Object s : coll) {
q.bind("__" + key + "_" + idx++, s);
}
}
};
}
}
}
We see that @BindIn is actually both a binding annotation and a statement customizing annotation. This allows it to manipulate the statement and to control how arguments get bound.
As a statement customizing annotation,
public SqlStatementCustomizer createForParameter(Annotation annotation,
Class sqlObjectType,
Method method,
Object arg)
{
Collection<?> coll = (Collection<?>) arg;
BindIn in = (BindIn) annotation;
final String key = in.value();
final List<String> ids = new ArrayList<String>();
for (int idx = 0; idx < coll.size(); idx++) {
ids.add("__" + key + "_" + idx);
}
StringBuilder names = new StringBuilder();
for (Iterator<String> i = ids.iterator(); i.hasNext();) {
names.append(":").append(i.next());
if (i.hasNext()) {
names.append(",");
}
}
final String ns = names.toString();
return new SqlStatementCustomizer()
{
public void apply(SQLStatement q) throws SQLException
{
q.define(key, ns);
}
};
}
This customizer has access to the collection passed as a parameter, so it traverses it building up the expanded binding list. In this case, if it is passed the list [2, 3] it will build the string :__names_0, :__names_1 and define it on the statement context.
The StringTemplate3 statement locator uses values defined on the statement context as arguments to the template, so in this case the template will receive that string as the names argument, and the actual SQL will be select name from something where id
in (:__names_0, :__names_1).
The binding annotion piece,
public static class BindingFactory implements BinderFactory
{
public Binder build(Annotation annotation)
{
final BindIn in = (BindIn) annotation;
final String key = in.value();
return new Binder()
{
public void bind(SQLStatement q, Annotation bind, Object arg)
{
Iterable<?> coll = (Iterable<?>) arg;
int idx = 0;
for (Object s : coll) {
q.bind("__" + key + "_" + idx++, s);
}
}
};
}
}
Receives the argument being bound, which will be what was passed in, and it performs the same operation to generate the binding names, then it actually binds them.
Which To Use?
In general, if your database supports it I would suggest using the database specific mechanism which works. Sadly, the only database which supports it well, which I have used enough to have an opinion on, is PostgreSQL. If anyone knows similar techniques for other databases, please tell us in the comments.
In all other cases, templating your SQL is the only real option. This can cause problems with statement caches (each different SQL is usually a different statement (though you can fuzz this in Oracle)), you need to align your statement locator with your templating, and you need to be careful about how big your in clause literal gets, but it works pretty well in the 80% or cases.