I tried a couple of things, with varying degrees of success.
1) Use a FileOutputStream in append mode instead of a RandomAccessFile to get a FileChannel in ChannelDescriptor.open(). This makes an auto-append FileChannel, but the speed was the same.
Its not surprising, since looking at the internals of FileChanelImpl, it does exactly what ChannelDescriptor#internalWrite was doing - sets the file position to FileChannel#size(), then does the write.
2) Use the absolute position version of FileChannel#write(). This produced about a 10% improvement.
3) Wrote a FileChannel implementation that only handles append-mode files. It assumes that the current file position is the end of the file, so doesn't bother seeking when doing a relative write. Any call to a method that will change the current position (e.g. read, position(long)), sets a flag, which causes FileChannel#write to reset the file position to the end of file.
Option #3 is what is in append.patch - with it, and jruby --server, jruby is about 10-20% slower than MRI on MacOS.