mjl blog
April 16th 2014

Including files in go binaries

I had some fun writing code to add assets to go binaries. I.e. arbitrary files the program needs at runtime, e.g. HTML templates or JS/CSS files. After too little research on existing solutions, I jumped in and wrote some code. Of course, it turns out this has been done before: See Carlos Castillo’s comment in this golang-nuts discussion, or Donovan’s comment in this other golang-nuts thread. No code was provided though, and a web search didn’t turn it up either.

Anyway, have a look at my asset package. An example is worth a thousand words, so here we go:

/*
go build -o testasset testasset.go
./testasset

# mkdir assets; echo test >assets/test.txt
(cd assets && zip -r0 ../assets.zip .) && cat assets.zip >>testasset
./testasset
*/

package main

import (
    "code.google.com/p/go.tools/godoc/vfs"
    "bitbucket.org/mjl/asset"
    "log"
    "io"
    "os"
)

func main() {
    fs := asset.Fs()
    if err := asset.Error(); err != nil {
            log.Println("using local assets")
            fs = vfs.OS("assets")
    }

    // note: paths must start with a slash!
    f, err := fs.Open("/test.txt")
    if err != nil {
            log.Fatal(err)
    }
    defer f.Close()
    io.Copy(os.Stdout, f)
}

You build your program, create a zip file with your assets, and simply append it to the binary:

cat myprog.zip >>myprog

The binary will still run. The asset package will look for the appended zip file and open it as a vfs.FileSystem (from the godoc package). This lets you call Open() and friends. Very familiar, no need to learn something new. Godoc also provides a vfs.FileSystem that simply uses the local file system. This means your code can easily fallback to read local files instead of embedded files. Quite handy during development: no need to recompile after changing an HTML template. Only when you’re ready to test for a release will you append the assets to the binary.

There are two things potentially tricky to this approach:

  1. Arbitrary data is appended to binaries. Picky OS’es could refuse such binaries. Luckily, it appears they don’t care. Did a quick test on a Linux (Ubuntu 12.04), Mac OS X 10.9 and Windows 8.1 system, and they all worked.
  2. The appended zip file cannot be parsed by the archive/zip package without further help: The zip file contains a file-index at the end of the file, pointing to offsets within the zip file. These offsets are offset (hah) by the size of the binary. To work around this, the asset package reads a few fields at the end of the zip file to determine the original zip file size, and uses an io.SectionReader to present archive/zip with just the zip-portion of the fat binary. The tricky part is that it assumes the file-index (“central directory” in zip parlance) comes immediately before the end-of-central directory marker. AFAIK, this is not strictly necessary in the zip file format, but will be the case for all zip files created in practice.

There are other approaches to includings assets. Such as generating go code that contains the asset data. The advantage of that approach is that you can also guarantee at compile time that some files will be available. Meaning you can get away with less error handling. In contrast, my asset package can return errors for file operations.

However, with the asset package approach, you can easily switch between local file system operation and embedded zip file operation. Easy for testing (no recompiles) and provides the same interface (i.e. no different code paths during development and production, so fewer surprises).

Comments

edit