Real-World Go Reflection with `flags`

by Ulysse Carion 23 Jul 2020

This article is a brief little exploration of Golang’s reflection system (exposed through the reflect stdlib package). We’re going to add a flag:"..." tag you can put on struct members, and then use reflection to automatically convert those into CLI flags using the stdlib’s flag package.

In other words, we’ll make a function that lets you generate from this:

type config struct {
  FirstName string `flag:"first" usage:"first name"`
  LastName  string `flag:"last" usage:"last name"`
}

func main() {
  // These default values should be shown to the user if they do --help.
  config := config{
    FirstName: "john",
    LastName:  "doe",
  }

  LoadFlags(&config) // LoadFlags is the function we'll write in this article
}

Some code like this:

// This is what the LoadFlags above "expands" into:
flag.StringVar(&config.FirstName, "first", "john", "first name")
flag.StringVar(&config.LastName, "last", "doe", "last name")

Simple enough, and pretty convenient, too – just a small function that makes the flag stdlib package much more usable at scale, when you start to have services with 10+ config options.

Baby steps

Let’s just get started simple — trivially simple. If you want to follow along, let’s put this in a main.go.

package main

import (
	"flag"
	"fmt"
)

func main() {
	config := struct {
		FirstName string `flag:"first" usage:"first name"`
		LastName  string `flag:"last" usage:"last name"`
	}{
		FirstName: "john",
		LastName:  "doe",
	}

	LoadFlags(config)
	flag.Parse()
}

func LoadFlags(config interface{}) {
	fmt.Println(config)
}

Now you can run this as:

go run main.go --help

{john doe}
Usage of /var/.../exe/main:
exit status 2

Ok, so pretty useless so far. But this is the starting point. Now that we have an instance of config passed in, let’s begin exploring what we can do with it.

Introducing `reflect.ValueOf`

Almost always, your entrypoint into the reflect system is going to be reflect.ValueOf, which gives you an instance of Value.

Let’s have our LoadFlags method dump information on the config it’s given:

func LoadFlags(config interface{}) {
	fmt.Printf("%#v\n", config)
	fmt.Printf("%#v\n", reflect.ValueOf(config))
}

go run main.go --help

struct { FirstName string "flag:\"first\" usage:\"first name\""; LastName string "flag:\"last\" usage:\"last name\"" }{FirstName:"john", LastName:"doe"}
struct { FirstName string "flag:\"first\" usage:\"first name\""; LastName string "flag:\"last\" usage:\"last name\"" }{FirstName:"john", LastName:"doe"}
Usage of /var/.../exe/main:
exit status 2

Whoops, that’s not very useful. This is one of the most annoying aspects of reflect if you’re a fan of fmt.Println-driven-development. Printing a Value always returns the same thing as printing the underlying thing it’s wrapping. Value is basically useless for printing if you’re trying to understand reflect, because it’s a documented feature of fmt that it always “unwraps” a reflect.Value:

If the operand is a reflect.Value, the operand is replaced by the concrete value that it holds, and printing continues with the next rule.

We’re just gonna have to live with that. Let’s press on.

Using `Field` and `NumField`

At some point in making this package, we’re going to need to be able to extract the john and doe string values we passed as input to LoadFlags, because we’re going to pass those as the default values to flag.StringVar. Let’s use reflect to iterate over all the fields in a struct.

To find what you need out to reflect, having some understanding of Go’s formal terminology is helpful. In a struct like:

type Foo struct {
  A string
  B int
  C bool
}

The A string, B int, and C bool are called “fields”. So we should look in reflect.Type for field-related stuff, and find some promising options:

Slightly confusingly, Field is “by index” just as much as FieldByIndex is. Basically, FieldByIndex is mostly just a loop that goes like:

for _, i := range index {
  v = v.Field(i)
}

In other words, it behaves like a utility function, except under the hood it will handle dereferencing pointers inside structs for your convenience.

The other methods are about field names, which we don’t have in advance. So let’s focus on Field. Unfortunately, Field takes an index but gives no indication of what the valid indexes are. However, a simiarly-named method on reflect.Type, which we’ll see again later, has better docs:

    // Field returns a struct type's i'th field.
    // It panics if the type's Kind is not Struct.
    // It panics if i is not in the range [0, NumField()).
    Field(i int) StructField

So NumField seems to be the key. Thankfully, there’s a NumField on Value too, so let’s try to use it:

func LoadFlags(config interface{}) {
	v := reflect.ValueOf(config)
	for i := 0; i < v.NumField(); i++ {
		fmt.Println(v.Field(i))
	}
}

# I'm going to stop showing this invocation from now on. It'll stay the same
# unless noted otherwise.
go run main.go --help

john
doe
Usage of /var/.../exe/main:
exit status 2

It works! Now, note well: those outputted lines correspond to Value instances, not plain old string instances. It’s just that fmt hides that from us.

So fow now, we have the “default” values of our flags. We still need to figure out the names of the flags we want to pass to flag.StringVar, as well as their usage strings.

Introducing `reflect.Type`

Let’s try to see if we can figure out how to print out FirstName and LastName given config – in other words, figure out how to get the type of struct that config is, and then iterate over its field names. This isn’t quite what we ultimately need — we’ll want to read the first out of flag:"first", but it’s a start.

It turns out that a Value doesn’t hold information on its members directly. Instead, you need to use reflect.Type, using the Value.Type method. Let’s just print out config's Type to see what we’re dealing with:

func LoadFlags(config interface{}) {
	fmt.Printf("%#v\n", config)
	fmt.Printf("%#v\n", reflect.ValueOf(config).Type())
}

struct { FirstName string "flag:\"first\" usage:\"first name\""; LastName string "flag:\"last\" usage:\"last name\"" }{FirstName:"john", LastName:"doe"}
&reflect.rtype{size:0x20, ptrdata:0x18, hash:0x4ed3bd54, tflag:0x2, align:0x8, fieldAlign:0x8, kind:0x19, equal:(func(unsafe.Pointer, unsafe.Pointer) bool)(0x10aa930), gcdata:(*uint8)(0x10f7e70), str:104632, ptrToThis:0}

Following the lead we already started in the previous section, let’s iterate from 0 to NumField() - 1, but this time using Type instead of Value, to see what we have to play with:

func LoadFlags(config interface{}) {
	v := reflect.ValueOf(config)
	t := v.Type()

	for i := 0; i < t.NumField(); i++ {
		fmt.Println(t.Field(i))
	}
}

{FirstName  string flag:"first" usage:"first name" 0 [0] false}
{LastName  string flag:"last" usage:"last name" 16 [1] false}

Look at that, we have great information here! We can see FirstName and LastName, as well as their types (string) and the value we put in the struct tags, aka the “backticks”.

So here’s our cutesy field-name-iterator working:

func LoadFlags(config interface{}) {
	v := reflect.ValueOf(config)
	t := v.Type()

	for i := 0; i < t.NumField(); i++ {
		fmt.Println(t.Field(i).Name)
	}
}

FirstName
LastName

So far, nothing useful yet, but we’re definitely cooking.

Parsing struct tags

Ever wondered how the encoding/json package understands how to parse out the foo in a struct like:

type User struct {
  Thing string `json:"foo"`
}

The answer is almost disappointingly simple. It iterates over the fields just like we do, and then it gets the Tag property of a StructField. Tag is just an alias over string, but has a convenience method assuming the tag is formatted in the conventional way:

// This prints "foo"
fmt.Println(reflect.StructTag(`json:"foo"`).Get("json"))

So encoding/json does Get("json"). But we need two things: the flag’s name (flag), and the flag’s usage (usage). Let’s update our field-iterator to print that information:

func LoadFlags(config interface{}) {
	v := reflect.ValueOf(config)
	t := v.Type()

	for i := 0; i < t.NumField(); i++ {
		f := t.Field(i)
		fmt.Println(f.Tag.Get("flag"), f.Tag.Get("usage"))
	}
}

first first name
last last name

Perfect! So now we have the flag names, flag usages, and flag default values. All we have left to figure out is how to to pass the first argument to flag.StringVar – the pointer to a string.

Converting back into a `*string`

Without resorting to using unsafe, the typical way you convert data back out of Value is with the Interface method, which returns an ordinary interface{} with the Value's underlying data in it. Then, you can cast that data into the type you need with the usual method (i.e. the v.(T) syntax).

If we just do the obvious thing though, you’ll run into some issues:

func LoadFlags(config interface{}) {
	v := reflect.ValueOf(config)
	t := v.Type()

	for i := 0; i < t.NumField(); i++ {
		fieldT := t.Field(i)
		fieldV := v.Field(i)

		name := fieldT.Tag.Get("flag")
		usage := fieldT.Tag.Get("usage")
		value := fieldV.Interface().(string)
		addr := fieldV.Interface().(*string)

		flag.StringVar(addr, name, value, usage)
	}
}

panic: interface conversion: interface {} is string, not *string

You could probably tell this panic would happen, just from the fact that we’re casting fieldV.Interface() into both a string and a *string. But if we instead return a pointer to the casted interface, that won’t do the right thing:

func LoadFlags(config interface{}) {
	v := reflect.ValueOf(config)
	t := v.Type()

	for i := 0; i < t.NumField(); i++ {
		fieldT := t.Field(i)
		fieldV := v.Field(i)

		name := fieldT.Tag.Get("flag")
		usage := fieldT.Tag.Get("usage")
		value := fieldV.Interface().(string)
		addr := fieldV.Interface().(string)

		flag.StringVar(&addr, name, value, usage)
	}
}

Because if we print out our config at the end of main, and then call our program like this:

go run main.go -first foo

We get the wrong output. FirstName isn’t overridden.

{john doe}

That’s because the pointer we passed just updates the address of the data we allocated in the LoadFlags, not the data in config. We need to have the reflect system give us a pointer to the data we got passed in.

Promisingly, Value has a promising method named Addr that we can try. So let’s try that:

func LoadFlags(config interface{}) {
	v := reflect.ValueOf(config)
	t := v.Type()

	for i := 0; i < t.NumField(); i++ {
		fieldT := t.Field(i)
		fieldV := v.Field(i)

		name := fieldT.Tag.Get("flag")
		usage := fieldT.Tag.Get("usage")
		value := fieldV.Interface().(string)
		addr := fieldV.Addr().Interface().(*string)

		flag.StringVar(&addr, name, value, usage)
	}
}

This time, we get an error on the line where we do Addr():

panic: reflect.Value.Addr of unaddressable value

What’s going on here that reflect actually re-implements a lot of compile-time logic around whether a value “addressable”, except at runtime. So it will actually follow with the punches of how you got a Value (in our case, via ValueOf then Field), and knows whether the equivalent non-reflect code would let you take the address of the variable.

This is why, for example, json.Unmarshal requires a pointer. In other words, you always call Unmarshal like this:

var data []string
json.Unmarshal(input, &data) // pass a *pointer* to Unmarshal

Under the hood, encoding/json is plain Golang code doing a bunch of reflect. So even though the Unmarshal signature takes an open-ended interface{}, it only actually works if you give it a pointer. That’s because of the exact same limitation we’re running into here.

The fix is to change our call to LoadFlags from:

LoadFlags(config)

Into:

LoadFlags(&config)

Then, we can do an Elem call to the result of ValueOf. Elem corresponds to dereferencing a pointer, and if you Elem something you can usually Addr back out of it later. So our code becomes:

func LoadFlags(config interface{}) {
	v := reflect.ValueOf(config).Elem()
	t := v.Type()

	for i := 0; i < t.NumField(); i++ {
		fieldT := t.Field(i)
		fieldV := v.Field(i)

		name := fieldT.Tag.Get("flag")
		usage := fieldT.Tag.Get("usage")
		value := fieldV.Interface().(string)
		addr := fieldV.Addr().Interface().(*string)

		flag.StringVar(addr, name, value, usage)
	}
}

This code actually works. The help text works fine:

go run main.go --help

Usage of /var/.../exe/main:
  -first string
    	first name (default "john")
  -last string
    	last name (default "doe")

And setting flags works too:

go run main.go -first foo

{foo doe}

The rest is considerably less interesting, but to flesh this out into a really useful package, you’d probably want to add:

Support for all types that flag supports: bool, float64, int, uint, int64, uint64, and time.Duration.
Support for structs within structs.
Support for inferring a flag name from a field name, even if there isn’t a flag tag. Just like how encoding/json can infer one.
Support for skipping a field if it has a flag:"-", similar to how encoding/json lets you do that.

If you have a desire for such a package, or are curious what fleshed-out code and tests for such a thing looks like, check out:

https://github.com/ucarion/structflag

That’s all I got. Hope this helps.

Real-World Go Reflection with flags

Baby steps

Introducing reflect.ValueOf

Using Field and NumField