Luca Sepe

Using HCL in your Go application

HCL configuration design golang go

What is HCL Config Language Toolkit?

HCL is a toolkit for creating structured configuration languages that are both human and machine friendly, for use with command-line tools, servers, etc.

It includes an expression syntax using variables and functions for more dynamic configuration languages.

HCL-based configuration is built from two main constructs: attributes and blocks.

HCL is the low level syntax of the Terraform language. Terraform usually uses the word “argument” instead of “attribute”.

# Grid rows and columns
rows = 10
cols = 10

# Define the base uri for all the images.
var "baseUri" {
   value = "/Pictures/AWS-Architecture-Icons/PNG"
}

# AWS API Gateway icon
tile "icon" "agw" {
   row = 3
   col = 4
   fit = true
   uri = "${mkURI(var.baseUri, "aws_api_gateway.png")}"
}	

tile "horizontal_line" "hl1" {
   row = "${move("agw", 2, "north")}"
   col = 4
}	

Attributes

An attribute assigns a value to a particular name:

rows = 10

The identifier before the equals sign is the attribute name, and the expression after the equals sign is the attribute's value.

Blocks

Blocks are containers for any other content:

var "baseUri" {
   value = "/Pictures/AWS-Architecture-Icons/PNG"
}
  • a block has a type (var in this snippet)
  • a block type defines how many labels must follow the type keyword - the var block type expects one label, the variable identified, which is “baseUri” in the example above
  • a block type may have zero or any number of labels
  • if a block has labels, then these are required
  • the block body is delimited by the { and } characters
  • in a block body, further arguments and blocks may be nested

Comments

HCL configuration language supports different syntaxes for comments:

  • single-line comment - begins with # or // as alternative
  • multiple lines comment - delimited by /* (start) and */ (end)

Expressions

Expressions are used to compute or refer to values within a configuration.

uri = "${mkURI(var.baseUri, "aws_api_gateway.png")}"
  • the result of an expression is a value whose type can be can be: string, number, bool, tuple or object

If you are curious and want to learn more about how HCL handle different types, please check the go-cty library.

Parsing HCL files in Go

Enough theory! let's see how to parse HCL files containing our custom domain language. I'll refer to the custom HCL file defined above, at the beginning of this post.

Define an helper struct for parsing our HCL file

// rootHCL is the helper struct for parsing our HCL file.
type rootHCL struct {
	Rows       int    `hcl:"rows"`
	Cols       int    `hcl:"cols"`
	Margin     int    `hcl:"margin,optional"`
	Background string `hcl:"background,optional"`

	Variables []*struct {
		Name  string         `hcl:"name,label"`
		Value hcl.Attributes `hcl:"value,remain"`
	} `hcl:"var,block"`

	Tiles []*struct {
		Kind    string   `hcl:"type,label"`
		ID      string   `hcl:"id,label"`
		HCLBody hcl.Body `hcl:",remain"`
	} `hcl:"tile,block"`

}

As you can see, each attribute of our helper struct is tagged - gohcl supports the following keywords:

  • attr: the default, indicates that its value comes from an attribute
  • block: indicates that its value cames from a block
  • label: indicates that its value cames from a block label
  • optional: is the same as attr, but the field can be omitted (attr implies required)
  • remain: indicates that its value cames from the remaining body after populating other fields

Moreover…

  • remain's corresponding HCL type should be hcl.Body or hcl.Attributes
  • all fields are mandatory except those marked with optional
  • if there is no remain field, any attributes or blocks not matched with our helper struct, will cause an error

Instantiate an HCL parser with the source byte slice

Given a buffer (data) with our HCL content and a string (uri) just for debugging purposes:

parser := hclparse.NewParser()

srcHCL, diags := parser.ParseHCL(data, uri)
if diags.HasErrors() {
   return Config{}, fmt.Errorf("error parsing HCL file: %w", diags)
}

All hcl functions will return Diagnostics instead of error, we should check for errors by diags.HasErrors() instead of err != nil.

Rendering Diagnostics into a terminal

The hcl package contains a default implementation for rendering Diagnostics to a terminal:

wr := hcl.NewDiagnosticTextWriter(
    os.Stdout,      // writer to send messages to
    parser.Files(), // the parser's file cache, for source snippets
    78,             // wrapping width
    true,           // generate colored/highlighted output
)
wr.WriteDiagnostics(diags)

Decoding With The Low-level API

HCL decoding in the low-level API has two distinct phases:

  • structural decoding: analyzing the arguments and nested blocks present in a particular body
  • expression evaluation: obtaining final values for each argument expression found during structural decoding

Since HCL supports incremental decoding, let's do a first pass in order to acquire all user variables - parsing var blocks:

// Start the first pass of decoding
var root rootHCL
if diags := gohcl.DecodeBody(srcHCL.Body, nil, &root); diags.HasErrors() {
   return Config{}, fmt.Errorf("error decoding HCL configuration: %w", diags)
}

// Decode all variables
variables := map[string]cty.Value{}
for _, v := range root.Variables {
   if len(v.Value) == 0 {
      continue
   }

   val, diags := v.Value["value"].Expr.Value(nil)
   if diags.HasErrors() {
      return Config{}, fmt.Errorf("error decoding HCL configuration: %w", diags)
   }

   variables[v.Name] = val
}

Expression evaluation

Now we have all user variables but to give our custom HCL a “real power”, we need a Context and eventually some cool function.

Let's create an helper function that build a evaulation context:

// createContext is a helper function that creates 
// an *hcl.EvalContext to be used in decoding HCL. 
// It add all user variables to the eval context 
// and also creates custom functions.
func createContext(vars map[string]cty.Value) (*hcl.EvalContext, error) {
   return &hcl.EvalContext{
      Variables: map[string]cty.Value{
         "var": cty.ObjectVal(vars),
      },
      Functions: map[string]function.Function{
         "mkURI": funcs.MakeURIFunc, 
      },
   }, nil
}

Decoding all the remaining blocks

Using the evaluation context, we can proceed with our decoding:

// Call a helper function which creates an HCL context for use in
// decoding the parsed HCL.
evalContext, err := createContext(variables)
if err != nil {
   return Config{}, fmt.Errorf("error creating HCL evaluation context: %w", err)
}

// Decoding all remaining 'tile' blocks
for _, tile := range root.Tiles {
	switch t := tile.Kind; t {
	case "icon":
		el, err := decodeIcon(tile.HCLBody, evalContext)
		if err != nil {
			return Config{}, err
		}
	case "label":
		el, err := decodeLabel(tile.HCLBody, evalContext)
		if err != nil {
			return Config{}, err
		}
	case "frame":
		el, err := decodeFrame(tile.HCLBody, evalContext)
		if err != nil {
			return Config{}, err
		}
	// ...
	}
}

for the sake of brevity I will show you only one of the decodeXXX methods, they are all the same except for the “struct specialization” (Icon, Label and Frame).

// decodeIcon decode the HCL 'icon' block
func decodeIcon(body hcl.Body, ctx *hcl.EvalContext) (jumble.Icon, error) {
   // Temporary helper struct
   var tmp struct {
      Row int    `hcl:"row"`
      Col int    `hcl:"col"`
      Fit bool   `hcl:"fit,optional"`
      URI string `hcl:"uri"`
   }
   // Perform the decoding
   if diags := gohcl.DecodeBody(body, ctx, &tmp); diags.HasErrors() {
      return jumble.Icon{}, fmt.Errorf("error decoding HCL configuration: %w", diags)
   }
   // do something with the decoded struct ...
   return jumble.Icon{
      Row: tmp.Row, Col: tmp.Col,
      Fit: tmp.Fit,
      URI: tmp.URI,
   }, nil
}

HCL custom Functions

In the expression evaluation section we created an hcl.EvalContext with all the user defined variables and our custom language functions.

Functions are expected to behave as pure functions, and not create any visible side-effects.

Let's see how to create our custom ‘mkURI’ function:


// MakeURIFunc constructs a function that joins a baseUri with a path.
var MakeURIFunc = function.New(&function.Spec{
	// Params is a description of the positional parameters for the function.
	Params: []function.Parameter{
		{
			Name: "elem1",
			Type: cty.String,
		},
		{
			Name: "elem2",
			Type: cty.String,
		},
	},
	// Type is the TypeFunc that decides the return type of the function
	Type: function.StaticReturnType(cty.String),
	// Impl is the ImplFunc that implements the function's behavior
	Impl: func(args []cty.Value, retType cty.Type) (ret cty.Value, err error) {
		var elem1, elem2 string
		if err := gocty.FromCtyValue(args[0], &elem1); err != nil {
			return cty.UnknownVal(cty.String), err
		}
		if err := gocty.FromCtyValue(args[1], &elem2); err != nil {
			return cty.UnknownVal(cty.String), err
		}

		if strings.HasPrefix(elem1, "http") {
			u, err := url.Parse(elem1)
			if err != nil {
				return cty.UnknownVal(cty.String), err
			}
			u.Path = path.Join(u.Path, elem2)

			return cty.StringVal(u.String()), nil
		}

		return cty.StringVal(filepath.Join(elem1, elem2)), nil
	},
})

That's it for now! I hope it was useful to you.

HCL is great. I use it in all my Go commandline tools. Right now I'm writing a 2D command line CAD using HCL!

References