codehakase's logs

Notes mostly about software engineering and what I’m working on.

Working with JSON in Go

Posted at — May 3, 2021

Summary: I’ve worked with JSON in various programming languages in context to data exchange/communication between applications. In this article, I’ll give a brief overview of the encoding/json package in Go, and point some gotchas I’ve encountered.

JSON (JavaScript Object Notation), is a popular data interchange format commonly used for communication between applications.

Working with JSON in Go is stress-free thanks to the encoding/json package from the standard library.

Encoding and decoding data

To encode or decode data we use the Marshal and Unmarshal functions from the json package. Both methods have the following signatures:

func Marshal(v interface{}) ([]byte, error)
func Unmarshal(data []byte, v interface{}) error

Encoding

Here’s a basic example of retrieving the JSON-encoded value of a Config struct instance:

type Config struct {
  BuildDirPath string
  MaxRefreshRate int
}

c := Config{BuildDirPath: "/bin"}
b, err := json.Marshal(c)
fmt.Println(string(b), err)
// output:
// {"BuildDirPath":"/bin","MaxRefreshRate":0}

By default, the Marshal function will include all zero-value exported fields as it treats the field names as the key of the JSON element. The code snippet above elaborates this – the MaxRefreshRate field had no value set but the JSON-encoded output included the field set to zero.

Custom JSON keys

From the previous code snippet, the JSON-encoded output uses the name of the struct fields. To control this we use the struct tag literal – here’s an example:

type Config struct {
  BuildDirPath string `json:"build_dir_path"`
  MaxRefreshRate string `json:"max_refresh_rate"`
}

Printing the JSON-encoded instance of the update Config struct will yield the following:

{"build_dir_path":"/bin","max_refresh_rate":0}

Decoding

To decode JSON data we need to define a data structure (struct or map) where the value will be decoded to. Here’s what it looks like:

configJSON := `{"build_dir_path":"/bin","max_refresh_rate":0}`
// create struct to decode JSON string to
var config Config

err := json.Unmarshal([]byte(configJSON), &config)
// handle err
Destination field identification

Unmarshal will decode only the fields that it can find in the destination type (config). In our example, only the build_dir_path and max_refresh_rate field of m will be populated, ignoring any other fields not defined in the destination type.

The Unmarshal function identifies each JSON key on the destination struct by checking for: - An exported field with a json tag of KEY - An exported field named KEY or - An exported field matching KEY (case-insensitive)

Decoding an unknown JSON structure

There are cases where you wouldn’t know the structure of the JSON data you’re decoding, or the data source is subject to changing its structure – telemetry data for instance. We can decode the JSON data into an interface{} or map[string]interface{} value, and use type assertions to acess the underlying data structures. Here’s an example:

unknownData := `{"status": "online", "available_items": 200, "stats": [{"p1":
"-2", "p2": "0"}]}`
var data interface{}
_ = json.Unmarshal([]byte(unknownData), &data)

// access underlying data
m := f.(map[string]interface{})
status := m["status"]
fmt.Println(status) // online

HTTP Handler Example

The json package enables encoding and decoding json over HTTP hassle-free. Here’s an example of a HTTP POST request handler.

type BoxhubSync struct {
  IP string `json:"ip"`
  LastSyncUpdate string `json:"last_sync_update"`
  // ... other fields omitted for brevity
}

func readBoxhubSyncRequest(w http.ResponseWriter, r *http.Request){
  var reqArgs BoxhubSync
  err := decodeReqBody(r, &reqArgs)
  if err != nil {
    log.Fatal(err)
  }

  // process reqArgs

  // send response to client
  resJSON(w, http.StatusCreated, map[string]interface{}{
    "status": "success",
    "message": "sync requested processed",
  })
}

// decodeReqBody decodes from [r.Body] to [v]
func decodeReqBody(r *http.Request, v interface{}) error {
  defer r.Body.Close()
  b, _ := ioutil.ReadAll(r.Body)
  err := json.Unmarshal(b, v)
  if err != nil {
    return err
  }
  r.Body = ioutil.NopCloser(bytes.NewBuffer(b))
  return nil
}

// resJSON writes a response to the http client
func resJSON(w http.ResponseWriter, code int, payload interface{}) {
  res, err := json.Marshal(payload)
  if err != nil {
    log.Fatal(err)
  }
  w.Header().Set("Content-Type", "application/json")
  w.WriteHeader(code)
  w.Write(res)
  return
}

JSON Streaming

The json package provides NewEncoder and NewDecoder functions to encode/decode JSON data from/to a stream. This is helpful if your data is coming from an io.Reader stream (eg. HTTP request body), or you need to decode multiple values from a stream of data.

Encoder

The json.NewEncoder function returns a new encoder that writes to a io.Writer stream:

func NewEncoder(w io.Writer) *Encoder

The Encoder.Encode function is responsible for writing JSON-encoded data to the stream. Each time the Encode function is called, JSON is marshalled from v and appended to the io.Writer with a trailing newline.

func (enc *Encoder) Encode(v interface{}) error

Here’s an example of how to encode JSON data to a buffer:

func main() {
  buf := new(bytes.Buffer)
  config := Config{
    BuildDirPath: "/bin",
    MaxRefreshRate: 80,
  }
  err := json.NewEncoder(buf).Encode(u)
  if err != nil {
    log.Fatal(err)
  }
  fmt.Print(buf.String())
  // output:
  {"build_dir_path":"/bin","max_refresh_rate":80}
}

Decoder

The json.NewDecoder function returns a new decoder that reads from an io.Reader.

func NewDecoder(r io.Reader) *Decoder

The Decoder.Decode function decodes the JSON-encoded data. Decode reads the next JSON-encoded value from its input and stores it in the value pointed to by v.

func (dec *Decoder) Decode(v interface{}) error

Here’s an example of how to decode JSON-data from a Stdin stream:

func main() {
	decoder := json.NewDecoder(os.Stdin)
	output := make(map[string]interface{})
	for {
		err := decoder.Decode(&output)
		if err != nil {
			log.Fatal(err)
		} else {
			fmt.Println("JSON output:")
			fmt.Println(output)
			output = make(map[string]interface{})
		}
	}
}

Running the above snippet and passing JSON to Stdin will print the decoded version:

# sample json gotten from: https://jsonapi.org/examples
$ go run stream.go
{
  "data": [{
    "type": "articles",
    "id": "1",
    "attributes": {
      "title": "JSON:API paints my bikeshed!",
      "body": "The shortest article. Ever.",
      "created": "2015-05-22T14:56:29.000Z",
      "updated": "2015-05-22T14:56:28.000Z"
    },
    "relationships": {
      "author": {
        "data": {"id": "42", "type": "people"}
      }
    }
  }],
  "included": [
    {
      "type": "people",
      "id": "42",
      "attributes": {
        "name": "John",
        "age": 80,
        "gender": "male"
      }
    }
  ]
}

JSON out:
map[data:[map[attributes:map[body:The shortest article. Ever. created:2015-05-22T14:56:29.000Z title:JSON:API paints my bikeshed! updated:2015-05-22T14:56:28.000Z] id:1 relationships:map[author:map[data:map[id:42 type:people]]] type:articles]] included:[map[attributes:map[age:80 gender:male name:John] id:42 type:people]]]

Read and Write JSON to file

I’ve needed to read and write json to and from a file. This can come in handy when you have some sort of state stored in files you want to read and write to. Here’s an example of reading and writing a config file:

// using Config struct from previous examples
var config Config

// read file
b, err := io.ReadFile("path/to/config.json")
if err != nil {
  log.Fatal(err) // handle err
}

err = json.Unmarshal(b, &config)
if err != nil {
  log.Fatal(err)
}

// write file
config.MaxRefreshRate = 300
b, err := json.Marshal(config)
if err != nil {
  log.Fatal(err)
}

err = io.WriteFile("path/to/config.json", b, FILE_PERM)
if err != nil {
  log.Fatal(err)
}

JSON Gotchas in Go

While working with JSON inputs, I’ve come across things that surprised or confused me, some of which I’ve read about on the encoding/json documentation. I’ll list a few I’ve noted down below:

Unexported fields can’t have data decoded as values

When decoding JSON data to a struct, if there are fields unexported by the struct present in the json blob, the field is skipped:

type User struct {
  FirstName string `json:"first_name"`
  LastName string `json:"last_name"`
  photoURL string `json:"photoURL"`
}

data := `{"first_name": "Jane", "last_name": "Doe", "photoURL": "xx.io/xx"}`
var user User
_ = json.Unmarshal([]byte(data), &user)
fmt.Println(user)

// output:
{Jane Doe }

Nil and empty slices are decoded differently

A JSON-encoded nil slice value is null, while for an empty slice, is an empty JSON array:

d := map[string]interface{}{
  "nil_slice": []int,
  "empty_slice": []int{},
}

// encoding d will output:
{"empty_slice":[],"nil_slice":null}

Maps are sorted alphabetically

Encoding a map to JSON will sort its keys alphabetically:

d := map[string]interface{}{
  "status": "success",
  "code": 201,
  "0": nil,
}

// output:
{"0":null,"code":201,"status":"success"}

[]byte is encoded as base64 string

When encoding a []byte to JSON, the value is converted to a base64-encoded string:

d := map[string]interface{}{
  "public_key": []byte("public_key"),
  ...
}

// output:
{"public_key":"cHVibGljX2tleQ=="}

Trailing zeros are removed from floats

When encoding a floating-point number, any trailing zeroes will not appear in the JSON-encoded value:

d := map[string]interface{}{
  "starting_balance": 80.0,
  "upgrade_perc": 30.23,
  ...
}

// output:
{"starting_balance":80,"upgrade_perc":30.23}

Decoding numbers to interface{} resolves to float64

When decoding a JSON number into an interface{}, the value will have the type float64:

data := `{"type": "tx", "amount": 230}`
var out map[string]interface{}
_ = json.Unmarshal([]byte(data), &out)

fmt.Printf("%T", out["amount"]) // float64

Wrapping Up

There’s much more to say about the json package and its various applications, but hopefully, this is a helpful introduction. To learn more, I highly recommend the package documentation found HERE.

Let me know your thoughts or feedback, or send ideas for improvements. You can also go the discussions on Twitter, [Reddit](), and [Hacker News]()

comments powered by Disqus