XML To JSON in Golang Demystified // Keiran.SCOT

There comes a point in every developer’s life where XML data is painfully unavoidable to work with. RSS Feeds try to make this easier but everyone prefers good ol’ JSON, unless you are a sadist.

In Golang we can attack this in a number of ways like using an XML to JSON Library, but why not use the Go standard libraries and save some vendoring issues.

Building our own XML to JSON function

I was recently working on something that required a Medium RSS feed published to a website and thought why not use a GoLang function running on OpenFaaS to accomplish this.

Create a new function

To create our new function we need to use the OpenFaaS CLI

$ faas new medium2json --lang=go

Now we have a super simple Golang handler for OpenFaaS.

Import our Libraries

There are a few libraries required by our function. So let’s import them. Open *function/handler.go *and add the imports under package

1
2
3
4
5
6
7
8


import (
	"encoding/json"
	"encoding/xml"
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
)

You will notice I added net/http so let’s make use of it.

GETing our RSS Feed

Regular Medium users will know that an RSS feed is exposed for each user at https://medium.com/feed/@affix

We can utilize this feed to get what we need using a simple HTTP GET with the net/http package in Go. Let’s add it to our Handle function:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


func Handle(req []byte) string {
	resp, err := http.Get("https://medium.com/feed/@" + string(req)) // Use the 'req' variable to build our feed URL

        // Standard Error Handler
	if err != nil {
		log.Fatal(err)
	}
  
	defer resp.Body.Close() // Defer closing until the Handle function returns
	body, _ := ioutil.ReadAll(resp.Body) // Read the response body and ignore errors
}

The above code is pretty simple, but I always get asked about the defer statement.

What exactly can happen if you don’t defer?

The answer is pretty simple. Without defering the Close(), you create a resource leak. While the application is running, the resource used to create the request can never be closed and the file descriptor will not be released.

Do I need to Close the Request?

If the Body is not both read to EOF and closed, the Client’s underlying RoundTripper (typically Transport) may not be able to re-use a persistent TCP connection to the server for a subsequent “keep-alive” request.

Parse that XML

Now that we have some data, we should parse it. But before we do that we need a structure to unmarshal the data into.

In the handler.go file above the function lets create a struct to match the data we require. The beauty of the GoLang unmarshal function allows us to specify in the struct only the fields we require.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


type PostData struct {
	Channel struct {
		Posts []struct {
			Title       string   `xml:"title"`
			Link        string   `xml:"link"`
			Category    []string `xml:"category"`
			Creator     string   `xml:"creator"`
			PubDate     string   `xml:"pubDate"`
			Updated     string   `xml:"updated"`
			License     string   `xml:"license"`
			Encoded     string   `xml:"encoded"`
			Description string   `xml:"description"`
		} `xml:"item"`
	} `xml:"channel"`
}

// Handle a serverless request
func Handle(req []byte) string {
  //...
}

If you look at the XML returned by the Medium Feed, you will notice I have omitted a lot of unnecessary fields like Channel.Title and Channel.Image so the xml.Unmarshal function will not attempt to add those fields to the struct.

You may also notice I have a Posts []struct that doesn’t appear on the RSS Feed. This comes from the items element of the XML Document specified at the end of the struct with xml:”item”* *the unmarshal function will take all item elements and parse them into the Posts []struct.

So let’s parse that data into the struct in the Handle function:

1
2
3
4
5


func Handle(req []byte) string {
	// ...
	data := &PostData{} // Create and initialise a data variable as a PostData struct 
	err = xml.Unmarshal(body, data) // Parse the XML into the structure
}

The Unmarshal function parses the XML-encoded data and stores the result in the value pointed to by v, which must be an arbitrary struct, slice, or string. Well-formed data that does not fit into v is discarded.

Because Unmarshal uses the reflect package, it can only assign to exported fields. Unmarshal uses a case-sensitive comparison to match XML element names to tag values and struct field names.

Convert to JSON and Return

Now we have our structure we can Marshal it to JSON.

1
2
3
4
5



func Handle(req []byte) string {
	json, _ := json.Marshal(data.Channel.Posts) // Encode our Struct as JSON
	return fmt.Sprintf("%s", string(json)) // Return a String to the OpenFaaS Gateway
}

Marshal returns the JSON encoding of the input interface.

Marshal traverses the value v recursively. If an encountered value implements the Marshaler interface and is not a nil pointer, Marshal calls its MarshalJSON method to produce JSON.

Since the Marshal function produces a []byte we need to convert this to a string when returning from our Handle function using string().

Deploy to OpenFaaS

Now we can deploy our function to our OpenFaaS cluster:

$ faas up -f medium2json.yml

$ faas up -f medium2json.yml [0] > Building medium2json. Clearing temporary build folder: ./build/medium2json/ Preparing ./medium2json/ ./build/medium2json//function Building: affixxx/medium2json:latest with go template. Please wait.. Sending build context to Docker daemon 7.68kB

Once you have your function deployed you can test it using curl:

curl -X POST -d “affix” [https://gateway.faas.kfj.io/function/medium2json](https://gateway.faas.kfj.io/function/medium2json)

curl -X POST -d “affix” https://gateway.faas.kfj.io/function/medium2json [{“Title”:”#FaasFriday — Building a Serverless Microblog…

The above URL is currently deployed, feel free to see the results using it.

TLDR; The Full Function Code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41



package function

import (
	"encoding/json"
	"encoding/xml"
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
)

type PostData struct {
	Channel struct {
		Posts []struct {
			Title       string   `xml:"title"`
			Link        string   `xml:"link"`
			Category    []string `xml:"category"`
			Creator     string   `xml:"creator"`
			PubDate     string   `xml:"pubDate"`
			Updated     string   `xml:"updated"`
			License     string   `xml:"license"`
			Encoded     string   `xml:"encoded"`
			Description string   `xml:"description"`
		} `xml:"item"`
	} `xml:"channel"`
}

func Handle(req []byte) string {
	resp, err := http.Get("https://medium.com/feed/@" + string(req))

	if err != nil {
		log.Fatal(err)
	}
	defer resp.Body.Close()
	body, _ := ioutil.ReadAll(resp.Body)
	data := &PostData{}
	err = xml.Unmarshal(body, data)
	json, _ := json.Marshal(data.Channel.Posts)
	return fmt.Sprintf("%s", string(json))
}