Testing external API calls in Go

Intro

I’ve been writing a lot of Go tests lately. If anyone’s actually been following this blog, you probably remember my writing a lot of Go in my hobby projects. One of my favorite things about Go was how easy it made writing tests. Over the last year and a half I’ve been writing Go both at home and at work, which only made me love Go’s test tooling even more.

One topic I’ve seen come up a few times is how to test external API calls in Go tests. There are two main options I’ve used: wrapping the external calls in a mockable interface and using Go’s native httptest package to start up a test http server. I figured I’d do a quick overview of both here.

Let’s say we want to make a Get request to https://bookbeta.com/api/releases to get a list of all new releases available to book beta readers from our service. You have a method, getAvailableReleases() that makes the call and unmarshals the data into a struct, to then process it and recommend certain releases to the authenticated users (the story isn’t important, the point is that we get the releases, do stuff with them, and return some relevant ones to the caller).

import (
	"encoding/json"
	"fmt"
	"net/http"
)

const (
	apiURL      = "https://bookbeta.com/api/v1"
	releasesURL = apiURL + "/releases"
)

// Let's define a single error for the purposes of this example
var ErrFailedAPICall = errors.New("bad response from BookBeta API")

type Release struct {
	ID          int64  `json:"id"`
	BookName    string `json:"bookName"`
	AuthorName  string `json:"authorName"`
	IsAvailable bool   `json:"isAvailable"`
}

func GetAvailableReleases() ([]Release, error) {
	res, err := http.Get(releasesURL)
	if err != nil {
		return nil, fmt.Errorf("failed to get releases: %v: %w", err, errFailedAPICall)
	}
	defer res.Body.Close()
	if res.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("unexpected status code: %d - %s: %w", res.StatusCode, res.Status, errFailedAPICall)
	}
	var releases []Release
	if err := json.NewDecoder(res.Body).Decode(&releases); err != nil {
		return nil, fmt.Errorf("failed to decode body into release slice: %w", err)
	}
	var availableReleases []Release
	for _, r := range releases {
		if r.IsAvailable {
			availableReleases = append(availableReleases, r)
		}
	}
	return availableReleases, nil
}

We want to write a test for your GetAvailableReleases() method, but we need to simulate the call to the externa API (which we don’t want to do for real in the test…and in some cases can’t - what if the API requires user authentication or some other parameter not available at test-time?)

Method 1: Mockable interface around the external call

You can define an interface that looks something like this:

type Communicator interface {
	GetNewReleases() (*http.Response, error)
}

And then create a small struct that implements Communicator. All it will do is send the same Get request we’re making up above to the BookBeta API:

type bbCommunicator struct {}

func(c *bbCommunicator) GetNewReleases() (*http.Response, error) {
	return http.Get(releasesURL)
}

But now we need to decide where to instantiate that struct. We don’t really want to make it a global variable (well, I don’t). If we make it a global, we won’t be able to run parallel standalone tests because we’ll keep having to override this one global var and get test case pollution.

So I’m going to create another struct that will actually contain the GetAvailableReleases() method inside it, which will also hold the communiator. I’ll then make a function that takes a Communicator and instantiates a bbCommunicator.

// ReleasePopulator recommends new book releases to users.
type ReleasePopulator struct {
	communicator Communicator
}

func NewReleasePopulator() *ReleasePopulator {
	return &ReleasePopulator{communicator: &bbCommunicator{}}
}

So what we’ll end up with is having ReleasePopulator call out to the implementer of Communicator to make the Get request instead of doing it directly as it was above:

func (rp *ReleasePopulator) GetAvailableReleases() ([]Release, error) {
	res, err := rp.communicator.GetNewReleases()
	if err != nil {
		return nil, fmt.Errorf("failed to get releases: %v: %w", err, ErrFailedAPICall)
	}
    ...
    

We can now have GoMock generate a mock implementation of the above interface by either running mockgen manually, or by using a go:generate comment:

//go:generate mockgen --destination="mocks/mock_communicator.go" . Communicator
type Communicator interface {
	GetNewReleases() (*http.Response, error)
}

We can now run go generate ./... to have the mock source generated for us.

Then, in our test, we can mock the call to the external API and its return:

func TestGetAvailableReleases(t *testing.T) {
	t.Parallel()
	cases := []struct {
		name             string
		bbStatusCode     int
		bbBody           string
		wantReleaseCount int
		wantErr          error
	}{
		{
			name:         "bad status code",
			bbStatusCode: http.StatusInternalServerError,
			wantErr:      ErrFailedAPICall,
		},
		{
			name:         "one available book",
			bbStatusCode: http.StatusOK,
			// This example body was retrieved from the BookBeta API docs
			bbBody: `[
			  {
				"id": 2355,
				"bookName": "The Little Giraffe",
				"authorName": "G. Neckton",
				"isAvailable": false
			  },
			  {
				"id": 123,
				"bookName": "The Big Pelican",
				"authorName": "P. Birdster",
				"isAvailable": true
			  }
			]`,
			wantReleaseCount: 1,
		},
	}
	for _, tc := range cases {
		tc := tc
		t.Run(tc.name, func(t *testing.T) {
			t.Parallel()
			mockCtrl := gomock.NewController(t)
			mockCommunicator := mocks.NewMockCommunicator(mockCtrl)
			body := io.NopCloser(strings.NewReader(tc.bbBody))
			mockCommunicator.EXPECT().GetNewReleases().Return(&http.Response{
				StatusCode: tc.bbStatusCode,
				Body:       body,
			}, nil).Times(1)

			rp := NewReleasePopulator()
			rp.communicator = mockCommunicator
			gotReleases, gotErr := rp.GetAvailableReleases()
			require.ErrorIs(t, gotErr, tc.wantErr)
			if gotErr == nil {
				require.Len(t, gotReleases, tc.wantReleaseCount)
			}
		})
	}
}

Above, we define two test cases: one which will test the expected return of the error we defined when the API returns a bad response of any kind and another for a successful response.

We first prepare our mock and tell it what to respond with and how many times it’s expected to be called:

	mockCtrl := gomock.NewController(t)
	mockCommunicator := mocks.NewMockCommunicator(mockCtrl)
	body := io.NopCloser(strings.NewReader(tc.bbBody))
	mockCommunicator.EXPECT().GetNewReleases().Return(&http.Response{
		StatusCode: tc.bbStatusCode,
		Body:       body,
	}, nil).Times(1)

Then, we just create our ReleasePopulator instance with the mock communicator and run the function we’re wanting to test:

	rp := NewReleasePopulator()
	rp.communicator = mockCommunicator
	gotReleases, gotErr := rp.GetAvailableReleases()

Since the test is in the same package we can access the unexported communicator field. But if you prefer putting the test in a separate package you could also do this via an Option or passing the mock to the constructor func, for example. Although that would also expose that override functionality to other callers, which you may or may not want to do.

Afterwards, all that’s left to do is check whether we get the return we expect from the function:

	require.ErrorIs(t, gotErr, tc.wantErr)
	if gotErr == nil {
		require.Len(t, gotReleases, tc.wantReleaseCount)
	}

Method 2: httptest

In some situations, defining an interface for mocking can feel a bit like overkill. What if we want to test this without defining and overriding a Communicator?

Go provides a solution!

We can spin up a local http server and redirect our calls there instead.

This time, though, we will need to allow the ability to override the API URL (to redirect the request to the http server).

Just like we did above, I’ll stick with the ReleasePopulator struct. Only this time instead of holding an implementation of our Communicator interface (which doesn’t exist in this example), we’ll have it hold the API URL:

type ReleasePopulator struct {
	apiURL string
}

func NewReleasePopulator() *ReleasePopulator {
	return &ReleasePopulator{apiURL: defaultAPIURL}
}

We can no longer have releasesURL in a const because it will change depending on the apiURL. So we can rename apiURL to defaultAPIURL and define a small function that takes this URL to construct the releases URL:

const (
	defaultAPIURL = "https://bookbeta.com/api/v1"
)

func releasesURL(apiURL string) string {
	return fmt.Sprintf("%s/releases", apiURL)
}

In GetAvailableReleases(), we make the http call and pass in the relevant URL:

func (rp *ReleasePopulator) GetAvailableReleases() ([]Release, error) {
	res, err := http.Get(releasesURL(rp.apiURL))
	if err != nil {
		return nil, fmt.Errorf("failed to get releases: %v: %w", err, ErrFailedAPICall)
	}
    ...

To test it, we’ll keep the test cases exactly the same as the mocking method above. The only thing that will change is the fact that instead of setting up our mock Communicator, we will spin up a test server for each test case and define a tiny handler returning the data we want:

	testServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(tc.bbStatusCode)
		_, err := w.Write([]byte(tc.bbBody))
		require.NoError(t, err)
	}))

	defer testServer.Close()

We’ll create our ReleaseCommunicator and override the api URL with that of our test server above:

rp := NewReleasePopulator()
rp.apiURL = testServer.URL

And then we can get our releases and do our validation:

	gotReleases, gotErr := rp.GetAvailableReleases()
	require.ErrorIs(t, gotErr, tc.wantErr)
	if gotErr == nil {
		require.Len(t, gotReleases, tc.wantReleaseCount)
	}

Conclusion

In conclusion, Go makes testing fun and easy for the whole family.

© - 2021 · Liza Shulyayeva ·