Adding context and options to your Go client package

This is a follow-up on my previous post where I showed how you can write user-friendly client packages for REST APIs that will make it even more enjoyable for users to integrate with your services. And for many services, the patterns I showed will probably suffice. This time though, we’ll take a look at two additional patterns. First, I’ll show you how you can bring your client package up-to-date by adding support for the context package. Then we’ll look at how we can pass optional parameters for things like pagination and filtering. Let’s jump right in.

Cancelation and deadlines

The context package was introduced in Go 1.7 and offers request-scoped values, cancelation signals and deadlines. By adding the context object to our client methods the users maintain control even when a request runs too long. Now, there are already quite a few articles that explain the details better than I’ll be able to. I like this one by Jack Lindamood. And of course, there’s the official blog post.

Since the inclusion to the standard library, any function doing any networking shenanigans should take a context object as its first argument, so why don’t we take a look at how to add this to an already existing client method? I’m going to borrow one of the examples from the previous post, namely one that lists users from a fictional chat service. The only real change we need to make, is to change the method to take a context object as an argument and then pass it on to our do method:

The do method is a helper method for making API requests to our service. We’ll change it to take a context object as well. Next thing we need to do is to wrap our request with the context object, using the WithContext method, before we send it to the HTTP client. If the request timed out or was canceled, the HTTP client will return context.DeadlineExceeded or context.Canceled. Also, if the channel returned by ctx.Done() is closed, ctx.Err() will indicate why the context was canceled. In this case, we return that error instead of the one returned from the HTTP client, as it’s likely to be more informative:

Note: In the pre-context era, you would cancel a request by calling CancelRequest on the Transport. Since the introduction of WithContext, this has now been deprecated, which is also mentioned in the docs.

Optional parameters

Many REST APIs support pagination and filtering through optional query parameters, and so we’d like to have a way of letting users define the ones they want. While you could pass them all as argument, because they’re optional there’s going to be a lot of parameters that the user simply won’t care about.

So, instead of dealing with each option separately, we’ll send a struct to our method, encode the fields as query parameters and then append them to the URL that we’re using to build the request. While you could encode each field manually, I like the google/go-querystring package, which encodes struct fields that have a url tag, into query parameters. Like this one:

This struct can then be encoded into something like page=2&per_page=10. We can use this to create a helper function that appends the options to a URL:

This way, we can reuse it in all of our client methods; let’s go back and modify the ListUsers method so that it supports optional parameters. The only thing we need is to add an additional argument for passing an options object, and then wrap the URL path using our new addOptions helper function:

Putting it all to use

Finally, let’s take a look at how the users would interact with our new and improved API:

With this, your users can now explicitly check if the request timed out and deal with it accordingly. Along with the added support for options, and the patterns outlined in the previous post, we’ve covered the essentials of writing client packages in Go. Using what I’ve showed you, you’ll be able to create a modern, idiomatic and easy-to-use API that is easy to extend even as your RESTful API grows.

If you have any other patterns or suggestions on writing client packages, please share them in the comment section!

Writing a Go client for your RESTful API

(This article was originally posted on Medium)

If you are providing a RESTful API for your product, you are already doing developers around the world a huge favor by enabling them to do amazing things, things that you may not even had imagined were possible. By providing your users with a client SDK in their favorite language you are making it even easier to integrate their services with yours. In this blog post I will show you how to write your own client package in Go!

The example we will be using throughout this post is a generic chat service — we will call it The Great Chatsby — which resembles a chat service you might already be using in your organization. The Great Chatsby provides a RESTful API that allows us to create channels and send chat messages to them. It also supports listing messages sent to a channel as well as all the registered users. For the remainder of this post we will write a client package, github.com/thegreatchatsby/chatsby-go, so that developers can write integrations with their products.

Your first Go client SDK

First off, you are going to need a Client struct to hold information about where to find the API you are going to consume. It will also contain a reference to the http.Client used to make request to our API.

Pro tip: By making the base URL configurable we can make it testable by passing the URL of a httptest.Server.

Next, let us create a method for listing users. We create a complete URL from our base URL and relative path, and use it to build a request that we can send using our http.Client.

Note: Keep in mind that the Transport is only able to reuse the HTTP connection if the response body has been drained completely. We can ensure this is the case by calling io.Copy(ioutil.Discard, resp.Body) right before resp.Body.Close is called. Check out this thread to learn more.

Edit: As pointed out by some, in practice you probably want to use io.CopyN in order to not drain an unbounded number of bytes. Note though that draining the body is really only necessary if you can realistically expect junk data to follow your response.

When we implement the rest of the API operations however, we see that there are some things that we can extract to their own methods. By extracting our own newRequest() and do() methods, we can reuse them for all our API operations:

You can even provide Go methods for each HTTP method as a further convenience. Depending on how consistent your requests and responses are, you can conveniently encode/decode them in the newRequest() and do()methods, or where you do the actual request. When we are done, the complete API will look something like this:

Often, a Client with a bunch of methods is all you are going to need, especially if the REST API is fairly small. Both docker/docker and nlopes/slack are examples of two decent-sized API clients that are taking this pattern to the limit. However, when the API grows larger and we start exposing several services, you might want to consider using service objects.

Service objects

At some point, you are going to want to break up your package into multiple files, e.g. client.go, chat.go, channels.go, and users.go. docker/docker deals with this by communicating services in the filename, e.g. container_list.go, and by prefixing each method with the service name, e.g. client.ContainerList() (check out their Client object on godoc.org).

If you do not like spreading methods across multiple files, another approach is to break down your API into service objects that group related operations. Each service object would then contain a reference to the Client. This approach is used by for example google/go-github.

A common practice is to have service objects as fields in the Client. When creating the client, you can then pass a reference to the Client to each service object in the NewClient function:

Configuring the http.Client

Depending on how customizable you want the API to be, you can go for simply passing an http.Client, or you could go for the functional optionsapproach, letting your users pretty much customize all they want. For example, if you provide a library for a hosted service you might rely on a defaultEndpoint, while if you are talking to an on-site installation you might provide a func BaseURL(u string) func(*Client) option.

Regardless of the approach you go with, you should provide a means of letting users configure their own http.Client, e.g. setting sensible timeout values. This also allows you to handle authentication outside your Client. Here is an example from google/go-github of how you can authenticate using OAuth2 access tokens.

Pro tip: If you have several ways of letting users authenticate themselves, consider providing helper functions for creating authenticated http.Clients.

Keeping a reference to a http.Client lets us handle authentication, timeouts and other things independently from your API. Allowing your users to pass their own HTTP client used by your library lets them configure settings that make sense in their context.

Tips and tricks

Dealing with errors

Arguably the simplest way to deal with errors coming from you API is to simply check whether the status code in the response is anywhere between 200≤ and <400, or otherwise return an error.

Depending on how your RESTful API returns errors, consider converting them into proper Go errors instead of generic API errors, e.g. check for 404 and instead return a instance of ErrChannelNotFound. Also make sure to document that ErrChannelNotFound is returned when channel is not found. This makes for a more pleasant and consistent experience for your users.

Pointer fields

To distinguish between unset fields and zero-value fields, structs can use pointer values.

This can be helpful but it can also make for a rather unsightly code when using it. In this case make sure to provide helper functions for creating pointers to common types.

This is obviously more tedious than using non-pointer values so do this only if it makes sense for your API. Check out google/go-github and aws/aws-sdk-go to see this in action. Compare with tambet/go-asana where values are just as effective.

Summary

Go makes it easy writing production-ready client packages that allow your users to consume your RESTful services safely and efficiently from their Go applications. In this blog post you have seen how you can get started writing your own. If you have experience writing client SDKs in Go and you feel I left something out, please let me know by commenting on this post!

Togetherness

You know that feeling when you’re part of a focused team working together towards a common goal? It’s great. It’s productive and engaging. But it’s kind of hard to get there.

Maybe it’s because it requires quite a few things. For instance, the possibility to focus on one thing at a time, with few disturbances. This can be difficult to achieve working in an open-plan office space within a company managing a few of the most visited web sites in Sweden. People tend to make sounds, technology tends to brake down, problems arises from seemingly nowhere and on top of that there’s a future to plan. Also you’re often requested to estimate how long something you’ve never done before will take. Etc.

So we try to find ways to make this manageable. We apply different processes, organize and reorganize, invest in new technology, use deck of cards to help us estimate. Etc.

Da Mob

At the moment many of us at Bonnier Broadcasting think that the best tool at hand right now is mob programming. The web teams take on it is simple. A group of people. One computer. One screen, a really big one. One problem. And then we attack it. From start to production.

We’ve created a few mob spaces where we manage to be part of the open-plan, but still isolated enough to keep an all-day conversation going without upsetting our co-workers too much.

We enforce hard restrictions on our business surrounding when it comes to prioritizing. We can handle one thing a time, pick the one you wish for the most. That is of course not entirely true. We are still responsible for a fairly complex product that needs tending to. Meetings needs to be attended and so forth. But we can manage that. For instance, one person can leave the mob to focus on a problem that emerged or attend a meeting. The work still goes on in the mob.

The mob becomes a natural place for communication and cooperation with people and functions outside the team. For instance we have a great UX team. Unfortunately they don’t have time to hang around our team all day long. But with the mob approach it becomes natural for them to swing by when they have an opening. They can become part of the mob for a while, and together we can discuss problems and develop solutions. If a stakeholder wants to now how things are going, they can simple stop by. Maybe just watch for a while, as the result of our work is almost always right there on the screen, the really big one.

imagesliderw800

A few weeks ago the top priority was an initiative to enhance the news section of TV4 Play. This has been in the making for a while. Lots of discussions, UX work, wire framing and so on. But now it was finally time for us to build something. It had been agreed that the most wished for feature was a new player module. Where the live news stream would be rolling right from the start but with no sound. A single click on the video should start the sound and on the right hand side there should be a list of the latest news clips that could be started quickly.

imagesliderw800-2

So the mob started to work. Things went great. Seemingly, our previous work reactifying our views paid off. At the beginning the UX team spent a lot of time in the mob. We identified problems and found solutions. Together. A few days in, everything started to feel production-ready. At that time we brought in people from SEO and Analytics who helped us to set up reasonable tracking and make sure google would still be able to find our news content. We were ready for production. One day ahead of the US election.

The sound of commercials

One thing you simply do not do, when developing an online video service all day long in an open-plan office, is having your volume set to more than a bare minimum. Especially when we’re sharing one big screen and not using headphones. Maybe that’s the reason we didn’t notice earlier that we had a huge problem with our video player. When switching between the latest news clips at a swift, but reasonable pace, the player could end up playing the sound of the commercial from one clip while playing sound and video from another. That’s a horrible user experience.

Into the mob enters one of our developers from the video player team. With common effort we could determine the problem was mainly caused by a bug in third party code. But we could also see a way around it. So we pulled up the code for the player and together we managed to fix the problem, and at the same time refactor parts of the player to make it more manageable and more easily integrated with TV4 Play. Now, we were really ready for production. We only missed the US election by two days. Well. At least we hadn’t estimated, “sync up meeted” and “moved stuff around in jira” our way into missing, what seemed to be, a perfect launch opportunity.

The bug we found in our third party code is acknowledged in their support system. We’re told that a fix is planned to be included in a release that will be available in two weeks from now.

C More 2.5.0 – made by Tegeluddsvägen

About a week ago Bonnier Broadcasting had the privilege to announce the release of C More’s new mobile apps for iOS and Android. At first sight you will see that the UI is brand new and it is aligned with the new design guidelines for C More and Bonnier Broadcasting. The new apps are much more responsive and easier to use, and we have put a lot of effort into making them robust. But the biggest transformation lies beneath this pretty surface – a brand new engine!

Nowadays, we have full control of the development process in-house. We have implemented a new modern code base, which give us better flexibility and capability to be more effective in our future product development. This project delivery is the first one out for Bonnier Broadcasting’s new development department, and specialists from various teams have been involved and contributed.

Our Android client programmers have been working with new innovating techniques such as: Model-View-Presenter design pattern, Data Binding, RxJava, Lambda expressions, ExoPlayer, annotations and dependency injections with Dagger. The iOS application is written in the Swift programming language.

During the first week we have already reached over half a million downloads for both iOS and Android.

Some of the fabulous people who made this happen.
This pic is taken shortly after release to App Store and Google Play.

Engineers, engineers, engineers

Today and tomorrow on the 22-23 of November Bonnier Broadcasting will attend the engineering career fair THS Armada at KTH Royal Institute of Technology

We will talk to engineering students about how we are building the next generation TV-experience on TV4 and C More through our tech development.

Ask us questions and share your views on our technologies and platforms. Talk to us about our cloud platforms, programming languages and how our engineers build our streaming services TV4 and C More on one shared tech platform.

We will also have fun quizzes with nice prices and some great cupcakes that you definitely want to taste.

Also students will have an opportunity to refine our suggestions or pitch own ideas for exciting master thesis projects next year at Bonnier Broadcasting.