Build a Go RESTful API using GOA that scales effortlessly with Google App Engine and are beautifully documented by Swagger

Prelude

If you are interested in the guide, just jump to the building section below. Otherwise, I will pace you through my thought process.

In the previous months, I have found myself spending copious amount of time with Go. There are a multitude of reasons, but the most prominent being its portability and the speed it provides. Recently I had a need to write a Web service, which ideally would be powered by an API. Instead of picking my usually tools (python/flask) for the job I decided to tackle this one in Go. After spending a few days defining the requirements for the API I settled on the following:

• Needs to scale with usage automatically, requires little maintenance and must be impervious to usage spikes as much as possible.
• Needs to be highly fault tolerant
• Needs to be well documented, ideally via a interactive UI
• Lastly, since time is of the essence, any tech stack selected needs to give me as much scaffolding out of the box I can use for CRUD operations. Then, allow me to iterate on it as fast as possible (add/edit) endpoints.

The first two requirements are easily met by using a container based platform like Kubernetes. Having some experience with it, it was my obvious go to choice. My greatest aversion to using it was the overhead that it would have with managing such a deployment. One of the key things I had in mind was keeping the maintenance cost (labor and monetary) to a minimum for this API. This made Google App Engine (GAE) an obvious choice. App engine would provide me with:

• Infinitely scale based on usage, not paying for CPU time that’s not used but it will scale up if there is a spike in usage
• Fault tolerance across multiple regions in Google Cloud Platform automatically
• Access to all other google components like data store natively (I needed a NOSQL store as well)

Now that I had selected a platform to deploy to, my next search was for a Go web framework that would serve as a great CRUD API platform. My top two Choices were Gin and Echo. Personally, I gravitate to the simplicity and documentation of Echo but found better examples (specifically tutorials that describe how to deploy to GAE) for Gin. After much searching I settled on GOA for the following reasons:

• To get started with GOA I do not have to write any scaffolding code. GOA generates a full project structure for me from a design file.
• It also generates a Swagger specification to help me document which was one off my requirements.
• Using its DSL it allows me iterate on the API functions in an extremely quick fashion.
• Lastly, has security for the API built in.### Building

Before we jump in, I want to point out that we will walk though the cellar example as a base to show how a fully functioning example is meant to work. We will then make a few changes to support hosting in Google App engine with a Swagger UI for documentation.

Requirements

• Install cellar example and have it running locally
• Install Google Cloud SDK and configure a project
• Install Goa go get -u github.com/goadesign/goa/...

As the goa-cellar example states the only non-generated directories are:

• the controllers package, which contains the API business logic
• the store package, which contains the database operations.
• the design package itself, which contains the DSL the describes the API and which is used to generate the code

I encourage to peak into each directory and look at their contents. This is the code that you will be writing for your app.

If we look at the design/resources.go component, you will notice that the API URL paths are defined. We also see that the only path defined is that for a [swagger] (https://github.com/goadesign/goa-cellar/blob/master/public/swagger/swagger.yaml)spec file but not its UI. Let’s change that!

In the Resource definition for swagger, lets change the file directive to include everything under the public folder not just the swagger.json file. Change your resource file swagger section to look like this:

var _ = Resource("swagger", func() {  
 Origin("*", func() {  
  Methods("GET", "OPTIONS")  
 })  
 Files("/swagger/*filepath", "public/swagger/")  
})

Save, edit and exit out of resource.go

Now that we have a path (/swagger) that loads all files placed in it.

The next step is to regenerate our app code based on the newly modified resource and for same of example, lets also regenerate our, app, main, and swagger code base.

cd $GOPATH/src/github.com/goadesign/goa-cellar/  
goagen app -d github.com/goadesign/goa-cellar/design  
goagen main -d github.com/goadesign/goa-cellar/design  
goagen swagger -d github.com/goadesign/goa-cellar/design -o public

Now we are ready to move the Swagger UI into it. Run:

cd ~/tmp && git clone https://github.com/swagger-api/swagger-ui

Then move the dist folder into cellar-goa public folder:

mv ~/tmp/swagger-ui/dist/* $GOPATH/src/github.com/goadesign/goa-cellar/public/swagger/

Then edit the $GOPATH/src/github.com/goadesign/goa-cellar/public/swagger/index.html and change the URL from this to point to your local swagger url: “./swagger.json”

You should end up with something similar to this tree ( brew install tree if you do not have the tool):

~/go/src/github.com/goadesign/goa-cellar/public   
└── # tree  
.  
├── html  
│   └── index.html  
├── js  
│   ├── axios.min.js  
│   ├── client.js  
│   ├── example.go  
│   └── index.html  
├── schema  
│   └── schema.json  
└── swagger  
    ├── favicon-16x16.png  
    ├── favicon-32x32.png  
    ├── index.html  
    ├── oauth2-redirect.html  
    ├── swagger-ui-bundle.js  
    ├── swagger-ui-bundle.js.map  
    ├── swagger-ui-standalone-preset.js  
    ├── swagger-ui-standalone-preset.js.map  
    ├── swagger-ui.css  
    ├── swagger-ui.css.map  
    ├── swagger-ui.js  
    ├── swagger-ui.js.map  
    ├── swagger.json  
    └── swagger.yaml
    4 directories, 20 files

We now have a swagger UI.

And if we were to run this example locally, everything should work perfectly. Try it out now:

cd $GOPATH/src/github.com/goa-design/goa-cellar/  
go build main.go  
./main

You should see a service start up in port 8081, then just visit http://localhost:8081/swagger/

API should be fully functional too.

curl -s localhost:8081/cellar/accounts | jq  
[  
  {  
    "href": "/cellar/accounts/1",  
    "id": 1,  
    "name": "account 1"  
  },  
  {  
    "href": "/cellar/accounts/2",  
    "id": 2,  
    "name": "account 2"  
  }  
]

Now to deploy in Google’s App Engine. The gcloud tool-chain ships with dev_appserver.py a tool to allow you to run a local development environment for GAE standard. To run it, we must pass it the app.yml which contains our GAE definitions.

cd $GOPATH/src/github.com/goadesign/goa-cellar/
dev_appserver.py app.yml

And … you will see the go build fail with an error similar to:

2018/07/19 11:48:22 go-app-builder: Failed parsing input: app file account.go conflicts with same file imported from GOPATH

This is primarily due to how App Engine expects the location of imports. Let’s create a folder and move our GAE configuration over to it. We also need to move the public folder as we are changing the root location the app is initiating from.

mkdir appengine  
mv app.yaml appengine/  
mv appengine.go appengine/   
mv public appengine/

Then, lets try to start our app engine development environment again.

dev_appserver.py appengine/app.yaml

You should see a similar output to below:

WARNING  2018-07-19 15:52:51,940 devappserver2.py:114] *** Notice ***  
In a few weeks dev_appserver will default to using the Cloud Datastore Emulator. We strongly recommend you to enable this change earlier.  
To opt-in, run dev_appserver with the flag --support_datastore_emulator=True  
Read the documentation: [https://cloud.google.com/appengine/docs/standard/python/tools/migrate-cloud-datastore-emulator](https://cloud.google.com/appengine/docs/standard/python/tools/migrate-cloud-datastore-emulator)  
Help us validate that the feature is ready by taking this survey: [https://goo.gl/forms/UArIcs8K9CUSCm733](https://goo.gl/forms/UArIcs8K9CUSCm733)  
Report issues at: [https://issuetracker.google.com/issues/new?component=187272](https://issuetracker.google.com/issues/new?component=187272)
INFO     2018-07-19 15:52:51,945 devappserver2.py:158] Skipping SDK update check.  
INFO     2018-07-19 15:52:52,219 api_server.py:274] Starting API server at: [http://localhost:52855](http://localhost:52855)  
INFO     2018-07-19 15:52:52,329 dispatcher.py:270] Starting module "default" running at: [http://localhost:8080](http://localhost:8080)  
INFO     2018-07-19 15:52:52,342 admin_server.py:152] Starting admin server at: [http://localhost:8000](http://localhost:8000)  
lvl=info msg=mount ctrl=Account action=Create route="POST /cellar/accounts"  
lvl=info msg=mount ctrl=Account action=Delete route="DELETE /cellar/accounts/:accountID"  
lvl=info msg=mount ctrl=Account action=List route="GET /cellar/accounts"  
...  
lvl=info msg=mount ctrl=Public files=public/html/index.html route="GET /ui"  
lvl=info msg=mount ctrl=Js files=public/js route="GET /js/*filepath"  
lvl=info msg=mount ctrl=Js files=public/js/index.html route="GET /js/"  
lvl=info msg=mount ctrl=Swagger files=public/swagger/ route="GET /swagger/*filepath"  
lvl=info msg=mount ctrl=Swagger files=public/swagger/index.html route="GET /swagger/"

Note that our listening port changes to 8080 in this case but we can test the API again:

└── # curl -s localhost:8080/cellar/accounts | jq  
[  
  {  
    "href": "/cellar/accounts/1",  
    "id": 1,  
    "name": "account 1"  
  },  
  {  
    "href": "/cellar/accounts/2",  
    "id": 2,  
    "name": "account 2"  
  }  
]

Now we are ready to deploy to your project run.

cd $GOPATH/src/github.com/goadesign/goa-cellar/appengine && gcloud app deploy

Now you have a REST API example that scales with usage, is documented automatically using swagger spec, has a CLI tool generated and an API client objects. This foundation makes a perfect framework for deploying micro services across your organization as well as the basis for an API project. I hope you enjoyed this tutorial and if you run into any questions following these steps feel free to message me via twitter.