Lessons from building the Go SDK for Dropbox. Slides from Golang UK 2017

1. How to build SDKs in Go
Lessons from the
Go SDK for Dropbox

2. Diwaker Gupta
● Infrastructure @
Dropbox
● Metadata Services,
Multihoming, Search,
Messaging, Email,
Notifications, Machine
Learning etc
● Authored Dropbox Go
SDK and dbxcli
● Twitter/Github:
@diwakergupta

3. users businesses

4. ● Gophercon ‘17 Keynote by @tammybutow:
https://bit.ly/gophercon17dbx
● Dozens of Go services, many with millions of RPS!
● We open source at: https://github.com/dropbox
● We are hiring! dropbox.com/jobs or email me (diwaker@)

5. Lessons from the Dropbox Go SDK

6. #1: Use a Code Generator

7. #1: Use a code generator
● Language-agnostic API definition (DSL, JSON,
YAML, etc), Language-specific generators
● Dropbox Stone (spec + generators)
○ https://github.com/dropbox/stone
● Google
○ JSON Spec https://developers.google.com/discovery/
○ Codegen
https://github.com/google/google-api-go-client/
● AWS Go SDK has a very similar setup
● OpenAPI: https://github.com/OAI/OpenAPI-Specification

8. #2: Avoid Surprises

9. #2: Avoid Surprises
● Principle of Least Surprise
● No external dependencies
○ Only standard library imports
● No vendored dependencies
● What you need is what you get
○ Scoped sub-packages for larger APIs (e.g. AWS)
○ Make it `go get` friendly

10. #3: Make Configuration Simple

11. #3: Make Configuration Simple
● Don’t use cmdline flags
○ Stdlib limitations
● Environment variables
○ OK, but beware pitfalls
● Use a config struct
○ Same pattern in AWS SDK
● Persistence outside SDK
○ Let applications choose
○ dbxcli load/stores json
configs
// Config contains parameters for
configuring the SDK.
type Config struct {
// OAuth2 access token
Token string
// Enable verbose logging in SDK
Verbose bool
...
// For testing only
Client *http.Client
}

12. #4: Provide Visibility

13. #4: Provide Visibility
● Allow verbose logging
● Allow configurable
logging targets
● Limited by stdlib :(
type Config struct {
// Enable verbose logging
Verbose bool
// Optional logging target
Logger *log.Logger
}
// TryLog will, if Verbose is set, log to
// the config's logger or the default logger
// if Config.Logger is nil.
func (c *Config) TryLog(format string, v
...interface{}) {
if !c.Verbose {
return
}
if c.Logger != nil {
c.Logger.Printf(format, v...)
} else {
log.Printf(format, v...)
}
}

14. #5: Use Go idioms for unsupported types

15. #5: Unions
Consider (in Stone):
union DeleteError
path_lookup LookupError
path_write WriteError
In JSON (note LookupError):
{
".tag": "path_lookup",
"path_lookup": {
".tag": "malformed_path",
"malformed_path": "/some/path"
}
}
Equivalent struct in Go
type DeleteError struct {
dropbox.Tagged
PathLookup *LookupError `json:"path_lookup"`
PathWrite *WriteError `json:"path_write"`
}
Problem: (de)serialization

16. func (u *DeleteError) UnmarshalJSON(body []byte)
error {
type wrap struct {
dropbox.Tagged
PathLookup json.RawMessage
PathWrite json.RawMessage
}
var w wrap
switch w.Tag {
case "path_lookup":
err = json.Unmarshal(w.PathLookup,
&u.PathLookup)
...
case "path_write":
err = json.Unmarshal(w.PathWrite,
&u.PathWrite)
...
}
#5: Unions
type DeleteError struct {
dropbox.Tagged
PathLookup *LookupError
`json:"path_lookup,omitempty"`
PathWrite *WriteError
`json:"path_write,omitempty"`
}
{
".tag": "path_lookup",
"path_lookup": {
".tag": "malformed_path",
"malformed_path": "/some/path"
}
}

17. #5: Inherited Types
In Stone
struct Metadata
// Common fields: Name, Path etc
union
file FileMetadata
folder FolderMetadata
Idiomatic Go: Embedding
type Metadata struct {
// Common fields
}
type FileMetadata struct {
Metadata
// FileMetadata specific fields
}
type FolderMetadata struct {
Metadata
// FolderMetadata specific fields
}

18. Solution: Use a dummy interface
type IsMetadata interface {
IsMetadata()
}
// Subtypes get this via embedding
func (u *Metadata) IsMetadata() {}
// Use IsMetadata where you need
// the union type
#5: Inherited Types
Problem: Polymorphism
func List(...) []Metadata
Can return FileMetadata or
FolderMetadata

19. Similar trick as Unions
type metadataUnion struct {
dropbox.Tagged
File *FileMetadata
Folder *FolderMetadata
}
func IsMetadataFromJSON(data []byte) (IsMetadata,
e) {
var t metadataUnion
if err := json.Unmarshal(data, &t); err !=
nil {
return nil, err
}
switch t.Tag {
...
}
}
#5: Inherited Types
Problem: (de)serialization
{
".tag": "file",
"name": "Prime_Numbers.txt",
"id": "id:a4ayc_80_OEAAAAAAAAAXw",
"content_hash": "e3b0c44298fc"
}
OR
{
".tag": "folder",
"name": "math",
"id": "id:a4ayc_80_OEAAAAAAAAAXz",
"path_lower": "/homework/math",
}

20. #6: Do NOT authenticate

21. #6: Do NOT authenticate
● Apps authenticate, SDK accepts OAuth Token
● Beware of known OAuth pitfalls
○ Changed OAuth Endpoint from api.dropbox.com to
api.dropboxapi.com
○ App started failing with oauth2: cannot fetch token:
400 Bad Request
○ Gem from oauth2/internal/token.go
○ func RegisterBrokenAuthHeaderProvider(tokenURL string)
○ Google, Stripe and yes, Dropbox are “broken”
● More information:
○ On debugging OAuth2 in #golang

22. #7: Auto Generate Tests

23. #7: Auto Generate Tests
● Tests should be
comprehensive and
up-to-date
● Dropbox SDK is NOT a
good example!
● Model after AWS SDK
○ Example input/output
defined in JSON
○ Tests are auto-generated
● Not a panacea, still
need to write tests!
○ Especially negative tests

24. #8: Handle errors the Go way

25. #8: Handle errors the Go way
Dropbox SDK
res, err := dbx.ListFolder(arg)
if err != nil {
switch e := err.(type) {
case files.ListFolderAPIError:
...
AWS SDK
output, err := s3manage.Upload(...)
if err != nil {
if reqerr, ok := err.(RequestFailure);
ok {
...
● Have SDK errors implement the `error` interface
● Use type assertions to extract errors
● Good packages exist for combining / unrolling errors --
wish stdlib had more support!

26. Lessons Recap
1. Use a Code Generator
2. Avoid Surprises
3. Make Configuration Simple
4. Provide Visibility
5. Use Go Idioms for Unsupported Types
6. Do Not Authenticate
7. Auto Generate Tests
8. Handle Errors the Go Way

27. Where to get it?
● SDK:https://github.com/dropbox/dropbox-sdk-go-unofficial
● dbxcli: https://github.com/dropbox/dbxcli/
○ Command line tool for Dropbox
○ Pre-compiled binaries for Windows, Mac, Linux, ARM
○ Useful for end users as well as team administrators
● Support for Dropbox Paper coming soon!
○ https://www.dropbox.com/paper