Naming is hard

FOSDEM 2016, Brussels - Day Two

Jan 31, 2016

Anand Babu ‘AB’ Periasamy - ‘Minio - Amazon S3 alternative in Go’

  • Worked previously on GlusterFS
  • Minio written in Go, open source
  • Apache v2 license
  • Object storage
  • Amazon S3 compatible API
  • when building command-line tools, think of the user and keep the interface simple
    • use colour in the terminal
    • use human-readable numbers (e.g. 10MB versus 1024000 bytes)
  • minio server ~/share-this-directory to start server
  • Why Go?
    • prototype was in C++ and Google V8; Mozilla SpiderMonkey
    • Rust came too late for this project; Go community was great
      • Rust’s strengths weren’t important for the design of Minio
    • Haskell has small community
    • C and Python
    • C and Guile
    • Java: didn’t want JVM dependency and startup time
    • goroutines and message passing (channels)
  • Minio project goals
    • minimalist design
    • community
    • easy
  • Experimenting with Go tooling for Minio
    • first added support for YASM to Go tool (patch, not adopted upstream); approach abandoned
      • Go tool rewritten in Go from C
  • Go error handling
    • Minio probe package (replaces Go’s standard error interface)
    • debug
    • Logrus logging library
  • Web interface in Go and React

Jonathan Boulle (CoreOS) - ‘etcd: the cornerstone of distributed systems using Go’

  • etcd is a distributed key-value store
    • like /etc but distributed
    • clustered
  • Building block for higher order systems
    • primitives for building reliable distributed systems
  • Started as an intern project
  • 2013: alpha; v0.1-v0.4
  • 2015: stable release, v2.0+
    • new Raft implementation
    • stable v2 API
  • 2016: v3.0
    • highly scalable backend
    • efficient, powerful API
  • Production ready
    • used widely
    • long running failure-injection tests
    • no known data loss issues
    • no known inconsistency issues
    • used in critical CoreOS systems like locksmith and fleet
  • Why build etcd?
    • CoreOS: ‘secure the Internet’
    • updating servers = reboot servers
    • move towards new application container paradigm
    • need a:
      • shared configuration store (for service discovery etc)
      • distributed lock manager (to co-ordinate reboots)
  • Why use etcd?
    • highly available
    • highly reliable
    • strong consistency guarantees
    • simple, fast HTTP API
    • open source
    • use only for most critical data; not designed for storage of lots of data
      • e.g. store file metadata but not file contents in etcd
  • How does etcd work?
    • replicated log to model a state machine
    • Raft consensus algorithm: “In search of an understandable consensus algorithm” (Ongaro, 2014)
      • three key concepts
        • leaders
        • elections
        • terms
    • written in go
    • etcd server binary
    • etcdctl client binary
  • typical cluster has 3 or 5 machines
    • ensure a quorum
  • simple HTTP API (v2)
    • GET /v2/keys/foo
    • GET /v2/keys/foo?wait=true for HTTP long-polling
  • compare-and-swap
    • atomic
    • change value if current value is equal to a known value
    • useful primitive for distributed locks
  • etcd client package
  • etcd apps
    • locksmith (machine reboots)
      • ‘semaphore for reboots’
    • CoreOS updates happen automatically
      • stop all the machines starting at once
    • skydns
      • service discovery and DNS server
      • backed by etcd for all configuration/records
    • vulcand
      • “programmatic, extendable proxy for microservers”
      • HTTP load balancer
      • etcd for all configuration
    • confd
      • simple configuration templating
      • watch etcd for changes and render templates with new values, reload applications
    • Kubernetes
  • Recent improvements (v2)
    • asynchronous snapshotting (used to be synchronous, stop-the-world problem)
      • log-based system
      • snapshot before purging log
    • raft pipelining
      • etcd previously used synchronous RPCs
  • Future improvements (v3)
    • scale to thousands of nodes (currently hundreds)
    • efficient and powerful API
      • flat binary key space instead of current directory hierarchy
      • multi-object transaction (multiple keys at once)
      • native leasing API
      • native locking API
      • gRPC (HTTP2 and protobuf)
    • range operation for multiple keys at once
      • prefix
      • range
    • delete a range of keys
    • mini transaction
      • compare and swap
      • multiple object transaction
        • check against multiple values at once
    • watch
      • support multiple keys and prefixes per stream
        • watchKey(foo)
        • watchPrefix(coreos)
      • support watch from historical point
    • gRPC
      • efficient
      • multiplexed; multiple streams share one TCP connection
      • protobuf, encapsulated in HTTP/2 binary protocol
      • rich generated libraries in many languages
    • incremental snapshot
      • only save the delta instead of full data set
      • less I/O and CPU cost per snapshot
    • disk backend
      • keep the cold historical data on disk
      • keep the hot data in memory
      • support ‘entire history’ watches
      • user-facing compaction API
      • disk space bounds how much history is kept
      • etcd becomes an MVCC (multi-version)
    • recipes for common functionality exposed as new packages
      • native lease
        • l := lease.Create(10*time.Second)
      • watch primitive
        • watcher.Watch("foo")
      • expose leader election
      • locking package
      • barrier package
  • etcd and Go
    • easy to iterate
    • fast
    • robust standard libraries
    • healthy, active ecosystem
    • simple but powerful concurrency
    • the less good:
      • unpredicable GC (latency spikes): github.com/etcd/issues/4111
      • scheduler starvation

Luis Pabón (Red Hat Storage) - ‘From prototype to deployment: Building a REST application using Go’

  • experiences learned during the development of Heketi
  • Heketi = a service to manage the lifecycle of GlusterFS volumes across multiple clusters
  • https://github.com/heketi/heketi
  • Requirements:
    • HTTP REST service with authentication
    • manage multiple nodes concurrently
    • maintain information on the clusters
    • simple deployment
    • allow concurrent requests
    • non-blocking
  • Design
    • middleware for authentication
    • endpoints, DB models
    • BoltDB
    • SSH executor
    • simple ring allocator
  • Other technologies considered
    • Python
      • CherryPy, Bottlepy, Django
      • hard to install (need Python runtime and libraries)
      • concurrency more difficult
    • Ruby
      • Rails, Sinatra, RESTRack
    • Java
      • RESTx, Jersey
      • JVM dependency
  • Why Go?
    • easy JSON integration
    • concurrency
    • simple deployment
    • simple HTTP package
    • integrated testing package (part of stdlib)
  • HTTP/REST standards
    • endpoint
    • GET, PUT, HEAD, POST, DELETE
    • response body contains data requested by client
    • could be either XML or JSON format
    • HEAD: metadata about content
    • return status codes:
      • 2xx successful
      • 3xx redirection
      • 4xx client error
      • 5xx server error
  • HTTP routing
    • really simple in Go (code sample)
    • Gorilla routing library (gorillatoolkit.org)
  • JSON
    • supported by the language
    • easy to set up using struct tags
    • remember to export your struct’s fields
    • JSON model using embedded types for easier changes to JSON
  • Logging
    • Go provides simple log API
    • create your own or use others like go-logging
  • Asynchronous HTTP
    • avoid timeouts
    • provide a more responsive service
    • server provides a temporary resource to poll for completion
    • 5-10 minutes completion time for some requests in Heketi
    • no standard for this; convention implemented was:
      • HTTP 202 Accepted with location of URL to poll
      • GET on URL to poll
      • HTTP 200 request still in progress (with X-Pending header)
      • HTTP 500 request terminated and has failed
      • HTTP 303 request completed successfully, use it on this URL
      • HTTP 204 Done: completed successfully, nothing to return
  • Authentication
    • JSON web tokens (JWT): https://jwt.io/

Derek Parker (CoreOS) - ‘Debugging Go programs with Delve’

  • A response to lack of a Go debugger at the time
  • Go-specific debugger
    • knowledge of runtime, scheduler, goroutines as a first class concept
  • Runs on Linux, Windows and OS X (Windows support very recent)
    • about 90% parity between Windows and other platforms
  • Written mostly in Go
    • can take advantage of standard library, i.e. parsing the AST
    • cgo used for OS X/Windows support
      • used to talk to low-level APIs not supported by stdlib
    • simple to build (through difficult to install on OS X because of codesigning)
      • uses a Makefile to build
  • Aims for simplicity of use
    • like to go tool
    • if you need a debugger, you’re probably not in the best mood
  • Supported by IDEs:
    • has JSON-RPC API
    • can be invoked in headless mode
    • already used by IntelliJ Go plugin, Microsoft VS Code, emacs, etc.
  • Future clients?
    • web client
    • more editor support
  • How does it work?
    • parses DWARF info and symbol table from binary
    • uses OS-specific syscalls for manipulating processes
      • Linux: ptrace family of syscalls does most of the work
      • OS X: mock syscalls; not documented at all
        • digging into header files
        • open source versions of new kernel
        • also has ptrace syscall
        • man page only has half the story
      • Windows: debug API
    • reads goroutine info from thread-local storage
      • uses goroutine info to track across context switches
        • stay locked to goroutine when stepping through program
    • context switching could be:
      • a blocking syscall
      • a channel operation
      • runtime.Gosched
      • invoking a goroutine
  • Usage
    • dlv debug
      • compiles your program
      • disables compiler optimisations
      • invokes compiled binary, immediately attaches to it
      • lands you at a prompt, ready to debug
      • run it in the same place you’d run go build
  • Demo!
    • runtime.Breakpoint() in code
    • step through
    • print variables
    • step into function
    • restart
    • set certain variables types
    • evaluate variables using expressions in Go syntax (print i == 9)
    • goroutines to list goroutine
  • dlv trace <regexp>
    • allows you to spy on your program (like strace/dtrace)
    • will set tracepoint on any function that matches the regexp passed into it
    • will display arguments to functions
    • can optionally display stack trace leading up to function that triggered tracepoint
    • can also trace running programs
  • dlv attach <pid>
    • attach to a running process
    • difficult on optimised binaries
  • The future
    • still pre 1.0
    • improved hardware support
      • 32-bit Linux
      • ARM
      • FreeBSD
      • anything that Go supports
    • ability to call functions
    • display disassembly of function (low hanging fruit)
  • Go compiler blockers
    • go linker emits DWARF debugging info
      • for historical reasons, instead of compiler
      • info can be wrong or lack info, e.g. variable scope
  • Check the issue tracker: github.com/derekparker/delve/issues
  • Other plans
    • split out various packages so code can be reused by other projects
    • continue growing community support around project
  • Feedback welcome!

Kushal M (Red Hat) - ‘Plugins and Go’

  • Works on GlusterFS (management daemon)
    • where interest in Go plugins started
  • What’s a plugin?
    • a software component that adds or extends features to an existing computer program without modifying the original program
    • easily add new features
    • allow third-part development
    • flexibility for deployment
  • Implementing a plugin system
    • define an interface
      • (optionally) a plugin request interface
    • define a location to drop plugins into
    • load plugins and execute them
  • Runtime loading and execution of plugins
    • requires support for dynamic loading
  • Two options currently in Go(?)
    • standalone programs using IPC
    • embedded scripting languages
  • Standalone
    • possibly more secure
    • host process can’t be compromised directly
    • plugin failures shouldn’t affect host process
    • performance a concern
    • e.g. Packer (Hashicorp go-plugin package: https://github.com/hashicorp/go-plugin)
    • https://github.com/natefinch/pie
  • Embedded scripts
    • Plugins are scripts
    • Embed a scripting language interpreter
    • Use interpreter method to communicate
    • Relatively more complex
    • Lua, Javascript
  • Package plugin
    • https://golang.org/s/execmodes
    • runtime dynamic loading for Go
    • Go-API and C-API -style API plugins

Francesc Campoy (Google) - ‘The state of Go’

  • Where we are in January 2016
  • Time flies
    • Go 1.6 release candidate 1 on January 28th
    • Go 1.6 probably ready in February
  • Changes since Go 1.5
    • no language changes
    • changes to standard library
      • net/http
      • text/template, html/template
      • sort
      • more speed
      • less bugs
    • net/http now uses HTTP/2 by default
  • text/template
    • new delimiters in template tags
      • allows whitespace to be trimmed
      • like ERB in Ruby
    • new block action
      • avoid repetition in templates
      • executes in-place
      • why do we need it?
        • it’s more compact
  • sort.Sort is faster; less interface calls
    • could change order but order was never guaranteed (use sort.Stable)
  • time.Parse is now aware of leap years
  • Changes to runtime
    • faster
    • detection of concurrent map writes will now panic at runtime
      • keep using the race detector so that it can trigger potential issues
    • garbage collector
      • now usually no more than ~10ms GC pauses even up to 250GB heap size
  • New ports
  • Changes to tooling
    • cgo tool
      • defined rules for pointers used by cgo
      • checked by runtime at execution
      • checks can be disabled, but you probably shouldn’t
    • GO15VENDOREXPERIMENT now enabled by default \o/
      • flag will be removed in Go 1.7
    • go doc will search vendored packages first
      • defines priority of packages with import paths with less elements when searching by name
    • go vet warns if the code prints a function instead of its result
  • Community
    • code of conduct introduced on November 24th 2015
    • Go meetups: http://go-meetups.appspot.com
    • Women Who Go now has 7 chapters: http://www.womenwhogo.org
  • Many conferences coming up
    • GopherGala (just finished)
    • FOSDEM
    • GopherCon Dubai
    • GopherCon India
    • GopherCon Denver
    • dotGo
  • Go 1.6 release party on February 17th