One woman's quest to understand how to leverage the Go std library to build epic networking services.

1. Networking and Go:
an epic journey

2. software engineer @DigitalOcean
networking team
loves nature and cats
@snehainguva

3. My journey with Go
2016: building DOCC, abstraction layer on top of k8s
2017: working on hypervisor-level daemons to configure monitoring with
Prometheus
2018: working on DHCP-server implementation in Go
2018/2019: Experimenting with building network primitives outside of work

4. Why use Go to build networking services?
And how?

5. The Plan
● Why use go? ★
● Networking Review
● Layer 4+ Services
● Layer 2+ Services
● Conclusion

6. Go for Microservices
Goroutines: lightweight processes
Excellent concurrency support with sync package
Communication primitive known as channels
Low learning-curve

7. Go and Networking
net package: portable interface for network I/O, Unix sockets, etc.
net/http package: provides HTTP client/server implementations
syscall package: provides access to low-level system primitives
os package: provides platform-independent interface to OS system functionality

8. The Plan
● Why use go?
● Networking Review ★
● Layer 4+ Services
● Layer 2+ Services
● Conclusion

9. Networking Basics: OSI Model

10. Networking Basics: A Segment, Packet, and Frame
Ports --------------------------
IP ------------------
MAC--
network
transport
data link

11. Networking Basics: Sockets
internal endpoint to send or receive data in a network
Stream Socket: Data sent reliably and in-order. Used for TCP connections.
Datagram Socket: Used for connectionless data transmission.
Raw Socket: Packets not sent with any transport-layer formatting.
Often used for low-level data transmission.

12. Networking Basics: Protocols
HTTP: an application layer (7) protocol
TCP: a transport layer (4) protocol providing ordered delivery of bytes
UDP: a transport layer (4) protocol providing connectionless data transmission
IP: a network layer (3) protocol
ARP: an IPv4 protocol used to map IP to hardware addresses
NDP: an IPv6, a network layer (3) protocol used to map IP to hardware addresses

13. The Plan
● Why use go?
● Networking Review
● Layer 4+ Services ★
● Layer 2+ Services
● Conclusion

14. Layer 4+ Networking Services
Layer 7 load balancer:
Application-layer load balancer
Can look at URL for routing purposes
Layer 4 load balancer:
Accept TCP connections from frontend and open TCP connections to backends
Similar to IPVS - layer 4 lb built into the Linux networking stack
Port scanner:
Similar to nmap utility
Attempts to open TCP connections to check what is opened and closed

15. Layer 7 Load Balancer
// HTTP handler and server.
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
// Randomly select from list of backends.
n := rand.Intn(len(backends))
r.URL.Host = backends[n]
r.URL.Scheme = "https"
req, err := http.NewRequest(r.Method, r.URL.String(), r.Body)
if err != nil {
// TODO(sneha): fix how this returns later.
http.Error(w, "cannot process request", http.StatusBadGateway)
return
}
...
←-http handler is listening for
requests
←- create new http request to
backends

16. Layer 4 Proxy
func handleConn(clientConn net.Conn) {
n := rand.Intn(len(backends))
backendConn, err := net.Dial("tcp", backends[n])
if err != nil {
log.Printf("error opening backend conn %s: %v", backends[n], err)
return
}
var g run.Group
{
g.Add(func() error {
return copy(clientConn, backendConn)
}, func(error) {
clientConn.Close() // TODO(sneha): handle errors here
backendConn.Close()
})
}
{
g.Add(func() error {
return copy(backendConn, clientConn)
}, func(error) {
backendConn.Close()
clientConn.Close() // TODO(sneha): handle errors here
})
}
...
←------read from TCP streaming socket opened client-side
←------ open TCP streaming socket to one of the backends
←--- use runGroups to run routines to copy data bidirectionally

17. Port Scanner func main() {
hostname := "localhost"
conSema := make(chan struct{}, 10)
var wg sync.WaitGroup
for i := 1; i < 65535; i++ {
wg.Add(1)
go func(port int) {
conSema <- struct{}{}
addr := fmt.Sprintf("%s:%d", hostname, port)
conn, err := net.Dial("tcp", addr)
if err != nil {
fmt.Printf("port %d closed: %vn", port, err)
} else {
fmt.Printf("port %d openn", port)
conn.Close()
}
<-conSema
wg.Done()
}(i)
}
wg.Wait()
}
use channel to limit worker pool ------------->
←-- use waitgroup to block execution of program
use net.Dial to open streaming socket ------------------>
each port tested in new goroutine ------------------>

18. The Plan
● Why use go?
● Networking Review
● Layer 4+ Services
● Layer 2+ Services ★
● Conclusion

19. Layer 2+ Services
NDP/ARP proxy:
● NDP (neighbor discovery protocol) is used to map IPv6 to hardware
addresses
● ARP (address resolution protocol) is used to map IPv4 to hardware addresses
DHCP server:
● dynamic host configuration protocol is used by routers to allocate IP
addresses to network interfaces
● DHCPv6 uses NDP and DHCPv4 uses ARP

20. ARP Package (uses raw sockets)
type Client struct {...}
// Dial creates a new Client using the specified network interface.
func Dial(ifi *net.Interface) (*Client, error) {
// Open raw socket to send and receive ARP packets using ethernet frameswe build ourselves.
p, err := raw.ListenPacket(ifi, protocolARP, nil) ←-------------------- open raw socket to listen for ARP packets
if err != nil {
return nil, err
}
return New(ifi, p)
}
func (c *Client) Request(ip net.IP) error {
if c.ip == nil {
return errNoIPv4Addr
}
arp, err := NewPacket(OperationRequest, c.ifi.HardwareAddr, c.ip, ethernet.Broadcast, ip)
if err != nil {
return err
}
return c.WriteTo(arp, ethernet.Broadcast)
}

21. Raw Package
// listenPacket creates a net.PacketConn which can be used to send and receive data at the device driver level.
func listenPacket(ifi *net.Interface, proto uint16, _ Config) (*packetConn, error) { ←---open a connection based on ifi
*os.File
var err error
// Try to find an available BPF device
for i := 0; i <= 10; i++ {
bpfPath := fmt.Sprintf("/dev/bpf%d", i)
f, err = os.OpenFile(bpfPath, os.O_RDWR, 0666) ←------- use os open a raw socket to BPF device
if err == nil {
// Found a usable device
break
}
// Device is busy, try the next one
if perr, ok := err.(*os.PathError); ok {
if perr.Err.(syscall.Errno) == syscall.EBUSY { ←-------- check if the device is busy using syscall
continue
}
}
...

22. NDP Package (uses datagram sockets)
// Dial returns a Conn and the chosen IPv6 address of the interface.
func Dial(ifi *net.Interface, addr Addr) (*Conn, net.IP, error) {
addrs, err := ifi.Addrs()
if err != nil {
return nil, nil, err
}
ipAddr, err := chooseAddr(addrs, ifi.Name, addr)
if err != nil {
return nil, nil, err
}
ic, err := icmp.ListenPacket("ip6:ipv6-icmp", ipAddr.String()) ←------- listen for ICMP packets
if err != nil {
return nil, nil, err
}
pc := ic.IPv6PacketConn()
...

23. ICMP Package
func ListenPacket(network, address string) (*PacketConn, error) {
var family, proto int
switch network {
case "udp4":
family, proto = syscall.AF_INET, iana.ProtocolICMP
case "udp6":
family, proto = syscall.AF_INET6, iana.ProtocolIPv6ICMP
default:
...
}
var cerr error
var c net.PacketConn
switch family {
case syscall.AF_INET, syscall.AF_INET6:
s, err := syscall.Socket(family, syscall.SOCK_DGRAM, proto) ←syscall to listen from datagram socket
if err != nil {
return nil, os.NewSyscallError("socket", err) ←---- use os to check for syscall error
}
...

24. The Plan
● Why use go?
● Networking Review
● Layer 4+ Services
● Layer 2+ Services
● Conclusion ★

25. Conclusion
net package for transport layer (4) and higher
use syscall and os packages to go lower if needed
go has excellent concurrency primitives (goroutines, channels, sync package)

26. A special thanks
Matt Layher (@mdlayher)
Julius Volz (@juliusvolz)
Networking Pillar at DigitalOcean

27. Links
Port scanner: https://github.com/si74/portscanner
Layer 7 Load balancer: https://github.com/si74/layer7lb
Layer 4 Load balancer: https://github.com/si74/tcpproxy
Raw Package: https://github.com/mdlayher/raw
ARP Package: https://github.com/mdlayher/arp
NDP Package: https://github.com/mdlayher/NDP