An axiom of programming usually found in the Operational, DevOps, and SRE spaces is this:

Don’t write a socket server.

There’s good reason. There are a lot of edge cases to handle and, for lack of a better phrase, magic sauce to be efficient and secure. Not to mention there are a lot of libraries already written to do this for you.

However, this really applies to writing socket clients as well. Writing a correct client to work with a socket has its own challenges. However, this is really quite common when building automation around Unix services. Many expose a Unix Socket in the file system somewhere to issue administrative commands on the fly.

These automation tools are likely to be, say, Python running in a Container. One really good option is to shell out to socat. Its a great tool. But there is also reason to keep dependencies limited and container sizes small. Its just issuing a couple commands to a socket, and downloading some state data, right? You might even connect to this socket by hand to discover status, run administrative commands, and test the automation tools. Works just like SSH, it can’t be hard!

Turns out, every time I find myself here I’ve managed to forget how to write a proper socket client. Its always a different language and a different situation. Today’s example comes from being able to download and save HAProxy’s state, and issue the reload command to make a changed configuration effective.

The common problem here is that its impossible to tell the difference between a short read (read buffer wasn’t completely full) and the end of the data stream. (Unless the socket speaks a more advanced protocol than a standard command line client situation.) While you can use this socket to issue multiple commands by hand, in code, we treat the socket as an old fashioned HTTP-like server. The only way to know that the state data you’ve read is complete is either parsing it, or getting a 0 length read which means the server has closed the connection. Otherwise, a short read may be just that rather than an end of data marker. The next read may or may not block. (Non-blocking IO is definitely another blog post!)

The form to write a simple socket client that can download data and issue follow up commands looks like this. Remember socket.shutdown()? No? That’s the key to do this well and why it is different than socket.close().

def reload_haproxy(sock):
    state = os.path.join(os.path.dirname(sock), "haproxy.state")
    s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
    s.connect(sock)
    with open(state, "wb") as fd:
        # Dump server state.  This assumes this is a socket to the master
        # HAProxy daemon in master-worker mode
        s.sendall(b"@1 show servers state\n")
        s.shutdown(socket.SHUT_WR)
        while True:
            buf = s.recv(4096)
            if len(buf) == 0:
                break
            fd.write(buf)
    s.close()

    # This command will cause the server end to close the connection as
    # the master process is re-executed.
    s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
    s.connect(sock)
    s.sendall(b"reload\n")
    s.shutdown(socket.SHUT_RDWR)
    s.close()

Each command and response is executed in its own connection to the Unix Socket on disk. This ensures that each transaction is complete and the received data is intact.

The first shutdown call, s.shutdown(socket.SHUT_WR), informs the server side that we will not be sending any more data. So after the server writes the response it will close the connection. (As it will never receive more commands.) The connection closure creates the 0 length read on the client side that informs us that we have read all data.

We do similar for the second shutdown call, s.shutdown(socket.SHUT_RDWR). In this case we know that HAProxy will not send us any data and that it will close the connection as the process re-executes. So in this case we use shutdown() to inform the server end that the client will write no more data and will read no more data. We wait for the connection to formally be closed.

I know I’ll refer to this when I write the next socket client that I probably shouldn’t. You too might find it a good guide for writing simple, blocking, socket clients. This works for Unix Sockets on disk, or TCP connections as well.