Sockets with Privileges, or Avoiding Setuid Erlang

There are a few things which require, or at least are easier to handle, in process: PAM, GSS-API, file descriptors .... While it may be possible to handle these using a separate process, like an Erlang port, it's more comfortable to handle it within the virutal machine. Previously, the only way to do this would have been by using a linked-in driver; though I haven't tried writing one, they have the reputation of being complex and error prone, an easy way of reducing the reliability of the Erlang VM.

With the introduction of the Erlang NIF interface, there now exists a simple, though synchronous, method of interfacing with C libraries. Using the NIF interface is pretty easy.

One of the problems with using a virtual machine is dealing with actions that require heightened privileges. In a standalone program, you would simply run as root, do whatever needs to be done as root, then drop your privileges. With a language running in a virtual machine, the VM does way too much before your code would be able to drop its privs. The result is unpredictable.

Ways of Handling Privileged Sockets

One example is requesting a socket on a low port.

For completeness, there are many ways of handling privileged sockets without resorting to setuid:

1. Use the firewalling capability of the OS, like iptables, to forward packets from the low port to the unrestricted port on which your application is running.
2. Use a separate device, like a load balancer, router, or firewall, to forward packets from the low port to your service.
3. Run an application in userspace that binds the low port and forwards packets; this could be a dedicated proxy like squid or nginx or a small shim like socat.
4. Run as root. All the power of Erlang combined with complete system access! What could go wrong?
   
   For certain tasks, this may be the way to go though. It'd be interesting (and very likely awesome) to see a version of cfengine (or puppet or chef) written in Erlang.

There may be times that doing any of the above is not too convenient. procket provides a way for the Erlang VM to grab a privileged socket without too much hassle.

procket consists of an NIF shared library and a setuid executable. On Unix systems, file descriptors are allowed to be passed between processes using local (Unix) domain sockets. procket works by creating a local domain socket from within the Erlang VM using an NIF, listening on the socket, then spawning a setuid exectuable that binds the privileged socket. The setuid executable then drops its privs and passes the file descriptor back to the VM over the local socket.

To handle the actual setup of the local domain sockets, I used libancillary (which quite rightly bills itself as "black magic on Unix domain sockets").

The Erlang NIF

The NIF consists of a C file that is compiled into a shared library and an Erlang module to load the library. The NIF interface is setup using a macro:

ERL_NIF_INIT(procket, nif_funcs, load, reload, NULL, NULL)

The first argument is the name of the module to be called from within Erlang. The second argument points to the struct holding our exports. In this case:

static ErlNifFunc nif_funcs[] = {
        {"open", 2, sock_open},
        {"poll", 0, poll},
        {"close", 0, sock_close}
};

The export "open" in the procket module, called with an arity of 2, calls our C function sock_open.

The NIF can optionally call other functions based on certain events (as defined in the ERL_NIF_INIT() macro), such as on load, upgrade, etc. I've only handled load and reload to allocate memory for the data structures.

NIF's can hold state. While it would be more predictable and safer to require arguments to be returned and passed in again on each function call (the NIF holds the file descriptor and path to the Unix socket), for expediency, I didn't do it. And even in this small example, it led to a few annoying bugs. Let that be a lesson.

The module interface has three actions: open, poll and close.

"open" takes a path (a string) to the location of the Unix domain socket and a port number. For safety, the socket should probably reside somewhere only the user running the Erlang VM has access to (by default, the socket will be placed under a directory in /tmp or wherever you have TMPDIR set to). On some platforms, Unix sockets are created with mode 777, which could lead to hijinks on multi-user systems.

"open" creates the Unix socket, sets the socket to non-blocking, then listens on it.

"poll" attempts to accept a connection on the Unix socket. If there is a connected client, it will receive the message (our file descriptor) and return a tuple in the format {ok, FD}.

"close" removes the socket and cleans up the data structure and pipe file descriptor.

The socket path and Unix socket file descriptor are stored in a data structure within the NIF, so the caller doesn't need to provide them when using poll() or close(). Arguably, both poll() and close() should have an arity of 2 as well (path, pipe file descriptor); that way the module would be stateless and would be able to handle as many sockets as required by the program. Probably a mistake that should be corrected in the future.

The Setuid Binary

procket_cmd.c is spawned from the procket Erlang module with root privileges, using setuid or sudo. It essentially does:

1. parses command line arguments (potentially dangerous since we are parsing user input and performing actions, such as memory allocation, based on it)
2. opening the socket and performing any additional operations required by the protocol
3. dropping our privileges
4. connecting to the Unix socket and writing the socket file descriptor
5. exiting

On error, the "procket" command will print out some debug output.

The Erlang Module

The Erlang module's interface mirrors the C NIF: open/2, poll/0 and close/0. To make it easier to use, these exports are wrapped by listen/1 and listen/2.

listen(Port) ->
    listen(Port, []).
listen(Port, Options) when is_integer(Port), is_list(Options) ->
    Opt = case proplists:lookup(pipe, Options) of
        none -> Options ++ [{pipe, mktmp:dir() ++ "/sock"}];
        _ -> Options
    end,
    ok = open(proplists:get_value(pipe, Opt), Port),
    Cmd = make_args(Port, Opt),
    case os:cmd(Cmd) of
        [] ->
            poll();
        Error ->
            {error, procket_cmd, Error}
    end.

listen/2 composes the command line for the procket binary, atomically creates a unique path for the Unix socket if one was not provided (to prevent symlink race conditions), runs the setuid executable then polls the Unix socket.

Preparing command line arguments to be passed to an executable is, admittedly, pretty ugly and risky, in a sort of retro, pre-millenial CGI sort of way. Passing the arguments from the program is fine, but allowing any sort of user input is just nasty. I once hacked a web app by putting a ";some command here" into the password field of an account signup (account details, I discovered, were set up by running a command in the shell).

An Example

Here's how you would use procket:

$ cd procket
$ erl -pa ebin
Erlang R13B03 (erts-5.7.4) [source] [rq:1] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.4  (abort with ^G)
1> {ok, FD} = procket:listen(53, [{progname, "sudo priv/procket"},{protocol, udp}]).
{ok,9}

netstat should display a socket listening on port 53/udp.

2> {ok, S} = gen_udp:
open(53, [{fd,FD}]).
{ok,#Port<0.929>}

gen_udp:open/2 and gen_tcp:listen/2 both take a file descriptor as an option.

3> receive M -> M end.

Use netcat to send some data to the Erlang process:

nc -u localhost 53
blah
^C

At your Erlang prompt, you should see something like:

{udp,#Port<0.929>,{127,0,0,1},47483,"blah\n"}

There is also an example of an echo server using procket in the source, see the README.

At any rate, I hope this will be marginally useful to other Erlang n00bs. In the future, I am going to play with setting socket options and requesting raw sockets. This will allow creating ICMP packets, like ping, within Erlang, as well as doing some packet manipulation.