An Erlang Packet Sniffer Using ei and libpcap

After testing the erlang nif interface, I thought I'd try the ei library. ei is a way of serializing terms between Erlang and ports. Whereas nif is a function interface and is thus blocking, ports treat external processes as an Erlang process and so can use asynchronous message passing for communication.

Some packet dump utilities for Erlang already exist using linked in drivers and the Linux raw socket interface. It should be possible, as well, to use tcpdump as a port, although that involves a lot of text parsing.
epcap is a standalone binary written in C, using the pcap packet capture library. I'll go through building epcap in two parts: making a port in C and parsing the packets in Erlang. If you just want to start playing with epcap, you can get it from github. It's just a beginning, but it should be work.

If you are interested in the ei libary, I suggest reading the excellent tutorial on trapexit first.

Creating a port is a bit more work than using nif's. Since a port is just a system process, we have to handle all the bookkeeping ourselves. The way the epcap process works is as follows:

1. start with root privileges
2. parse command line arguments
3. open the pcap device
4. drop root privileges
5. loop and read packets
6. convert packets into Erlang binary term format and write to stdout

epcap does not receive any messages from the Erlang process. As a convenience, we have epcap fork, with the parent blocking on stdin, so that the epcap process will exit gracefully when the Erlang node closes the port.

To dump packets, we do the following:

1. open the pcap device
2. set a filter for the device and other initialization
3. read one packet from the device, process it and loop

Opening the pcap device involves retrieving a default interface to sniff, if one hasn't been specified (on Mac OS X, it seems to almost always pick the wrong interface) and returning a pcap handle. After the device has been opened, we can drop root privileges.

pcap_t *
epcap_open(char *dev)
{
    pcap_t *p = NULL;
    char errbuf[PCAP_ERRBUF_SIZE];

    if (dev == NULL)
        PCAP_ERRBUF(dev = pcap_lookupdev(errbuf));
    PCAP_ERRBUF(p = pcap_open_live(dev, SNAPLEN, PROMISC, TIMEOUT, errbuf));

    return (p);
}

Initializing the device involves compiling and setting a filter:

int
epcap_init(EPCAP_STATE *ep)
{
    struct bpf_program fcode;
    char errbuf[PCAP_ERRBUF_SIZE];

    u_int32_t ipaddr = 0;
    u_int32_t ipmask = 0;


    if (pcap_lookupnet(ep->dev, &ipaddr, &ipmask, errbuf) == -1) {
        VERBOSE(1, "%s", errbuf);
        return (-1);
    }

    VERBOSE(2, "[%s] Using filter: %s\n", __progname, ep->filt);

    if (pcap_compile(ep->p, &fcode, ep->filt, 1 /* optimize == true */, ipmask) != 0) {
        VERBOSE(1, "pcap_compile: %s", pcap_geterr(ep->p));
        return (-1);
    }

    if (pcap_setfilter(ep->p, &fcode) != 0) {
        VERBOSE(EXIT_FAILURE, "pcap_setfilter: %s", pcap_geterr(ep->p));
        return (-1);
    }

    return (0);
}

Finally, we can just sit in a loop, pulling one packet from the queue. We use pcap_next() instead of pcap_loop() so that we can apply our own flow control, if necessary, in the future (sort of like {active,once}).

When a packet is sniffed, pcap returns 2 values to us: the packet itself and a struct with a timestamp, the length of the packet that was returned to us and the actual length of the packet on the wire (since the packet given to us may have been truncated).

At this point, we can test if packet dumps are working by printing out the ethernet header:

#include <net/ethernet.h>

<...>

    void
epcap_loop(pcap_t *p)
{
    <...>
    struct ether_header *eh = NULL;

    <...>
    eh = (struct ether_header *)pkt;

    (void)fprintf(stderr, "[shost]%02x:%02x:%02x:%02x:%02x:%02x\n", eh->ether_shost[0],
            eh->ether_shost[1], eh->ether_shost[2], eh->ether_shost[3],
            eh->ether_shost[4], eh->ether_shost[5]);

    <...>
}

Pulling all of this together and compiling should give something like:

[shost]00:1c:b3:xx:xx:xx
[shost]0:16:3e:xx:xx:xx
[shost]00:1c:b3:xx:xx:xx

Now we're ready to get to the interesting part: creating data for consumption by the erlang process.

void
epcap_response(const u_char *pkt, struct pcap_pkthdr *hdr)
{
    ei_x_buff msg;

    u_int16_t len = 0;


    /* [ */
    IS_FALSE(ei_x_new_with_version(&msg));
    IS_FALSE(ei_x_encode_list_header(&msg, 2));

    /* {time, {MegaSec, Sec, MicroSec}} */
    IS_FALSE(ei_x_encode_tuple_header(&msg, 2));
    IS_FALSE(ei_x_encode_atom(&msg, "time"));

    IS_FALSE(ei_x_encode_tuple_header(&msg, 3));
    IS_FALSE(ei_x_encode_long(&msg, abs(hdr->ts.tv_sec / 1000000)));
    IS_FALSE(ei_x_encode_long(&msg, hdr->ts.tv_sec % 1000000));
    IS_FALSE(ei_x_encode_long(&msg, hdr->ts.tv_usec));

    /* {packet, Packet} */
    IS_FALSE(ei_x_encode_tuple_header(&msg, 2));
    IS_FALSE(ei_x_encode_atom(&msg, "packet"));
    IS_FALSE(ei_x_encode_binary(&msg, pkt, hdr->caplen));

    /* ] */
    IS_FALSE(ei_x_encode_empty_list(&msg));

    len = htons(msg.index);
    if (write(fileno(stdout), &len, sizeof(len)) != sizeof(len))
        errx(EXIT_FAILURE, "write header failed");

    if (write(fileno(stdout), msg.buff, msg.index) != msg.index)
        errx(EXIT_FAILURE, "write packet failed: %d", msg.index);

    ei_x_free(&msg);
}

I'd like to send a list to the Erlang process consisting of a proplist of 2 tuples:

[{time, {MegaSeconds, Seconds, MicroSeconds}}, {packet, Packet}].

The timestamp is in the same format as erlang:now().

Creating the data structure is quite easy. ei functions come in statically and dynamically allocated versions. For creating the Erlang terms, the dynamic version is ideal.

After allocating an ei_x_buff struct, we add a version header. If we want to add a tuple, we add a tuple header with the appropriate arity and begin adding elements. If we want to add a nested tuple, it's as simple as sequentially adding another tuple header.

When calling the epcap binary, I allowed my login to run the binary as root using sudo:

$ visudo
myuser ALL = NOPASSWD: /path/to/epcap/epcap

Alternatively, you can put epcap in a directory owned by root and make epcap setuid (chown root:yourgroup epcap; chmod 4550 epcap). If you decide make epcap setuid, use a group to which your login is the only user.

The erlang port driver should be familiar from the trapexit tutorial.

start(PL) when is_list(PL) ->
    Args = make_args(PL),
    Port = Args,
    spawn_link(?MODULE, init, [self(), Port]).

init(Pid, ExtPrg) ->
    register(?MODULE, self()),
    process_flag(trap_exit, true),
    Port = open_port({spawn, ExtPrg}, [{packet, 2}, binary, exit_status]),
    loop(Pid, Port).

We take a proplist to set up the command line arguments for the C binary. We also pass in the calling processes pid, so messages can be sent back.

loop(Caller, Port) ->
    receive
        {Port, {data, Data}} ->
            Caller !  binary_to_term(Data),
            loop(Caller, Port);
        {Port, {exit_status, Status}} when Status > 128 ->
            io:format("Port terminated with signal: ~p~n", [Status - 128]),
            exit({port_terminated, Status});
        {Port, {exit_status, Status}} ->
            io:format("Port terminated with status: ~p~n", [Status]),
            exit({port_terminated, Status});
        {'EXIT', Port, Reason} ->
            exit(Reason);
        stop ->
            erlang:port_close(Port),
            exit(normal)
    end.

Like any other erlang process, messages from ports are captured using receive. We convert the message to erlang term format and send the message to the calling pid.
A process using epcap could work as follows:

epcap:start([{chroot, "/tmp/epcap"},{interface, "en1"}, {filter, "tcp and port 80"}]).
receive Any -> Any end.

Parsing and pretty printing the packets is covered here.