Wednesday, December 30, 2009

SoDS; Care and Feeding of,

SoDS provides a covert method of bypassing firewall restrictions, obscuring traffic analysis and hiding your data by tunnelling it over DNS. SoDS was intended to be small, resource efficient, reasonably portable with few dependencies (at least on Unix) as well as taking basic precautions against misuse.

Test Ride

To get a feel for sods, its simple to try out:
git clone git://
cd sods/sods
cd ../sdt

Depending on your OS, you might need to adjust the Makefiles.

To test sods, you'll need to have an SSH server available somewhere and root privs on your test host. Start up the socket server:
$ sudo ./sods -d /tmp -L localhost:22 -vvv

In another window, open an SSH session:
$ ssh -o ProxyCommand="./sdt -r"

Real World SoDS

To use sods, you'll need a few things:
  1. A publicly addressable IP
  2. Something on which to run sods. The server doesn't need to be dedicated or high powered.
  3. A registered domain name
  4. A DNS server to host your domain

Well, that may seem to be waaaayyy too much bother, but it might not be as much work as it seems.

Publicly Available IP Address

You need an IP address that is reachable from the Internet or control of a device that can forward packets to your sods server.

The IP addresses the ISP assigns you tend to fall into 3 groups:
  1. the IP address is static and never (or rarely) changes
  2. the IP address changes infrequently, but the reverse lookup for the address is static
  3. the IP address and, consequently, the reverse address, changes frequently

You can work with any of these.

The IP address must be able to receive UDP packets on port 53 and send packets from port 53. If your ISP blocks port 53 (DNS), then you'll have to look around for another ISP, use a hosted service or maybe piggyback off of a friend.

A SoDS Server

You'll need sods to be running any time you might need it. SoDS doesn't require much in the way of resources, so I suggest using a low powered server that you can leave running all the time. The SoDS server might be the same device that you use for your gateway.

I use a Linksys WRT54gl as my home router and sods server. But any small, lower power device will do: I've also used a Linksys NSLU2, and, more recently, a Sheeva Plug.

A Registered Domain Name

If you're even thinking about setting this up, you probably already own a domain or two.

Another DNS Server

Finally, you will need another DNS server to delegate your domain name. This could be a DNS server you run on another IP address, a service provided by your registrar or one of the many free DNS services on the internet (there are a bunch, any of them should work, google for them).

Assuming your domain name is with an IP address of, here is what you would have to set up.
  • If you have a static IP address (or one that doesn't change too frequently to be inconvenient), delegate a name server for a subdomain (in this example, "a"):    IN  A
    a               IN  NS

  • Some ISP's will change your IP, but maintain a consistent reverse DNS lookup for your IP address (something like a DHCP name). You can use this by setting a CNAME.

    For example, if your IP address always resolves to:

    You could set up your DNS entry as follows:  IN  CNAME

  • Finally, if your ISP gives you a dynamic IP address, you can use one of the dynamic DNS services. Many of these services are free and will provide a consistent reverse DNS entry that you can exploit as with the previous example.

Set Up Your SoDS Server

Set up your sods server and point it to whatever SSH or tcp servers you're interested in. These servers don't have to be publicly exposed; they could be running somewhere behind your firewall.

SoDS is configured from the command line, but its easy to wrap it in a script for system startup. Here is an example for OpenWRT:
OPTIONS="-D -L ssh.server1:22 -L ssh.server2:22 -L"

#!/bin/sh /etc/rc.common
# Copyright (C) 2006


start() {
include /lib/network
config_load /var/state/network
config_get ipaddr wan ipaddr

[ -f $DEFAULT ] && . $DEFAULT
[ ! -d $CHROOT_D ] && mkdir -p $CHROOT_D

$BIN $OPTIONS -i $ipaddr $DOMAIN

stop() {
killall sods

Accessing Your DNS Tunnel

So next time you are enjoying a venti americano in your local coffee shop, you may think to pause from chatting with the cute barista to fire up your laptop or other mobile device and login to IRC. We're geeks, it happens.
$ ssh -o ProxyCommand="./sdt"

Since UDP is innately unreliable, for best results, wrap the ssh command in a script that reconnects on failure. On the server side, if you are using ssh, running everything within a GNU screen session provides seamless access to your console, in case your connection drops. See the example scripts for more details, but basically, all you need to do is use a named GNU screen session on your ssh server:
$ while [ "1" ]; do ssh -t -C -o ProxyCommand="./sdt" "screen -drR sshdns"; done

SoDS Client

  • sdt will try not to overwhelm the local DNS server and will back off. Some DNS servers throttle chatty clients.

    To add some reliability to the protocol, both the client and server track sent packets and will re-send if they are lost.
  • If you are having problems connecting, run with multiple "-v" switches to see what's going on. It'll fill your screen with debug messages but will eventually stop. To control the number of debug messages displayed, use the "-V" switch (it defaults to 100).
  • You can bounce your connection off of other DNS servers, beside your local DNS server, if they support recursion by using the "-r" flag. To see which servers sdt knows about, run:
    sdt -v -h

    To use a particular recursive server, choose it by name:
    sdt -r google

    To pick a random server, run:
    sdt -r random
  • sods supports tunnelling using TXT, CNAME and NULL records. The default is TXT records. I've found that some ISP's re-write TXT records and many DNS servers in the wild do not support NULL records at all. Using CNAME's seems to work with most networks.

    NULL records have the lowest encoding overhead and the specification allows for a large packet size, reducing overhead further. Since sods does not support TCP, large packets aren't supported.
  • If you are on a mobile device, you probably want to conserve your battery. sdt polls for data at regular intervals by sending a request to the local DNS server to flush any pending data from your SoDS server.

    You can control this polling (without much effect on latency) by using the "-b" option. Every time data is received polling ramps up and slowly decays as no new data is recieved.

    If you happen to be on linux, powertop is a great tool for fine tuning the polling intervals.
  • Files can be transferred securely over DNS by using sftp, see the "sdftp" script in the examples. If you're brave, you can also run a fuse filesystem using sshfs.

Thursday, December 3, 2009

epcap: An Erlang Packet Sniffer

In the last entry, I talked about using erlang ports to pass messages between a Unix and an Erlang process. The data was encoded using the ei library.

As we saw, parsing the packets in C isn't hard: we cast the packet to the header structures. For example, to get the source address of the packet, we would do:
struct *iph = NULL;
const u_char *pkt; /* the packet given to us by pcap */
struct in_addr src;

iph = (struct ip *)(pkt + sizeof(struct ether_header));

src = iph->ip_src;
Erlang has a reputation for elegant and robust binary parsing. Is that true?

Parsing the ethernet header from a packet looks like this:
ether(<<Dhost:6/bytes, Shost:6/bytes, Type:2/bytes, Packet/binary>>)
We specify bytes for convenience (the default is bits). Each variable corresponds to the C structure found in net/ethernet.h.
#define ETHER_ADDR_LEN      6

struct  ether_header {
    u_char  ether_dhost[ETHER_ADDR_LEN];
    u_char  ether_shost[ETHER_ADDR_LEN];
    u_short ether_type;
The binary is stored in a record:
-record(ether, {
dhost, shost, type, crc
We return a tuple containing the header and the payload.
ether(<<Dhost:6/bytes, Shost:6/bytes, Type:2/bytes, Packet/binary>>) ->
Len = byte_size(Packet) - 4,
<<Payload:Len/bytes, CRC:4/bytes>> = Packet,
dhost = Dhost, shost = Shost,
type = Type, crc = CRC
}, Payload}.
We follow the same pattern for IPv4, TCP, UDP and ICMP. I took the code from an old project to write an Erlang TCP/IP stack, so the implementation is incomplete and very likely buggy, but it's good enough for demonstration purposes. I'll clean it up over time.

Finally, we have a port, a port driver and a parser. We can put the pieces together quickly.
start(L) ->

loop() ->
[{time, Time},{packet, Packet}] ->
try epnet:decapsulate(Packet) of
P -> dump(Time, P)
error:Error ->
io:format("~s *** Error decoding packet (~p) ***~n~p~n", [timestamp(Time), Error, Packet])
stop ->

timestamp(Now) when is_tuple(Now) ->

iso_8601_fmt(DateTime) ->
{{Year,Month,Day},{Hour,Min,Sec}} = DateTime,
lists:flatten(io_lib:format("~4.10.0B-~2.10.0B-~2.10.0B ~2.10.0B:~2.10.0B:~2.10.0B",
[Year, Month, Day, Hour, Min, Sec])).

dump(Time, [Ether, IP, #tcp{} = Hdr, Payload]) ->
io:format("~s =======================================~n", [timestamp(Time)]),
io:format("SRC:~p:~p (~s)~nDST:~p:~p (~s)~nFlags [~p] seq ~p, ack ~p, win ~p, length ~p~n", [
IP#ipv4.saddr,, string:join(epnet:ether_addr(Ether#ether.shost), ":"),
IP#ipv4.daddr, Hdr#tcp.dport, string:join(epnet:ether_addr(Ether#ether.dhost), ":"),
Hdr#tcp.seqno, Hdr#tcp.ackno,,
io:format("~s~n", [epnet:payload(Payload)]).
Note that I hardcoded the chroot directory and the default options. Running the above from the shell will display something like this:
2009-12-02 22:14:46 =======================================
SRC:{207,97,227,239}:80 (0:16:B6:xx:xx:xx)
DST:{192,168,1,100}:59341 (0:1C:B3:xx:xx:xx)
Flags ["ack"] seq 3306458405, ack 3526830431, win 46, length 492
HTTP/1.1 301 Moved Permanently..Server: nginx/0.7.61..Date: Thu, 03 Dec 2009 03:14:46 GMT..Content-Type: text/html; charset=utf-8..Connection: close..Status: 301 Moved Permanently..Location: 0ms..Content-Length: 93..Set-Cookie: _github_ses=BAh7BiIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNoSGFzaHsABjoKQHVzZWR7AA%3D%3D--884981fc5aa85daf318eeff084d98e2cff92578f; path=/; expires=Wed, 01 Jan 2020 08:00:00 GMT; HttpOnly..Cache-Control: no-cache
Occasionally, sniff will have an error parsing a packet. I haven't looked too deeply into this as yet. Check github, maybe it's already been fixed. I'll be cleaning up the code over the next few days.

So what's next for epcap? It'd be interesting building a monitoring system around Erlang or an intrustion detection system. Erlang's pattern matching would be awesome for signatures. Combining epcap with an Erlang libnet port could be the basis of a very cool vulnerability scanning engine; combined with erlang's seamless distribution, excellent web servers and frameworks, and distributed databases, it could certainly be the basis for something remarkable.

An Erlang Packet Sniffer Using ei and libpcap

After testing the erlang nif interface, I thought I'd try the ei library. ei is a way of serializing terms between Erlang and ports. Whereas nif is a function interface and is thus blocking, ports treat external processes as an Erlang process and so can use asynchronous message passing for communication.

Some packet dump utilities for Erlang already exist using linked in drivers and the Linux raw socket interface. It should be possible, as well, to use tcpdump as a port, although that involves a lot of text parsing.
epcap is a standalone binary written in C, using the pcap packet capture library. I'll go through building epcap in two parts: making a port in C and parsing the packets in Erlang. If you just want to start playing with epcap, you can get it from github. It's just a beginning, but it should be work.

If you are interested in the ei libary, I suggest reading the excellent tutorial on trapexit first.

Creating a port is a bit more work than using nif's. Since a port is just a system process, we have to handle all the bookkeeping ourselves. The way the epcap process works is as follows:
  1. start with root privileges
  2. parse command line arguments
  3. open the pcap device
  4. drop root privileges
  5. loop and read packets
  6. convert packets into Erlang binary term format and write to stdout
epcap does not receive any messages from the Erlang process. As a convenience, we have epcap fork, with the parent blocking on stdin, so that the epcap process will exit gracefully when the Erlang node closes the port.

To dump packets, we do the following:
  1. open the pcap device
  2. set a filter for the device and other initialization
  3. read one packet from the device, process it and loop
Opening the pcap device involves retrieving a default interface to sniff, if one hasn't been specified (on Mac OS X, it seems to almost always pick the wrong interface) and returning a pcap handle. After the device has been opened, we can drop root privileges.
pcap_t *
epcap_open(char *dev)
    pcap_t *p = NULL;
    char errbuf[PCAP_ERRBUF_SIZE];

    if (dev == NULL)
        PCAP_ERRBUF(dev = pcap_lookupdev(errbuf));
    PCAP_ERRBUF(p = pcap_open_live(dev, SNAPLEN, PROMISC, TIMEOUT, errbuf));

    return (p);
Initializing the device involves compiling and setting a filter:
epcap_init(EPCAP_STATE *ep)
    struct bpf_program fcode;
    char errbuf[PCAP_ERRBUF_SIZE];

    u_int32_t ipaddr = 0;
    u_int32_t ipmask = 0;

    if (pcap_lookupnet(ep->dev, &ipaddr, &ipmask, errbuf) == -1) {
        VERBOSE(1, "%s", errbuf);
        return (-1);

    VERBOSE(2, "[%s] Using filter: %s\n", __progname, ep->filt);

    if (pcap_compile(ep->p, &fcode, ep->filt, 1 /* optimize == true */, ipmask) != 0) {
        VERBOSE(1, "pcap_compile: %s", pcap_geterr(ep->p));
        return (-1);

    if (pcap_setfilter(ep->p, &fcode) != 0) {
        VERBOSE(EXIT_FAILURE, "pcap_setfilter: %s", pcap_geterr(ep->p));
        return (-1);

    return (0);
Finally, we can just sit in a loop, pulling one packet from the queue. We use pcap_next() instead of pcap_loop() so that we can apply our own flow control, if necessary, in the future (sort of like {active,once}).

When a packet is sniffed, pcap returns 2 values to us: the packet itself and a struct with a timestamp, the length of the packet that was returned to us and the actual length of the packet on the wire (since the packet given to us may have been truncated).

At this point, we can test if packet dumps are working by printing out the ethernet header:
#include <net/ethernet.h>


epcap_loop(pcap_t *p)
    struct ether_header *eh = NULL;

    eh = (struct ether_header *)pkt;

    (void)fprintf(stderr, "[shost]%02x:%02x:%02x:%02x:%02x:%02x\n", eh->ether_shost[0],
            eh->ether_shost[1], eh->ether_shost[2], eh->ether_shost[3],
            eh->ether_shost[4], eh->ether_shost[5]);

Pulling all of this together and compiling should give something like:
Now we're ready to get to the interesting part: creating data for consumption by the erlang process.
epcap_response(const u_char *pkt, struct pcap_pkthdr *hdr)
    ei_x_buff msg;

    u_int16_t len = 0;

    /* [ */
    IS_FALSE(ei_x_encode_list_header(&msg, 2));

    /* {time, {MegaSec, Sec, MicroSec}} */
    IS_FALSE(ei_x_encode_tuple_header(&msg, 2));
    IS_FALSE(ei_x_encode_atom(&msg, "time"));

    IS_FALSE(ei_x_encode_tuple_header(&msg, 3));
    IS_FALSE(ei_x_encode_long(&msg, abs(hdr->ts.tv_sec / 1000000)));
    IS_FALSE(ei_x_encode_long(&msg, hdr->ts.tv_sec % 1000000));
    IS_FALSE(ei_x_encode_long(&msg, hdr->ts.tv_usec));

    /* {packet, Packet} */
    IS_FALSE(ei_x_encode_tuple_header(&msg, 2));
    IS_FALSE(ei_x_encode_atom(&msg, "packet"));
    IS_FALSE(ei_x_encode_binary(&msg, pkt, hdr->caplen));

    /* ] */

    len = htons(msg.index);
    if (write(fileno(stdout), &len, sizeof(len)) != sizeof(len))
        errx(EXIT_FAILURE, "write header failed");

    if (write(fileno(stdout), msg.buff, msg.index) != msg.index)
        errx(EXIT_FAILURE, "write packet failed: %d", msg.index);

I'd like to send a list to the Erlang process consisting of a proplist of 2 tuples:
[{time, {MegaSeconds, Seconds, MicroSeconds}}, {packet, Packet}].
The timestamp is in the same format as erlang:now().

Creating the data structure is quite easy. ei functions come in statically and dynamically allocated versions. For creating the Erlang terms, the dynamic version is ideal.

After allocating an ei_x_buff struct, we add a version header. If we want to add a tuple, we add a tuple header with the appropriate arity and begin adding elements. If we want to add a nested tuple, it's as simple as sequentially adding another tuple header.

When calling the epcap binary, I allowed my login to run the binary as root using sudo:
$ visudo
myuser ALL = NOPASSWD: /path/to/epcap/epcap
Alternatively, you can put epcap in a directory owned by root and make epcap setuid (chown root:yourgroup epcap; chmod 4550 epcap). If you decide make epcap setuid, use a group to which your login is the only user.

The erlang port driver should be familiar from the trapexit tutorial.
start(PL) when is_list(PL) ->
    Args = make_args(PL),
    Port = Args,
    spawn_link(?MODULE, init, [self(), Port]).

init(Pid, ExtPrg) ->
    register(?MODULE, self()),
    process_flag(trap_exit, true),
    Port = open_port({spawn, ExtPrg}, [{packet, 2}, binary, exit_status]),
    loop(Pid, Port).
We take a proplist to set up the command line arguments for the C binary. We also pass in the calling processes pid, so messages can be sent back.
loop(Caller, Port) ->
        {Port, {data, Data}} ->
            Caller !  binary_to_term(Data),
            loop(Caller, Port);
        {Port, {exit_status, Status}} when Status > 128 ->
            io:format("Port terminated with signal: ~p~n", [Status - 128]),
            exit({port_terminated, Status});
        {Port, {exit_status, Status}} ->
            io:format("Port terminated with status: ~p~n", [Status]),
            exit({port_terminated, Status});
        {'EXIT', Port, Reason} ->
        stop ->
Like any other erlang process, messages from ports are captured using receive. We convert the message to erlang term format and send the message to the calling pid.
A process using epcap could work as follows:
epcap:start([{chroot, "/tmp/epcap"},{interface, "en1"}, {filter, "tcp and port 80"}]).
receive Any -> Any end.
Parsing and pretty printing the packets is covered here.

Saturday, November 28, 2009

Simple Mutable Variables using Erlang NIF's

I wrote a simple mutable variable data store to experiment with the new Erlang native implemented function (NIF) interface. Like Erlang itself, the NIF interface is simple, well thought out and robust.

I should mention that I like the immutable variables in Erlang. I only find them to be an annoyance when I want to write a quick program. In this sense they are sort of like static types in other languages; they force you to design your program before you start coding. The sort of complaints I've read about non-destructive variable updates (e.g., writing tests) seem to reinforce this view.

A quick example of using wat: you want to benchmark various checksum functions and prove the results are evenly distributed for a given set of inputs. We can benchmark a version using arrays and another using NIF's.

The array version:

start(Hash,File) ->
    {ok, Fh} = file:open(File, [
    {MicroSec, L} = timer:tc(
        ?MODULE, loop,
        [Fh, Hash, array:new([3, {default,0}])]
    T = lists:sum(L),
    P = [ N / T * 100 || N <- L ],
    {{elapsed_time, MicroSec / 1000000},
        {result, L}, {total, T}, {perc, P}}.

loop(Fh, Hash, Count) ->
    case file:read_line(Fh) of
        eof ->
        {error, Error} ->
            io:format("error:~p~n", [Error]);
        {ok, Line} ->
            N = erlang:Hash(Line) rem
            V = array:get(N, Count),
            loop(Fh, Hash, array:set(N, V+1, Count))

The NIF version:

start(Hash, File) ->
    {ok, Fh} = file:open(File, [
    {MS, A} = timer:tc(
        ?MODULE, loop,
        [Fh, Hash, 3]),
    T = lists:sum(A),
    P = [ N / T * 100 || N <- A ],
    {{elapsed_time, MS / 1000000}, {result, A}, {total, T}, {perc, P}}.

loop(Fh, Hash, Count) ->
    case file:read_line(Fh) of
        eof ->
            [ wat:get(N) ||
                N <- lists:seq(0,Count-1) ];
        {error, Error} ->
            io:format("error:~p~n", [Error]);
        {ok, Line} ->
            N = erlang:Hash(Line) rem Count,
            wat:add(N, 1), loop(Fh, Hash, Count)

Running csum/scum on a MacBook Pro with 2 cores and SMP enabled:

4> csum:start(crc32,"test.txt").
5> scum:start(crc32,"test.txt").

Using "+A 10":
5> csum:start(crc32,"test.txt").
6> scum:start(crc32,"test.txt").

Thursday, November 26, 2009

sods: A Socket over DNS Tunneling Service

So, once -- almost 2 years ago now! -- I planned on taking about a month's vacation to travel around Vietnam, Cambodia and Thailand. Roaming cell phone data plans are expensive and unreliable, and I wanted a way of quickly reading emails, browsing those important web sites (what if Linus had announced this was finally the year of the Linux desktop? what if I missed out on LOLCATS?) and checking up on my servers.

I expected wired and wireless access to be available pretty much everywhere I went, and, I was right, it was ubiquitous, though pricy. For internet access, hotels in Phnom Penh charged about $20/day, in a country where the average teacher makes around $80/month. Even for a tourist the price is a bit steep for a few minutes to chat with your friends.

Well, the alternative was figuring out a way of tunneling IP over whatever protocols these "walled garden" ISP's allowed. In the past, data could be hidden in the payload of a ping packet. Another option was piggy backing on the domain name system. Most public internet services will assign a new client an IP address, then re-direct whatever domain their browser has requested to a gateway page, where they can sign up for access. There have been many clients and tutorials that exploit this to let you tunnel your data across the internet.

But the clients I found were too complicated to setup or required specific platforms or had strange dependencies. What would be ideal would be a client that:
  • required very little in the way of additional software
  • ran completely in user space and was small and portable enough to run on a Nokia n770
  • supported different DNS message formats to get around DNS access controls
  • took basic security precautions, like using chroot and dropping privileges
  • was fast enough to minimize lag and frustration
OzymanDNS, while awesome, had a number of dependencies and required a threaded Perl -- way too complicated to install on an embedded device.

So I decided to see if I could come up with something that worked better. sods is a small client and server written in C that will tunnel ssh connections over DNS. If I remember correctly, it was at least 4 times as fast as the Perl version, but such is its awesomeness, I wouldn't be surprised if it were 10 or 20 times faster. Let me emphasize this: this client may even approach dial up line speeds.

Of course, being written in C, it has many other advantages as well: lower latency, lower memory usage, efficient use of CPU and tweaks for power saving, segmentation faults, memory leaks, the possibility of amusing buffer overflows that entail an enjoyable evening cleaning up root'ed servers. But you may just get a discount on Viagara from all of that spam you've been sending out!

I'll discuss how to set up sods in another post. If you want to download and play with sods, it can be found on github.

Thursday, November 5, 2009

Testing Syntax Highlighting with a Code Snippet

As a way of testing whether the syntax highlighting works, an Erlang example of an HTTP client for a RESTful service, adapted from the awesome RESTful Web Services.

-define(TERM, "jellyfish").

r() ->

r(Term) ->
    {ok, {_Response, _Headers, Body}} =
    http:request(get, {?URI ++ edoc_lib:escape_uri(Term), []}, [], []),

parse(Body) ->
    {Xml, _Rest} = xmerl_scan:string(Body),
    Titles = xmerl_xpath:string("/ResultSet/Result/Title/text()", Xml),
    Urls  = xmerl_xpath:string("/ResultSet/Result/ClickUrl/text()", Xml),
    [ [{title, M#xmlText.value}, {url, N#xmlText.value}] || {M,N} <- lists:zip(Titles, Urls) ].
inets needs to be started first.
Erlang R13B02 (erts-5.7.3) [source] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false]

Eshell V5.7.3  (abort with ^G)
1> inets:start().
2> wr:r("erlang").