Simple Mutable Variables using Erlang NIF's

I wrote a simple mutable variable data store to experiment with the new Erlang native implemented function (NIF) interface. Like Erlang itself, the NIF interface is simple, well thought out and robust.

I should mention that I like the immutable variables in Erlang. I only find them to be an annoyance when I want to write a quick program. In this sense they are sort of like static types in other languages; they force you to design your program before you start coding. The sort of complaints I've read about non-destructive variable updates (e.g., writing tests) seem to reinforce this view.

A quick example of using wat: you want to benchmark various checksum functions and prove the results are evenly distributed for a given set of inputs. We can benchmark a version using arrays and another using NIF's.

The array version:

-module(csum).
-export([start/2]).
-export([loop/3]).

start(Hash,File) ->
    {ok, Fh} = file:open(File, [
            binary,
            read,
            read_ahead,
            raw
        ]),
    {MicroSec, L} = timer:tc(
        ?MODULE, loop,
        [Fh, Hash, array:new([3, {default,0}])]
    ),
    T = lists:sum(L),
    P = [ N / T * 100 || N <- L ],
    {{elapsed_time, MicroSec / 1000000},
        {result, L}, {total, T}, {perc, P}}.

loop(Fh, Hash, Count) ->
    case file:read_line(Fh) of
        eof ->
            array:to_list(Count);
        {error, Error} ->
            io:format("error:~p~n", [Error]);
        {ok, Line} ->
            N = erlang:Hash(Line) rem
            array:size(Count),
            V = array:get(N, Count),
            loop(Fh, Hash, array:set(N, V+1, Count))
    end.

The NIF version:

-module(scum).
-export([start/2]).
-export([loop/3]).

start(Hash, File) ->
    wat:init(),
    {ok, Fh} = file:open(File, [
            binary,
            read,
            read_ahead,
            raw
        ]),
    {MS, A} = timer:tc(
        ?MODULE, loop,
        [Fh, Hash, 3]),
    T = lists:sum(A),
    P = [ N / T * 100 || N <- A ],
    {{elapsed_time, MS / 1000000}, {result, A}, {total, T}, {perc, P}}.

loop(Fh, Hash, Count) ->
    case file:read_line(Fh) of
        eof ->
            [ wat:get(N) ||
                N <- lists:seq(0,Count-1) ];
        {error, Error} ->
            io:format("error:~p~n", [Error]);
        {ok, Line} ->
            N = erlang:Hash(Line) rem Count,
            wat:add(N, 1), loop(Fh, Hash, Count)
    end.

Running csum/scum on a MacBook Pro with 2 cores and SMP enabled:

4> csum:start(crc32,"test.txt").
{{elapsed_time,0.668354},
{result,[49001,48963,48743]},
{total,146707},
{perc,[33.40058756569216,33.37468559782423,
33.224726836483605]}}
5> scum:start(crc32,"test.txt").
{{elapsed_time,0.642855},
{result,[49001,48963,48743]},
{total,146707},
{perc,[33.40058756569216,33.37468559782423,
33.224726836483605]}}

Using "+A 10":

5> csum:start(crc32,"test.txt").
{{elapsed_time,3.597195},
{result,[49001,48963,48743]},
{total,146707},
{perc,[33.40058756569216,33.37468559782423,
33.224726836483605]}}
6> scum:start(crc32,"test.txt").
{{elapsed_time,2.839503},
{result,[49001,48963,48743]},
{total,146707},
{perc,[33.40058756569216,33.37468559782423,
33.224726836483605]}}

sods: A Socket over DNS Tunneling Service

So, once -- almost 2 years ago now! -- I planned on taking about a month's vacation to travel around Vietnam, Cambodia and Thailand. Roaming cell phone data plans are expensive and unreliable, and I wanted a way of quickly reading emails, browsing those important web sites (what if Linus had announced this was finally the year of the Linux desktop? what if I missed out on LOLCATS?) and checking up on my servers.

I expected wired and wireless access to be available pretty much everywhere I went, and, I was right, it was ubiquitous, though pricy. For internet access, hotels in Phnom Penh charged about $20/day, in a country where the average teacher makes around $80/month. Even for a tourist the price is a bit steep for a few minutes to chat with your friends.

Well, the alternative was figuring out a way of tunneling IP over whatever protocols these "walled garden" ISP's allowed. In the past, data could be hidden in the payload of a ping packet. Another option was piggy backing on the domain name system. Most public internet services will assign a new client an IP address, then re-direct whatever domain their browser has requested to a gateway page, where they can sign up for access. There have been many clients and tutorials that exploit this to let you tunnel your data across the internet.

But the clients I found were too complicated to setup or required specific platforms or had strange dependencies. What would be ideal would be a client that:

• required very little in the way of additional software
• ran completely in user space and was small and portable enough to run on a Nokia n770
• supported different DNS message formats to get around DNS access controls
• took basic security precautions, like using chroot and dropping privileges
• was fast enough to minimize lag and frustration

OzymanDNS, while awesome, had a number of dependencies and required a threaded Perl -- way too complicated to install on an embedded device.

So I decided to see if I could come up with something that worked better. sods is a small client and server written in C that will tunnel ssh connections over DNS. If I remember correctly, it was at least 4 times as fast as the Perl version, but such is its awesomeness, I wouldn't be surprised if it were 10 or 20 times faster. Let me emphasize this: this client may even approach dial up line speeds.

Of course, being written in C, it has many other advantages as well: lower latency, lower memory usage, efficient use of CPU and tweaks for power saving, segmentation faults, memory leaks, the possibility of amusing buffer overflows that entail an enjoyable evening cleaning up root'ed servers. But you may just get a discount on Viagara from all of that spam you've been sending out!

I'll discuss how to set up sods in another post. If you want to download and play with sods, it can be found on github.

Testing Syntax Highlighting with a Code Snippet

As a way of testing whether the syntax highlighting works, an Erlang example of an HTTP client for a RESTful service, adapted from the awesome RESTful Web Services.

-module(wr).
-include_lib("xmerl/include/xmerl.hrl").

-define(URI,
    "http://api.search.yahoo.com/"
    "WebSearchService/V1/webSearch"
    "?appid=restbook&query=").
-define(TERM, "jellyfish").
-export([r/0,r/1]).

r() ->
    r(?TERM).

r(Term) ->
    {ok, {_Response, _Headers, Body}} =
    http:request(get, {?URI ++ edoc_lib:escape_uri(Term), []}, [], []),
    parse(Body).

parse(Body) ->
    {Xml, _Rest} = xmerl_scan:string(Body),
    Titles = xmerl_xpath:string("/ResultSet/Result/Title/text()", Xml),
    Urls  = xmerl_xpath:string("/ResultSet/Result/ClickUrl/text()", Xml),
    [ [{title, M#xmlText.value}, {url, N#xmlText.value}] || {M,N} <- lists:zip(Titles, Urls) ].

inets needs to be started first.

Erlang R13B02 (erts-5.7.3) [source] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false]

Eshell V5.7.3  (abort with ^G)
1> inets:start().
ok
2> wr:r("erlang").