speak, it ain't illegal yet

August 9, 2010

ErlangCamp Chicago 2010

Filed under: 2010,camp,chicago,conference,erlang,erlangcamp,otp,Uncategorized — Jordan Wilberding @ 1:29 pm


I am happy to to announce ErlangCamp Chicago 2010 (http://erlangcamp.com/) has gone live and is ready to take registrations. This has been in planning for quite a while between Martin Logan, Eric Merritt, Tristan Sloughter, and myself. We are trying to help build the Erlang community here in the U.S. We hope by providing an affordable conference that focuses on the fundamentals of production-grade development, we can foster a solid base of strong Erlang programmers. If you have used Erlang and are interested in learning more, I encourage you to attend. With the prevalence of new Erlang jobs becoming available here in the United States, having advanced OTP knowledge will prepare you for working for them. Hope to see you all there!

April 3, 2007

Writing a duplicate file finder in Erlang

Filed under: duplicate files,edupe,erlang,programming — Jordan Wilberding @ 2:38 pm

A couple of days ago I decided I wanted to go through my collection of music in order to get rid of duplicates. I would have assumed that this problem had been solved already, but when I googled “linux file duplicate finder” the results were not so great. The first result was a program called FSlint. It wasn’t in portage, and I didn’t feel like finding out what its dependencies were, so I skipped to the next result, which was someones duplicate file finder that they wrote in Java. It was amazingly slow.

Therefore, I decided, what the heck, I will just write my own in Erlang! It sounded simple enough, just get a complete file listing, sort the files by their file size, then go through and compare files that have the same size(since a different file size would mean a different file), if you find files with the same size, get the MD5 sum, then compare those.

It turned out to be slightly more complicated than that. First of all, the filelib:fold_files function gets stuck in an infinite loop if you have a symlink that points to a higher parent directory in which that symlink is contained. Wine actually does this when you set it up. So I had to write my own function to generate a file listing that just ignores symlinks for now.

Another thing I found out was the amazement of how many files I had that were the same size. Taking hashes of all these files is a pain, especially if they are large. So I decided I would instead just take the first 256 bytes and the last 256 bytes of the file and hash that. If the file was less than 512 bytes, I just hash the whole file. Using this method increased the performance by 400% and maintained the same accuracy, though is technically still a small possibility of a false positive, but it is good enough for what I need.

The funny thing was, after I was almost done, a friend on the #chiglug@irc.oftc.net channel referred me to a perl program called fdupe. I tried it out and found out it actually did what I wanted and used close to the same methods. So I decided I would benchmark it against my implementation.


For a collection of 35,042 files that totaled 168GB, my implementation was 23.77% faster
fdupe: 2m23s
edupe: 1m49s

For a collection of 220,463 files that totaled 85GB, my implementation was 16.46% faster
fdupe: 35m39s
edupe: 29m47s

It is good to see that Erlang can hold its own.

I have released the code under the name EDupe, and you can download the latest code here.

November 15, 2006

Adding AES Encryption To Erlang Chat

Filed under: aes,chat,cryptography,dh,diffie-hellman,echat,encryption,erlang,irc — Jordan Wilberding @ 5:01 am

I have been working on adding encryption to an IRC clone that I am working on with Tristan Sloughter called EChat. Just like IRC, in normal operation EChat sends all communications to the server, then the server figures out who needs to get what. So, for my initial stab at adding encryption to EChat, I decided to encrypt the transmissions between clients and the server, instead of end to end.

Client1 Server Client2

So basically, the server does all the key management. Since AES requires a symmetric 128-bit key, we need a method of having each client the server generate a key. That is where Diffie-Hellman(DH) comes in!

DH is a quick and easy way to create symmetric keys between two people without ever having to reveal what the key actually is in plain text.

DH can be explained alot better here, than I can do, so I won’t spend the time going over it. However I will talk about a few specifics of my implementation in Erlang.

In the algorithm, a and b both need to be a certain size in order to guarantee there are 128 bits in the final key. Therefore, to create them I simply did the following:

gen_DHa() -> gen_DHb().
gen_DHb() -> crypto:rand_uniform(170141183460469231731687303715884105729,
                    340282366920938463463374607431768211455).

For the curious, 2^127+1 = 170141183460469231731687303715884105729,
and 2^128-1 = 340282366920938463463374607431768211455. Generating a number in this range assures us we get a number that is at least 128 bits.

Also, to generate g and h, I just did the following:

gen_DHp() -> gen_DHg().
gen_DHg() -> make_prime(50).

On the client side, we calculate A as follows:

DHg = ecrypt:gen_DHg(),
DHp = ecrypt:gen_DHp(),
DHa = ecrypt:gen_DHa(),
A = crypto:mod_exp(DHg, DHa, DHp).

On the server side we calculate B as follows:

DHb = ecrypt:gen_DHb(),
B = crypto:mod_exp(DHg, DHb, DHp).

Now the client and server exchange A and B, and each can calculate the key
appropriately:

Client: Key = crypto:mod_exp(B, DHa, DHp).
Server: Key = crypto:mod_exp(A, DHb, DHp).

Now we have a key that is at least 128 bits. We want to make sure it is exactly 128 bits though, so I wrote the following function that ensures this. All it does is generate exactly 16 bytes from the stream of bits:

integerlist_to_key([], Key, _) -> lists:concat(lists:sublist(Key, 16));
integerlist_to_key([Head | Tail], Key, Tmp) ->
Test = list_to_integer(lists:reverse([Head | Tmp])),
 if
     Test > 255 -> integerlist_to_key(Tail, [list_to_atom([list_to_integer(Tmp)])| Key], [Head]);
     true -> integerlist_to_key(Tail, Key, lists:reverse([Head | Tmp]))
 end.

Now that we have a 128-bit symmetric key for both the client and server, we can start using AES encryption!

To find out about AES, read here. Basically AES requires an initialization vector(IV) for each message, as well as a symmetric key, and whatever text you want to encode.

To create the Encrypt and Decrypt functions, I decided just to append the IV to the front of each message sent. The IV does not need to be kept secret, and is only used to deter statistical attacks on the encrypted message. Thus, the following code shows how I did the encrypt and decrypt:

encrypt(Message, Key) ->
IV = crypto:rand_bytes(16),
list_to_binary([IV] ++ [crypto:aes_cfb_128_encrypt(Key, IV, Message)]).

decrypt(Message, Key) ->
{IV, Crypt} = lists:split(16, binary_to_list(Message)),
binary_to_list(crypto:aes_cfb_128_decrypt(Key, list_to_binary(IV),
             list_to_binary(Crypt))).

That’s all there is to it! Now you can add AES encryption to your application, whether it is for messaging or file storage. If you would like to find out more about the Erlang AES implementation, go here.

October 31, 2006

Serializing Erlang Tuples For Network Transmission

Filed under: erlang,network,serializing,ssl — Jordan Wilberding @ 7:44 am

I have been playing around a bit with setting up an SSL client/server connection between nodes using Erlang. As you have guessed, the SSL module in Erlang expects you to send and receive binary data. You may also know, that the typical way of sending data between nodes in Erlang is by using the ‘!’ operator. For example:

Pid ! {data, SomeList}

Now since the SSL uses sockets to send data, we need to convert from the tuple to a binary form. Luckily, Erlang provides a facility to do this!

On the encoding side all you need to do is use the Erlang term_to_binary Built In Function(BIF):

Data = term_to_binary({data, SomeList})

Then we send it away.

ssl:send(CSock, Data)

Now we receive the data:

{ok, DataRecv} = ssl:recv(SSock, 0)

Then we do the decoding. ssl:send automaticlly converts the binary stream to a list when sending, so we need to convert it back, then go from binary_to_term to reverse our initial encoding.

{ok,OrigData}=binary_to_term(list_to_binary(DataList))

That’s all there is to it!

October 30, 2006

ECal First Beta Release!

Filed under: calendar,ecal,erlang,event planner — Jordan Wilberding @ 11:18 am

I have made my first release of ECal! It is a personal calendar and event organizer written in Erlang. Right now it is just meant for the command line, but later on I will probably make a nice little GUI for it with wxErlang. The program really is nice and handy for those of us who live and breathe on the command line and want a simple way to stay organized and remember meetings.

October 27, 2006

Visualizing problems in Erlang

Filed under: erlang,mergesort — Jordan Wilberding @ 7:31 am


Ever since I started learning Erlang almost 10 months ago, I have always just had this deep down unexplainable gut feeling that Erlang is neat. Erlang just feels so right. The flow from idea to implementation in Erlang is so smooth, sometimes I second guess myself, "can it really be this easy?!"

Last night I couldn't sleep, so naturally I stayed in bed thinking about Erlang, and I think I might have figured out why Erlang is so nice: visualization.

Let's just take an example. Let's say I tell you to write Quick Sort in C.

The first thing you would probably do is look up the algorithm which is:

function quicksort(q)
    var list less, pivotList, greater
    if length(q) = 1
        return q
    select a pivot value pivot from q
    for each x in q except the pivot element
        if x
To code the algorithm, you need to start thinking about the implementation details, such as how you will manage the pivot list, what type of values will your quick sort support, etc. It is very difficult to go from the algorithm directly to C, there are too many C things to worry about first.
void quickSort(int numbers[], int array_size)
{
 q_sort(numbers, 0, array_size - 1);
}
void q_sort(int numbers[], int left, int right)
{
 int pivot, l_hold, r_hold;
 l_hold = left;
 r_hold = right;
 pivot = numbers[left];
 while (left = pivot) && (left  pivot)
   q_sort(numbers, pivot+1, right);
}

Now let's say I tell you to think about writing a Merge Sort in Erlang. You instantly realize this problem is dead easy using a list a few recursion calls and Erlang's powerful list operators. There are no memory issues to think about, no type checking to worry about, no extra temporary values to handle, no having to "translate" your programs calls to fit the algorithm. With Erlang you can actually imagine in your head, having a long list of things, doing a few list operations, then using recursion and being done.

sort([Pivot|T]) ->
   sort([ X || X = Pivot]);
sort([]) -> [].

While I admit it takes a functional mindset to get used to seeing things this way, once you do have it, it is an almost instantaneous process of going from algorithm to code. Being able to visualize both the algorithm and the Erlang code together at the same time without much difficulty and implementing them immediately makes programming a lot more enjoyable and fulfilling. Gone are the days of tedious details and mindless translations.

References:
1.http://en.wikipedia.org/wiki/Quicksort
2.http://www.erlang.org/doc/doc-5.4.12/doc/programming_examples/list_comprehensions.html

October 21, 2006

How to compile wxErlang on Linux

Filed under: erlang — Jordan Wilberding @ 7:34 am

wxErlang is an interface for Erlang to the popular wxWidgets GUI library. Combining Erlang and wxWidgets creates an unbeatable combination for GUI writing. First of all we inherit the power of having a true concurrent functional programming with Erlang and add to that the fact that we can write a program once but have it use the native GUI for Linux, MacOSX, Windows, etc. This means if you execute the program in Linux, you will get a nice Gnome look, if you execute in MacOSX, you will net the nice Mac look, all with one single program.

wxErlang is still in an early version, but alot of documentation is provided how to use it, the only trouble one might face is actually compiling the thing.

Before you compile, make sure to have the following programs installed on your computer:
1. Erlang
2. wxGTK
3. Doxygen

Now download wxErlang and let’s start.

1. tar zxvf wxe-*.tar.gz
2. cd wxe
3. Edit config.mk and change the following values:
a. PLATFORM=MACOSX
b. ERLI_VSN=`echo /usr/lib/erlang/lib/erl_interface-* | awk ‘BEGIN{ FS=”erl_interface-” } { print $2 }’`
c. WXCONFIG=`which wx-config`
4. cd gen
5. make xml
6. cd ..
7. make

Now if that worked, you should be able to try the example program:
1. cd etop
2. make
3. erl -pa ../ebin -run erltop start

If that worked, congratulations! You are now well on your way to learning more and writing some real applications.

Theme: Rubric. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.