Erlang: Supervising two processes from a single supervisor

Only because this ate over an hour of my day today.

Below is a shell example of a supervisor that starts two child processes which are also supervisors.

-module(arti_sup).

-behaviour(supervisor).

%% API
-export([start_link/0]).

%% Supervisor callbacks
-export([init/1]).

%% Helper macro for declaring children of supervisor
-define(CHILD(I, Type), {I, {I, start_link, []}, permanent, 5000, Type, [I]}).

%% ===================================================================
%% API functions
%% ===================================================================

start_link() ->
    supervisor:start_link({local, ?MODULE}, ?MODULE, []).

%% ===================================================================
%% Supervisor callbacks
%% ===================================================================

init([]) ->
    Hip_Sup = ?CHILD(parser_sup , supervisor),
    ServiceSup = ?CHILD(service_sup , supervisor),

    Children = [ Hip_Sup, ServiceSup ],
    RestartStrategy = { one_for_one , 4, 9600},
    {ok, { RestartStrategy, Children } }. 
 
I have to other supervisors parser_sup and  and service_sup.

If you find that no matter what you change in this file doesn't seem to make a difference to which supervisor is being called, check that this supervisor itself is being started from the application filename_app.erl ,

If all goes well, you can use the sasl application to check the supervisor / launch information.

1> application:start(sasl).
ok

=PROGRESS REPORT==== 11-Oct-2011::01:59:07 ===
          supervisor: {local,sasl_safe_sup}
             started: [{pid,<0.41.0>},
                       {name,alarm_handler},
                       {mfargs,{alarm_handler,start_link,[]}},
                       {restart_type,permanent},
                       {shutdown,2000},
                       {child_type,worker}]
 

<removed for brevity...>

2> application:start(myapplication).
INIT parser supervisor
INIT service supervisor

=PROGRESS REPORT==== 11-Oct-2011::01:59:14 ===
          supervisor: {local,human_input_parser_sup}
             started: [{pid,<0.51.0>},
                       {name,human_input_parser_srv},
                       {mfargs,{human_input_parser_srv,start_link,[]}},
                       {restart_type,permanent},
                       {shutdown,5000},
                       {child_type,worker}]

=PROGRESS REPORT==== 11-Oct-2011::01:59:14 ===
          supervisor: {local,arti_sup}
             started: [{pid,<0.50.0>},
                       {name,human_input_parser_sup},
                       {mfargs,{human_input_parser_sup,start_link,[]}},
                       {restart_type,permanent},
                       {shutdown,5000},
                       {child_type,supervisor}]

=PROGRESS REPORT==== 11-Oct-2011::01:59:14 ===
          supervisor: {local,service_sup}
             started: [{pid,<0.53.0>},
                       {name,service_srv},
                       {mfargs,{service_srv,start_link,[]}},
                       {restart_type,permanent},
                       {shutdown,5000},
                       {child_type,worker}]

=PROGRESS REPORT==== 11-Oct-2011::01:59:14 ===
          supervisor: {local,arti_sup}
             started: [{pid,<0.52.0>},
                       {name,service_sup},
                       {mfargs,{service_sup,start_link,[]}},
                       {restart_type,permanent},
                       {shutdown,5000},
                       {child_type,supervisor}]

=PROGRESS REPORT==== 11-Oct-2011::01:59:14 ===
         application: arti
          started_at: nonode@nohost
ok

You can also use appmon to get a visualization of the supervisor tree with the command

3> appmon:start().
{ok,<0.55.0>}

This should present a graphical application which shows the supervisor/process trees currently running and is updated in real time.

Rust basics: Numbers

Rust is a systems level language, it is my understanding that it provides the basic "types" of numbers that most system languages do.

There are two types of numbers that can be used by rust:
 

Integer literals, floating point literals.

Integer Literals

I figured that quoting from the authoritative spec means I can't be too wrong, it says:


An integer literal has one of three forms:

1. A decimal literal starts with a decimal digit and continues with any mixture of decimal digits and underscores.
2. A hex literal starts with the character sequence U+0030 U+0078 ("0x") and continues as any mixture hex digits and underscores.
3. A binary literal starts with the character sequence U+0030 U+0062 ("0b") and continues as any mixture binary digits and underscores.

I don't understand the interest of allowing underscores,  I guess this means programmers can make values like 0xDEAD_BEEF_BABE or 0b_1111_0000 more readable.

Ah, but you can also specify the container (bitsize) size  of the object by  appending an integer suffix.  

• The u suffix gives the literal type uint.
  The g suffix gives the literal type big.
• "Signed" types i8, i16, i32 and i64
• "unsigned" types u8, u16, u32, u64.

An example would be 0xDEAD_u32 or 18_i64 .  If you believe it the underscores can be placed anywhere that the programmer likes to make the values more readable.

Floating Point Literals

And again from the spec:

 A floating-point literal has one of two forms:

1. Two decimal literals separated by a period character U+002E (’.’), with an optional
   exponent trailing after the second decimal literal.
2. A single decimal literal followed by an exponent.

Nothing too exciting here, things like 0.01 and 99.999999 and 0.99.  You can also add two optional floating suffix values for your enjoyment.

• The f32 suffix
• The f64 suffix

These are  unsurprisingly specify the size of the container storing your value.



Example time !

So for a basic example:

use std;
import std::io;

fn main() {

    let stdout = io::stdout();

    let x = 127u;

    stdout.write_line(#fmt["X = %u", x]);
}

This as you'd expect outputs the number "127".  Nothing too complex.  We can see that the type of "x" is inferred as an unsigned integer.

So, lets do the usual thing and test assignment of a different type, does in infer correctly ?

$ rustc assignment.rs -o assignment
assignment.rs:10:6: 10:7 error: mismatched types: expected uint but found int (types differ)
assignment.rs:10         x = 5;

Rust fails the compile, and graciously gives you the opportunity to fix this bug.  The quick and dirty solution is to change the assignment value to 5u, but if this value was created by another function you'd have to do what other languages consider casting, using the as keyword.

So you'd do it like this:

use std;
import std::io;

fn main() {

    let stdout = io::stdout();

    let     x = 127u8;
       
            x = (x + 128u8 ) as u8;

    stdout.write_line(#fmt["X = %u", x as uint]);
}

The reason why the equation 127 + 128 was chosen is because 255 is the largest value that can be stored in a 8 bit unsigned value. 

The first "as" in the above example is superfluous, but it remains as an example of how you can cast/convert a value that may be converted or passed by another function or module that you didn't write.

Note that in the above example no data was lost.  However if the final value of x is over 255, in the assignment line ( x = (x + 128u8 ) as u8 ) this is where the x value will be 'overflowed' not in the final stdout.write_line() function.

Much like C, Fortran, Algol or any other language when you use specific types you must be careful about overflowing the maximum.  I was reading the source code and came across the auto-expanding "big" type, but I'll explore that for another day.

I guess that is as good of a start as any.  I probably should put something about "using crates" but I don't know enough about them to be useful yet.

Famicom Final Fantasy II Japanese to English Conversion

I have always been a big fan of native hardware. Sure, emulators to a certain extent would give you a "taste" of what the device can do. But as they say, nothing beats the real thing. It could be the grinding sound a floppy disk drive makes as the head tries to seek past track 0, or little things such as millisecond delay of a game controller compared to native hardware.

I have played a few of the Final Fantasy games. One day I decided I'd play through all of the games in the main series. Not on an emulator though. On the real thing.

For some reason some of the games in the series were only released in Japan. Since my grasp of the language is quite limited, I thought I would look for a fan translated version of the game.

In order to begin I set up my Family Computer (Japanese version of the NES). Found a copy of Final Fantasy II. First thing to do of course, was to make sure this cart is still working (it was). And after confirming that, it's time to take apart the cart. Famicom carts are held together by 4 plastic tabs, and it is very difficult to open up a cart and keep them intact. Super glue can restore broken off tabs.

Next we need to desolder the ROM chip(s). Final Fantasy II only uses 1 ROM chip. I carefully desoldered every pin and got the chip out. Next is to find some pinouts of the mask ROM, and how different it is from the EPROM I was planning to use (AM27C20). I found them here:

http://www.raphnet.net/electronique/nes_cart/nes_cart_en.php

http://nesdev.parodius.com/NES%20ROM%20Pinouts.txt

http://nintendoallstars.w.interia.pl/romlab/nesmod.htm

After modifying the board, this is how it looks:

And after burning the eprom and soldering it on:

Of course, I have to test it first before closing the case:

Moment of truth: (it works!)

The save files are still there too, and the names are now garbled:

Rust vim syntax and indent files.

I'm a bit of a vim junkie, and if you're like me, you'll almost definitely want to have your vim configured for the correct rust syntax.  Not just for the pretty colors but to let you know when you're missing a quote. 

Patrick Walton must be a vim user as he seems to have written the vim syntax file for rust, You can pull it from the rust source code or from the git repository, and put it in the relevant vim settings folder (.vim on unix)


After copying the files to this folder, add the following  line to your vimrc:

au BufRead,BufNewFile *.rs setfiletype rust

 (usually $HOME/.vimrc or $HOME/.gvimrc ).  Reload your vim settings or restart vim and open a rust source code file.

You should see your rust files in all their syntactic highlighted glory.

Hello World in Rust

Rust is a low level systems language developed by Mozilla for future embedding into their browser and other tools.  I was first made aware of it a year ago on the programming subreddit, but it looked a little too abstract to be useful.

I stumbled across it again yesterday and decided to compile the language tool chain and give it another go.

The language seems to have some very nice functionality, a cross between erlang and C, which may suit my fancy.

The toolchain was a bit annoying to set up, and I might go harrass the #Rust irc channel to see if there is a simpler way to get started.

In the mean time, here is hello world.

-- hello-world.rs -- 

use std;
import std::io;

fn main(argv: [str]) {

    let out = io::stdout();

    out.write_line("Hello world");
   
}

I compiled it with the command:

$ rustc hello-world.rs -o hello-world -L /usr/local/lib/rust/

$ ./hello-world

Hello world

Fin.

Converting to and from hex in Erlang.

$ erl
Erlang R14B03 (erts-5.8.4) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false]
Eshell V5.8.4  (abort with ^G)



%% The base (16) followed by the hex value (0F), number is returned as a decimal.

1> 16#0F.
15


%% lets convert our value back,

2> integer_to_list(15, 16).
"0F"


This is of course covered in the erlang manuals, but Its nice to have it in one place.

I regularly work in crash so, I need to do basic math in hex, which I usually do something like this

3>  integer_to_list(16#0F + 16#FF, 16).
"10E"

Starting with rebar, live upgrades and packaging erlang code RPM

Rebar is something that I wish I was taught when I first started using erlang, it will help new programmers get started with erlang without having to do the boring "book-keeping" behind the setup/compile/test/release system.  It really is a good way to get started.

Some readers may note that there are many rebar tutorials out on the net, however nothing for Red Hat style Linux, and many of them more complicated than they need be.

Installing Rebar

I'm lazy and always use a relatively modern Linux distribution. If you're using Fedora 15/16 or Red Hat Enterprise Linux 5 or 6 (with EPEL) you can install rebar with the command

$ sudo yum install erlang-rebar

This will also install the necessary erlang runtime/vm and other related tools, correct for whatever architecture your system is running.

Creating your first erlang project

[wmealing@mutalisk wmealing] mkdir myfirstapp  
[wmealing@mutalisk wmealing] cd myfirstapp 

[wmealing@mutalisk myfirstapp] rebar create-app appid=something
==>; myfirstapp (create-app)
Writing src/something.app.src
Writing src/something_app.erl

Writing src/something_sup.erl

If you get the error "ERROR: Template simpleapp not found." when attempting to try the above commands, copy the templates into the user-specific template directory.

[wmealing@mutalisk myfirstapp] mkdir -p $HOME/.rebar/templates/

[wmealing@mutalisk myfirstapp] cp /usr/lib*/erlang/lib/rebar-2/priv/templates/* $HOME/.rebar/templates/

Backup cowboy, what just happened ? Rebar created an app based on the templates that are included.  These templates are located in the /usr/lib64/erlang/lib/rebar-2/priv/templates/ directory if you're on 64 bit  or if you're still using 32 bit erlang, /usr/lib/erlang/lib/rebar-2/priv/templates/ directory.  So it creates from the "simpleapp*" named files substituting whatever you supplied for the appid where it is mentioned in the templates.

Unsurprisingly this created the src directory and a simple "application" with the appid provided on the command line.  Can I suggest you be a little bit more creative than my example.

Even though you have not written any code yet, run compile and test that the skeleton that you have created compiles.

Compiling

[wmealing@mutalisk myfirstapp] rebar compile
==> myfirstapp (compile)
Compiled src/something_app.erl
Compiled src/something_sup.erl

And yeah! your first template-style erlang app compiled, and the "beam" code is in the ebin directory.  See the project layout so far in the example below.

[wmealing@mutalisk workspace] tree myfirstapp
myfirstapp
|-- ebin
|   |-- something.app
|   |-- something_app.beam
|   `-- something_sup.beam
`-- src
    |-- something_app.erl
    |-- something.app.src
    `-- something_sup.erl

2 directories, 6 files

The project so far is pretty much a basic supervisor for your application, this by itself isn't quite enough to be useful.  Usually you'd want your application to do something useful.  This typically involves creating  a "server" of some kind.  The most common is the gen_server.

Many of the other rebar tutorials seem to either enjoy making additional complex files however modern releases of rebar already include templates for a basic server.

Creating a server

[wmealing@mutalisk myfirstapp]$ rebar create template=simplesrv srvid=mine_srv
==> myfirstapp (create)
Writing src/mine_srv.erl

This creates a gen_server style server in the src/mine_srv.erl file that you can modify to how you see fit.

Lets make it greet the user in a stereotypical Australian greeting, for use later when we show hot-code-upgrading in the section "Running the code".

So, modify src/mine_srv.erl and its existing handle_cast function to print the Aussie greeting to the stdout in the gen_server cast.

-export([greet/0]).

greet() ->
    gen_server:cast(?MODULE, {matchallthethings}).

handle_cast(_Msg, State) ->
  io:format("Gday Mate, how are ya ?~n"), 

  {noreply, State}.

We added the exported "greet" call so we can abstract away the implementation from the caller.  We wont test it now, but we'll test it later.

To have mine_srv be supervised/started by the application, the something_sup (supervisor) will need to be instructed to start mine_srv as a worker.

In src/something_sup.erl lets change the init function to start the mine_srv worker.

init([]) ->
    MineSrvWorker = ?CHILD(mine_srv , worker),
    Children = [ MineSrvWorker ],
    RestartStrategy = { one_for_one , 4, 9600},
    {ok, { RestartStrategy, Children } }.

Cleaning up (make clean)

Just so we can test how it works, issue the command rebar clean and rebar compile to compile the supervisor and your new gen_server server.  Just as you can compile with erlang, you can also use rebar to  "make clean" and recompile as you would when any project is changed.

[wmealing@mutalisk myfirstapp]$ rebar clean
==> myfirstapp (clean)
[wmealing@mutalisk myfirstapp]$ rebar compile
==> myfirstapp (compile)
Compiled src/something_app.erl
Compiled src/mine_srv.erl
Compiled src/something_sup.erl

Running tests

You may or  of running unit tests or integration tests, rebar supports a few kind of tests, but i'll kick it off with eunit to get things started.

Rebar looks at eunits configuration options through a file named rebar.config  file. The erlang-rebar package puts one in /usr/share/doc/erlang-rebar-2/rebar.config.sample by default, but you should be able to make an empty one with just the following lines in it.

[wmealing@mutalisk myfirstapp]$ cat rebar.config

%% Erlang compiler options

{erl_opts, [{i, "test"}, {src_dirs, ["src"]},

            {platform_define,

            "(linux|solaris|freebsd|darwin)", 'HAVE_SENDFILE'},

            {platform_define, "(linux|freebsd)", 'BACKLOG', 128}]}.

{eunit_opts, [verbose, {report, {eunit_surefire, [{dir, "."}]}}]}.

{cover_enabled, true}.

This line ensures that eunit will include the test directory in its locations to search for "included" files.

Create a test directory

[wmealing@mutalisk myfirstapp]$ mkdir test

For each of the parts of your project you wish to test, add something similar to this at the end.  For my example I append the below to mine_srv.erl.

-ifdef(TEST).
-include("something_tests.hrl").
-endif.

The HRL file extention is uncommon and is intended to show that the contents of the file should be included in another erl file.

Next create the test/something_tests.hrl file.  This should contain standard eunit tests.  The name something_tests.hrl should accurately reflect what the "something" is that you are testing. It is a good practice to name it something related to the module or server  and type of testing you are doing. Here is a quick example of a very simple test in the the test/something_tests.hrl file.

-include_lib("eunit/include/eunit.hrl").

my_test() ->
    ?assert(0 + 0 =:=  0).

my_second_test() ->
    ?assert(0 + 1 =:= 1 ).

simple_test() ->
            ?assert(1 + 2 =:= 3).

To run these tests run the command:

[wmealing@mutalisk myfirstapp]$ rebar eunit
==> myfirstapp (eunit)
======================== EUnit ========================
module 'something_sup'
module 'mine_srv'
  mine_srv: my_test...ok
  mine_srv: my_second_test...ok
  mine_srv: simple_test...ok
  [done in 0.008 s]
module 'something_app'
=======================================================
  All 3 tests passed.

Cover analysis: /home/wmealing/Documents/workspace/myfirstapp/.eunit/index.html


A source code coverage analysis can be found in the the url above.
I'll leave it as an exercise to the reader to figure out what it shows when tests go wrong.

Creating Documentation

Rebar can build documentation on your API, or whatever additional information you wish to provide assuming you make it happen in the edoc format.

Edoc requires writing documentation in code. It uses the  @something style tagging and immediately documents the function right below it.  For example:

  %% @doc Greets the user on the standard output, returns nothing

  greet() ->

       gen_server:cast(?MODULE, {matchallthethings}). 

The tags must follow a comment (%%) followed by a space starting on the first character of the line.

There are a bunch of supported tags, the edoc welcome page has a bunch more on the topic.  Rebar makes it easy to generate html documentation from your code by running the command

[wmealing@mutalisk myfirstapp]$ rebar doc
==> myfirstapp (doc)


Stylized html format pages will  appear in the docs subdirectory, you can regenerate them at any time, but this will destroy any changes made to the files in the doc directory.

Running your code.

One thing that rebar does not do, is run your application for you.   You will need to run it yourself.  Fortunately its actually pretty easy.

[wmealing@mutalisk myfirstapp]$ erl -pa ebin/
Erlang R14B03 (erts-5.8.4) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false]
1> application:start(something).
ok

Now to test the greet function we wrote earlier:

2> mine_srv:greet().
Gday Mate, how are ya ?
ok


Building the first release

It is rarely the case that the software will only run on the system that it is developed for.  The smart cookies at Ericsson and rebar know this and have a release mechanism that can be used to deploy the "whole stack" as a release.  This includes all the files, applications and libraries that would be used to run the application.

Rebar simplifies the release requirements, but its still a little involved to "get right".   Lets walk through using the existing sample to build a first release to deploy on a system.

[wmealing@mutalisk myfirstapp]$ mkdir rel
[wmealing@mutalisk myfirstapp]$ cd rel/

[wmealing@mutalisk myfirstapp]$ rebar create-node nodeid=something_node

==> myfirstapp (create-node)
Writing reltool.config
Writing files/erl
Writing files/nodetool
Writing files/something_node
Writing files/app.config
Writing files/vm.args

The name here shouldn't be labelled after a release codename or version but the "service" or daemon name that you'd like to start the script with.


Hot upgrades

@fixme This is something I also plan to cover another day.


Wrapping it up.

So rebar is a very useful tool to add to your arsenal, and if it is used for all your projects it will make a good head start on creating standardized, simple maintainable code that other (seasoned) erlang programmers should be familiar with.


Footnotes

I know there could be a better project layout for this, but with the mindset of keeping it simple, I've negated mentioning multiple applications and some of the finer points of migrating state in the gen_server between updates, this should be considered when you update your erlang application.