blog.rastageeks.org

Unix.open_process* and file descriptors

Toots — 2012-03-15T00:23:26Z

Hi fellow OCamlers, long time no see! :-)

I would like to mention here an issue with Unix.open_process* that may concern some of you here.

Processes opened using Unix.open_process in POSIX systems are spawned using fork(). Therefore, they inherit all file descriptors opened by the calling process.

This leads to many unfortunate minor issues. For instance, if the calling process has opened a port and crashes after calling the external process, then that port remains open until the external process terminates..

More importantly, this is a source of potentially important security issues: by default, processes spawned with Unix.open_process* will have access to any opened file or socket, allowing them to those read files' content or sniff network traffic..

The problem is not easy to fix using the standard POSIX API. If you have control over all opened file descriptor in your code, you can use close-on-exec on each of them, which makes sure that they are closed when forking. However, for large programs this quickly turns out to be impractical, in particular if your program is using libraries that open file descriptors.

Another common solution is to emulate BSD's closefrom() function, which closes all file descriptors higher than a given number. You can find a portable but possibly very slow implementation of this function in openssh's code: http://anoncvs.mindrot.org/index.cgi/openssh/openbsd-compat/bsd-closefrom.c?view=markup

OCaml developers are aware of the situation but haven't found a suitable fix for them. Here is a comment from Xavier Leroy:

Feel free to publicize this PR so that Unix experts out there can chime in.

This is an old issue that popped up during the development of Cash (the Caml shell), in particular. The "ideal" solution is to have all file descriptors in close-on-exec mode, but it requires discipline from the programmers. (File descriptors created by Pervasives.open_in and Pervasives.open_out are close-on-exec, but not those created by Unix.openfile or Unix.socket, for compatibility with the Unix specs.)

Also, a long time ago we experimented with a closeall()/closefrom() function that did not use /proc//fd, and it was really slow to call close() on thousands of potential file descriptors. The /proc//fd trick is nice but not terribly portable.

This issue can be a problem for some of you so I wanted with this post to properly document it. It is particularly crucial if you implement any kind of delegation to a less powerful system user using forked processes, for instance when implementing a CGI interface..

Take care y'all!

Type fascism..

Toots — 2011-08-26T01:06:44Z

I used to think of me as a libertarian.. Maybe it is because I am getting old and grumpy but I've found myself lately being more of a totalitarian rigid conservative.. concerning typing and APIs.. :-)

Liquidsoap being a stream-oriented software, we had felt the need for a long time to be able to manipulate mp3 id3 tags without them being attached to an actual file.. This is certainly possible in theory: id3 tags are not part of the mpeg I/II audio layer 3 format but an addition in the form of a binary blob, either at the beginning or at the end of the file. However, in practice there are very few libraries that actually support that — in fact there are very few implementations of the id3 tag specifications..

Anyway, I realize lately that, twisting TagLib's arm, it was actually possible to implement this using this clean, maintained and well-designed library.. So I went on and upgraded our binding, adding what I called inline tags support. This allows to parse existing id3 binary tags or generate a binary representation of an inline tag that can be written to a file, or a stream in our case.

However, there are some constraints to consider in order to assure a proper use of the library, namely:

The user should be able to allocate empty inline tags.
An empty inline tag can be filled by parsing a binary blob representing an actual id3 tag
An can also be filled directly with id3 frames.
An initial empty inline tag is valid iff it has been filled with at least one frame or parsed from a valid id3 binary tag
Only valid inline tags should be rendered as binary blobs.

To make things more tricky, parsing a binary representation is done in two steps:

Parsing of the header, some initial 10 bytes. Only after this step is the total size of the binary data known.
Parsing of the full tag, using the size returned during header parsing.

And, of course, one should not do anything after parsing the header expect parsing the full tag, as the current state of the tag after parsing only the header is not consistent yet..

So, with all these considerations in mind, I wondered how it could be possible to restrict the freedom of the users while adding as few of a burden as possible, both for me and them.. As it turned out, phantom polymorphic types provide a very neat way of doing so!

Reading from above, an inline tag has 3 possible states:

type state = [`Invalid | `Parsed | `Valid]

`Invalid is the initial state of an empty tag
`Parsed is the state of a tag after parsing its header
`Valid is the state of a tag which is valid for rendering

Then, the full signature is as follows:

 type state = [ `Invalid | `Parsed | `Valid ]
 
 type 'a t 
 
 val init : unit -> [`Invalid] t
 
 val header_size : int
 
 val parse_header : [`Invalid] t -> string -> [`Parsed] t
 
 val tag_size : [< `Parsed | `Valid] t -> int
 
 val parse_tag : [`Parsed] t -> string -> [`Valid] t
 
 val attach_frame : [< `Invalid | `Valid ] t -> frame_type -> frame_text -> [`Valid] t
 
 val render : [`Valid] t -> string

As you can see, the tag's state is used as a phantom type in 'a t and is updated as the tag passes through the various API calls. This makes it impossible to actually hit render without either adding a single frame or successfully parsing a binary tag.

And it comes at zero cost, both for me and the user! As for myself, I just write the implementation bearing in mind that I need to return the given tag after each call. Likewise, the user only has to redefine its tag after each call, much like in the plain pure functional good ole fashion.. For instance:

let tag = init () in
 let tag = attach_frame tag "TIT2" "The times They Are A-Changin'" in
 render tag

I am too lazy to draw it but this can be generalized: if you think of your API as a diagram, where nodes represent states and arrows represent the various calls, then each state can be mapped to a polymorphic type and each call can be annotated with similar polymorphic phantom types in order to make sure that one can only call them from a correct state..

I think I really appreciate static typing when it comes at such a low cost. It makes you feel comfortable with both the code you write and the code people will write using your API and removes the need to constantly check boring things at runtime..

Moving out of the 70s...

Toots — 2011-07-25T22:12:35Z

Dear OCamlers,

The seventies were a cool time.. Progressive rock, pacifist movements and, of course, free love. However, the seventies were also the dark ages for modern computing.. and the time of POSIX..

On this note, I am sick and tired of fixing over and over the same bugs caused by corner cases and shitty aspects of the POSIX norm. The top two is the following:

SIGPIPE

A sigpipe, depending on some obscur run-time considerations, may silently terminate your program or raise an exception. If you want to be sure that you only have an exception, you need to explicitely ask for this signal to be caught.

Of course, this signal does not exist under win32 and if you try to catch it under windows, you will get an exception thrown to your face which most likely will also crash your program..

Sipgipe may be received when communicating through a network socket. Thus, any time you write a server, you need to make sure the signal is properly registered for catching. And if you do not do it, your code may work perfectly for you but crash randomly on another system. In this case, I wish you good luck to trackback the crash to this issue..

To sumarize:

In any server-oriented program or, more generally, any program which expects an exception for sigpipe, you need to make sure the signal is registered for catching
If you plan on porting your code to windows, you also need to not execute the registration code under windows.

Now, why is that hapenning? This is well explained in this bug report: sigpipe is also received when a program is piped to something that failed, for instance if you pipe a cat to less and the user terminates the less part.

How many OCaml programs/programmers expect their program to silently crash when receiving a sigpipe? I bet not so many.... Back to the seventies, piping programs was an essential feature of Unix. But what about the 21st century...

End of stream

Another classic of the POSIX semantics is the way that end of streams are detected. There may not be any error or exception raised when a connection has ended. Instead, the system will simply report 0 characters read when invoking Unix.read.

This means that in every place where you call this function, you need to think about what should happen if the returned number is zero and raise an appropriate exception if needed.

This also means that any failure to do so may result in a severe bug in your application.

What now?

The reason I am writing this is that although I totally understand that a "serious" programming language should adequately reflect the POSIX api — after all, skilled programmers and dinosaurs expect the POSIX behaviour — it does not seem to make sense to me that this should be the only alternative.

What about having a global flag in the Unix module that would allow to switch to a 21 century semantics? It could not fix all the shitty POSIX aspects (like having to call select to know if a socket is ready for reading or writing) but it could at least turn on some workaround, for instance making sure an exception is raised for each sigpipe or end of stream..

So, fellow OCamlers, am I the only one to struggle with those issues? Do you have other similar stories to report? Do you think that OCaml should at least have a non-default behaviour that reflects 21st century concerns and not plain old seventies hippie programming?

Dynlink as dlopen..

Toots — 2011-05-13T23:18:16Z

Dear Lazyweb,

One big issue we constantly face with liquidsoap is that most of our users have to recompile the whole software just to have support for mp3 encoding using liblame and ocaml-lame.

We have been devising two different approaches with Samuel to address this issue. I would like to expose mine here, both for documentation purposes — finding documentation and examples of how to use the dynlink module is not easy — and to maybe get some feedback..

The whole issue with dynlink is that it is capable of loading a module at run-time but cannot expose its API to the running program. Therefore, a shared module has to be defined so that it registers some functionalities when loaded.

In our case, we would like to be able to activate mp3 encoding by loading the Lame module at run-time if detected. Our requirements are the following:

We should be able to compile liquidsoap on a system which does not have ocaml-lame installed
We would like to expose the whole ocaml-lame API once loaded dynamically

The first requirement is important because we would like to distribute a canonical build of liquidsoap through debian for instance, so that if a user wants to dynamically load Lame, she only has to compile ocaml-lame. On the one hand, this reduces our workload by only having to work on one package and on the other hand, this the only case which makes sense.

Indeed, the other solution would be to compile a liquidsoap plugin against a compiled Lame module. Practically, this means that we would have to compile a liquidsoap package on a system which has lame installed. But in this case, we should simply build the sotware with lame included and forget about the shared module..

The second requirement is a natural consequence of the first one: if the shared module is built independently of liquidsoap, then it has to be provided by ocaml-lame. Then in this case, it should be more general than just exposing the functionalities needed by liquidsoap..

So, after much poking around, here is the solution I came to. I will explain it with a simple example.

Imagine that you have a module Foo which has the following interface.

(** An abstract type for the encoder. *)
 type encoder
 
 (** A function to instanciate an encoder. *)
 val create : unit -> encoder
 
 (** A function to encode. *)
 val encode : encoder -> float array array -> string

The idea here is that we want to expose the whole API and make it possible for any software to be compiled without having the module Foo installed and still be able to dynalically load it at run-time. Therefore, we define the Foo_dynlink interface:

type encoder
 type encoder_handler = 
 { create : unit -> encoder;
 encode : encoder -> float array array -> string }
 type handler = { mutable encoder_handler : encoder_handler option }
 val handler : handler

But we do not implement the corresponding .ml module. This mli specifies what the program that will load dynamically Foo will have to implement.

Now, we implement a loader for Foo, called Foo_loader:

open Foo_dynlink
 let () = 
 let create () = 
 Obj.magic (Foo.create ())
 in
 let encode enc data = 
 Foo.encode (Obj.magic enc) data
 in
 handler.encoder_handler <- 
 Some { create = create;
 encode = encode }

Here, we need to use Obj.magic as the two abstract types Foo.encoder and Foo_dynlink.encoder cannot be unified: Foo_dynlink will be implemented by the calling program, which will not have Foo available..

Finally, we can now write a program that dynamically loads Foo. It consists of at least two modules, Foo_dynlink, which implements the corresponding interface and another module that uses it. Let's see foo_dynlink.ml first:

type encoder
 type encoder_handler = 
 { create : unit -> encoder;
 encode : encoder -> float array array -> string }
 type handler = { mutable encoder_handler : encoder_handler option }
 let handler = { encoder_handler = None }

Now main.ml

let foo_dyn = "/path/to/foo.cmxs"
 let foo_loader = "/path/to/foo_loader.cmxs"
 
 open Foo_dynlink
 
 let () =
 (** First load Foo. *)
 Dynlink.loadfile foo_dyn;
 (** Now execute the loader. *)
 Dynlink.loadfile foo_loader;
 (** At this point, Foo_dynlink.handler should be populated. *)
 match handler.encoder_handler with
 | None -> Printf.printf "Dynamic loading failed..\n"
 | Some _ -> Printf.printf "Dynamic loading succeeded!!\n"

Let's try it now!

First. make a seperate directory for Foo:

mkdir /tmp/foo
 cd /tmp/foo

Then compile the module and its loader:

ocamlopt -c foo.mli
 ocamlopt -shared foo.ml -o foo.cmxs
 ocamlopt -c foo_dynlink.mli
 ocamlopt -shared foo_loader.ml -o foo_loader.cmxs
 # We do not need the compile interface for foo_dynlink at this point..
 rm foo_dynlink.cmi

Now, let's prepare another directory for the main program:

mkdir /tmp/bar
 cd /tmp/bar
 cp /tmp/foo/foo_dynlink.mli .

And compile it:

ocamlopt -c foo_dynlink.mli
 ocamlopt -c foo_dynlink.ml
 ocamlopt -c main.ml
 ocamlfind ocamlopt -linkpkg -package dynlink foo_dynlink.cmx main.cmx -o bar

Of course, the paths to foo.cmxs and foo_loader.cmxs in main.ml have been changed to their actual location..

At this point, bar has been compiled without any reference to Foo. The only requirement is that it implements a module that has the interface required by foo_loader.cmxs. Now is the time to try it!

./bar 
 Dynamic loading succeeded!!

:-)

What do you think? Ugly or interesting?

A Monad for OCaml Duppy

Toots — 2011-05-07T19:42:37Z

The context

Duppy is the OCaml module that we have developped with the Savonet Team as a solution for all scheduling tasks in Liquidsoap.

Duppy was initially intended as a nice way to create a simple scheduler that supports the following:

Delayed tasks, that is tasks to be exectued after a given delay
Reading and writing on Unix.fd file descriptors without blocking the calling thread
Custom priority system whith a pool of threads running concurrent queues processing tasks according to their priorities

For instance, for a HTTP server, a client acessing a local file such as an image, should be processed with maximum priority but a client requesting a php page should be processed with low priority, because executing a php script can potentially take a long time whereas reading from a local file is fast. Also, one php page may be using many images so serving the images faster is important to ensure a smooth data flow to the client.

Duppy was initially considered as a slight abstraction — namely a functional abstraction over Unix.select, but not necessarily of a very general scope. However, I realized lately that a much more interesting use of Duppy could be done.

Servers in Liquidsoap

The main use of Duppy in Liquidsoap is for implementing the various servers (telnet and harbor, our native implementation of the icecast source protocol) as well as implementing delayed tasks within the main language.

Concerning the server part, the code was written using exception catching as follows:

let serve_task socket task = 
 (..code for serving task..)
 if (..some condition, for instance that the task is recognized..) then
 (..code processing task..)
 raise (Answer "P = NP!")
 else
 raise (Answer "Invalid request")
 
 let login socket data = 
 (...code checking authentication..)
 if not (..login correct..) then
 raise (Answer "Wrong Authentification data") 
 
 let process_client socket request = 
 try
 (..parse login data..)
 (* Check login *)
 login socket data;
 (.. parse task request..)
 (* Process data *)
 serve_task socket task
 with
 | Answer s -> reply s; close socket

When implementing a complex protocol, such as the Icecast source protocol, this allows to have very readable code, using an exception to stop the execution flow. The ultimate goal being the possibility to seperate more clearly the protocol logic from the control flow of the execution, in order to be able to proof-read the code more conveniently.

However, this approach has a major drawback: it messes with the semantics of exceptions, which are usually used to raise errors in OCaml.

Although not necessarily an issue for simple flows, in the case of protocols such as Icecast source protocol, where two totally different version of the protocol have to be supported, with different authentication schemes, etc.. this quickly became a nightmare.

In order to cope with the complex execution flow when using exception-based programming, it becomes necessary to add multiple type of exceptions and to catch and process them accordingly. This impacts the readability of the code very quickly, to the point where using exceptions is no longer really interesting.

Furthermore, OCaml does not offer the possibility to know which exceptions may be raised by a given function. which does not make it easy to verify that all possible catching cases are covered..

What about a Monad!

Seperating execution flow from the computaiton logic and encapsulating side-effects are the two main advantages of using monads in functional programming. Therefore, why about implementing those using a monad!

Basically, processing a request from a client consists of composing computations which either return immediatly with a response or return a result to use with the next one.

Furthermore, we would like those computations to be implemented using the Duppy scheduler, which has great advantages for writing and reading on network sockets and also to prioritize the different steps of the computation.

We consider thus computations of type ('a,'b)t. A computation ('a,'b)t either:

returns a value of type 'a, which is passed to the next computation
returns a value of type 'b which is sent back to the previous computation

This monad does the same as an exception monad, except that is is implemented using Duppy tasks.

The main fonctions of the monad are:

module Monad :
 sig
 (** Type representing a computation
 * which returns a value of type ['a]
 * or raises a value of type ['b] *)
 type ('a,'b) t
 
 (** [return x] create a computation that
 * returns value [x]. *)
 val return : 'a -> ('a,'b) t
 
 (** [raise x] create a computation that raises
 * value [x]. *)
 val raise : 'b -> ('a,'b) t
 
 (** Compose two computations.
 * [bind f g] is equivalent to:
 * [let x = f in g x] where [x]
 * has f's return type. *)
 val bind : ('a,'b) t -> ('a -> ('c,'b) t) -> ('c,'b) t
 
 (** [>>=] is an alternative notation
 * for [bind] *)
 val (>>=) : ('a,'b) t -> ('a -> ('c,'b) t) -> ('c,'b) t
 
 (** [catch f g] redirects values [x] raised during
 * [f]'s execution to [g]. The name suggests the
 * usual [try .. with ..] exception catching. *)
 val catch : ('a,'b) t -> ('b -> ('a,'c) t) -> ('a,'c) t
 
 (** [=<<] is an alternative notation for catch. *)
 val (=<<) : ('b -> ('a,'c) t) -> ('a,'b) t -> ('a,'c) t
 
 (** [run f ~return ~raise ()] executes [f] and process
 * returned values with [return] or raised values
 * with [raise]. *)
 val run : return:('a -> unit) -> raise:('b -> unit) -> ('a,'b) t -> unit
 end

Mutexes and Conditions are also implemented within the monad, but we do not describe them here.

Finally, there is a camlp4 syntax extension, which allows to write code using this monad in a natural syntax:

duppy check_http x = 
 if x = "HTTP" then
 duppy_return Http
 else
 duppy_raise Not_http
 in
 duppy_try
 check_htttp x
 with
 | Not_http -> (...)

Priorities and I/O operations

The main advantage of the Duppy monad is the handling of priorities and I/O operations.

Concerning priorities, a duppy computation can be rescheduled at any time with a new priority and optionally a delay. For instance the following computation:

duppy f x =
 duppy_do
 (..computation code..)
 with
 { priority = Slow;
 handler = h }

When f is called, its execution is sent to a queue processing tasks of Slow priority.

Likewise, reading or writing a value from and to a Unix.file_descriptor is very simple:

duppy read_crlf () = 
 duppy_read
 Duppy.Split "\r\n"
 with
 { handler = h;
 priority = Fast }

In this code, f_crlf is a computation that reads from a file descriptor specified in handler until the first \r\n marker has been seen. It is scheduled with a Fast priority because all read/write operations in Duppy are executed only when they will not block.

Code examples

Several examples using the Duppy monad are available. The module has two examples, a telnet and a HTTP server, which can be browsed online:

The monad is also used in liquidsoap, for the internal telnet server, which is very similar to the Telnet server above, but also for the harbor server, which implements the source part of the icecast protocol, and for the new output.harbor, which implements the listener part of the icecast protocol. These two examples can also be browsed online:

Difference with Lwt

The reader familiar with Lwt may wonder what is the difference with this module. The essential difference is that Duppy does not implement light-weight threads but rather a pool of queues picking up different tasks one by one.

Therefore, it allows to specify priorities for the queues and does not require that you rewrite all your code using compatible versions of the blocking functions.

Conclusion

Monads can be very useful in order to encapsulate specific computation flows. In this case, we have used a monad to have a notion of exception raising that is compatible with the Duppy scheduler. Eventually, this allows to write code that is easy to proof-read and that takes advantage of all advanced functionalities of the scheduler without being obscured by many irrelevant technical details..

Liquidsoap at ON2: Test signals, October, 22-23 2010 in Berlin

Toots — 2010-09-28T02:14:05Z

It is my pleasure to announce that the Savonet Team [1] will be presenting Liquidsoap [2] at the ON2: Test signals festival [3], taking place on Friday, Oct. 22 and Saturday, Oct. 23 in Berlin, Germany.

We will be holding a workshop session on Friday, during which we will explain how to use Liquidsoap in many details, and a more general talk about Liquidsoap the next Saturday.

We are also expecting to prepare and release a second beta for the occasion, which we are planning to use during the event.

Any interested user or contributor to Liquidoap that is available is warmly invited to join and meet with us! The festival is free but requires to signup (see above link).

[1] http://savonet.sourceforge.net/savo...

[2] http://savonet.sourceforge.net/

[3] http://wiki.sourcefabric.org/displa...

Liquidsoap 1.0 beta1 and 0.9.3 released!

Toots — 2010-09-11T01:38:54Z

We have finally released two new versions of Liquidsoap [1]!

This is a very good news for the project because the 1.0 branch contains huge and exciting improvements in particular video support, dynamic source creation/destruction and support for different clocks/time flow. 1.0 beta is also the first release that runs natively on windows, when compiled using the ocaml cross-compiler [2].

1.0 is still some time away but the beta is now considered solid enough to be considered for production. That's also where bugfixing will happen now.

0.9.3 is a bugfix release for 0.9.2 and will be the last 0.9.x release.

The release also includes a bunch of new binding releases, namely:

ocaml-alsa: added resume and recover function, reworked exception handling
ocaml-cry: removed an extra crlf sent when using icecast source protocol
ocaml-dtools: added support for syslog
ocaml-duppy: minor win32 fixes..
ocaml-gavl: fixed Gc issue.
ocaml-mad: win32-related fixes.
ocaml-ogg: stricter parsing of ogg streams
ocaml-theora: support for theora 1.1 API

The custom debian packages have been updated for 0.9.3. For the beta, you can try the SVN daily package [3]

Official Debian packages should come, but through experimental while squeeze is frozen..

The release notes and changelog can be read in the announcement mail [4].

Yeah!

[1] http://savonet.sf.net/

[2] https://fedoraproject.org/wiki/Feat...

[3] the automatic build needs to be fixed, the cron script does not work now for some reasons although it is perfectly fine manually..

[4] https://sourceforge.net/mailarchive...

Mingw32-ocaml 3.12.0

Toots — 2010-08-11T23:02:54Z

An updated version of the ocaml cross-compiler package, based on ocaml 3.12.0, has just been uploaded to Debian experimental !

Any report and test on the package would be very welcome ! I have personally tested it with Liquidsoap and built a win32 version of the software.

Since this build implies many external modules as well as C objects, I am pretty confident in the cross-compiler uploaded to experimental..

About the cross-compiler: the ocaml cross-compiler is the result of the hard work done by Richard Jones for Fedora. The Debian package is merely a backport (and adaptation to ocaml 3.12.0) of his patches.

If you care about the future of the cross-compiler, the best you can do is work with upstream to find how to push the needed changes there in order to have a plain support for it.

I have personally no time for starting this process but I could try to describe the patches to an intereste contributor. Warning: some are REALLY hacky :-)

I have a dream...

Toots — 2010-06-21T21:22:49Z

I have a dream that one day, serious programming language theoriticians, which would gather and realize that LaTeX has done a great job but is basically just a pile of rewriting rules built on top of postscript.

I have a dream that one day, one would get the fundings and rewrite from scratch a new language.

A language with a clear syntax for describing the beautification constraints.

A language which would also be flexible enough to be able to override those constraints.

A language that would allow to write mathematic formula, macros and other tools without arcane tricks and hacks to obtain the required rending....

And much more...

Private records in OCaml

Toots — 2010-06-15T00:23:43Z

In Private type abbreviations, what are they good for? [1] and then in Another use for private type abbreviations [2], yminsky wrote about the use of private type in OCaml.

Among the various proposed examples, the idea of read-only value was mentioned in the second post. I may have not followed the whole discussion after this, but I came on to an example in ocaml-ao today.

In ao, a driver is handled with an int, its id. It also has several read-only values, name, short_name etc..

Using a private record, it is possible to create a driver type this way:

(** Driver type (private). *)
 type driver_t = private
 { id : int ;
 kind : driver_kind_t ;
 short_name : string ;
 name : string ;
 comment : string ;
 author : string ;
 priority : int ;
 preferred_byte_format : byte_format_t ;
 options : string list
 }

That way, one can access directly the fields of the record without being able to write a fresh one. In our case for instance, we then make sure that the id field, used when opening a device, is always a valid entry.

Another suggested use was, for bindings, when you have an extra internal field, like a state represented by a C void pointer. In this case, you may declare an object with n+1 element in the C function an only export the n first one as a private record type.. Its a bit hacky but it may be useful as well...

[1] http://ocaml.janestreet.com/?q=node/59

[2] http://ocaml.janestreet.com/?q=node/77