blog.rastageeks.org

Mingw32-ocaml 3.12.0

Toots — 2010-08-11T23:02:54Z

An updated version of the ocaml cross-compiler package, based on ocaml 3.12.0, has just been uploaded to Debian experimental !

Any report and test on the package would be very welcome ! I have personally tested it with Liquidsoap and built a win32 version of the software.

Since this build implies many external modules as well as C objects, I am pretty confident in the cross-compiler uploaded to experimental..

About the cross-compiler: the ocaml cross-compiler is the result of the hard work done by Richard Jones for Fedora. The Debian package is merely a backport (and adaptation to ocaml 3.12.0) of his patches.

If you care about the future of the cross-compiler, the best you can do is work with upstream to find how to push the needed changes there in order to have a plain support for it.

I have personally no time for starting this process but I could try to describe the patches to an intereste contributor. Warning: some are REALLY hacky :-)

I have a dream...

Toots — 2010-06-21T21:22:49Z

I have a dream that one day, serious programming language theoriticians, which would gather and realize that LaTeX has done a great job but is basically just a pile of rewriting rules built on top of postscript.

I have a dream that one day, one would get the fundings and rewrite from scratch a new language.

A language with a clear syntax for describing the beautification constraints.

A language which would also be flexible enough to be able to override those constraints.

A language that would allow to write mathematic formula, macros and other tools without arcane tricks and hacks to obtain the required rending....

And much more...

Private records in OCaml

Toots — 2010-06-15T00:23:43Z

In Private type abbreviations, what are they good for? [1] and then in Another use for private type abbreviations [2], yminsky wrote about the use of private type in OCaml.

Among the various proposed examples, the idea of read-only value was mentioned in the second post. I may have not followed the whole discussion after this, but I came on to an example in ocaml-ao today.

In ao, a driver is handled with an int, its id. It also has several read-only values, name, short_name etc..

Using a private record, it is possible to create a driver type this way:

(** Driver type (private). *)

type driver_t = private
 { id : int ;
 kind : driver_kind_t ;
 short_name : string ;
 name : string ;
 comment : string ;
 author : string ;
 priority : int ;
 preferred_byte_format : byte_format_t ;
 options : string list
 }

That way, one can access directly the fields of the record without being able to write a fresh one. In our case for instance, we then make sure that the id field, used when opening a device, is always a valid entry.

Another suggested use was, for bindings, when you have an extra internal field, like a state represented by a C void pointer. In this case, you may declare an object with n+1 element in the C function an only export the n first one as a private record type.. Its a bit hacky but it may be useful as well...

[1] http://ocaml.janestreet.com/?q=node/59

[2] http://ocaml.janestreet.com/?q=node/77

Liquidsoap-full 0.9.2-2 and ocaml-cry 0.1.2

Toots — 2010-03-08T23:13:43Z

An issue with the new ocaml-cry module that replaces the shout library has been recently discovered.

This issue was related to the data sent to the icecast server and could result in a higher load and loss of data when sending data to the server.

We have released a new version of ocaml-cry that fixes this issue, numbered 0.1.2.

We have also released a new liquidsoap-full tarball, numbered 0.9.2-2 and now available in the download section on sourceforge.net.

Additionally, our custom stable packages have been rebuild against this new version, both for the i386 and the amd64 architecture.

The package available in Debian sid/unstable is now built against the fixed ocaml-cry as well.

We recommend that any user that uses liquidsoap 0.9.2 updates its liquidsoap to make sure it uses ocaml-cry 0.1.2.

Please note that ocaml modules are not dynamically loaded, so you need to rebuild liquidsoap against the new ocaml-cry if you are building from source.

OCaml cross compiler for win32 in Debian !

Toots — 2010-02-11T16:01:20Z

Thanks to the great work from Richard Jones on building a OCaml cross-compiler for windows, we now have a similar package in Debian !

The package is called mingw32-ocaml and has just been accepted yesterday.

We have already used it to build a win32 version of liquidsoap.

I hope that this cross-compiler will be useful to OCaml developers, in particular in porting their application to the world of windows and fences, in order to spread the word about how great OCaml is :-)

Liquidsoap 0.9.2 is out !

Toots — 2009-10-31T03:45:23Z

The savonet team is proud to announce the release of liquidsoap [1] 0.9.2. This release is a snapshot of upcoming features, but it also brings several important bugfixes. As a snapshot, it contains experimental and unpolished features, and also slightly breaks compatibility with previous 0.9.x versions.

The interesting features in this release are:

Switched to a custom implementation of the shout protocols. This allows in particular AAC+ output support, using either the external aacpluenc [2] binary or the native libaacplus [3] [4].
New scheduling algorithm for request-based sources which avoids queueing multiple songs at once.
Support for libre.fm [5].
Built-in support for extracting replay gain [6] metadata.
Supports for fallible outputs.
MANY bugfixes !

You can read a detailed changelog there [7].

This release is a milestone on our way to liquidsoap 1.0.0. Major changes are going to be merged into the main development branch now. We keep getting closer!

The Debian package for sid/unstable will soon be available too.

Additionally, a stable debian backport should be available anytime soon.

[1] http://savonet.sourceforge.net/

[2] http://teknoraver.net/software/mp4tools/

[3] http://tipok.org.ua/ru/taxonomy/term/36

[4] Output using libaacplus is still experimental and bug prone and both outputs suffer from the limitations of the same codebase, in particlar when streaming at low bitrate such as 32Kbps..

[5] http://libre.fm/

[6] http://en.wikipedia.org/wiki/Replay_Gain

[7] http://savonet.rastageeks.org/brows...

Ocaml-cry

Toots — 2009-10-15T05:42:15Z

Coming along with the next release of liquidsoap, we have just released a new module for ocaml, namely ocaml-cry.

Ocaml-cry is a native implementation of the protocols used for sending source data to icecast and shoutcast servers.

The main difference with the famous libshout implementation is about flexibility. This module allows to set any source header, which in particular allows to use any custom format which is compatible with the servers.

However, contrary to libshout, ocaml-cry does not handle the synchronisation of the audio data sent to the servers. Hence, the task of controling the rate of the data sent is up to the programmer.

You can find more informations on this module there:

http://www.rastageeks.org/cry.html

Debian packages are available there [1]:

http://packages.debian.org/libcry-ocaml-dev

[1] Many thanks to the ftp-masters for dealing with the NEW package so fast !

DJ mix using liquidsoap

Toots — 2009-06-08T16:26:48Z

Recently, I wanted to generate a single audio file containing the mix of several others, in order to publish it as a podcast.

The purpose of this is, based on an playlist of songs, to generate a single audio file containing an enhanced sequence of these songs. We would also like to normalize the volumes between the songs and have nice crossfaded transitions.

As you can imagine, I used my tool of choice for that purpose: liquidsoap !

In the following is presented the script that performs that. It takes a playlist, located at ./playlist.m3u, and introductory jingle, located at ./intro.mp3 and generates mix.mp3, mix.ogg and mix.aac.

The script is commented. There were minor issues with the playlist parsing not preserving the ordering, so it should be run with a recent SVN code.

Also, the script uses a perl handler to extract replay gain information. This script is:

#!/usr/bin/perl -w
 # A perl script that get the replay gain
 # from either a vorbis or an mp3 file
 # and returns it.
 
 use strict ;
 
 my $file = $ARGV[0] || die ;
 
 if ($file =~ /\.mp3$/i) {
 
 my $out = `nice -n 20 mp3gain "$file" 2> /dev/null` ;
 $out =~ /Recommended "Track" dB change: (.*)$/m || die ;
 print "$1 dB\n" ;
 
 } elsif ($file =~ /\.ogg$/i) {
 
 system("nice -n 20 vorbisgain -f \"$file\" \
 2>/dev/null >/dev/null") ;
 my $info = `ogginfo "$file"` ;
 $info =~ /REPLAYGAIN_TRACK_GAIN=(.*) dB/ || die ;
 print "$1 dB\n" ;
 
 }

Eventually, the final script is:

# Liquidsoap script that generates the mix of a playlist
 # in a single audio file, with normalization using
 # replay gain, and smart crossfading.
 # Author: Romain Beauxis 
 # License: DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
 # License URL: http://sam.zoy.org/wtfpl/COPYING
 
 
 ########### Settings ###########
 # In this section, we declare
 # the settings for this script.
 
 # Log to stdout
 set("log.file",false)
 set("log.stdout",true)
 # Uncomment to allow verbose logging..
 #set("log.level",5)
 # Disable real-time processing, to process with the maximun speed
 set("root.sync",false)
 
 ########### Replay Gain Backend ###########
 # In this section, we register the replay 
 # gain backend, to add replay gain information
 # to each resolved file.
 
 # Function to add replay_gain information 
 # to the metadata
 def replaygain_metadata(~format,file)
 x = get_process_lines("./replaygain.pl #{quote(file)}")
 if list.hd(x) != "" then
 [("replay_gain",list.hd(x))]
 else
 []
 end
 end
 
 # We register the above function as a new metadata resolver.
 add_metadata_resolver("replay_gain",replaygain_metadata)
 
 ########### Source Declaration ###########
 # In this section, we declare the main source
 # that we want to output.
 
 # An introductory single
 intro = single("./intro.mp3")
 
 # The main playlist
 playlist = playlist.once("./playlist.m3u")
 
 # We normalize audio files, according to the
 # replay_gain information (see above).
 final = amplify(1.,override="replay_gain",sequence([intro,playlist]))
 
 # We use the smart crossfade, to crossfade between tracks, 
 # paying attention to the relative volumes.
 # It is of course advised to place this operator *after* 
 # the normalization !
 final = smart_crossfade(final)
 
 ########### Shutdown fallback ###########
 # In this section, we wrap the main source
 # in order to shutdown liquidsoap when
 # it has finished to play.
 
 # We define a special transition, that
 # triggers the shutdown..
 def end_trans(a,b) =
 shutdown ()
 b
 end
 
 # The default transition
 def_trans = fun (a,b) -> b
 
 # The fallback source: play the whole mix, and 
 # then fallback to a blank. The end_trans transition 
 # makes sure that liquidsoap stops when falling back to the blank
 # source, i.e. at the end of the mixed playlist.
 final = fallback(transitions=[def_trans,end_trans], 
 track_sensitive=false, [final,blank()])
 
 ########### Outputs ###########
 # In this section, we define the output
 # that we want.
 
 # Output in mp3 format
 output.file.mp3("mix.mp3",final)
 
 # Output in ogg/vorbis format
 output.file.vorbis("mix.ogg",final)
 
 # Output in AAC+ format in 64kbps 
 # (needs aacplusenc in the path)
 output.pipe.external(process=fun (_) -> "aacplusenc - mix.aac 64",
 final)

Lastfm no longer free as in free beer (and some bits about xml in OCaml)

Toots — 2009-04-23T07:42:34Z

Lastfm no longer free as in free beer

As I was trying move the code of ocaml-lastfm [1] from the unmaintained xml-light [2] to xmlm [3], I discovered that it now fails to request track in anonymous mode.

Then, I went on the lastfm [4] site, and discovered that now I cannot find any full content available for anonymous users. Some more researches and I found this [5]:

Today we are making the changes to the radio that were previously announced here.

This means that from today, listeners to Last.fm Radio outside of the USA, UK and Germany will be asked to subscribe for €3.00 per month, after a 30 track free trial period. In the USA, UK and Germany, where it's feasible to run an ad-supported radio service, there won't be any changes. Everything else on Last.fm (scrobbling, recommendations, charts, biographies, events, videos etc.) will remain free in all countries, like it is now.


Alcool ! Voilà l'ennemi.: Poster by French painter and missionary Frédéric Christol (1850-1933) warning of the dangers of absinthe and other alcoholic drinks.

Although I will not comment this with the same violence as in the comments of the above message, this is not a good news at all. I totally understand how it can be difficult to find financial resources for this kind of business, and how complicated it can be after few years to maintain an activity that initially was breaking new and attracting investors.

However, given the current global [6] propaganda [7] campaign [8] that is organized [9] by the major music companies, I do not believe this decision has only to do with lastfm's financial resources. In particular, also the legality of Deezer [10] was challenged by universal [11] such that they had to require registration [12] and also limit drastically the available titles.

The current situation is now really becoming worse and worse. Not only the music companies are trying to push for dangerous laws for the civil rights while pretending to fight against illegal music sharing, but also they are trying to shutdown all the new competitors that were successful in doing exactly what they refused to do during the same time.

All of this is just simply pathetic, and I strongly hope there will soon be an end to this, which will surely mean for these companies adapt or perish.. Or perhaps they plan to impose their restricting and dangerous laws in any country in the world ?

Another remark about all this is that it clearly demonstrate the importance of having the right to copy and store for your own usage any copyrighted material. Indeed, these are not only products but also artistic productions, and for this reason it is important to be able to save them in some place in order to not loose track of it if the streaming company was to be shutdown, as it seems to be the trend now.


Viktor Oliva: The Absinthe Drinker.: The original painting can be found in the Café Slavia in Prague.

Moving from xml-light to Xmlm

The other part of this post is about moving from xml-light to xmlm. This is in fact very easy, and should only be a matter of adding a piece of code like this:

type xml =
 | Element of (string * (string * string) list * xml list)
 | PCData of string
 
 let parse_string s =
 let source = `String (0,s) in
 let input = Xmlm.make_input source in
 (* Map a tag representation in xmlm to
 * (name, attributes list) where attribute = string*string. *)
 let make_tag (x,l) = 
 (* Forget about the uri attribute *)
 let l = 
 List.map (fun ((_,y),z) -> (y,z)) l 
 in
 snd x,l
 in
 let rec get_elems l =
 if Xmlm.eoi input then
 l
 else
 match Xmlm.input input with
 | `El_start tag -> 
 let elem = get_elems [] in
 let (name,attributes) = make_tag tag in
 get_elems ((Element (name, attributes, List.rev elem)) :: l)
 | `El_end -> l
 | `Data s -> 
 get_elems ((PCData s) :: l)
 | `Dtd _ -> get_elems l
 in
 let elems = get_elems [] in
 Element ("", [], List.rev elems)

This is a very simple code that surely needs more fixes, but starting from that, you can parse a string into an equivalent representation of the xml data, and then use it as before in your code..

[1] Ocaml-lastfm: http://www.rastageeks.org/lastfm.html

[2] Xml-light: http://tech.motion-twin.com/xmlligh...

[3] Xmlm: http://erratique.ch/software/xmlm

[4] Lastfm: http://www.last.fm/

[5] "Radio Subscriptions": http://blog.last.fm/2009/04/22/radi...

[6] "New Zealand: safe from Big Music. Or is it?": http://www.p2pnet.net/story/19074

[7] " La Quadrature du Net discréditée auprès des députés anti-Hadopi": http://www.numerama.com/magazine/12.... Link is in french. It explains how, after filling a so-called petition with hundreds of signatures from employees presented as artists, or artists abused by the presentation of the content, these companies complain about the "totalitarian methods" used to verify the validity of the signature, which were simply based on the public available information on these names on the web, while they propose a system that would automatically cut the internet access to probably 1.000 people a day in France...

[8] "Faux procès : Les pirates paient": http://www.ecrans.fr/Faux-proces-Le.... again, in french, the article reports a Danish study that prove once again that people who tend to download a lot of music are also much more likely to spend their money in the music business, being concert, records or else..

[9] ""Three strikes" for Ireland - Eircom, music industry settle filtering case": http://tjmcintyre.com/2009/01/three...

[10] Deezer: http://www.deezer.com/

[11] "Universal Music challenges the legality of Deezer, a free streaming website": http://french-law.net/universal-mus...

[12] "Now Deezer Required Registration": http://forums.techarena.in/technolo...

Liquidsoap now supports AAC+ encoding.

Toots — 2009-04-21T18:11:17Z

The plot

Following many user requests , we were considering adding this format to the list of supported encoders.

As for mp3 encoding, this support was subject to some legal issues. These issues were indeed the initial motivation for implementing icecast2 using the patent-free ogg codecs and container for streaming data.

However, it seems that many users and net-radios are still asking for these formats. There can be several motivations for that, among which I can list the following major ones:

These formats are supported by more players and, in particular, embedded devices, which are the main target for net-radios
For the case of AAC+, the codec supports a very good audio quality for low bandwidths, such as 64 and 32kb/s.

The philosophy

There is a constant complain from the open source communities about the use of patent-encumbered formats. In particular, those formats are not allowed in the Wikimedia commons and the corresponding encoders, in particular mp3 encoders, are not supported by for instance the Debian distribution.

In a similar intent to push for a wider usage of patent-free codecs, the icecast project reimplemented its streaming server in order to use the ogg container and codecs.

All these actions are very important. In particular, the support for ogg/theora video files in firefox, and hopefully the major HTML 5 browser is mainly the result of wikipedia refusing other video formats. Indeed, wikipedia is now a major website, and it would be insane to not support the videos there.

However, I for myself do not like when these actions become too restrictive to the user. For this matter, I personally prefer freedom of choice among other considerations. Also, as a matter of fact, when trying to change the situation about these issues, one has to take into account the relative importance of each actor. It is because wikipedia is already huge that it has managed to change the things.

Hence, in the case of net-radios, a majority of potential users are asking for AAC+ and other patent-encumbered formats. Although this is not the ideal situation we would like to see, there comes a point were either the free-software projects have something to propose, or the users will just go for a proprietary software, which is even worse to the situation.

For this purpose, I believed it was important to add such support into our software. However, as you will see later, we do not implement ourself the encoding algorithm, but only added support for external encoders. That way, we simply let the user choose what he wants to use, but remain a patent-free and open-source software.

The implementation

Since mp3 was, and still is, the major target for net-radios, icecast2 still has the possibility to stream this format. In fact, this format, as well as AAC+, does not have headers, such that it is possible to decode a stream at any point in it. Hence, in this case, icecast only acts as a proxy, and the only constraint is to send it the encoded data at real rate.

Hence, the main library for streaming to icecast, libshout, parses the audio data that it receives, and schedules the streaming according to real rate. Additionally, it also supports a raw mechanism, for which there would be not such mangling.

In the case of liquidsoap, we did not need this feature of the libshout since the software is already working in real time. Furthermore, the source connection protocol provides a content-type header field, which allows icecast to display the correct mime for the data format that is streamed.

All in all, in order to send AAC+ data to icecast, one needs to send the correct mime-type in the source connection headers, and then send audio data at the correct rate.

However, the libshout did not allow to set an arbitrary content-type header. Hence, I proposed a patch. It was proposed for an inclusion into main code, since I believe it is interesting not only for our specific need, but also for any other generic data format support. Although the patch was accepted, it was never committed.

So, we decided to go on: since liquidsoap need not the real rate mangling feature of the libhsout, and since the source protocol is mainly a basic HTTP, we decided it was worth a custom implementation. This implementation was quite easy, thanks to the powerful capabilities of the OCaml language, and resulted in a custom implementation, ocaml-cry, that does the same as libshout, except that it does not control the rate of the data flow sent to the icecast server.

Then, we also needed to support encoding data to the AAC format. After looking at the only available open-source encoder, aacplusenc, I tried to propose some changes in order to export a library and be able to use it in liquidsoap. However, these propositions did not make it into upstream's code, so I decided to also move on and use directly the encoder binary.

Hence, using only the good old unix standard input/output mechanism, it is possible to start an encoder as a child process, send it the raw PCM data to its standard input and read the encoded data from its standard output. This lead to the development of the external output operators, which include output.icecast.external.

Although this implementation was not easy, it now appears to work quite well in the current SVN version. In particular, the encoder process exits correctly for the FLAC encoder, which is needed in order to stream correctly, since the ogg container needs a restart for every track. Hence, we also gained support for encoding to the FLAC audio format, which was not binded for OCaml (yet ?) because its callback-driven API makes it tricky to do.

Additionally, the support for external processes as encoders allows to use the lame binary for encoding mp3 data. Hence, liquidsoap can now encode mp3 data without the need of a built-in support for it. And, as always, the idea of using external processes spread over several other operators, including the support for external file and stream decoders, enabling support for many audio format (in fact almost all supported in linux, using mplayer).

In conclusion, liquidsoap now has a fairly good support for AAC+ audio streaming, as well as a many new formats for decoding files and streams. It also support arbitrary formats and encoders using a custom variant of the output.icecast.external. Hence, provided your format can be decoded from any point in the stream, you should be able to use liquidsoap to stream data to icecast as soon as you have a binary which is able to receive RAW or WAV PCM data to its standard input and send encoded data to its standard output.

As an illustration, here is an implementation of the output.icecast.aacplusenc:

def output.icecast.aacplusenc(
 ~id="output.icecast.aacplusenc",
 ~start=true,
 ~restart=true,
 ~restart_delay=3,
 ~host="localhost",
 ~port=8000,
 ~user="source",
 ~password="hackme",
 ~genre="Misc",
 ~url="http://savonet.sf.net/",
 ~description="OCaml Radio!",
 ~public=true,
 ~dumpfile="",
 ~mount="Use [name]",
 ~name="Use [mount]",
 ~protocol="http",
 ~aacplusenc="aacplusenc",
 ~bitrate=64,
 ~restart_on_crash=false,
 ~restart_on_new_track=false,
 ~restart_encoder_delay=3600,
 ~headers=[],
 s)
 # Metadata update is set by ICY with icecast
 def aacplusenc_p(m)
 "#{aacplusenc} - - #{bitrate}"
 end
 output.icecast.external(id=id,
 process=aacplusenc_p,
 bitrate=bitrate,
 start=start,
 restart=restart,
 restart_delay=restart_delay,
 host=host,
 port=port,
 user=user,
 password=password,
 genre=genre,
 url=url,
 description=description,
 public=public,
 dumpfile=dumpfile,
 name=name,
 mount=mount,
 protocol=protocol,
 header=true,
 restart_on_crash=restart_on_crash,
 restart_on_new_track=restart_on_new_track,
 headers=headers,
 restart_encoder_delay=restart_encoder_delay,
 format="audio/aacp",
 s)
 end