Using Camlp4 for conditional compilation

Hi,

so I have been wondering for quite some time: how to do conditional compilation in Ocaml as simple as in C?

As a reader of a library development log, you probably know that one would typically use preprocessing to modify a code before it gets compiled. There are a few cases in particular when one wishes to do so:

  1. When code on various platform would differ: a typical OCaml example would be the use of the Unix module which is not fully implemented in Win32 OSes. Hence we might want to have an alternative implementation on Win32 but keep the simpler or faster Unix implementation when relevant.
  2. When optimization is more important than genericity of code, in particular if we can use material specificities of the underlining architecture: for instance by having specialized 32 bits or 64 bits version, more generally direct access to a material interface, and so on.
  3. To create environment-specific behavior. This is mostly used during implementation where we might want to have a “debug mode” providing development information, disabling it for release without having to edit the code
  4. To provide options, in particular if this prevents from linking some non-mandatory or problematic library.

Unfortunately though C/C++ make it easy, Objective Caml preprocessing feature (Camlp4) wants us to make our own grammar! Actually the various documentation and tutorial found on the subject compares Camlp4 rather to lex/yacc (flex/bison in the Free Software world) lexer and parser, than to the C preprocessor.

Though it is most certainly nice and powerful (I admit, I didn’t read the whole Camlp4′s documentation), I only needed a very simple conditional compilation for my current use case.

I’ll discuss below how I did it for people needing this kind of feature as well as the limitation of this “simple” method.
Then I’ll discuss more generally on conditional compilation, why it can be a good idea; yet why we should be careful about this; and finally alternatives.

OCaml and simple conditional compilation

Documentation Flaws

Let’s admit, while OCaml is great, and has a good enough official documentation and a few great alternative ones out there, most of it is rather outdated, or very limited and imprecise. Camlp4 follows the rule. The official manual and tutorial both date from 2003, Ocaml version 3.07 (we are in 2011 and version 3.12 has been published for 7 months now!) and all other information I found were in similar state. Moreover documentation really focuses on extensive use of Camlp4, barely explaining how to use the easy pre-implemented grammars.

Default Preprocessing Grammar for Macros

There is indeed one grammar for macro support, written with Camlp4 and provided with the official OCaml release, which one can optionally load during compilation: pa_macro. That is what I got interested into. Sadly the best documentation to see how to use it and the full range of possibilities was… comments in the compiler source code.
Not to clutter this log anymore, I uploaded the relevant comment in a separate file for people to easily check on this.

What it can do

It allows to:

  1. define MACROs to be tested for existence or to be replaced by code;
  2. compile some code depending on the existence of macros.

And that’s basically all (nearly).

What it cannot do

In particular it cannot do these very basic uses of macros:

Test values of macros.
You cannot test the value of a macro and run some code depending on it. The only testable condition is existence/absence.
In my case for instance, I wanted to use a different type in a module depending whether I was on a 32 bits or a 64 bits platform. My first idea was to run:
ARCH=`uname -m`
in the Makefile, then add this as a macro and check value while preprocessing. That’s not possible.
Instead I had to make the test in the Makefile:
ifeq(`uname -m`,x86_64)
and depending on this condition, I set or not the ARCH_64 macro. Then I test its existence.
Replace macros with nothing
On C, you could have macros set to nothing so that would simply disappear when embedded in the code. This is impossible with pa_macro.
If I try DEFINE MY_MACRO=, it would raise a preprocessing error.
If I simply define the macro (without ‘=’), and try to use it in the code, it would not be replaced, hence this time, the compiler raises a constructor error.
There is a predefined macro called NOTHING but apparently it can be used only as a function argument that you want to erase (and I did not really understand the use case for it). I cannot set:
DEFINE MY_MACRO=NOTHING

Hence my solution was this:
DEFINE MY_MACRO(x)=x

You could say: but why would I want this? In my code, I have a TO_INT macro. On a 32 bits architecture, I want it to be Int32.to_int whereas on 64 bits, I would use the int type, then TO_INT is the identity function (and if I can avoid a useless function call…).

Flawed check on constants
In the algorithm I implemented (SHA1), there are some constants. Some of them are very big and are over max_int on a 32 bits platform.
But as I said, in such a case, I use Int32, so all is fine. So I had this kind of test:

IFNDEF ARCH_64 THEN
    0x8F1BBCDCl
ELSE
    0x8F1BBCDC
ENDIF

The problem I encountered was that camlp4o was raising an error while compiling on a 32 bits machine, telling me that 0x8F1BBCDC was bigger than max_int!
I don’t know why it tried to check the validity of the constant while this code was not going to be compiled. In my opinion, it should stick to its “syntax work”, then pass the result to the compiler. This one will check validity of constants and if I made an error (which I did not), it would tell me. No need to make this check twice, especially when the check is flawed! That’s a clear limitation in pa_macro that I could only work around by making this constant as a macro that I pass from my Makefile. That’s not clean, clearly, but I did not find a better solution.

“parser” is a keyword
I had a class called XML.parser. I had to rename it because — when using Camlp4 — parser becomes a keyword (used to filter streams) and cannot be used anymore to name entities. This is not an error, but just so that you know that camlp4o can cause this kind of issues. Yet it does not change the original syntax. On the other hand, camlp4r (‘r‘ as “revised”) does change the basic syntax. So use only camlp4o if you wish to continue using the original OCaml!

How it looks

So in the end, my file has this kind of code:

IFNDEF ARCH_64 THEN
  DEFINE LOGAND=Int32.logand
  DEFINE LOGOR=Int32.logor
  DEFINE TO_INT=Int32.to_int
  DEFINE OCTET=0xFFl
[…]
ELSE
  DEFINE LOGAND=(land)
  DEFINE LOGOR=(lor)
  DEFINE TO_INT(i)= i
  DEFINE OCTET=0xFF
[…]
ENDIF
[…]
let rem = TO_INT (LOGAND div OCTET) in
  w.[i] <- Char.unsafe_chr rem;
[…]

How to compile this

And my Makefile contains such code:

ARCH64_PP_FLAGS=-DARCH_64 -DHEX_8F1BBCDC=0x8F1BBCDC
# You remember the ugly workaround I told about because
# Camlp4 was forbidding me to write big integers? Here it is.

ifeq ($(shell uname -m),x86_64)
PP_FLAGS=$(ARCH64_PP_FLAGS)
else ifeq ($(shell uname -m),ia64)
PP_FLAGS=$(ARCH64_PP_FLAGS)
[…]
else
PP_FLAGS=-UARCH_64
endif

all:
    ocamlopt -pp "camlp4o pa_macro.cmo $(PP_FLAGS)" sha1.mli sha1.ml

Here we simply tell the compiler to use camlp4o as a preprocessor (-pp) with the grammar pa_macro.cmo and we set (or unset) a few macros.

Conditional Compilation: when should it be used?

Here are a few examples of bad and good use of conditional compilation.

Creates parallel Code versions

In particular, it generate “logical” code versions (not the physical code, as I will show, when well done, it can in fact factorize it, which is good) which makes as though you had several different programs to maintain. Indeed in my example, I already have 2 versions of my code: 32 and 64 bits. What it means in particular is that my code compiling on a 32 bits architecture does not mean anymore it compiles on a 64 bits one (and reciprocally) because the compiler is not given anymore the whole code at once but a part which has been “dumped” by Camlp4 (hence “pre-processing”).

Here that’s still ok because I have only 2 versions. But when a software multiplies conditional compilations instead of writing generic code, it can easily end up with dozens of outputs depending on platforms or user options (and many software out there are this way).

Pass through files

Variables, constants… they all have limitations imposed by the language: they cannot get out of a file, a module, a class, etc. That’s good. That’s one reason why we use a strict language: to have better control of what we want in and out.

Macros on the other hand are not controlled by the language. They have their own syntax and are managed by the preprocessor. It means they usually pass files. As I could not find any documentation on this point for Camlp4, I made some tests and I can gladly say pa_macro hasn’t got this issue, but I will explain it as general information about macros.

Let’s suppose that I create a ZERO macro (which I did) to be worth either the integer 0 or the Int32 0l. In a further file, I might make some day another ZERO, which is actually not a macro, but some constructor and I completely forgot that I had this macro set. Then it is replaced at compilation time by the macro value and fails; or worse, for some reason the replacement is a correct or possible type, it compiles, but later it fails at runtime making it difficult to diagnose.

Fortunately pa_macro‘s macros do not get out of files (external files of macros must be explicitly INCLUDEd). But the issue can still occur from command-line macros.

So if some of them are supposed to be global with specific name (like ARCH_64), it is usually ok. But if ever you were passing macros on the command line with very generic names, that you wanted to be used only in a single module, it might become a problem.

Comparison to Runtime tests

Some people oppose the macro solution to in-code conditional tests.
But this has flaws too:

Code forbidden

I told it, sometimes some code is forbidden on some platform. For instance if my very big integer constant was to make it up to the compiler, it would fail to compile (and this time, with a good reason). In such cases, runtime tests are impossible.

Performance

Why I made a conditional compilation was because logical computation on int was faster than on Int32, but I could not use int on a 32 bits platform (Ocaml has 31 bits int on 32 bits). So even supposing it were possible to do runtime condition tests (but it’s not as I demonstrated in the previous point), doing so would impact performance for no reason while I was doing it for the opposite reason! Fully interpreted language have no other solutions, whereas we use a compiled language. So let’s make good use of the tools we are given.

Code clarity

Sometimes heavy use and/or bad use of macros can lead to very obscure code. But it can also lead to clearer code when well used.

For instance my example above avoids any code duplication with many conditional everywhere.
I think we will agree that having only this:
let rem = TO_INT (LOGAND div OCTET) in
is far nicer than something like this:

let rem =
  if arch_64
  then
    div land 0xFFFFFFFF
  else
    Int32.to_int (Int32.logand div 0xFFFFFFFFl)
  in

The first code is factorized and the whole logics of conditional is hidden in the macros which are simply switched at compilation time. Moreover if I change my code, I need to change only one line instead of 2 (code duplication is a common source of bug when we fix something only at one place while the same code is copied over somewhere else).

Alternate Solutions

Other solutions exist to run different piece of code in different cases. Nearly none of them are all good or all bad, they may depend mostly on use case.

Again runtime tests

Yes it strikes again! Though I seem to have said a lot of bad things of it, it is still a pretty good solution in many cases.
In particular, if you want to be able to switch behavior as often as you wish, it is far better to make it runtime.

Debug for instance is a common feature which you can see either set as a compilation option or a runtime. Making it runtime allows you to never have debug but when you want it, you just switch a “-debug” option.
Note that if you have a lot of debug set in your program, it means there will be a lot of useless tests on the boolean value of some debug variable (if debug then). I told that’s a slight performance issue maybe not suitable for all use. A common solution is then to have debug both compilation AND runtime. At compilation, you can decide whether or not you have the “-debug” option, and if not, all tests won’t be compiled.

Makefile

The Makefile can deal with selecting between 2 files which have a different implementation of a single feature. Then depending on the platform, it can compile against one or the other implementation.
The advantage is obvious: 2 files completely not encumbered with macros or conditional code of anysort (neither preprocessing code, nor runtime code). The drawback is that you may have code duplication (unless the 2 implementations are actually very different logics, which can happen).

Code generation

Another solution is code generation, which is often some kind of code completion. This is actually a logic close to the macro logic. A common tool is autoconf, which replaces variables on the fly. You can often see some code with @VERSION@ in it. These are usually variables replaced by autoconf before compilation (check for AC_SUBST and AC_CONFIG_FILES in autoconf documentation). Though this is a pretty useless example, this kind of behavior can be pretty well extended to generate conditional code depending on a platform in a way similar to what the macros did in my example.

To get the full power of autoconf, think of how it generates a full Makefile from some very basic template.

Camlp4

I described the easy way with Camlp4, which is using the existing pa_macro. But of course if that’s not enough for you, you can write your own macro grammar.

Conclusion

So we saw that conditional compilation can be good for several reason, among them optimization, or even portability when no common code exist to do a single action between 2 platforms.
But as all good things, it must be used parsimoniously. In particular:

  • Is optimization always necessary? My example was a cryptographic example, the kind of code which can be run thousands of time every seconds in some kind of software and where the gain of speed can be from 1 to 1000 depending on the implementation. Optimization can be critical on cryptography algorithms.
    On other kind of feature, when you have a piece of code that you will run only once in a while, it is unconditionally better to have a very readable code than a fast and complicated one, even more if the gain in time you would obtain with optimization is so small that none will make the difference when it is run once every hour.
  • When portability is the goal, aim rather for generic code than for conditional compilation. So your current code is platform specific? Before looking for the equivalent platform-specific code in that other platform you support, try to look if there does not exist a generic code which works on both.
  • So there is no generic code possible? Optimization is too critical here? Or else the generic code exists but makes thing more ugly/complicated or bring other issues? Fine, conditional compilation is for you. But try to encapsulate it in a single file. If this is for a function you’ll want to use in many places, then place this function in its own file and hide any conditional compilation there so that it is seen as a black box. Once done, you won’t have to worry a single bit anymore about how it actually works. Doing this makes the “parallel logical code” much less an issue.

It is sad Ocaml does not advertize better some “easy” solution for this kind of use case. But multiple solutions still exist and they are not that difficult, as long as there is some documentation.
So I hope my little ticket here will help other people to write great program in Ocaml (or whatever other language, as long as your program is great… and Free… as in Free Speech!).

This entry was posted in Howto and tagged , , . Bookmark the permalink.

11 Responses to Using Camlp4 for conditional compilation

  1. gasche says:

    > 0x8F1BBCDC [...] That’s not clean, clearly, but I did not find a better solution.

    I think you can work around the issue rather simply by using `int_of_string “0x8F1BBCDC”`.

    • Jehan says:

      gasche,

      I could do that but I’ve worked a great deal to optimize the SHA1 algorithm, so this kind of solution is not suitable. Not only it would generate a useless call which will go on the stack, but also this call will make some string parsing and int computation (see the C function parse_intnat in the file byterun/ints.c of the OCaml distribution), unless the compiler is able to make some optimization by seeing the string is a constant and transforming it to int at compilation time (but I doubt it does this).

      Actually I was doing what you proposed in my very first version for all Int32 constants, and I was really annoyed by this, always thinking « they are constant, this sucks! ». That was before I discovered one could write Int32 constants by appending a lowercase ‘l’ (the documentation of the Int32 module does not write anything about this! And that was the first time I actually needed this module, so…).

      So that’s not a solution. But thanks. :-)

      • gasche says:

        I don’t understand. Why don’t you compute it as a global constant ?

        let const_0x8F1BBCDC = IFNDEF ARCH_64 THEN .. ELSE … ENDIF

        • Jehan says:

          Hi,

          that’s what I said would be the natural way but camlp4 is bugged and does not allow this, as Kim confirmed below.

          The only 2 (that I found up to now) solutions to work this around are either to define the macros on the command line (what I was doing) or in a separate file that I INCLUDE (as proposed by Kim, and what I indeed do now. That’s indeed slightly nicer than in the Makefile).

          The fact what you propose is not possible was the whole point of raising this camlp4 issue/bug in this post.

  2. Kim says:

    Indeed the pa_macro extension is buggy in the sense that writing:
    IFDEF b THEN
    m1
    ELSE
    m2
    ENDIF
    m1 and m2 are parsed, regardless of the result of b (which is problematic if b is testing
    the kind of architecture you are on and m1 and m2 define large constants (63 bits constant would then fail on 32bits). On solution si to define all your constants in two files cst32.inc and cst64.inc and do the following:

    IFDEF ARCH32 THEN
    INCLUDE “cst32.inc”
    ELSE
    INCLUDE “cst64.int”
    ENDIF
    only the correct file will be included.

    • Jehan says:

      Hi,

      yes I thought about this one though I didn’t try to check if it would work. Also that makes additional files (I liked that my implementation was so small but that’s obviously not very relevant).

      Still that’s basically equivalent to write them on the command line in the Makefile as I currently do, in the end. Both systems are just workaround of a bug of camlp4 unfortunately. :-/
      Thanks for the idea.

  3. Andrej Bauer says:

    You probably know this, but for the benefit of other readers I will recite the official line. Ocaml and other language from the ML family has a powerful module mechanism. In particular, there are Functors, which allow you to parametrize a piece of code by values and types (int vs Int32, the value 0x8F1BBCDCl vs the value 0x8F1BBCDC, etc). You can achieve things with functors which you cannot with macros. Macros work at the syntactic level, so the compiler does not know what they are about. Consequently, they end up being confusing and difficult to debug. Code with lots of ifdef’s is hard to read. End of official line.

    So, did you try doing your thing with functors?

    If performance is really such a big deal you should have implemented the critical code in C or assembler and link it into Ocaml.

    • Jehan says:

      Hi,

      I know functors. My HMAC and my SCRAM modules are functors for instance (because they are both generic algorithm which relies on a cryptographic hash function, which one can pretty well be anything: MD5, SHA1, SH256, etc.).

      As for your proposition of using functors instead of macros, it seems to me inappropriate to the problematics. You use functors for genericity (like my HMAC and SCRAM implementation which are actually really generic and accept any kind of hash function) because you will want them to be extended very easily (if I add a new crypto hash function, I generate the HMAC for this function in one line from my functor!).

      For the current problematics though, we don’t expect any extension (the implementation won’t even get out of the file, that’s all private!). There is only 2 possibilities in our mind: int or Int32. Using functors here is like a deviant usage from what it exist for. And it means that to make it work (if possible), it will imply a lot of even uglier hacks.
      Plus, I think that will not be as efficient (or at least it definitely won’t be faster by definition) because it implies some runtime check which is useless considering the fact that the architecture of the machine is not supposed to change. The macro system does all adequate check and adaptation at compilation, once and for all.
      Using functors here is using the wrong tool for the task. There are most probably other way to do it efficiently, functor is not one.

      And for the second question: no performance is not a big deal. I wrote this for the fun, the challenge, because I wanted to see up to which point one could “push” OCaml, if it would be able to compete natively with C on cryptography topics, and also if I would be capable to do this all… all these kinds of reason. If I really wanted performance, I would probably use a third party library in C, in fact.
      But here at first, I needed a SASL library for Ocaml with SCRAM (SCRAM-SHA1 is the recommended authentication mechanism in XMPP). I could not find one (note that SCRAM is pretty recent and not implemented that much yet). Then I found all these RFCs interesting so I thought I would like to try and implement all the RFC chain myself. So I began. Then I saw it was slow (extremely slow even at first). Thus I thought “ok let’s optimize it”. And each time I found something to optimize, I was excited; plus, it taught me a lot of things on the language.

      That’s all. And now I share some of the things I learned during this whole process. Not as a general verity, more as experience sharing because there definitely is not much documentation out there (or not easy to find) for this side of OCaml (the “actual use case in daily programming” side, and the “human” side also).

      Note also that’s not a payed work, nor anything and I do it on free time. So no, performance is definitely not a big deal. It is only a sweet candy for my brain.

  4. Andrej Bauer says:

    There’s no problem with having a functor just so that you can instantiate it twice, once with int and once with Int32. And there won’t be any runtime checks (maybe if you use recursive functors), why do you think there will be runtime checks?

    • Jehan says:

      There is at least one runtime check: the code will have to check if I am on a 32 or 64 bits platform and then instantiate the functor with the right implementation (int32 or int).

      That’s not a big deal (that’s why I said just before “at least it definitely won’t be faster by definition” because that might not do much a difference in performance viewpoint) but that clearly shows that functors are not made for this. They are made for genericity, to easily deal with features based on a common template at runtime. They are not made to load something which is “written in stone”.

      Indeed this is so aberrant to do it runtime while we should always know from the day of compilation anyway if we are 32 or 64 bits. Your solution would load 2 implementations in memory while we will always only use one of them and never the other. We know it from compilation time, but the executable will still check it each time and will still load the 2 implementations.

      That’s a design inconsistency which implies we would be making a mistake if we were doing so.

      • Andrej Bauer says:

        Oh, I was rather thinking of this differently. In Functor.ml you put your functor. Then you have two separate FooInt.ml and FooInt32.ml which apply the Functor to int and Int32 separately. But then you don’t link them both into your executable. You do a little bit of Makefile magic to just link the one that’s relevant for the architecture you’re compiling. (I agree though that this is kind of clumsy. I mean, clearly, we are discussing things which the machine should do by itself.)

Leave a Reply to gasche Cancel reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>