Miou, a simple scheduler for OCaml 5

Miou is a small library that facilitates asynchronous and parallel programming in OCaml. It is a project of the Robur cooperative, aimed at developing system and network applications. This library only requires OCaml 5 and can be obtained via opam.

$ opam install miou

Miou offers three key features:

a multi-domain runtime for executing asynchronous code.
flexibility in defining interactions between the system and Miou.
essential components for asynchronous and/or parallel programming.

Miou's role in your project

When developing an application that heavily interacts with the system to offer services like an HTTP server, embracing asynchronous programming is recommended. However, OCaml lacks built-in support for asynchronous programming, necessitating the use of a library such as Miou, which provides the required runtime.

Such a runtime emerges as a pivotal element of your application, orchestrating and executing tasks to ensure continued service availability despite varying workloads. Miou caters to diverse systems, ranging from unikernels to large servers with numerous cores or even small embedded devices.

While pivotal, this runtime represents the final frontier between your application's intended functionality and its current execution by interacting with the system. Hence, we advise users to defer selecting the scheduler, such as Miou, until the application's design phase is complete.

A multi-domain runtime

Since OCaml 5, it has been possible to execute tasks in parallel. Miou provides this capability by solving the inter-domain synchronization problems involved. Miou allocates multiple domains that are available to the user to manage in parallel, for example, clients.

We recommend referring to the OCaml manual to learn more about domains. Indeed, Miou manages domains itself because they can be a costly resource. As such, Miou handles their allocation, transfers your tasks to them, manages synchronization when you want to obtain the results of your tasks, and ultimately deallocates these domains properly.

Agnostic to the system

Miou only requires OCaml to operate. This choice stems from our ambition to integrate Miou as a scheduler for our unikernels, which are highly specialized systems. However, more generally and based on experience, we understand that interactions with the system are inherently complex and cannot be standardized through a common interface.

Therefore, we believe that the best person to determine how to interact with the system is you! Miou thus provides this capability so that you can leverage the full potential of your system.

However, Miou offers a small extension allowing interaction with your system through the miou.unix library. While rudimentary, it is adequate for most system and network applications.

Essential components for asynchronous/parallel programming

Finally, Miou provides essential elements for parallel and/or asynchronous programming. These components help address synchronization challenges inherent in parallel and/or asynchronous programming.

It is worth noting that these elements may seem somewhat rudimentary. However, we would like to caution the user that the topic of synchronization is a vast realm of solutions and research, and we do not claim to have omniscience over it. Therefore, we prefer to leave this space open for the user.

When not to use Miou

Although Miou is useful for many projects that need to do a lot of things simultaneously, there are also some use-cases where Miou is not a good fit: speeding up CPU-bound computations by running them in parallel on several domains. Miou is designed for IO-bound applications where each individual task spends most of its time waiting for IO. If the only thing your application does is run computations in parallel, you should use moonpool. That said, it is still possible to "mix & match" if you need to do both.

A practical counter-example: a synchronous server

The benefits of asynchronous programming aren't immediately apparent to everyone, and its understanding can be equally challenging. Therefore, we've chosen to illustrate asynchronous programming with Miou through a counterexample demonstrating its value: the implementation of a synchronous server. Our goal is to then transform our synchronous server into an asynchronous one capable of handling billions of connections simultaneously.

This tutorial presupposes that the reader is proficient in OCaml. While we aim to provide comprehensive explanations of each step but we won't delve into basic OCaml concepts.

The goal of this tutorial is to implement an "echo" server. This server simply echoes back whatever the user sends to it. While this may seem straightforward, several challenges arise, including the issue of synchronicity.

Socket

To facilitate communication between a client and a server, we utilize sockets. These are fundamental components provided by the system for handling communication, and in OCaml, we can manipulate them effectively. Let's delve deeper into their functionality.

A socket acts as an endpoint for communication, enabling two computers to connect and exchange data. It follows a client-server model, where one side initiates communication (the client), and the other side responds (the server). In our example, we'll be focusing on implementing a server. To start, we need to initialize a socket that's ready to accept connections.

val socket : socket_domain -> socket_type -> int -> file_descr

This function returns a file descriptor¹ representing the new socket. Initially, this descriptor is "disconnected," meaning it's not yet set up for reading or writing.

Several arguments are required, including the domain (determining whether the socket communicates locally or over the Internet), the type of communication (such as packet or stream communication), and the protocol used². For our purposes, establishing a TCP/IP connection suffices. Thus, we create our socket as follows:

let server () =
  let socket = Unix.socket Unix.PF_INET Unix.SOCK_STREAM 0 in
  ...

Establish a service

After creating the socket, we need to assign a specific address to it so that it can be reached from the network. This is done using the bind system call:

val bind : file_descr -> sockaddr -> unit

Our address must consist of an IP and a port since we intend our socket to communicate over the Internet. In OCaml, the sockaddr value represents this address. For our server, we want it to be available on our local network at port 3000:

let server () =
  let socket = Unix.socket Unix.PF_INET Unix.SOCK_STREAM 0 in
  let sockaddr = Unix.ADDR_INET (Unix.inet_addr_loopback, 3000) in
  Unix.bind socket sockaddr

Next, we specify that the socket can accept connections using the listen system call:

val listen : file_descr -> int -> unit

The listen function requires our file descriptor to begin accepting client connections. It also needs a second argument, specifying the maximum number of pending incoming connections. Our server not only handles incoming connections but also echoes back what clients transmit. It's possible that a client may want to connect simultaneously, so the system keeps these clients on hold until we can manage the new incoming connection.

let server () =
  let socket = Unix.socket Unix.PF_INET Unix.SOCK_STREAM 0 in
  let sockaddr = Unix.ADDR_INET (Unix.inet_addr_loopback, 3000) in
  Unix.bind socket sockaddr;
  Unix.listen socket 64

With everything initialized, we can now handle incoming connections using the accept syscall:

val accept : file_descr -> file_descr * sockaddr

This function will block until a new connection arrives. It returns a new file descriptor representing the client along with its address. To communicate with the client, we use this new file descriptor.

We're going to implement our client handler. Its goal is to read what the client sends us and then echo it back. Transmitting bytes via a socket is done using two functions:

val read : file_descr -> bytes -> int -> int -> int
val write : file_descr -> bytes -> int -> int -> int

These functions are blocking as well. The aim here is to store the bytes received from the client to echo them back. We'll repeat this process as long as we receive bytes from the client. Finally, we'll need to release our file descriptor; we won't need it anymore. We'll use Unix.close to inform the system that it can free all resources associated with this file descriptor.

let rec echo client =
  let buf = Bytes.create 0x100 in
  let len = Unix.read client buf 0 (Bytes.length buf) in
  if len = 0 then Unix.close client
  else let _ = Unix.write client buf 0 len in echo client

Now, let's complete the implementation of our service. This involves calling our echo function with the file descriptor of our client as soon as we receive it.

let server () =
  let socket = Unix.socket Unix.PF_INET Unix.SOCK_STREAM 0 in
  let sockaddr = Unix.ADDR_INET (Unix.inet_addr_loopback, 3000) in
  Unix.bind socket sockaddr;
  Unix.listen socket 64;
  let client, address_of_client = Unix.accept socket in
  echo client;
  Unix.close socket;
  print_endline "Server terminated"

let () = server ()

Compilation & usage

Let's start testing our program! We need to compile it first before we can use it. A tool like netcat/nc is sufficient to act as a client:

$ ocamlfind opt -linkpkg -package unix main.ml
$ ./a.out &
[1] 4347
$ echo "Hello World"|netcat -q0 localhost 3000
Hello World
Server terminated
[1]  + 4347 done       ./a.out

Our server worked well! It handled only one client, but it correctly echoed back what netcat sent to it (as seen in echo "Hello World"). It's also noteworthy that the server terminated correctly. However, at this point, there are still several aspects to describe.

A step back

There are several concepts we need to clarify in this exercise that are crucial when it comes to implementing a system and network application.

Synchronicity

While it may be obvious to some, it's important to clarify this concept to fully understand the following steps. If we review our code, we can describe what it does:

It creates a socket.
It defines an address.
It "binds" this socket to our address.
It instructs our system to make our socket available on the network.
It waits for a new connection.
It executes the echo function with our new incoming connection.
It displays "Server terminated".

It's important to note that your system processes the program line by line, in the order we wrote it. At each step, the system waits for the current line to finish its execution before moving on to the next one. This is necessary because each line depends on the work performed in the preceding lines.

This characteristic makes our program synchronous. Even if we introduce the echo function, the program remains synchronous because the caller must wait for the function to complete its task and return a value before continuing.

Blocking function

Previously, we introduced the accept function, which waits for a connection to arrive. It's worth noting that if no connection arrives, your program will wait indefinitely! We say the function blocks, meaning it's waiting for an external event (like the arrival of a client). And we can't do anything else as long as this function is blocked. This type of function makes it impossible for a program to carry out other computations while waiting for the former to finish.

The reason for this is that OCaml programs are single-threaded³. A thread is a sequence of instructions that a program follows. Because the program consists of a single thread, it can only do one thing at a time: so if it is waiting for our long-running synchronous call to return, it can't do anything else.

Handle multiple clients

Our goal now is to handle more than one client. We could simply repeat our accept call every time a client arrives.

let rec echo client =
  let buf = Bytes.create 0x100 in
  let len = Unix.read client buf 0 (Bytes.length buf) in
  if len = 0 then Unix.close client
  else let _ = Unix.write client buf 0 len in echo client

let server () =
  let socket = Unix.socket Unix.PF_INET Unix.SOCK_STREAM 0 in
  let sockaddr = Unix.ADDR_INET (Unix.inet_addr_loopback, 3000) in
  Unix.bind socket sockaddr;
  Unix.listen socket 64;
  while true do
    let client, address_of_client = Unix.accept socket in
    echo client
  done;
  Unix.close socket;
  print_endline "Server terminated"

let () = server ()

Let's try our code:

$ ocamlfind opt -linkpkg -package unix main.ml
$ ./a.out &
[1] 8592
$ echo "Hello World"|netcat -q0 localhost 3000
Hello World
$ echo "Hello World"|netcat -q0 localhost 3000
Hello World
$ kill -9 8592
[1]  + 8592 killed     ./a.out

It seems to work! However, let's try a specific scenario: when two clients try to connect "at the same time". First, we'll launch a background client that will simply connect, and then we'll launch another client that will attempt to send some text.

$ ./a.out &
[1] 8711
$ netcat localhost 3000 &
[2] 8728
[2]  + 8728 suspended (tty input)  netcat localhost 3000
$ echo "Hello World"|netcat -q0 localhost 3000
^C
$ kill -9 8728
$ kill -9 8711

Our second client, after our first one is connected, gets stuck. In reality, when our first client connected, it made our server unavailable. This goes back to our explanation of synchronicity: our program can strictly only do one thing at a time. So, our program is currently handling our first client, and it can't handle our second client until the first one is finished. In practice, our echo function must finish so that our server can handle other clients.

Asynchronicity

We're starting to see the fundamental problem of synchronicity in implementing a system and network application: the ability for our service to respond to all clients "at the same time." In our specific case, what we want is to be able to background our echo function so that our server can wait for a new connection again with accept. However, the concept of backgrounding a task is not so straightforward:

We know that we only have one thread available, so we can strictly only do one thing. Which thread could execute our background task?
We know that some functions put us in a waiting state (waiting for a new connection or waiting for data sent by the client). Instead of waiting, could we seize this opportunity to do "something else"?
Ultimately, we primarily want to respond to events coming from the system.

Several solutions exist for this. They vary even more depending on the language used and what it can offer to address these issues. Regarding OCaml 5, two elements can help us:

Effects
Domains

For the next chapter, we'll focus on effects and follow our second intuition. Namely, taking the opportunity to do something else as soon as we're waiting for an event such as the arrival of a connection with accept.

It is a unique identifier used by your system to represent an input/output resource. In concrete terms, it's a non-negative integer that serves as a reference to I/O channel within a process.

These parameters may seem daunting for a newcomer. If you're interested in delving deeper into system programming with OCaml and Unix, we recommend checking out this tutorial.

The more curious will say that OCaml is "multicore", and that's true. However, it is if you want to use the Domain/Thread module and allocate a domain/thread that can do a task in parallel or concurrently. But we'll explain all this in detail later.

A simple scheduler

In our previous chapter, we discussed the idea of backgrounding our echo function to continue accepting incoming connections. Let's stick with the idea of being able to background functions:

let fn () = print_endline "World"

let () =
  do_this_in_background fn;
  print_endline "Hello"

Here, do_this_in_background would be a function that takes a task and adds it to a kind of hidden to-do list. Instead of being literally synchronous and waiting to display "World" before displaying "Hello", we could imagine directly displaying "Hello" and letting something execute our fn function to display "World".

This "something" is what we call a scheduler. It holds our list of tasks to do and attempts to execute them all in a specific order. In our example, we would notify our scheduler that a task fn needs to be done, display "Hello", and then wait for our scheduler to complete all tasks (including displaying "World").

In our explanation, we subtly introduced a new concept: waiting. In reality, what we want is to wait for all our tasks to finish. To do this, we could have a "witness" for our tasks so that we can wait for these tasks through their witnesses:

let fn () = print_endline "World"

let () =
  let witness = do_this_in_background fn in
  print_endline "Hello";
  await witness

We're starting to get closer to what every scheduler aims to provide and what asynchronous programming is all about. Let's keep this example in mind and move on to another concept necessary for implementing our scheduler.

Effects

Since OCaml 5, it has been possible to utilise effects. An effect allows you to pause the execution of a function and enter a handler, which, depending on the effect, would execute a specific operation to resume the paused function with the result of that operation.

Here, we're delving into the flow of execution in your program. If we revisit our definition of synchronicity, we understand that our system processes the program line by line. However, effects (as well as exceptions) break this flow; they're known for breaking the linear progression of execution. Consider the example of an exception:

exception My_exception

let () =
  try print_endline "Hello";
      raise My_exception;
      print_endline "World"
  with My_exception -> print_endline "My_exception"

In this scenario, our code will print "Hello" and then trigger an exception. Consequently, the subsequent line won't execute. Instead, this exception will be "caught" by our handler with .... This mechanism attempts to identify the raised exception and, based on that, execute certain code — in our example, printing "My_exception".

Raising exceptions or triggering an effect can be likened to a jump in our code. The key distinction between an exception and an effect lies in the ability, for the latter, to return to the point where the effect was initiated.

open Effect.Deep

type _ Effect.t += My_effect : unit Effect.t

let run fn v =
  let rec retc x = x
  and exnc = raise
  and effc
    : type c. c Effect.t -> ((c, 'a) continuation -> 'b) option
    = function
    | My_effect ->
      print_endline "My_effect";
      Some (fun k -> continue k ())
    | _ -> None
  and handler = { retc; exnc; effc; } in
  match_with fn v handler

let my_program () =
  print_endline "Hello";
  Effect.perform My_effect;
  print_endline "World"

let () = run my_program ()

The mechanics are a bit more intricate, but the principle remains consistent. Upon executing the above code, we'll observe that we indeed enter our handler (and display "My_effect" akin to using a with ... block for exceptions), but we return to the precise point where the effect was initiated and then proceed to display "World".

When considering our core challenge of implementing a scheduler, the utility of effects becomes apparent in obtaining the value k as a representation of our suspended function at a specific point — where we triggered our effect. For managing tasks, with each task as a function, this k allows us to suspend and resume functions, maintaining them in a suspended state in the background.

Indeed, for our scheduler, maintaining this suspension is crucial. Rather than simply performing an operation and continuing with the result, we aim to:

keep the suspension in background
allow other tasks to execute.
resume the suspension after giving the opportunity for other tasks to execute.

A task

Now, we need to define what a task is. Earlier, we mentioned this value k, which would represent a suspended state of our function. We could define what a task is (an OCaml function) and in what state this task is:

an initial state
a suspended state that can be resumed
a termination state

open Effect.Shallow

type 'a state =
  | Initial of (unit -> 'a)
  | Suspended : ('c, 'a) continuation * 'c Effect.t -> 'a state
  | Resolved of 'a

Just like with exceptions, we need to "attach" a handler to catch effects and obtain that famous k. We can envision a generic handler that generates this state from the effects produced by a function:

let handler =
  let open Effect.Shallow in
  let retc v = Resolved v in
  let exnc = raise in
  let effc
    : type c. c Effect.t -> ((c, 'a) continuation -> 'b) option
    = fun eff -> Some (fun k -> Suspended (k, eff)) in
  { retc; exnc; effc }

Finally, we can define a task simply as our state.

type task = Task : 'a state -> task

At this stage, we have tasks, and it is now essential to define the operations that we can perform with them. If we revisit our code from the very beginning, which encapsulates the idea that we want to put tasks in the background, we essentially perform two operations:

we notify the scheduler of a new task (with our do_this_in_background).
we wait for our task with a witness of it.

A promise

This witness holds a term commonly found in asynchronous programming: a promise. Indeed, this witness is a promise that our task will be executed (but is not executed yet). Waiting for a task via its promise simply corresponds to obtaining the result of our task. In our example, this result is () : unit because we are merely displaying text, but we could easily imagine a hefty computation (such as finding a prime number) that we would want to run in the background.

In short, the promise would allow us to obtain this result. We could define it as a mutable value that changes as soon as our task is completed:

type 'a promise = 'a option ref

Now we can define our effects that will interact with our scheduler:

type _ Effect.t += Spawn : (unit -> 'a) -> 'a promise Effect.t
type _ Effect.t += Await : 'a promise -> 'a Effect.t

Our scheduler

Our types are defined, and we know how to obtain them. Now, all we need to do is implement our scheduler. As mentioned, the scheduler simply maintains a list of tasks to be done. Therefore, it's a task list that we'll be manipulating. The action of Spawn will enlarge this list, while Await will observe the state of our promise, which will change as soon as its associated task is completed. Waiting can result in two different situations:

the case where, indeed, the promise has been resolved. In this case, we simply transmit its result.
the case where it is not yet resolved. In this particular situation, we give the opportunity for other tasks to execute (which can help in resolving our initial task). This is referred to as yielding.

let perform
  : type c. task list ref -> c Effect.t -> [ `Continue of c | `Yield ]
  = fun todo -> function
  | Spawn fn ->
    let value = ref None in
    let task = Initial (fun () -> value := Some (fn ())) in
    todo := !todo @ [ Task task ] ;
    `Continue value
  | Await value ->
    begin match !value with
    | Some value -> `Continue value
    | None -> `Yield end
  | _ -> invalid_arg "Invalid effect"

Finally, it's just a matter of iterating over this list to gradually complete all our tasks. This iteration involves observing the state of each of our tasks.

For the initial state, we simply launch the task and see what we obtain through the handler we defined earlier (i.e., whether the task is resolved or suspended).
For the resolved state, there's nothing to do; our task has finished.
Lastly, for the suspended state, we need to determine what operation our effect produces (using perform). The case of yielding is interesting because it involves keeping our suspension in our to-do list and attempting to execute our other tasks first.

let step todo = function
  | Initial fn ->
    Effect.Shallow.(continue_with (fiber fn) () handler)
  | Resolved v -> Resolved v
  | Suspended (k, effect) ->
    match perform todo effect with
    | `Continue v -> Effect.Shallow.(continue_with k v handler)
    | `Yield -> Suspended (k, effect)

let run fn =
  let result = ref None in
  let rec go = function
    | [] -> Option.get !result
    | Task task :: rest ->
      let todo = ref rest in
      match step todo task with
      | Resolved _ -> go !todo
      | (Initial _ | Suspended _) as task -> go (!todo @ [ Task task ]) in
  let task = Initial (fun () -> result := Some (fn ())) in
  go [ Task task ]

Let's play!

Have you kept in mind our initial code and our primary goal? It was about putting a task in the background and executing it afterward. Returning to our basic problem, we wanted to manage our clients as background tasks while effectively handling the reception of new connections. Let's revisit the original code:

let fn () = print_endline "World"

let () =
  let witness = do_this_in_background fn in
  print_endline "Hello";
  await witness

With our scheduler, this code would become:

let spawn fn = Effect.perform (Spawn fn)
let await prm = Effect.perform (Await prm)

let fn () = print_endline "World"

let () = run @@ fun () ->
  let prm = spawn fn in
  print_endline "Hello";
  await prm

If we compile all of this and run the code, we get:

$ ocamlopt main.ml
$ ./a.out
Hello
World

Et voilà! Our task displaying "World" was successfully put into the background, and we indeed displayed "Hello" first. We now have the basics of our scheduler. You now understand the core concepts of all schedulers (whether in OCaml or JavaScript). Several (perhaps suboptimal) choices were made, but the most important thing is to grasp the concept of asynchronous programming through a concrete example.

Now, it's time to address our initial problem: managing our clients while also accepting new connections. At this stage, you might think that simply "spawning" our echo function will make it work in the background. However, even though we've addressed the issue of synchronicity by offering an asynchronous library, we deliberately overlooked mentioning blocking functions! That's what we'll explore in our next chapter.

Interacting with the system

In the implementation of our echo server, beyond the question of synchronicity, there were also considerations about interactions with the system and the resources it provides, such as sockets.

We noticed that to manage these resources, we had functions described as "blocking", meaning they waited for specific events (such as a new client connection) before proceeding. Apart from wanting to delegate tasks in the background, we also aimed to leverage these situations to perform other tasks.

We have this opportunity with Await, which observes the state of our promise and then decides to continue if it contains the result of our task or to "yield" (i.e., execute other tasks) if the associated task is not yet complete.

We could reproduce the same approach for these blocking functions: continue if they have an event to notify us about, or "yield" if we know they will block. The crucial question then is to predict in advance whether they will block. Fortunately, the system can provide us with this information.

File-descriptors

In our first chapter, we introduced the concept of file descriptors. These are system resources used to manage I/O operations such as handling client connections, transmitting bytes, and more. It's essential to monitor the state of these resources and determine beforehand whether functions like accept() (for managing a new client) will block or not.

Typically, we can consider that all our functions interacting with the system block by default. However, we can periodically check our active file descriptors to determine if we can safely resume functions that will perform these blocking system calls.

Monitoring the state of our active file descriptors and determining if an event (which would unblock our functions) occurs is done using the select() function:

val select :
    file_descr list -> file_descr list -> file_descr list ->
      float -> file_descr list * file_descr list * file_descr list

This function takes several arguments, but only 3 are of interest to us. The first and second arguments pertain to monitoring file descriptors that are awaiting "read" and "write" operations, respectively. Typically, when we want to wait for a client connection, we are waiting for a "read" operation on our file descriptor. If we intend to transmit bytes to the client, we are waiting to be able to "write" to our file descriptor. The last argument that concerns us is the timeout for this observation. A reasonably short time is sufficient; let's say 10ms.

For example, let's consider our accept() function. We want to determine whether we should execute accept() without blocking:

let rec our_accept file_descr =
  print_endline "Monitor our file-descriptor.";
  match Unix.select [ file_descr ] [] [] 0.01 with
  | [], _, _ -> our_accept file_descr
  | file_descr :: _, _, _ -> Unix.accept file_descr

let server () =
  let socket = Unix.socket Unix.PF_INET Unix.SOCK_STREAM 0 in
  let sockaddr = Unix.ADDR_INET (Unix.inet_addr_loopback, 3000) in
  Unix.bind socket sockaddr;
  Unix.listen socket 64;
  let client, sockaddr = our_accept socket in
  Unix.close client;
  Unix.close socket

Let's test this code and see what happens.

$ ocamlfind opt -linkpkg -package unix main.ml
$ ./a.out &; sleep 1; netcat -q0 localhost 3000
Monitor our file-descriptor.
Monitor our file-descriptor.
Monitor our file-descriptor.
...
[1]  + 6210 done       ./a.out

Testing this code reveals the repeated "Monitoring our file-descriptor." messages until the program ends (after receiving a connection using netcat after 1 second). What's interesting here is that instead of blocking on accept(), we execute it only if select() informs us that our file descriptor is indeed ready (meaning it has received an incoming connection). If not, we retry the observation by calling select() again.

In more concrete terms, we are no longer in a situation where we indefinitely wait for an event to unblock us, but rather we wait for just 10ms to retry an observation or execute our accept() if ready. We've found a way to determine in advance whether our function will block or not.

Integration into our scheduler

Now, the advantage of select() is that it can observe multiple file descriptors (not just one as in our example). Our goal is to provide an our_accept function that doesn't block. In case our file descriptor isn't ready (which is the default case, as a reminder, all our system functions block), we'll reuse our Await to suspend the execution before actually performing our accept(). This suspension will give us the opportunity to execute other tasks.

let waiting_fds_for_reading = Hashtbl.create 0x100

let our_accept file_descr =
  let value = ref None in
  Hashtbl.add waiting_fds_for_reading file_descr value;
  Effect.perform (Await value);
  Hashtbl.remove waiting_fds_for_reading file_descr;
  Unix.accept file_descr

Finally, periodically, we'll observe all the file descriptors that are waiting. select() will inform us about those that can be unblocked. We just need to fulfill our promise so that our scheduler can resume our suspended function.

let fullfill tbl fd =
  let value = Hashtbl.find tbl fd in
  value := Some ()

let our_select () =
  let rds = List.of_seq (Hashtbl.to_seq_keys waiting_fds_for_reading) in
  let rds, _, _ = Unix.select rds [] [] 0.01 in
  List.iter (fullfill waiting_fds_for_reading) rds

Ultimately, we just need to call our_select() periodically. We previously mentioned that our scheduler tries to resolve our tasks step by step. We'll interleave these steps with this observation. This way, we'll be almost immediately aware of the occurrence of events (within 10ms and a snippet of a task execution).

let run fn =
  let result = ref None in
  let rec go = function
    | [] -> Option.get !result
    | Task task :: rest ->
        let todo = ref rest in
        let todo =
          match step todo task with
          | Resolved _ -> !todo
          | (Initial _ | Suspended _) as task -> !todo @ [ Task task ]
        in
        our_select (); go todo
  in
  let task = Initial (fun () -> result := Some (fn ())) in
  go [ Task task ]

Let's try!

Let's revisit our example with accept():

let server () =
  let socket = Unix.socket Unix.PF_INET Unix.SOCK_STREAM 0 in
  let sockaddr = Unix.ADDR_INET (Unix.inet_addr_loopback, 3000) in
  Unix.bind socket sockaddr;
  Unix.listen socket 64;
  let client, sockaddr = our_accept socket in
  Unix.close client;
  Unix.close socket

let () = run server

If we execute our server with our scheduler:

$ ocamlfind opt -linkpkg -package unix main.ml
$ ./a.out &; netcat -q0 localhost 3000
[1] 38255
[1]  + 38255 done       ./a.out

We notice that our program does not indefinitely block. It only blocks periodically for 10ms¹ and observes the active file descriptors (in our waiting_fds_for_reading table). Finally, as soon as netcat connects, we can resume our our_accept function and continue executing our program. With the ability to put tasks in the background, we can now attempt to reimplement our server asynchronously. However, we need to provide, just like our_accept, our_read and our_write. The first one will reuse our waiting_fds_for_reading table, while the second one will use a new table to determine if our file descriptors are ready to transmit bytes.

let our_read file_descr buf off len =
  let value = ref None in
  Hashtbl.add waiting_fds_for_reading file_descr value;
  Effect.perform (Await value);
  Hashtbl.remove waiting_fds_for_reading file_descr;
  Unix.read file_descr buf off len

let waiting_fds_for_writing = Hashtbl.create 0x100

let our_write file_descr buf off len =
  let value = ref None in
  Hashtbl.add waiting_fds_for_writing file_descr value;
  Effect.perform (Await value);
  Hashtbl.remove waiting_fds_for_writing file_descr;
  Unix.write file_descr buf off len

let our_select () =
  let rds = List.of_seq (Hashtbl.to_seq_keys waiting_fds_for_reading) in
  let wrs = List.of_seq (Hashtbl.to_seq_keys waiting_fds_for_writing) in
  let rds, wrs, _ = Unix.select rds wrs [] 0.01 in
  List.iter (fullfill waiting_fds_for_reading) rds;
  List.iter (fullfill waiting_fds_for_writing) wrs

Now, we can both await new connections and manage in background our clients:

let rec echo client =
  let buf = Bytes.create 0x100 in
  let len = our_read client buf 0 (Bytes.length buf) in
  if len = 0 then Unix.close client
  else let _ = our_write client buf 0 len in echo client

let server () =
  let socket = Unix.socket Unix.PF_INET Unix.SOCK_STREAM 0 in
  let sockaddr = Unix.ADDR_INET (Unix.inet_addr_loopback, 3000) in
  Unix.bind socket sockaddr;
  Unix.listen socket 64;
  while true do
    let client, address_of_client = our_accept socket in
    ignore (spawn @@ fun () -> echo client)
  done;
  Unix.close socket;
  print_endline "Server terminated"

let () = run server

To test this code, simply launch your server and run 2 netcat instances simultaneously (in 2 different terminals). You'll notice that our server no longer blocks and can handle these 2 clients "simultaneously". We have finally succeeded in creating an asynchronous server with effects in OCaml.

$ ocamlfind opt -linkpkg -package unix main.ml
$ ./a.out &; \
  echo "Salut"|netcat -q0 localhost 3000; \
  echo "Hello"|netcat -q0 localhost 3000
[1] 40381
Salut
Hello
$ kill -9 40381
[1]  + 40381 killed     ./a.out

At this point, all the basic concepts of a scheduler and asynchronous programming have been explained. It's time to take a look back at what we've learned and, most importantly, start comparing it with Miou in the next chapter.

The purpose of the 10ms interval is to prevent our program from falling into what is known as a "busy-loop". Indeed, these 10ms intervals notify our system that our program will do nothing during this time period unless an event occurs. Our system is then able to put our program to the sleep mode and also take the opportunity to do something else. What is certain is that this sleep mode allows our program not to monopolize the processor. In the case of a "busy-loop," our program would be the only one able to run, and you would likely hear your processor fan whirring loudly.

Retrospective

At this stage, you're familiar with all the concepts of a scheduler, asynchronous programming, and interactions with the system. Of course, as you might suspect, we've omitted a whole bunch of details, and Miou offers much more than our simple scheduler. However, at this point, we can describe in detail what Miou brings to the table (including its subtleties). That's what we'll explore in this chapter.

A task as a resource

Let's revisit our example with the echo server, where we aimed to handle client management in the background:

    ignore (spawn @@ fun () -> echo client)

You can achieve the same thing with Miou.call_cc. This function essentially does will more what our spawn does: it creates a new task that will run on the same thread using our scheduler. This type of task which coexists with others on the same thread is called a fiber. And, just like spawn, Miou.call_cc also returns a promise for this task. In Miou, you're creating a child of your task, a subtask.

The key difference with Miou, though, is that you can't forget your children!

let () = Miou.run @@ fun () ->
  ignore (Miou.call_cc @@ fun () -> print_endline "Hello World!")
Exception: Miou.Still_has_children.

In Miou, we treat a task as a resource. You allocate it (using Miou.call_cc), but you also have to release it with Miou.await. A fundamental rule governs Miou programs: all tasks must either be awaited or canceled. Forgetting a task will result in a fatal error.

Background tasks

This brings up another question: what should we do with our subtasks that manage clients? If this rule exists, it's because these children can misbehave. And you need to be notified of these abnormal situations. What matters isn't the existence of these tasks (since your goal is to put them in the background) but their results to ensure everything went well.

In this regard, Miou offers a way to save your tasks somewhere and retrieve them once they're completed. This is mainly done using the orphans value:

let rec clean_up orphans = match Miou.care orphans with
  | None | Some None -> ()
  | Some (Some prm) ->
    match Miou.await prm with
    | Ok () -> clean_up orphans
    | Error exn -> raise exn

let server () =
  ...
  let orphans = Miou.orphans () in
  while true do
    clean_up orphans;
    let client, address_of_client = Miou_unix.accept socket in
    ignore (Miou.call_cc ~orphans @@ fun () -> echo client)
  done

The advantage of this approach is that it treats a task as a resource that must be released at some point in your program's execution. Our experience in implementing protocols at Robur has convinced us of the importance of not forgetting about our children. Developing system and network applications involves creating programs with long execution time (6 months, 1 year, etc.). Tasks consume memory and possibly processor resources. Forgetting tasks can lead to memory leaks, which can hinder your program's long-term viability (the system might terminate your program due to an out-of-memory error).

Structured concurrency

Managing tasks and their promises can be a real challenge when implementing a large application. Indeed, conceptualizing tasks running in the background leaves room for practices (like forgetting these tasks) that can lead to significant maintenance overhead in the long run. At Robur, through certain projects we maintain, we've encountered situations where the time to fix bugs becomes disproportionately large given our resources because we need to re-establish the mental model of task management, which isn't all that obvious.

Thus, when developing Miou, it was essential from the outset to establish rules to prevent repeating past mistakes. We've already introduced one rule: never forget your children.

There's a second rule: only the direct parent can wait for or cancel its children. For instance, the following code is incorrect:

let () = Miou.run @@ fun () ->
  let a = Miou.call_cc @@ fun () -> Miou.yield () in
  let b = Miou.call_cc @@ fun () -> Miou.await_exn a in
  Miou.await_exn a;
  Miou.await_exn b
Exception: Miou.Not_a_child.

The purpose of such a constraint is to maintain a simple mental model of the active tasks in your application: their affiliations form a tree with the root being your main task (the one launched with Miou.run). Therefore, if your main task terminates, it invariably means that all sub-tasks have also terminated.

Cancellation

One aspect deliberately left out in the implementation of our small scheduler is cancellation. It can be useful to cancel a task that, for example, is taking too long. Miou provides this mechanism for all tasks using their promises.

let () = Miou.run @@ fun () ->
  let prm = Miou.call_cc @@ fun () -> print_endline "Hello World" in
  Miou.cancel prm

The rules of parentage explained earlier also apply to cancellation. You can only cancel your direct children:

let () = Miou.run @@ fun () ->
  let a = Miou.call_cc @@ fun () -> Miou.yield () in
  let b = Miou.call_cc @@ fun () -> Miou.cancel a in
  Miou.await_exn a;
  Miou.await_exn b
Exception: Miou.Not_a_child.

Cancellation overrides any promise state. You can cancel a task that has already completed. In this case, we lose the result of the task, and it's considered canceled:

let () = Miou.run @@ fun () ->
  let prm = Miou.call @@ fun () -> 1 + 1 in
  let _ = Miou.await prm in
  Miou.cancel prm;
  match Miou.await prm with
  | Ok _ -> assert false
  | Error Miou.Cancelled -> assert true
  | Error exn -> raise exn

Of course, to be consistent with our other rules, canceling a task implies canceling all its sub-tasks:

let rec infinite () = infinite (Miou.yield ())

let () = Miou.run @@ fun () ->
  let p = Miou.call_cc @@ fun () ->
    let q = Miou.call infinite in
    Miou.await_exn q in
  Miou.cancel p

Lastly, let's delve into the behavior of Miou.cancel. It's said that this function is asynchronous in the sense that cancellation (especially that of a task running in parallel) may take some time. Thus, Miou seizes the opportunity to execute other tasks during this cancellation. For example, note that p0 runs despite the cancellation of p1:

let () = Miou.run @@ fun () ->
  let p1 = Miou.call @@ fun () -> Miou.yield () in
  let v = ref false in
  let p0 = Miou.call_cc @@ fun () -> print_endline "Do p0" in
  print_endline "Cancel p1";
  Miou.cancel p1;
  print_endline "p1 cancelled";
  Miou.await_exn p0

$ ocamlfind opt -linkpkg -package miou main.ml
$ ./a.out
Cancel p1
Do p0
p1 is cancelled

The advantage of this asynchronicity is to always be able to handle system events even if we attempt to cancel a task.

Multiple domain runtime

In the introduction, it was mentioned that it's possible to use multiple domains with Miou. Indeed, since OCaml 5, it has been possible to launch functions in parallel. This parallelism has become possible only recently because these functions have their own minor heap. Thus, synchronization between domains regarding allocation and garbage collection is less systematic.

To launch a task in parallel with Miou, it's sufficient to use:

let () = Miou.run @@ fun () ->
  let prm = Miou.call @@ fun () ->
    print_endline "My parallel task." in
  Miou.await_exn prm

Miou takes care of allocating multiple domains according to your system's specifics. These domains will be waiting for tasks, and Miou.call notifies them of a new task to perform. Just like Miou.call_cc, Miou.call also returns a promise, and the same rules apply: you must not forget about your children.

A task in parallel explicitly means that it will run in a different domain than the one it was created in. That is, this code, which returns the domain in which the task is executing, is always true:

let () = Miou.run @@ fun () ->
  let p =
    Miou.call @@ fun () ->
    let u = Stdlib.Domain.self () in
    let q = Miou.call @@ fun () -> Stdlib.Domain.self () in
    (u, Miou.await_exn q)
  in
  let u, v = Miou.await_exn p in
  assert (u <> v)

However, the choice of the domain responsible for the task is made randomly. Thus, this code is also true (meaning that two tasks launched in succession can then use the same domain):

let () = Miou.run @@ fun () ->
  let assertion = ref false in
  while !assertion = false do
    let p = Miou.call @@ fun () -> Stdlib.Domain.self () in
    let q = Miou.call @@ fun () -> Stdlib.Domain.self () in
    let u = Miou.await_exn p in
    let v = Miou.await_exn q in
    assertion := u = v
  done

It may happen then that we want to distribute a specific task to all our available domains. We cannot do this with Miou.call, which may, several times, assign the same domain for a task. However, Miou offers a way to distribute the workload evenly across all your domains:

let task () : int = (Stdlib.Domain.self () :> int)

let () = Miou.run ~domains:3 @@ fun () ->
  let domains =
    Miou.parallel task (List.init 3 (Fun.const ()))
    |> List.map Result.get_ok in
  assert (domains = [1; 2; 3])

Finally, one last rule exists regarding parallel tasks. There may be a situation called starvation. Indeed, like your number of cores, the number of domains is limited. It may happen that domains wait for each other, but it's certain that the main domain (the very first one that executes your code, known as dom0) will never be assigned a task via Miou.call.

This rule prevents a domain from waiting for another domain, which waits for another domain, which waits for dom0, which waits for your first domain - the starvation problem. Thus, it may happen that dom0 is no longer involved in the execution of your program and is only waiting for the other domains. However, we can involve it using Miou.call_cc:

let task () : int = (Stdlib.Domain.self () :> int)

let () = Miou.run ~domains:3 @@ fun () ->
  let prm = Miou.call_cc my_super_task in
  let domains =
    Miou.await prm
    :: Miou.parallel task (List.init 3 (Fun.const ())) in
  let domains = List.map Result.get_ok domains in
  assert (domains = [0; 1; 2; 3])

However, we would like to warn the user. Parallelizing a task doesn't necessarily mean that your program will be faster. Once again, Miou focuses essentially on managing system events. Domains are equally subject to observing these events, which means that all computations are interleaved with this observation (and can therefore have an impact on performance).

Having more domains is not a solution for performance either. If Miou takes care of allocating (as well as releasing) domains, it's because they require a lot of resources (such as a minor-heap). Thus, the domains are available, but it's Miou that manages them.

Finally, Miou mainly aims to facilitate the use of these domains, especially regarding inter-domain synchronization. Indeed, tasks as well as their promises concerning Miou.call do not exist in the same domains. An internal mechanism helps the user not to worry about the synchronicity between the task's state and its promise, even though they exist in two spaces that run in parallel.

For the next steps

This retrospective allows us to introduce the basic elements of Miou. We now need to see how to use Miou and also introduce you to some new concepts. The next chapter will consist of re-implementing our echo server with Miou. There should be only a few differences, but we will seize the opportunity to improve our server, especially with the use of parallel tasks and the notion of ownership.

An echo server with Miou

Passer de notre petit scheduler à Miou est assez simple. Mise à part la règle de ne pas oublier ses enfants que nous avons introduit dans le chapitre précédent, il s'agit principalement d'utiliser Miou et Miou_unix. Prennons la fonction echo qui gère nos clients:

Moving from our custom scheduler to Miou is quite straightforward. Apart from the rule of not forgetting our children that we introduced in the previous chapter, it mainly involves using Miou and Miou_unix. Let's take the echo function that handles our clients:

let rec echo client =
  let buf = Bytes.create 0x100 in
  let len = Miou_unix.read client buf 0 (Bytes.length buf) in
  if len = 0 then Miou_unix.close client
  else
    let str = Bytes.sub_string buf 0 len in
    let _ = Miou_unix.write client str 0 len in echo client

A subtlety lies in Miou_unix.write, which expects a string instead of bytes. In a concurrent execution, it may happen that buf could be modified concurrently. Using a string ensures that the buffer we want to transmit does not change in the meantime. Once again, this comes from our experience in protocol implementation.

Then, according to the rules introduced in the previous chapter, we need to re-implement our server so that it uses an orphans value and periodically cleans up our terminated clients to avoid forgetting our children:

let clean_up orphans = match Miou.care orphans with
  | None | Some None -> ()
  | Some (Some prm) -> match Miou.await prm with
    | Ok () -> ()
    | Error exn -> raise exn

let server () =
  let socket = Miou_unix.tcpv4 () in
  let sockaddr = Unix.ADDR_INET (Unix.inet_addr_loopback, 3000) in
  Miou_unix.bind_and_listen socket sockaddr;
  let orphans = Miou.orphans () in
  while true do
    clean_up orphans;
    let client, _ = Miou_unix.accept socket in
    ignore (Miou.call_cc ~orphans (fun () -> echo client))
  done;
  Miou_unix.close socket

let () = Miou_unix.run server

Note the use of Miou_unix.run to handle system events and the functions from this module. For more details, we will explain the interactions between the system and Miou more precisely in the next chapter.

And there we go, we've switched to Miou! All we need to do is compile our project like this:

$ opam install miou
$ ocamlfind opt -linkpkg -package miou,miou.unix main.ml
$ ./a.out &
[1] 436169
$ echo "Hello" | netcat -q0 localhost 3000
Hello
$ kill -9 436169
[1]  + 436169 killed     ./a.out

Parallelism

Since it's now possible to utilise multiple domains, let's take advantage of this to instantiate more than one server. Indeed, it's conceivable that multiple servers could exist at the same address (localhost:3000). In such a scenario, it's first come, first served. Therefore, we can envision managing multiple servers in parallel, each handling several clients concurrently.

To distribute the implementation of our server across multiple domains, we'll use Miou.parallel. We won't forget to involve dom0 (referring to our rule where dom0 would never be assigned a task from other domain) via Miou.call_cc:

let () = Miou_unix.run @@ fun () ->
  let domains = Stdlib.Domain.recommended_domain_count () - 1 in
  let domains = List.init domains (Fun.const ()) in
  let prm = Miou.call_cc server in
  Miou.await prm :: Miou.parallel server domains
  |> List.iter @@ function
  | Ok () -> ()
  | Error exn -> raise exn

To ensure that we're leveraging the full potential of our machine, we can check how many threads our program has (note that an OCaml domain always has 2 threads!). Thus, for a system with 32 cores:

$ ocamlfind opt -linkpkg -package miou,miou.unix main.ml
$ ./a.out &
[1] 438053
$ ls /proc/438053/task | wc -l
64
$ kill -9 438053
[1]  + 438053 killed     ./a.out

Almost for free, we've managed to launch multiple servers in parallel!

Ownership

When it comes to building system and network applications, we often deal with resources shared between the application and the system. One such resource we use here is the file descriptor. OCaml has the advantage of offering a garbage collector to handle memory management for us. However, we still need to consider releasing system resources, particularly file descriptors.

Another point to consider is the manipulation of these resources. We subtly mentioned, using Miou_unix.write, the possibility that a buffer could be concurrently modified. From our experience, the concept of resource ownership (like a buffer) specific to a particular task is lacking in OCaml and can lead to rather challenging bugs to identify and understand. In this regard, languages like Rust offer solutions that can help developers avoid a resource being manipulated by two tasks "at the same time". The problem is even more significant with tasks that can run in parallel. This is referred to as a data race.

Therefore, the best we can offer, Miou provides resource management that resembles that of Rust: a task has exclusive access to a resource once it has "proof" of ownership.

Miou offers an API, Miou.Ownership, where you can:

Create proof of ownership
Own a resource through this proof
Disown a resource through this proof
Transfer this resource to a child via this proof
Transfer this resource to the parent via this proof
Verify, before manipulating this resource, that you have exclusive access to it

Miou_unix extends this API to file descriptors. In this second part of this chapter, it's essential to ensure that we are indeed the owners of the file descriptor we are manipulating. This won't change the behavior of our server; it just allows us to sleep better tonight!

Let's start with the echo function:

let rec echo client =
  let buf = Bytes.create 0x100 in
  let len = Miou_unix.Ownership.read client buf 0 (Bytes.length buf) in
  if len = 0 then Miou_unix.Ownership.close client
  else
    let str = Bytes.sub_string buf 0 len in
    let _ = Miou_unix.Ownership.write client str 0 len in echo client

Miou_unix.Ownership.{read,write} perform the necessary checks, while Miou_unix.Ownership.close disown our file descriptor since we no longer need it. Forgetting this step would result in an error in your application, and Miou would notify you that a resource has been forgotten (via Miou.Resource_leaked). Once attached to a task, a resource must be transferred or released; otherwise, it's considered forgotten! The aim is truly to assist the developer in not forgetting anything they manipulate.

Another interesting aspect of resources is the case of an abnormal termination of our echo function via an exception. A resource is also associated with a finalizer that will be executed if the task in possession of the resource terminates abnormally. Again, the goal is to sleep well tonight.

Now, let's move on to our server function, where we need to transfer our client file descriptor to our echo task:

let server () =
  let socket = Miou_unix.Ownership.tcpv4 () in
  let sockaddr = Unix.ADDR_INET (Unix.inet_addr_loopback, 3000) in
  Miou_unix.Ownership.bind_and_listen socket sockaddr;
  let orphans = Miou.orphans () in
  while true do
    clean_up orphans;
    let client, _ = Miou_unix.Ownership.accept socket in
    ignore (Miou.call_cc
      ~give:[ Miou_unix.Ownership.resource client ]
      ~orphans (fun () -> echo client))
  done;
  Miou_unix.Ownership.close socket

And there you have it! Assuming everything goes well, our code is correct, and we are using our resources correctly. The Miou.Ownership module is not mandatory in Miou's usage but provides a value to dynamically verify the proper use and transfer of your resources. While it's not obligatory, we strongly recommend using it.

The finalizer associated with the resource can also be genuinely beneficial, especially when cancellation occurs: it ensures there are no leaks, even in abnormal situations.

Conclusion

If you've made it this far, you've likely seen a good portion of what Miou has to offer and delved into the intricacies of asynchronous programming and system interactions. You can continue experimenting and having fun with Miou or delve deeper into our tutorial.

If you recall our initial challenge with our echo server, we divided the subject into two parts: the scheduler and system interactions. Miou also maintains this separation between the Miou module and the Miou_unix module. The next chapter will revisit system interactions, but this time with Miou. The goal will be to implement sleepers (and replicate Unix.sleep).

Finally, the last chapter is an enhancement of our echo server using Mutexes and Conditions provided by Miou. This chapter explains in detail the benefits of using these modules over those offered by OCaml and presents, once again, a concrete case.

Interacting with the system with Miou

Miou is structured similarly to our small scheduler, but with a clear distinction between scheduling and system interaction.

The goal is to be able to inject a certain type of interaction based on the system used, which may not necessarily be Unix. As mentioned in the introduction, our cooperative also aims to implement services as unikernels. These are very specific systems where the concept of files may not even exist. In this sense, Miou_unix is specific to Unix.

Unix and Miou_unix may suffice for most use cases. However, they may not be suitable for unikernels and may not meet your criteria and system requirements. A system like Linux may offer alternative means such as epoll() for interacting with the external world, for example.

Beyond other possibilities, subtle differences may also exist. Indeed, systems like Linux or *BSD may not necessarily offer the same semantics in their handling of TCP/IP connections, for instance. These nuances make it difficult and even error-prone to try to standardize all these interactions. Overall, we believe that you are the best person to know how to interact with the system. As such, we offer this small tutorial to guide you on how to implement the glue between Miou and your system.

Sleepers

One of the simplest interactions we can implement with Miou is the sleeper. It's akin to what Unix.sleep does: waiting for a certain amount of time. We'll iterate through its implementation to:

Start by offering such a function.
Handle the scenario where multiple domains might use this function.
Finally, manage the cancellation of tasks that have called this function.

To guide our tutorial, here's the code we aim to execute. The module Chat is the one we need to implement.

let program () =
  Chat.run @@ fun () ->
  let a = Miou.call_cc @@ fun () -> Chat.sleep 1. in
  let b = Miou.call_cc @@ fun () -> Chat.sleep 2. in
  Miou.await_all [ a; b ]
  |> List.iter @@ function
  | Ok () -> ()
  | Error exn -> raise exn

let () =
  let t0 = Unix.gettimeofday () in
  program ();
  let t1 = Unix.gettimeofday () in
  assert (t1 -. t0 < 3.)

If we assume that tasks run "simultaneously," this program should execute in less than 3 seconds.

Our sleeper function

If you remember from our little scheduler, we reused our Await effect to implement our our_accept function. We then injected our our_select into our scheduler so that it could observe events and signal suspension points to resume them.

Similarly, Miou offers such mechanisms. You can suspend a function using a unique identifier called Miou.syscall:

type syscall
type uid = private int [@@immediate]

val syscall : unit -> syscall
val uid : syscall -> uid
val suspend : syscall -> unit

The equivalent of our Effect.perform (Await prm) from our little scheduler would be Miou.suspend. Therefore, our main goal is to save our syscall somewhere and attach the time we need to wait to it:

type sleeper =
  { time : float
  ; syscall : Miou.syscall }

module Sleepers = Miou.Pqueue.Make (struct
  type t = sleeper

  let compare { time= a; _ } { time= b; _ } = Float.compare a b
  let dummy = { time= 0.0; syscall= Obj.magic () }
end)

let sleepers = Sleepers.create ()

let sleep delay =
  let time = Unix.gettimeofday () +. delay in
  let syscall = Miou.syscall () in
  Sleepers.insert { time; syscall } sleepers;
  Miou.suspend syscall

Using a priority queue¹ will allow us to get the syscall that needs to resume its function as soon as possible among all our sleepers.

Now we need to find a way to inject a function that will be called periodically, in which we can resume our functions just like we did for our little scheduler with our our_select function. Fortunately, Miou offers this possibility of injecting such a function via Miou.run:

type select = block:bool -> uid list -> signal list
type events = { select: select; interrupt: unit -> unit }

val run : ?events:(Domain.Uid.t -> events) -> (unit -> 'a) -> 'a

At the very beginning, Miou tries to allocate the domains. For each domain, it will ask for an event value that contains a select function. This is the function we need to implement and provide to Miou. This function will be called whenever there is a syscall after executing a portion of our tasks and periodically. This function must return signals allowing Miou to resume functions that have been suspended by our syscalls.

type signal

val signal : syscall -> signal

So, one of its goals is to determine if one of our sleepers should resume one of our tasks or not.

let in_the_past ts = ts = 0. || ts <= Unix.gettimeofday ()

let rec remove_and_signal_older sleepers acc =
  match Sleepers.find_min sleepers with
  | None -> acc
  | Some { time; syscall } when in_the_past time ->
    Sleepers.delete_min_exn sleepers;
    remove_and_signal_older sleepers (Miou.signal syscall :: acc)
  | Some _ -> acc

let select ~block:_ _ =
  let ts = match Sleepers.find_min sleepers with
    | None -> 0.0
    | Some { time; _ } ->
      let value = time -. Unix.gettimeofday () in
      Float.max 0.0 value in
  Unix.sleepf ts;
  remove_and_signal_older sleepers []

let events _ = { Miou.select; interrupt= ignore }
let run fn = Miou.run ~events fn

As you can see, we are specializing our Miou.run function with our events. That's why whenever you use functions from Miou_unix, for example, you should use Miou_unix.run. Let's now try our code:

$ ocamlfind opt -linkpkg -package miou,unix -c chat.ml
$ ocamlfind opt -linkpkg -package miou,unix chat.cmx main.ml
$ ./a.out
$ echo $?
0

And there you go! We've just proven that our two tasks are running concurrently. Note the use of Unix.sleepf instead of Unix.select. Here, we are only interested in waiting rather than observing our file descriptors.

Domains & system

As mentioned earlier, Miou manages multiple domains. Therefore, if our select function uses a global variable like sleepers, we will definitely encounter an access problem with this variable across domains. There are several solutions to this issue, one of which involves "protecting" our global variable using a Mutex. However, we have specified that each domain manages its own select.

In general, a syscall is always local to a domain; it cannot be managed by another domain that did not suspend it. In this sense, we can consider allocating a sleepers for each domain. In OCaml, there is a way to consider values that exist and are accessible to each domain, and these domains have exclusivity over them: it's called Thread Local Storage.

So instead of having a global sleepers, we will use this API:

let sleepers =
  let key = Stdlib.Domain.DLS.new_key Sleepers.create in
  fun () -> Stdlib.Domain.DLS.get key

let sleep delay =
  let sleepers = sleepers () in
  let time = Unix.gettimeofday () +. delay in
  let syscall = Miou.syscall () in
  Sleepers.insert { time; syscall } sleepers;
  Miou.suspend syscall

let select ~block:_ _ =
  let sleepers = sleepers () in
  let ts = match Sleepers.find_min sleepers with
    | None -> 0.0
    | Some { time; _ } ->
      let value = time -. Unix.gettimeofday () in
      Float.max 0.0 value in
  Unix.sleepf ts;
  remove_and_signal_older sleepers []

We can now safely replace our Miou.call_cc with Miou.call. We know that each task will have its own sleepers, and there will be no illegal access between domains.

Cancellation

There is one last point remaining: cancellation. Indeed, a task that has suspended on a syscall can be canceled. We need to "clean up" our syscalls and consider some of them unnecessary to observe. The select function has a final argument corresponding to a list of canceled syscalls (with their unique identifiers). When cancellation occurs, Miou attempts to collect all canceled syscalls and then provides them to you so that you can clean up your variables from these syscalls (in this case, our sleepers).

Another subtlety concerns inter-domain synchronization. Cancellation may require synchronization between two domains, especially if we use Miou.call where the promise exists in a different domain than the executing task. The problem is that this synchronization may occur when one of the two domains performs Unix.sleepf: in this case, we would need to wait for our domain to complete its operation before continuing with cancellation. This is obviously not feasible, especially if cancellation involves cleaning up a myriad of promises and tasks across multiple domains (recall that Miou.cancel also cancels children).

We have not mentioned it yet, but events has another function called interrupt. This function precisely allows Miou to interrupt select because the state of a promise has changed (and this change must be taken into account in our syscall management).

So the question is: how do we interrupt Unix.sleepf? There are several solutions, such as sending a signal, for example (and generating the EINTR² exception). However, we could just as well reuse Unix.select. To remind you, this function can wait for a certain time (just like Unix.sleepf) but can also be interrupted as soon as an event occurs (such as the execution of interrupt by Miou). The idea is to create a file descriptor that Miou can manipulate and that Unix.select can observe.

Once again, the idea is not to interrupt the entire domain. An interruption does not necessarily mean that we want to wake up the domain because we know it is in a state where it cannot handle cancellation—actually, we cannot know that. The interruption just aims to unblock the domain only if it is performing a select. Thus, it may happen that Miou attempts to interrupt multiple times when it is unnecessary — because the domain in question is not operating on the select. But that's okay; we can simply ignore these interruptions.

So, we have two things to do to manage cancellation:

Clean up our sleepers from the syscalls given as arguments to our select function.
Be able to interrupt our select using a file descriptor that Miou could use.

Let's start by cleaning up our syscalls. The problem with Miou.Pqueue is that we cannot arbitrarily delete an element unless it is the smallest element. Indeed, we could cancel a sleeper that is not necessarily the next in terms of time. But we could tag our sleepers as canceled and simply ignore them when we should signal them:

type sleeper =
  { time : float
  ; syscall : Miou.syscall
  ; mutable cancelled : bool }

module Sleepers = Miou.Pqueue.Make (struct
  type t = sleeper

  let compare { time= a; _ } { time= b; _ } = Float.compare a b
  let dummy = { time= 0.0; syscall= Obj.magic (); cancelled= false }
end)

let sleepers =
  let key = Stdlib.Domain.DLS.new_key Sleepers.create in
  fun () -> Stdlib.Domain.DLS.get key

let sleep delay =
  let sleepers = sleepers () in
  let time = Unix.gettimeofday () +. delay in
  let syscall = Miou.syscall () in
  Sleepers.insert { time; syscall; cancelled= false } sleepers;
  Miou.suspend syscall

let rec remove_and_signal_older sleepers acc =
  match Sleepers.find_min sleepers with
  | None -> acc
  | Some { cancelled; _ } when cancelled ->
    Sleepers.delete_min_exn sleepers;
    remove_and_signal_older sleepers acc
  | Some { time; syscall; _ } when in_the_past time ->
    Sleepers.delete_min_exn sleepers;
    remove_and_signal_older sleepers (Miou.signal syscall :: acc)
  | Some _ -> acc

let rec clean sleepers uids =
  let f ({ syscall; _ } as elt) =
    if List.exists ((=) (Miou.uid syscall)) uids
    then elt.cancelled <- true in
  Sleepers.iter f sleepers

let rec minimum sleepers =
  match Sleepers.find_min sleepers with
  | None -> None
  | Some { cancelled; _ } when cancelled ->
    Sleepers.delete_min_exn sleepers;
    minimum sleepers
  | Some elt -> Some elt

let select ~block:_ uids =
  let sleepers = sleepers () in
  clean sleepers uids;
  let ts = match minimum sleepers with
    | None -> 0.0
    | Some { time; _ } ->
      let value = time -. Unix.gettimeofday () in
      Float.max 0.0 value in
  Unix.sleepf ts;
  remove_and_signal_older sleepers []

Now, we need to implement our interrupt and modify our select accordingly so that it can handle the interruption. As we explained, we will use Unix.select to both wait for the necessary time (like Unix.sleepf) and observe a specific event: whether we have been interrupted or not.

The interruption mechanism will be done using a file descriptor (because this is what we can observe). We need to transmit a signal of some sort via our interrupt function that select can handle. So we will create a pipe where interrupt writes to it, and select reads from it as soon as it has bytes available:

let consume ic =
  let buf = Bytes.create 0x100 in
  ignore (Unix.read ic buf 0 (Bytes.length buf))

let select ic ~block:_ uids =
  let sleepers = sleepers () in
  clean sleepers uids;
  let ts = match minimum sleepers with
    | None -> 0.0
    | Some { time; _ } ->
      let value = time -. Unix.gettimeofday () in
      Float.max 0.0 value in
  match Unix.select [ ic ] [] [] ts with
  | [], _, _ -> remove_and_signal_older sleepers []
  | ic :: _, _, _ ->
    consume ic;
    remove_and_signal_older sleepers []

let buf = Bytes.make 1 '\000'

let events _ =
  let ic, oc = Unix.pipe () in
  let interrupt () = ignore (Unix.write oc buf 0 (Bytes.length buf)) in
  { Miou.select= select ic; interrupt }

And there you have it! We can now allow Miou to interrupt a select in the case of cancellation. Interruption occurs quite frequently and is not limited to cancellation. Here, we are mainly interested in unblocking our Unix.select so that Miou can then handle its tasks immediately. It is also worth noting that we don't consume our pipe interruption by interruption. There may be unnecessary interruptions that we can ignore (hence reading 0x100 bytes instead of 1).

To demonstrate the validity of our implementation, we can try canceling a task in parallel that would take too long:

let () = Chat.run @@ fun () ->
  let prm = Miou.call @@ fun () -> Chat.sleep 10. in
  Miou.yield ();
  Miou.cancel prm

This code should not take 10 seconds but just the time it takes for cancellation.

Blocking indefinitely

As you can imagine, we have still omitted some details in our tutorial. Particularly, the block option. This option signals from Miou that there are no tasks left to handle, and only syscalls can unblock your application. This is typically what we expect from a system and network application: fundamentally waiting for system events as a top priority.

In this regard, if block:true, we can afford to wait indefinitely (while still considering possible interruptions) without handing control back to Miou until there is an event. To do this, Unix.select can have -1.0 as its last argument. However, in our example, this is not very relevant. But for our echo server, it's crucial that it does nothing but wait for system events. In this regard, we recommend reading the implementation of Miou_unix, which closely resembles what we have just done and handles the block option.

Conclusion

You've finally completed this little tutorial, which demonstrates what is arguably the most challenging part of Miou. It should be noted that we have mentioned other possible solutions in our interaction with the system here and there:

starting with epoll()
mentioning interruption mechanisms other than our Unix.pipe
explaining a completely different design with Stdlib.Thread that could be more efficient

In addition to these, there are subtle differences between systems (even Unix), and we realize that standardizing and homogenizing such a problem is a difficult task. What is most important to take away from this tutorial is Miou's ability to allow you to re-appropriate what is, for us, essential in the development of a system and network application: interactions with the system.

We could claim (and have the audacity to) offer solutions that would cover 90% of use cases in the development of such applications, but in our ambition to create unikernels as a service, we definitely fall into the margin (the remaining 10%). In this regard, we offer perhaps something that may be rudimentary but accessible and usable for truly all use cases.

Finally, this book is not finished. As mentioned in the previous chapter, we still need to cover Mutexes and Conditions. To do this, we will reuse our echo server and improve it!

It is worth noting that the Miou.Pqueue module comes from an extraction of an implementation of a proven priority queue using Why3 (see the vocal project).

Another solution, which would be more challenging to implement but more efficient, would be to use Stdlib.Thread to observe events. This observation would occur concurrently with our domain and transmit signals via a shared queue with our select. In this case, select would no longer be blocking at all, and we would no longer need to implement an interrupt.

Conditions & Mutexes

When it came to implementing our small scheduler and interacting with the system, the main challenge was to address the issue of suspending a function so that it could run in the background. However, it's not just syscalls that can suspend/block the execution of a function. There are also Mutexes and Conditions.

The real challenge of a scheduler is to be able to suspend functions without involving the system: in other words, to manage all suspensions. For novices, Mutexes and Conditions allow you to block and unblock the execution of a function (possibly based on a predicate).

The usefulness of such mechanisms lies in synchronizing tasks with each other. Whether they are in concurrency and/or in parallel, it is difficult, if not impossible, to know which task will execute before the others. However, we sometimes (and often) want to share information between these tasks. Miou only allows one type of information transfer between tasks: from children to their direct parents.

In all other cases (for example, between two tasks with no direct parent-child relationship and executing in parallel), we need to consider how to transfer this information correctly (meaning that this transfer would work regardless of the execution order of our tasks from both Miou's perspective — for Miou.call_cc — and the system's perspective — for Miou.call). It is in these cases that Mutexes and Conditions can be useful.

Mutexes

Mutexes allow obtaining exclusive access to manipulate information compared to other tasks. This means that we can manipulate a global resource, available to all tasks, securely using mutexes. To illustrate this example, let's revisit our echo server where we want to display incoming connections as logs:

let pp_sockaddr ppf = function
  | Unix.ADDR_UNIX v -> Format.pp_print_string ppf v
  | Unix.ADDR_INET (inet_addr, port) ->
    Format.fprintf ppf "%s:%d" (Unix.string_of_inet_addr inet_addr) port

let server () =
  let socket = Miou_unix.Ownership.tcpv4 () in
  let sockaddr = Unix.ADDR_INET (Unix.inet_addr_loopback, 3000) in
  Miou_unix.Ownership.bind_and_listen socket sockaddr;
  let orphans = Miou.orphans () in
  while true do
    clean_up orphans;
    let client, sockaddr = Miou_unix.Ownership.accept socket in
    Format.printf "new client: %a\n%!" pp_sockaddr sockaddr;
    ignore (Miou.call_cc
      ~give:[ Miou_unix.Ownership.resource clientr 
      ~orphans (fun () -> echo client))
  done;
  Miou_unix.Ownership.close socket

It may happen (and this is the difficulty of parallel programming) that an exception occurs seemingly out of nowhere if you run this code:

Fatal error: exception Stdlib.Queue.Empty

The real issue is that the Format module uses an internal queue to properly indent your output (especially according to the boxes). In our case, this queue ends up being manipulated by all our domains, and, as mentioned in the Stdlib.Queue documentation, the module is not thread-safe: the documentation explicitly mentions the use of a mutex¹.

So, we need to protect our output between domains. To do this, a simple mutex is necessary:

let mutex_out = Miou.Mutex.create ()

let printf fmt =
  let finally () = Miou.Mutex.unlock mutex_out in
  Miou.Mutex.lock mutex_out;
  Fun.protect ~finally @@ fun () ->
  Format.printf fmt

This way, we ensure that only one task executes our Format.printf and that the others must wait for the first one to finish. We say it has exclusive access to the resource.

Conditions

A major issue with our echo server is its termination. Currently, we are unable to terminate our server properly due to the infinite loop. However, we could handle a system signal that instructs all our domains to terminate gracefully. Since our main loop only accepts connections, we could implement a function accept_or_die that, upon receiving a signal such as SIGINT, initiates the process to terminate our domains.

Once again, a global resource comes into play — the signal sent by the system. We need to return a `Die value instead of waiting for a new connection. The purpose of a condition is to wait until a predicate (obtained using a global resource) becomes true. In the case of our echo server, if we receive a SIGINT signal, we return `Die; otherwise, we continue waiting for a new connection.

let condition = Miou.Condition.create ()
let mutex_sigint = Miou.Mutex.create ()

let accept_or_die fd =
  let accept () = `Accept (Miou_unix.Ownership.accept fd) in
  let or_die () =
    Miou.Mutex.protect mutex_sigint @@ fun () ->
    Miou.Condition.wait condition mutex_sigint;
    `Die in
  Miou.await_first [ Miou.call_cc accept; Miou.call_cc or_die ]
  |> function Ok value -> value | Error exn -> raise exn

let server () =
  let socket = Miou_unix.Ownership.tcpv4 () in
  let sockaddr = Unix.ADDR_INET (Unix.inet_addr_loopback, 3000) in
  Miou_unix.Ownership.bind_and_listen socket sockaddr;
  let rec go orphans =
    clean_up orphans;
    match accept_or_die socket with
    | `Die -> ()
    | `Accept (fd', sockaddr) ->
      printf "new client: %a\n%!" pp_sockaddr sockaddr;
      ignore (Miou.call_cc
        ~give:[ Miou_unix.Ownership.resource client ]
        ~orphans (fun () -> echo client));
      go orphans in
  go (Miou.orphans ())

We then need to "catch" the SIGINT signal. Signals are special in that they can execute a task outside of Miou. However, if these tasks have side effects, they won't be managed. Thus, Miou offers a way to attach functions to signals using Miou.set_signal:

let stop _signal =
  Miou.Mutex.protect mutex_sigint @@ fun () ->
  Miou.Condition.broadcast condition

let () = Miou_unix.run @@ fun () ->
  Miou.set_signal Sys.sigint (Sys.Signal_handle stop);
  let domains = Stdlib.Domain.recommended_domain_count () - 1 in
  let domains = List.init domains (Fun.const ()) in
  let prm = Miou.call_cc server in
  Miou.await prm :: Miou.parallel server domains
  |> List.iter @@ function
  | Ok () -> ()
  | Error exn -> raise exn

This simply signals all places where our condition is waiting. Consequently, all our domains are signaled to return `Die instead of continuing to wait for a new connection.

Ownership, sub-tasks and finalisers

If we try this code, it may not work, and Miou might complain with the Not_owner exception. This is because our accept task does not own the file-descritptor; we need to pass it the resource via the give parameter.

It's worth noting that this ownership is exclusive. Once we've performed Miou_unix.Ownership.accept, we need to:

transfer the file-descritptor back to the parent (so it can transfer it to the next accept).
transfer the new file-descriptor to the parent that was created in our accept task so that it can transfer it to our echo task.

The importance of finalizers in this situation should also be noted. Indeed, await_first will wait for one of the two tasks. If our condition unblocks and returns `Die, await_first will then cancel our accept task: we then finish it in an abnormal situation where our finalizers will be called on our file-descriptors. In other words, except for the active clients, all our resources have been properly released by Miou, and we no longer need to take care of them during the termination of our program.

Finally, even after these minor fixes, Miou may still return Still_has_children. Indeed, receiving a signal does not mean that we have finished all our children (we just cleaned up a few). However, we do know that:

we will not have any new children.
our echo task should terminate smoothly despite our signal.

So we need to await all our remaining children:

let rec terminate orphans =
  match Miou.care orphans with
  | None -> ()
  | Some None -> Miou.yield (); terminate orphans
  | Some (Some prm) ->
    match Miou.await prm with
    | Ok () -> ()
    | Error exn -> raise exn

The final version of `echo`

If we take all our previous comments into account, here is the final version of our echo server:

let condition = Miou.Condition.create ()
let mutex_sigint = Miou.Mutex.create ()
let mutex_out = Miou.Mutex.create ()

let printf fmt =
  let finally () = Miou.Mutex.unlock mutex_out in
  Miou.Mutex.lock mutex_out;
  Fun.protect ~finally @@ fun () ->
  Format.printf fmt

let rec echo client =
  let buf = Bytes.create 0x100 in
  let len = Miou_unix.Ownership.read client buf 0 (Bytes.length buf) in
  if len = 0 then Miou_unix.Ownership.close client
  else
    let str = Bytes.sub_string buf 0 len in
    let _ = Miou_unix.Ownership.write client str 0 len in echo client

let accept_or_die fd =
  let accept () =
    let fd', sockaddr = Miou_unix.Ownership.accept fd in
    Miou.Ownership.transfer (Miou_unix.Ownership.resource fd');
    Miou.Ownership.transfer (Miou_unix.Ownership.resource fd);
    `Accept (fd', sockaddr) in
  let or_die () =
    Miou.Mutex.protect mutex_sigint @@ fun () ->
    Miou.Condition.wait condition mutex_sigint;
    `Die in
  let give = [ Miou_unix.Ownership.resource fd ] in
  Miou.await_first [ Miou.call_cc ~give accept; Miou.call_cc or_die ]
  |> function Ok value -> value | Error exn -> raise exn

let pp_sockaddr ppf = function
  | Unix.ADDR_UNIX v -> Format.pp_print_string ppf v
  | Unix.ADDR_INET (inet_addr, port) ->
    Format.fprintf ppf "%s:%d" (Unix.string_of_inet_addr inet_addr) port

let clean_up orphans = match Miou.care orphans with
  | None | Some None -> ()
  | Some (Some prm) -> match Miou.await prm with
    | Ok () -> ()
    | Error exn -> raise exn

let rec terminate orphans =
  match Miou.care orphans with
  | None -> ()
  | Some None -> Miou.yield (); terminate orphans
  | Some (Some prm) ->
    match Miou.await prm with
    | Ok () -> ()
    | Error exn -> raise exn

let server () =
  let socket = Miou_unix.Ownership.tcpv4 () in
  let sockaddr = Unix.ADDR_INET (Unix.inet_addr_loopback, 3000) in
  Miou_unix.Ownership.bind_and_listen socket sockaddr;
  let rec go orphans =
    clean_up orphans;
    match accept_or_die socket with
    | `Die -> terminate orphans
    | `Accept (client, sockaddr) ->
      printf "new client: %a\n%!" pp_sockaddr sockaddr;
      ignore (Miou.call_cc
        ~give:[ Miou_unix.Ownership.resource client ]
        ~orphans (fun () -> echo client));
      go orphans in
  go (Miou.orphans ())

let stop _signal =
  Miou.Mutex.protect mutex_sigint @@ fun () ->
  Miou.Condition.broadcast condition

let () = Miou_unix.run @@ fun () ->
  Miou.set_signal Sys.sigint (Sys.Signal_handle stop);
  let domains = Stdlib.Domain.recommended_domain_count () - 1 in
  let domains = List.init domains (Fun.const ()) in
  let prm = Miou.call_cc server in
  Miou.await prm :: Miou.parallel server domains
  |> List.iter @@ function
  | Ok () -> ()
  | Error exn -> raise exn

You can compile it directly with ocamlfind, run it and, above all, test its load with parallel:

$ ocamlfind opt -linkpkg -package miou,miou.unix main.ml
$ ./a.out &
$ cat >echo.sh<<EOF
#!/bin/bash

send() {
  echo "Hello World" | netcat -q0 localhost 3000
}

export -f send

while true; do
  parallel send ::: $(seq 100)
done
EOF
$ chmod +x echo.sh
$ ./echo.sh

Our final command launches a myriad of clients, with 100 of them executing simultaneously. We can observe that all our domains are at work, and there are no conflicts on the console thanks to our mutex. Finally, to appreciate all our work, a SIGINT (with Ctrl+C) will terminate our server correctly and release all our file descriptors!

This little project broadly demonstrates what is possible with Miou and the insights that emerged during its development, particularly regarding system resources and I/O. We hope this tutorial has sparked your interest in using Miou in your applications. For the more adventurous, you can read our manifesto, which explains, in a more social than technical manner, the benefits of Miou.

It is worth noting that Miou offers a thread-safe queue: Miou.Queue. We use it internally for various purposes, particularly in inter-domain synchronization. However, it is essential to recognize that Miou.Queue may reveal other issues such as inter-domain contention.

Manifesto

This small manifesto aims to clarify our ambitions regarding Miou and our cooperative. Beyond the technical questions raised by a scheduler, there are also social realities to consider that justify Miou's existence in an ecosystem that can become fragmented on these issues. It is not about sharing our opinion on what might be found as a competing project to Miou or attributing statements and actions of which you alone can be the judge; it is about expressing what Miou means to us.

Protocols, services & Miou

For several years, we have been endeavoring to produce services in OCaml in the form of unikernels, as evidenced by this website. In pursuing this goal, the implementation of protocols and services in OCaml naturally comes into play within our cooperative. It is worth noting that we have implemented many protocols to date and intend to continue in this direction.

What we have learned over these years is the logical and indeed necessary separation between the protocol and the scheduler. Even if the sole application domain of implementing a protocol is I/O, we will continue to produce libraries that do not depend on Miou.

In this regard, we encourage our users to follow the same path. As mentioned in this tutorial, the scheduler is the final frontier between your application and reality: the aim is to push this question as far back as possible to ultimately make the best choice on which scheduler to use.

Furthermore, we will not impose the use of Miou in the majority of the software we develop and maintain. However, we still have a slight preference for it.

Miou Extension

For some, Miou is very, even too, minimal. It offers the bare minimum for developing system and network applications, and it can be tedious to reconsider (and reimplement) basic building blocks in the design of an application that Miou does not provide.

Once again, we do not shy away from our goal of integrating Miou into our unikernels, where this choice to restrict the application domain of our library becomes a necessity. In this regard, Miou will remain small.

However, this choice can be understood (and we hope our users can understand it this way) as our humility in not seeking to homogenize and/or standardize an ecosystem in which we believe you can find your place: a place that may involve extending or even rebuilding Miou. Thus, Miou's goal is not to be widely used.

All this is to inform our future users that extending Miou to facilitate application development can be done as third-party libraries but would not be integrated into Miou.

Collective Work

At several points in our tutorial, we mentioned that we are not necessarily omniscient about all possible techniques, for example, regarding synchronization between multiple tasks. In reality, Miou raises many challenges that can be daunting in terms of our skills and resources (we are only a small cooperative).

But we believe (and we hope to demonstrate publicly on a daily basis) in collective work and the contribution of our users to evolve Miou in a direction that can satisfy everyone. This collective work can be long, sometimes thankless (and we are the first to acknowledge it), but we would be delighted if Miou were the synthesis of a community rather than just a few people with very (too) strong opinions.

Moreover, this is already the case. Miou draws heavily from its "competitors" and builds upon the outstanding work of individuals who have pondered the same questions as us regarding scheduler implementation. It is a task that can be lengthy (both in assimilation and implementation) but can correspond, at least, to our cooperative structure.

Acknowledgments

Thus, this little manifesto concludes with thanks to all the individuals who, directly or indirectly, have contributed to the development of Miou as well as to the Robur cooperative, which has enabled the development of such a library. Hoping that our opinions and vision for the future of Miou resonate with you, we will be delighted, even if only to assist you in reclaiming the means of communication.