I'm thinking about how to write programs that do asynchronous I/O.

Most large programs I see do one of:

* Use callbacks relatively directly.
* Use userspace threads(/fibers/coroutines/continuations/whatever).
* Use some language "async" functionality.

All of these are ultimately some form of callback: You run some operation
(either a primitive like "read" or higher-level application operation like "send
http request" or "get key from on-disk database" or whatever) that completes
asynchronously, and you tell it what to do when it's done, by giving it some
code pointer to jump to once the operation has completed. The details are mostly
mechanisms to make the code more palatable, but all these schemes include an
indirect jump when the operation is done.

(In many cases this ends up being a lot like a function call, in that there's
a "call" and "return", though an important distinction is that you can start
multiple asynchronous operations and wait for any/all of them to finish, etc.)

I'm curious what a system that represents its state more explicitly would be
like. I think it would effectively be what people call an "actor model" system,
with something like message queues: When you want to do an operation, you write
to a queue, and then when it's done it puts a response in your queue. Your
program's main loop would look at all components that have anything in their
queues and run them.

I think there may be a lot of benefits to having the system state and
communication between components be legible, explicit data instead of a bunch of
function pointers. But I'm not really sure.


Concretely, imagine a distributed key-value program. Maybe you'd have
"components" like "key-value core", "LSM tree", "consensus", "storage", "RPC",
"frontend server", "I/O system calls", etc.

Each of these components (other than the last one) would effectively interact
with the world using only their queues. This would make for a very clear
interface and make it easy to test, which seems very valuable for something
tricky like a consensus protocol -- I'd love to be able to run a consensus
system by giving it a list of events and getting a list of commands.

Things I'm worried about:

* A benefit of the callback system is that things aren't *required* to use
callbacks -- if they don't need to respond asynchronously, they can return
a value immediately. Imagine looking something up in a cache, and doing
an asynchronous read only if there's a cache miss. If you require every
potentially-asynchronous operation to go through a queue you might add a lot of
overhead.

* Even ignoring that, for the fast path, each request might take a whole bunch
of trips through different queues, in a way that mirrors how you've broken up
your system into components, which seems unfortunate.

* If queues are bounded then you could imagine deadlock-type issues.

* Maybe code would end up being very awkward to write like this.


I think this kind of queue system is very valuable for the external interface to
a program (e.g. io_uring). I'm not sure whether it makes sense to use internally
between components of the same program, or as something for a library to expose.

I'm curious whether there's any system written like this, or reasons to do this
or not do this!