Selective Receive sounds great. Any idea how to handle messages that continue to pile up behind your actor when they're never handled? Do they get culled somehow after a period of time? If not, how do you handle the inherent memory leaking where every actor piles up messages that were never handled, and wastes processing time by replaying them every time you do handle a message? Works okay when you have lightweight processes that are completely independent, like in Erlang - on a monolithic process like the JVM, not so much.
Also, with the lightweight threads using CPS - how do you prevent the kernel from deciding to do some housekeeping tasks on your core? Using APIC or something in Linux to assign the JVM process exclusively to cores so you're guaranteed no interruptions? User-level hardware affinity is not exactly the JVM's strong point.
Regarding selective receive: the messages aren't replayed whenever a new message comes along. The receive operation keeps a pointer into the queue to the last message scanned. Whenever the inner receive returns, the outer receive continues to scan the queue wherever it left off. Now, in general it's a good idea to use bounded queues so messages don't pile up indefinitely. When the queue overflows, the queue's owning actor (the receiver) will get an exception.
When it gets the exception, it will either want to terminate or flush the queue.
If you're not replaying unhandled messages, you're not doing selective receive. To quote LYSEFGG, "Ignoring some messages to handle them later in the manner described above is the essence of selective receives" (http://learnyousomeerlang.com/more-on-multiprocessing). Erlang also doesn't limit mailbox size, so while it's great that you offer bound mailboxes (which is also great for performance since they can be array-based), it's not quite as flexible. And since you haven't begun to implement supervision or OTP, you'll have to handle failure as well when you break your bounds.
I'm a big fan of Martin Thompson's Mechanical Sympathy concepts, and I'm very intrigued in Peter Lawrey's work with Chronicle as well. That said, that hardware affinity library relies on native C code, and you better know what you're doing when you put it in. The topology of the CPUs and locality mean you have to be smart in your assignments, lest you end up message passing via QPI/Hypertransport between sockets at a latency of ~20ns/message. Point being, either be intimately familiar with your box and reconfigure for each kind on which you deploy, don't ever use a hypervisor, or pin and pray.
Are you able to introduce bulkheads and failure zones with your lightweight threads via CPS? If not, isolation of dangerous tasks on a thread that could impact other actors could be an issue. Akka does this by allowing you to specify what thread pool (preferrably forkjoin-based) you want to use for each actor.
Look, this is neat stuff you're doing. I'm not concerned that you don't like Scala, but Akka can be used from Java as well so that's a non-argument. It's merely another approach. And while you certainly CAN block in an Akka application, there are plenty of tools for asynchronous coding in Scala (Futures, Async) to help you avoid that and only block when you absolutely must.
The skipped messages will be replayed in the "outer" receive. Obviously, selective receive has its drawbacks, but it's part of what makes Erlang simple, and it can significantly help in modeling complex state transitions.
And yes, you can assign a fiber to a ForkJoinPool of your choosing (although I'm interested in what a "dangerous task" may be).
I agree that using selective receive helps in dealing with messages that arrive out of the order of a specific state transition. Akka gives users the ability to stash messages if they want to. On the JVM, a long-running actor-based application (which is one of the reasons for using actors in the first place) can struggle with it. It's one of the reasons the original Scala Actor library is no longer in use, though there are other important reasons - such as Akka's use of ActorRef, analogous to Erlang's PIDs, which mask the instance of an actor from those who wish to communicate with it, as well as it's physical location. As you scale actors across a cluster of machines, that becomes really useful.
That's great about assigning the fiber to a FJP. A dangerous task would be anything that could take down an actor, which can be worrisome depending on what state the actor is holding. There are varying kinds of such state, including that which can easily be retrieved again from an external source, that which is "scratch" data and inconsequential if lost, and that which cannot be recovered if lost. In actor-based applications, we want to encapsulate mutable state and use message-handling single-threaded interaction to prevent concurrency issues, right? If we're going to do something that could cause the actor to fail and risk losing data, we want to export that work along with the data needed to perform the task to another actor and let IT fail rather than risk the important one. There are ways to pass such data between incarnations of an actor on the JVM by carrying it with the Exception, but it's not free and you have to know when to use it.
So a dangerous task could be asking for data across an unreliable network or non-replicated source, it could be dividing by 0, anything that could cause typing errors (even in Erlang), you name it.
But how would a dangerous task affect the entire pool?
Also, I don't know if there should even be "important actors". Like in Erlang, we want to let it fail. Important data should be kept in a shared data structure that supports good concurrency, not in actor state. Like I said in the post, I don't think every aspect of the application should be modeled with actors.
With a thread pool shared by actors, even yours, if one of the actors fails, that thread is gone until the pool creates a new one (as needed). That's one less available thread until the recreation occurs. To minimize the impact on other actors, you put known dangerous tasks on their own small pool so that their probable thread death has no impact on others.
What you're not seeing is the relevance of a supervisor hierarchy and OTP. Yes, you want to let it crash. But you want isolation of failure as well, and only with failures of increasing criticality do you want it to be escalated through the hierarchy. There is a difference between a database call failing because a SQL command failed and all database interactions failing due to network partition. OTP via Erlang and Akka allow you to model that in your actor tree.
Important data kept in a shared data structure? Globally visible? Managed with what, STM? That won't fly at scale - STM is great only when you're dealing with small datasets that don't change very often. Immutable, persistent data structures? Also not good at scale due to fixed size allocations constantly happening as structural sharing is enforced. Allocations are cheap and hopefully the objects are short-lived and GC-friendly, but it's still far from a free ride.
The whole point of actors is a concurrency abstraction. They are meant for isolating mutable state from being affected by multiple threads at the same time. OTP and supervision is just a nice way of organizing them and learning when failure occurs on another thread.
Should an entire app be modeled with actors? Probably not. A subsystem certainly can be. Depends on what you're doing, of course.
Akka does that as well, and you're right, it's very important for writing declarative, distributed logic. Akka's ActorRef abstracts over the physical location of the actor, just as Erlang's PIDs do.
That's not entirely true. Erlang OTP allows for selective receive, and will stash unhandled messages to the side. When a received message is handled, all unhandled messages are retried because the actor may have changed what messages it can handle as a result. This is fine in Erlang - if the actor process crashes for exceeding allocated memory, it has no impact on other actors.
The original Scala Actors did this as well, with predictable results. Scala runs on the JVM, which is a monolithic process. The usage of selective receive meant that over time, actors would accumulate enough unhandled messages to result in a possible OutOfMemoryError, from which there is no recovery on the JVM. Akka does not do selective receive, so it does not entirely follow OTP.
That said, it is inspired by OTP, as evidenced by the original name of the project, Scala OTP. It's just optimized for the JVM.
Also, with the lightweight threads using CPS - how do you prevent the kernel from deciding to do some housekeeping tasks on your core? Using APIC or something in Linux to assign the JVM process exclusively to cores so you're guaranteed no interruptions? User-level hardware affinity is not exactly the JVM's strong point.