[prog.c++] A follow-up for basiliscos's article "Request Response Message Exchange Pattern"

Ivan Baidakou, the author of rotor actor framework for C++, wrote a blog post with a comparison of request-reply pattern support in different C++ actor frameworks: "Request Response Message Exchange Pattern". I've found the reference to that article on Reddit and started to write a comment but decided to write my own blog post because the answer started to grow rapidly.

I want to tell thanks to Ivan for his work. However, the topic of request-reply pattern is not as easy as it can look so I have to add some points to the table.

The first thing is the difference between async and sync request-reply approaches.

If we speak about async request-reply then the presence of some kind of Async Completion Token of ACT is vital. ACT can have different forms. It can be a simple request-id as in your example or it can be some complex struct or an opaque data type that structure is not known to anyone except request's issuer. But ACT should be an integral part of request and response messages. So for your example of async ping-pong interaction, the presence of `wrapped_ping` and `wrapped_pong` structures looks rather artificial. The `request_id` should be a member of `ping` and `pong` structures in the case of async message exchange.

Async request-reply processing in SObjectizer has another very important aspect: the presence of states in an agent. SObjectizer's agent is a state machine with an explicit declaration of states. When an agent has several states the reaction for the reply of timeout signal should be defined in different states (maybe differently). It makes handling of replies a rather complex task.

Because of that so5extra add-on already has async_op submodule with ready-to-use tools that significantly simplify that task. It worth to mention that this functionality was added to so5extra as a result of SObjectizer usage in different conditions.

The code of Ivan's example rewrote with the usage of async_op::time_limited can be seen here. I think that my version of `pinger` agent based on async_op::time_limited functionality is simpler and more robust:

class pinger final : public so_5::agent_t {    so_5::mbox_t ponger_;    request_id_t last_request = 0;    void on_pong(mhood_t<wrapped_pong> cmd) {       on_pong_delivery(*cmd);    }    void on_pong_delivery(const wrapped_pong &cmd) {       bool success = cmd.error_code == request_error_t::SUCCESS;       auto request_id = cmd.request_id;       std::cout << "pinger::on_pong " << request_id << ", success: " << success << "\n";       so_deregister_agent_coop_normally();    }    void on_timeout(mhood_t<timeout_signal> timedout) {       wrapped_pong cmd{timedout->request_id, request_error_t::TIMEOUT, std::nullopt};       on_pong_delivery(cmd);    }   public:    pinger(context_t ctx) : so_5::agent_t{std::move(ctx)} {}    void set_ponger(const so_5::mbox_t mbox) { ponger_ = mbox; }    void so_evt_start() override {       auto request_id = ++last_request;       auto cp = so_5::extra::async_op::time_limited::make<timeout_signal>(*this)          .completed_on(*this, so_default_state(), &pinger::on_pong)          .default_timeout_handler(&pinger::on_timeout)          .activate(200ms, request_id);       try {          so_5::send<wrapped_ping>(ponger_, request_id, ping{});       }       catch(...) {          cp.cancel();          throw;       }    } };

Another aspect is the possibility to receive messages of different types as the reply. For example, an issuer sends `process_request` message and expects back either `processing_result` or `processing_rejected` messages. And that aspect is also supported by so5extra's async_op.

So I have to say that SObjectizer already has rather powerful and robust support for asynchronous request-response interaction between actors. Maybe this support is not friendly for novices and requires a careful studying, but this is a consequence of the powerfulness of SObjectizer and the experience of SObjectizer's usage in different conditions.

The second thing is the applicability of sync request-reply pattern for actors.

From my point of view, the main advantage of sync request-reply over async one is the simplification of request-issuer code. In the case of synchronous interaction, we can write simple sequential code and the issuer can simply wait for the arrival of the reply right in the place where a request was sent:

void some_actor::some_event() {    ... // Some actions.    // Here we have to ask something on different actor and    // wait for the reply.    auto reply = make_sync_request(dest, ...);    ... // Some more actions with the reply. }

But the main question here is: what happened with the worker thread on that request's issuer works?

In principle that thread should be blocked. But it is not good because blocked thread can't handle other agents.

Unfortunately, if actors are represented as objects with callbacks we have no other possibilities except blocking the worker thread. In the case when actors are represented by stackful coroutines we can suspend coroutine with blocked actor and reschedule another actor on the current work thread. But no one of compared frameworks (CAF, SObjectizer, rotor) doesn't support actors in form of stackful coroutines yet AFAIK.

So the sync request-reply interaction can be used safely only if request's issuer and handler work on the different worker threads. And even in that case, there is a possibility of deadlocks. Because of that, I think that the usefulness of sync request-reply interaction of actors is greatly exaggerated.

Anyway, SObjectizer-5.6 still supports sync request-reply pattern via so5extra's sync submodule.

This support is not based on `std::future` anymore. Now a more complex and powerful mechanics is used inside so5extra's sync submodule. That mechanics allows doing several very important things like redirection of request processing to another actor or delaying request processing for some time, or redirecting the reply to a specific destination (like mbox or mchain) and handling replies from several requests in the first-received/first-handled manner:

using first_dialog = so_5::extra::sync::request_reply_t<first_request, first_reply>; using second_dialog = so_5::extra::sync::request_reply_t<second_request, second_reply>; // The single mchain for both replies. auto reply_ch = create_mchain(env); // Issuing requests. Please note the usage of do_not_close_reply_chain. first_dialog::initiate_with_custom_reply_to(    service, reply_ch, so_5::extra::sync::do_not_close_reply_chain,    ...); second_dialog::initiate_with_custom_reply_to(    service, reply_ch, so_5::extra::sync::do_not_close_reply_chain,    ...); // Waiting and handling of replies. receive(from(reply_ch).handle_n(2).empty_timeout(15s),    [](typename first_dialog::reply_mhood_t cmd) {...},    [](typename second_dialog::reply_mhood_t cmd) {...});

The third thing is the value of the composability of request-reply handlers.

If we speak about the cases when sync request-reply pattern is actually needed for a particular case (and sometimes it really is) the key point is the suspension of issuer until the reply arrives (or the timeout expires). I simply don't see any value in the composability of reply-handlers in the case of synchronous interaction. Will you write the code like that:

void some_actor::some_event() {    ... // Some actions.    // Here we have to ask something on different actor and    // wait for the reply.    try {       auto reply = make_sync_request(dest, ...);       ... // Some more actions with the reply.    }    catch(const some_error & x) {       ... // Some error handling.    } }

or:

void some_actor::some_event() {    ... // Some actions.    // Here we have to ask something on different actor and    // wait for the reply.    make_sync_request(dest, ...)       .on([](some_reply & r) {          ... // Some more actions with the reply.       })       .on_error([](const some_error & x) {          ... // Some error handling.       }); }

it's just a matter of a personal taste.

Maybe there is some value in the case of async request-reply interaction. But I personally don't see it. I hope someone can point me to any real-life examples where such composability is really useful. Anyway, I can suppose that such composability can be obtained via so5extra's async_op submodule mentioned above:

class pinger final : public so_5::agent_t {    so_5::mbox_t ponger_;    std::size_t pings_left;    std::size_t pings_success = 0;    std::size_t pings_error = 0;    void handle_results() {       if (pings_left == 0) {          std::cout << "success: " << pings_success << ", errors: " << pings_error                    << "\n";          so_deregister_agent_coop_normally();       }       else          next_ping();    }    void record_success() {       ++pings_success;       --pings_left;       handle_results();    }    void record_fail() {       ++pings_error;       --pings_left;       handle_results();    }    void next_ping() {       auto cp = so_5::extra::async_op::time_limited::make<timeout>(*this)          .completed_on(*this, so_default_state(),                [this](mhood_t<pong>) { record_success(); })          .default_timeout_handler(                [this](mhood_t<timeout>) { record_fail(); })          .activate(200ms);       try {          so_5::send<ping>(ponger_);       }       catch(...) {          cp.cancel();          throw;       }    }   public:    pinger(context_t ctx, std::size_t pings)       : so_5::agent_t{std::move(ctx)}       , pings_left{pings}    {}    void set_ponger(const so_5::mbox_t mbox) { ponger_ = mbox; }    void so_evt_start() override {       next_ping();    } };

The full code can be found here.

And the fourth point is the artificial separation of SObjectizer and so5extra in the original comparison.

so5extra can and should be seen as a part of SObjectizer functionality. So if something is present in so5extra then is it present in SObjectizer because it can be used in a SObjectizer-based project just out of the box.

If the comparison between frameworks is made on a technical basis -- features compared to features -- then the separation between so5extra and SObjectizer's core because of different licenses doesn't look fair.

If licenses are taken into the account then other non-technical aspects should be taken into the account too. Something like maturity and availability of commercial support. But in that case, we'll lose the sense of the comparison of frameworks' features.

So I think that speaking about SObjectizer's abilities without regarding the functionality of so5extra looks like an intentionally reducing of SObjectizer's powerfulness.

Instead of conclusion

It seems that SObjectizer is a rather complex and feature-rich framework despite the fact that it isn't a big one. So if you have any questions about SObjectizer's functionality or applicability of SObjectizer for specific conditions feel free to ask me either via issues section on GitHub, Google's Group or in comments here.