C# design for handling a high rate of async network operations that complete with a callback (hundreds per second)

I am working with a message broker technology, to which events will be published (following an “event carried state transfer” architecture) for consumption by other applications.

The vendor library uses a pattern where a message is handed to a Session object defined by the vendor’s client library code, which sends it over the network to the broker, and the vendor’s client code will call an OnSessionEvent method which includes data in the event args indicating whether the message was successfully published to the broker. A message may be initially accepted by the Session object but fail to be published to the broker if, for example, the broker’s buffer is full (typically a temporary state of affairs).

It could easily be the case that the original source of events is raising them at the rate of up-to-thousands of messages a second.

To further complicate matters, it may be the case that multiple different event sources may be publishing to the same Session, and so the OnSessionEvent response needs to be routed back to the appropriate publisher.

In any case, my struggle right now is trying to figure out an appropriate pattern to efficiently send messages and handle the callbacks. It would of course be less than ideal to send a single message and then wait for the callback result before sending the next message, since the network response may take several milliseconds.

I could create a Task for each message send attempt, collect up a bunch of these tasks, and wait on them as a batch of, say, 100. This is clearly faster than one-message-at-a-time. However it would mean generating hundreds or possibly up to thousands of Tasks every second. Note that the vendor code does not natively expose the network operation as an async operation using a Task. In order to use this pattern, I would add a TaskCompletionSource object to the (local) message object, and SetResult on that TaskCompletionSource when the (local) message object is made available via the callback argument. I am concerned about the rate of object construction that this could cause.

I am hoping for advice or articles which talk about situations like this. I am also curious as to the threading implications of asynchronous callbacks, so a comment or article that covers both would be idea.