I am trying to assess which of Apache Kafka, RabbitMQ, and Apache Pulsar will meet my throughput requirements for a system with multiple producers and multiple consumers for a single topic.
How can throughput even be compared if systems can be scaled to whatever throughput is needed? In benchmark tests I saw that RabbitMQ has the worst throughput and Kafka and Pulsar are much better (but which is best I’ve read conflicting things), but I don’t understand if these rates like “30 MB/s” are fixed or per-node. It has to be per CPU or per GB or RAM or per queue or something, and I’m confused which systems can be easily scaled as needed.
I read RabbitMQ queues are bound to one node. But for Quorum queues they are replicated. I’m guessing this type of replication helps with data safety but not throughput? Is there really a fixed total throughput RabbitMQ supports, and if you need to go beyond that then you’re out of luck?
I read Kafka has high throughput from a lot of optimizations like sequential disk reads, but does this just imply a fixed higher amount of throughput or can it be scaled to infinity?
Any significant differences between Kafka and Pulsar, or is one just a fixed ratio more efficient?
Basically wondering what resource limits the total possible throughput of a single queue/topic in each system, e.g. disk, number of nodes, or whatever.