I’m designing a distributed queue message queue and I’m not sure how to get the leader information to another component. This is my current design:
I have the following components:
- FrontEnd: request validation, authentication, SSL offloading, server-side encryption, etc
- Metadata Service: keeps queue metadata information
- BackEnd Nodes: store messages. For each queue, we’ll have one leader and several followers for replication
- Node Manager (for instance Zookeeper or etcd): takes care of leader/follower coordination in the BackEnd cluster
The picture shows the flow to handle a client request. The FrontEnd does the initial request processing and queries the Metadata Service to find out the leader for the given queue, and forwards the message to the corresponding leader.
My problems are with the Metadata Service. How should it fetch queue leader information? Directly querying/connecting to the Node Manager or should I introduce anoter component instead? Or should I get rid of the Metadata Service altogether and have the FrontEnd directly querying the Node Manager? The problem with the last option is that I still would need somewhere to store queue metadata information like creation date.