I have a project where there’s an external API which implements throttling. Roughly speaking, I’m allowed to perform
N requests per minute. I also have a message queue (Apache Kafka) whose consumers consume API requests: a consumer receives an API URL to call, calls it, then produces another message which is processed by another consumer (some business logic which updates internal databases).
I’m wondering if it’s possible to enforce
N API requests restrictions using the message queue alone. As far as I understand it’s impossible or contrary to the main purpose of message queues.
So far I’ve come up with the following alternative:
business logic which wants some API calls to be performed pushes the calls on a queue data structure (data structure which supports
dequeue/popmethods, for example Redis lists offers such semantics).
A job runs every minute and pops
NAPI calls from the queue and assigns the requests to a consumer which performs the calls concurrently and produces messages with the results of the calls, which are then processed by consumers. If any call needs to be retried it’s pushed back on the queue.
This way it’s easy to enforce
N requests per minute but this introduces certain complexity apart from the message queue itself.
Is my solution good architecture?