My task is to create new system (X) for realtime work tracking for internal use. You should think it like an app that you have opened on a separate monitor/ on phone all day and you are logging your work progress there. In fact, there is already a system (Y) that solves this and internally its working well, but it’s really annoying to use because of slowness, unresponsiveness, and terrible UX design. The legacy system cannot be replaced at this time (a lot of reasons) and data from app new must be synchronized with it. The legacy system Y is also integrated with a lot of other systems and plan is to take over these integrations and rebuild them to system X. There is intent to completely replace the legacy system with a new one in the future – not only because of unresponsiveness but also because of monster technical debt and obsolete tech stack.
I proposed to create new system X as a standalone software with its own database and integrate it with Y using its API. The problem is – API calls (http requests) to Y are slow (up to 10 seconds). System X should have to be very responsive, all requests have to be processed in < 1s. So my plan is to push events from X to some kind of queue and sent them to Y asynchronously. Additionally, it’s important to process responses from system Y in system X (example below) but it can be delayed. Also, other systems can eventually consume the same events from the queue.
Example of communication
User -> X : Create job request X -> User : Job(id_X=1,id_Y=null) created and Event1 pushed to queue Y : Starting processing event1 User -> X : Add comment to Job(id_X=1,id_Y=null) X -> User : Comment(id_X=2,id_Y=null) created Event2 and pushed to queue Y : Starting processing event2 Y -> X : Event1 processed, returning id_Y=8876 for job with id_x=1 X : Update job (id_X=1,id_Y=8876) internally from resnponse X -> User : Display updated job info
.. and so on.
- Is my proposed communication pattern meaningful or are there any design flaws?
- Should be queue functionality part of the system X or is it better to make it standalone?
- Can you suggest any existing software for the queue?
Thank you in advance for any advice