design – How to handle pagination in a stateless application having multiple components involved for the data?

It seems like the only problem is the slowness but its not clear why it is slow.

You have two possible approaches

Approach A, get all the data, but faster. Do this if the volume of data isn’t a problem, its just looping through all the calls which slows things down.

  1. Can you turn off pagination in the sub services?
  2. Can you requests all the pages at the same time asynchronously?
  3. Can you cache stuff?

Approach B, stream the data to the UI in smaller chunks. Do this if the data volume is a problem

  • get (n) items
  • get its children/extra data
  • send to UI
  • update UI
  • repeat

You will probably want a combination of approaches, it sounds from your question like you are processing the data synchronously, its a simple step to stop doing that and request pages 1,2,3,….. etc at the same time rather than processing page 1 before moving to page 2 but this will likely overload your services unless you have some sort of rate limiting.

Similarly taking approach B can lead to very synchronous code.

Your code needs to be aware of the rate limit at which the sub services can provide data.

If they have capacity, request the next n items and start work on them. Even if you haven’t sent the first ten yet.

If they don’t have capacity, don’t ask them for out of order data which you wont be able to send because the first chunk isn’t finished.

Avoid waiting for that one last child item before requesting the next ten items but also avoid requesting the next 1000 items when you haven’t finished with the first 10 yet!

Obviously you also have Approach C, request everything at once, send the main items without child items and then update the UI with the children later.

I wouldn’t normally go for this approach as presumably the UI needs that extra info to perform its function. Showing the parent and later showing the children is just a different kind of spinner