I have an application that is receiving a live stream of files, I am using active MQ for listening to the file arrival in s3. As soon as the file arrives in s3 my application downloads the file and starts processing for it.
Now, I require to pause the processing for certain types of files (*Xyz*.txt, pause window is 8 AM UTC to 9 AM UTC) and resume processing whenever the pause window is over. I know the pause time window for each file. For instance, If a file with the pattern as *Xyz*.csv is received between 8 AM UTC to 9 AM UTC then the application will not process that file during the pause window, but after 9 AM UTC, it will start processing that file.
What are the ways through which I can resume file processing? I was exploring the Kubernetes cronjob type application, which will poll my core application using some API in a 5 min window and if any files pause window is over it will pick that file and start processing it.
The one major problem I see with this approach is the cronjob application is polling every 5 min, which is making several calls to my core application.
Is there any better way to solve this problem?
Please pardon me if my question is very naive.