I've created a Python process that automates part of the process of editing support tickets in Salesforce. With selenium I extract all open support tickets and save them in an archive. From this list, I'll do some things (finding duplicate account records and finding additional account information) and save the results in 2 different directories: 1 for support tickets for non-duplicate accounts and another for potential duplicate records. And for each of these directories, I continue to edit them and save the results in another directory pair.
Here's an example of my directory structure:
ticket Processor | - ticket_archive | | - all_tickets_06_17_2019.pkl | | - all_tickets_06_18_2019.pkl | | - all_tickets_06_19_2019.pkl | `- all_tickets_06_20_2019.pkl | - non_duplicates | | - approved | | | - processing_06_20_2019.pkl | | | - processing_06_21_2019.pkl | | `- processing_06_22_2019.pkl | | - failures | | | - failed_06_20_2019.pkl | | `- failed_06_22_2019.pkl | `- unprocessed | | - staged_for_approval_06_19_2019.pkl | `- staged_for_approval_06_21_2019.pkl `- potential duplicates | - potential_duplicates_06_20_2019.pkl | - potential_duplicates_06_21_2019.pkl `- Potential_duplicates_06_22_2019.pkl
I'd like to know if there are some software paradigms for managing these ever-growing directories to determine the current state and subset of the data being processed.
My previous approach is to create a class TotalTicketScanner by simply extracting all support numbers (the unique ID for each support ticket) and making a record for each directory of support numbers. I'll make sure there's no new ticket in this pipeline so I do not duplicate my efforts.
I realized that this problem is not clear, and would be glad if someone pointed me to information about such problems
Thank you very much!