I have an S3 bucket with a bunch of top-level prefixes. I have a relatively inefficient way of summing the bytes over the top-level prefixes:
- Call the list of prefixes above
aws s3 ls [bucket name]followed by some sed / grep.
- Make a bash loop for these prefixes
aws s3api list-objects --bucket [bucket name] --prefix $ prefix --output json --query [sum(Contents.Size)],
This feels suboptimal to me for all the usual reasons that for-loops are sub-optimal. For one, I want it to be automatically parallelized on the AWS side.
I feel there has to be a JMESPath way to essentially return a list of (prefix, sum (size)) pairs across all top-level prefixes. But I can not figure out how to do it. Add
- delimiter & # 39; / & # 39; returns only the CommonPrefixes; There is no other information (including size) back so I could not use it to get the amount I wanted.
Is there a trick I miss? Or is the for loop the best option I have?