This is a script that I’ve written for personal use, and to educate myself. The script is a bit comment heavy to help future me remember all the details & design choices made at the time of writing.
The use case is simply to synchronize/copy files to an AWS S3 bucket using AWS CLI. There’s a separate CDK part of the project, which sets up the AWS infrastructure, but that isn’t really relevant here. There are some configuration items in a properties file, which are read, and then the script checks whether everything is in place on the AWS end, and if so, it reads through a config folder structure which folders to backup, and how (include & exclude patterns in respective files).
Going with Bash instead of a basic shell script was a deliberate choice, since this wouldn’t be run on any production server, and extreme portability wasn’t the main point here.
Folder structure of the overall project is:
- aws-infra -- (various things here, that are out of the scope of the question) - config -- backup --- Documents ---- includes.txt --- Pictures ---- includes.txt (example at the end) ---- excludes.txt (example at the end) --- (more files/folders following the same structure) -- configuration.properties - scripts -- sync.sh
Theoretically I could’ve just run
aws s3 sync on the base path, but since it’s a recursive command, and there are a lot (about 500k) of unnecessary files, it would take a lot of time to go through each of them separately.
Properties file (SSM parameter name censored):
# AWS profile to be used aws_profile=personal # Bucket to sync files to bucket_parameter_name=(my_ssm_parameter_name) # Config folder where backup folders & files are found. backup_folder=backup/ # Names of the files defining the include & exclude patterns for each folder. include_file_name=includes.txt exclude_file_name=excludes.txt
Example include file:
Example exclude file to pair with above:
The script works, but I’m interested on how to improve it. For instance, the error handling feels a bit clumsy.