amazon web services – Using Glue to Filter/Separate a JSON file into new categorized JSON files

I am trying to separate JSON objects out of JSON files with AWS glue. I am not sure it’s the right tool for the job. I want to filter out different JSON object out of the JSON file. The Gist below has 3 example messages but each message has a unique and dynamic schema. Before I send a crawler to build a catalog I want to separate the files into unique files.

The issue I am having is that glue’s

glueContext.create_dynamic_frame.from_options(
        connection_type='s3',
        connection_options={"paths": ("s3://<bucket_name>/raw/<partition_schema>"),
                            'recurse': True},
        format="json"
    )

creates a schema right away for the JSON files found in the directory(s).

Here is my goal.

Before:

s3://<bucket_name>/raw/<partition_schema>/json_file.json

After:

s3://<bucket_name>/raw/<partition_schema>/json_file.json
s3://<bucket_name>/filtered/<partition_schema>/message_json_file.json
s3://<bucket_name>/filtered/<partition_schema>/verification_json_file.json
s3://<bucket_name>/filtered/<partition_schema>/message_format_version_json_file.json

And then the filtered files would be crawled with a crawler and placed in a data catalog

Here is an example JSON file. The file would have thousands of message entries.

https://gist.github.com/Justin-productOps/e5afa19c8de87b4d122f0ff12e923e0a