amazon web services – FileNotFoundError occurred but the file exists in S3


I have a set of AWS lambda functions connected to each other using AWS Step Function.

My first lambda reads a DAT file from s3 into a dataframe, processes it, and writes it as parquet to an s3 location path. It then reads the file from path I just wrote into another dataframe and calculates the count of records in it.

My second lambda reads the parquet file in path and does some other processing.

There is no Wait state between these two Tasks

So during some of the StepFunction runs, I get a FileNotFoundError error when my second lambda tries to read the file in path. I have configured the second lambda to retry twice in case of failure, first after 10s and second after 20s. Both the tries fail in such runs.

I read about S3 eventual consistency which, when we try to overwrite the file in path and
try to read it, might give the old file. When I am able to read the file in path from inside the first lambda itself, how do I get FileNotFoundError error in the second lambda function?

Am I missing something? Thanks in advance.