I have a set of AWS lambda functions connected to each other using AWS Step Function.
My first lambda reads a DAT file from s3 into a dataframe, processes it, and writes it as parquet to an s3 location
path. It then reads the file from
path I just wrote into another dataframe and calculates the count of records in it.
My second lambda reads the parquet file in
path and does some other processing.
There is no
Wait state between these two
So during some of the StepFunction runs, I get a
FileNotFoundError error when my second lambda tries to read the file in
path. I have configured the second lambda to retry twice in case of failure, first after 10s and second after 20s. Both the tries fail in such runs.
I read about S3 eventual consistency which, when we try to overwrite the file in
try to read it, might give the old file. When I am able to read the file in
path from inside the first lambda itself, how do I get
FileNotFoundError error in the second lambda function?
Am I missing something? Thanks in advance.