Does anyone know what the problem is that when trying to pipe to write stream multiple large files and promise does not resolve
I have this method which returns a promise, I create multiple of these and use Promise.all to resolve them all. It works fine with a few files but when I am trying to write over 900 files and some of them are about 200mb in size it get stuck and the promise all never completes.
I can see that it creates all the files but then suddenly it just gets stuck and I can see that the file size are not increasing anymore
These are csv files and I removed piping to the csv2json package but it did not help
Any ideas what the problem might be ?
async writeFile(pathToFile: string, inputStream: ReadStream, count: number): Promise<void> {
return new Promise((resolve, reject) => {
const wStream = fs.createWriteStream(pathToFile);
inputStream
/*.pipe(csv2json({
separator: ',',
dynamicTyping: true
}))*/
.on('end', () => {
console.log('file read:', pathToFile);
})
.on('close', () => {
console.log('file closed:', pathToFile);
})
.on('error', (err) => {
console.log('read err: ', err);
inputStream.destroy();
reject(err);
})
.pipe(wStream)
.on('close', () => {
console.log('file written:', pathToFile);
})
.on('finish', () => {
resolve();
})
.on('error', (err) => {
console.log('write err: ', err);
wStream.close();
reject(err);
});
});
}
-Jani