Here is the thing, we are going to select a database to store about 5-10 GB of text files each. The content will be important later because we need to search through the whole lines and search for a specific string.
Sample data would be:
CTCTGTTACGCCTTCAGGAAGCGGAGGAAGCTCGGAGGGGGAAGTCCGGCCTCGATCTCTTCTTCTTCTC GCATCCTTGTAGAAAGAACTTTAGAAAATGGCAGATGCCGAGGATATCCAACCTCTTGTCTGTGACAACG GTACCGGAATGGTCAAGGCTGGATTTGCTGGTGATGATGCACCGAGGGCAGTATTTCCCAGTATTGTAGG CCGACCTCGTCACACGGGTGTCATGGTTGGCATGGGCCAAAAGGATGCATATGTTGGTGATGAAGCCCAG
Consider these lines with millions of them.
Is it possible to use NoSQL databases to insert a 10GB of a text file?
How can we approach the search feature for later?
We will definitely reach up to 50-100TB of data in the near future so we need to have scalability too.