I have a 300 TB Freenas server to back up multiple Linux nodes. Backup works with daily snapshot and rsync tasks.
The user often moves large amounts of data (2 to 5 TB) between servers. As a result, large files are often backed up multiple times on multiple servers.
Online deduplication would be too expensive (1.5TB of RAM …). Is there an offline deduplication software?
I mean, the files have the same name and often the same access times – fdupes would see them as identical with minimal effort …