Do I need to re-register all my images in the media library after a migration?

I’m moving a 6+ years running WordPress site to a new (Azure VM / LEMP) server, taking the time to upgrade Ubuntu, Nginx, etc, and trying to clean up years worth of grunge. ~20 authors, 40k posts, 6+GB of images, etc.
I’m moving things over manually, copying the uploads folder over, so the images aren’t registered in the Media Library. I’m looking at some plugins to help, but am wondering if it’s necessary to re-register all the old images into the Media Library? They’re all properly linked in the posts, etc.
So, wondering what kind of performance gains, if any, I would get from just not bringing (most of) the images back into the Media Library? Most of these will never be used again (the ones that will I will def get into the ML). Is it worth the bother?