The download button is actually rendered from a twig template found at
/views_data_export/templates/export-icon.html.twig. This means that you can override it in your theme by copying that file to
/themes/YOUR_THEME/templates/export-icon.html.twig and modifying it to what you need.
You can of course put that twig file into any subdir of
/templates/ that makes sense to you – e.g.
/templates/views_data_export/export-icon.html.twig will work as well.