Unfrothantly, there is no OOTB solution to control the list view toolbar functionality other than Managing Permissions.
So, I suggest sharing the list with can view permission would limit some toolbar functionality.
Otherwise, I think you have to use custom CSS.
