seo – pages with irrelevant content on a sitemap bad signals to search engines?

I have many WordPress sites with content that automatically generates a sitemap.xml file. By default, the sitemap includes pages such as "categories" or "author" that are either empty or of irrelevant content.

Is it harmful to allow these pages as part of the sitemap.xml file? Is this a bad signal for Google?

Removing them from the sitemap is possible, but requires some work and it wants to know if it's worth the effort or not.

I guess this looks like a stupid question, but in fact I've done some research and it seems that sending a sitemap does not mean that all pages are indexed. Google will decide which one is best, and the rest will be ignored. But I could not tell if ignoring is harmless or a bad signal.

Thank you, Mihai