You have a wrong idea of what a sitemap is.
The sitemap is used to monitor the crawling of the site by the search engine bot. The sitemap and crawling a site are two different and independent things. Google continues to search your site independently of the sitemap. The sitemap is used to check if Google can properly crawl your site. For example, if pages are found in your sitemap and Google has not seen the page, Google may add the page to the fetch queue to include.
The opposite is not true. If a page is not found in the sitemap, Google will not remove it from the index. Why? Because Google found it by crawling the site.
What you seem to believe is that the sitemap is the absolute authority Google uses to identify which pages exist on a particular site. That's not the case. The crawl is. The sitemap only helps Google to know if You can properly crawl your site, and if not, what pages Google is missing that should be added to the fetch queue.
Your expectation that Google will stop trying to access pages because these pages are no longer in your sitemap is wrong. Sitemaps are cached and checked at regular intervals. Why? Because it is a testing process.
You have a real problem that you need to solve.
You return a 500 error for pages that were not found. This is bad. Your website should return a 404 Not Found error. The 500 error is a system error and is treated as temporary by Google. If your site returns a 404 error, Google will still try the page a few times over a period of time, until it finds that the page no longer exists. If possible, you want to output a 410 Removed error for pages that you have removed. If this is too much work or not possible, the 404 will mean the same over time.
You must fix your 500 error.