Fast 404 throws page not found for sitemap

I am using fast_404 module. My site uses simple_sitemap module to generate sitemaps.
I have three sitemaps generated: sitemap.xml, /categories/sitemap.xml and /brands/sitemap.xml
The first sitemap is working as expected but the categories and brands sitemaps throws 404 error (fast_404 html page) when the fast_404 is enabled. Is there any way to prevent 404 on those two sitemap URLs?