8 – Default 404 page not working

In a Drupal 8.5.3 installation I have set up the following configuration:

  • I created a basic page, its node path is /node/42.
  • Under Administration » Configuration » System » Basic site settings (/admin/config/system/site-information) I have set the Default 404 (not found) page to /node/42.

Now I expect a non-sensical URL to show the contents of /node/42 for the 404 page. However, I still only get the generic 404 page.

The theme we are using does not have a custom 404 template or anything like that. What other reasons could there be, that the default 404 page is not the one set in the config?