bitcoin core – Regtest at height 120k took 600ms to generate a block

Just doing

while :
 time bitcoin-cli generatetoaddress 1 <non_local_wallet_address>
 bitcoin-cli getblockcount

brings regtest blockgen times up to 600ms when generating a block at the height 120k, even with all empty blocks.

I have noted down the following numbers:

height, gen-time
800 -> 7ms
5500 -> 18ms
11000 -> 31ms
22000 -> 65ms
37000 -> 110ms

and so on.

This makes my test suite slowing down a lot, over time.
Is this something we can deal with?