IPv6 iptables/nftables multi-wan NETMAP can’t access all hosts from outside

I have an ip6tables router with 3 WANs, each supporting IPv6. For some reason incoming connections only work for the default route, not ISPs #2 and #3. I have policy routing set up which works great for outgoing connections:

# ip -6 rule
209:    from all fwmark 0x70 lookup NW
210:    from all fwmark 0x6f lookup SPC
211:    from all fwmark 0x6e lookup VZ

But accessing the translated IPv6 IPs for internal hosts only work when I change the default ipv6 route on the router. What am I missing?

It’s as if I need some kind of connection tracking but I am not sure how to set that up?