Without miss, no problem. With a miss that can be filled from L2 or L3 cache, no problem. High end CPUs can have two paths to separate RAM chips, so they could run simultaneously. CPU and external hardware or multiple CPUs accessing the same RAM chip someone would have to wait.
If hardware designers determine that this causes a slowdown, it could be possible that RAM could be designed to have multiple accesses queued up to run faster.