rt.representation theory – Are there cases in which the Weyl group _does_ act on the flag variety/springer fiber?

In nearly every reference on the classical springer correspondence (for example Chriss/Ginzburg’s book on Complex Geometry) it is stated that the action of the Weyl Group on the homology of the springer fiber is not induced by an action of the Weyl group on the fiber itself. But I couldn’t find any reference or general statement that would make this precise (and I’m just learning about geometric representation theory so I don’t have any good intuition/working knowledge).

For concreteness (and I think this should be a simpler case), consider the springer fiber over zero — then $mathcal{B}_0 = mathcal{B} = G/B$ is the flag varitey (where $B$ is a Borel subalgebra). Now there is an action of the Weyl group $W$ on the quotient $G/T$ (where $Tsubseteq B$ is a maximal torus, which allows to write $W = N_G(T)/T$). Now in Yun’s notes, (1.5.4), he writes that the map $G/Tto G/B$ is an “affine space bundle”, that this implies that the their homologies are isomorphic and that under this isomorphism, the action of $W$ on $H(G/T)$ gives the springer action of $W$ on $H(mathcal{B})$ (I couldn’t find a reference for that though). But I don’t see why there should be no way to use this projection to define an action of $W$ on $mathcal{B}$.
Also, in “Schubert cells and cohomology of the spaces $G/P$” (1973), Bernstein Gelfand and Gelfand construct in chapter 5 a correspondence that gives rise to the springer action. I was hoping that this very explicit construction in the paper would shed some more light on what’s going on, but I haven’t been able to do that myself.

Any help, and also references, are really appreciated.