unity – Player rotation recalculation causes jittering on edges

When my player walks around a circular planet everything works well, but when my planet is a shape like a cube or a dodecahedron, even with smoothed edges, my character jitters if I walk slowly on the edge.

That is because I raycast downward and set my characters rotation to be perpendicular with the normal. If im on the edge, the ray hits a different side of the shape than the one I stand on, causing my player to rotate to that new side, even if the angle between the face I stand on and the one I hit is 179°. So when My character rotates, the face I hit with the ray becomes the one I was standing on before, causing me to rotate back and forth between 2 faces.

I tried using SphereCast instead of Raycast, but it didn’t work. How can I limit the back and forth rotations ?