unity – Character rotation jitters when looking around

This line extracts just 2 imaginary components out of a 4-dimensional quaternion:

Vector3 rotation = new Vector3( 
                          Quaternion.LookRotation(hit.normal).x,
                          0, 
                          Quaternion.LookRotation(hit.normal).z
                   );

In general, it is not safe to extract individual components of a quaternion like this. They’re not angles like Euler angle triplets. They’re individual components of a point on a unit sphere in 4 dimensions, so they don’t really make sense without all 4 components.

Each quaternion component ranges from -1 to 1, so it’s not really sensible to use as an angle in degrees here:

transform.eulerAngles = new Vector3(rotation.x,transform.eulerAngles.y,rotation.z);

It looks like instead you want to do something like this – using a trick I’ve covered in several past answers:

// Helper function to make an orientation whose local y+ direction matches "exactUp"
// and whose local z+ direction points as close as possible to "approximateForward". 
Quaternion TurretLookRotation(Vector3 approximateForward, Vector3 exactUp)
{
    Quaternion zToUp = Quaternion.LookRotation(exactUp, -approximateForward);
    Quaternion yToZ = Quaternion.Euler(90, 0, 0);

    return zToUp * yToZ;
}

// ...

// Start your raycast below your position relative to your current orientation.
Vector3 rayStart = Transform.TransformPoint(Vector3.down);
if (Physics.Raycast(rayStart, -transform.up, out RaycastHit hit, Mathf.Infinity))
{
    // Match our vertical orientation to the surface under us, while
    // preserving our forward facing direction as best that allows.
    Quaternion matchSurface = TurretLookRotation(transform.forward, hit.normal);

    // Compute the rotation about our vertical axis according to mouse movement.
    Quaternion twist = Quaternion.Euler(0, Input.GetAxis("MouseX") * sensitivityX, 0);
    
    // Compose these two rotations to get our resulting orientation.
    transform.rotation = matchSurface * twist;
}