First we’ll subtract the cylinder center from the ray origin, to translate our whole problem to be centered at (0, 0, 0).
Case 0: Inside
First, check if the ray origin is inside the cylinder from the start.
$$text{origin}_x^2 + text{origin}_z^2 leq text{radius}^2 quad text{and}quad
-frac {text{height}} 2 leq text{origin}_y leq frac {text{height}} 2$$
If so, it’s conventional to return no hit (we only count hits going into the cylinder). Though you could instead invert the following cases to check for a hit exiting the cylinder.
Case 1: Above / Below
If our shifted origin is above $frac {text{height}} 2$ or below $-frac {text{height}} 2$, then we check whether its direction takes it closer to the cylinder:
$$text{origin}_y times text{direction}_y < 0 ?$$
If it does not, we know the ray is too high/low to ever hit the cylinder, and we can early-out with no hit.
Otherwise, we can advance the ray to the time $t$ when it crosses the plane of the cylinder’s top (or bottom):
$$t_{text{plane}} = -frac {text{origin}_y} {text{direction}_y}$$
The ray’s position at that time is $vec p_1= vec {text{origin}} + t_text{plane} cdot vec {text{direction}}$
If that point is inside the cylinder’s radius,
$${p_1}_x^2 + {p_1}_z^2 leq text{radius}^2$$
Then we have a hit against the top/bottom cap of the cylinder, with a vertical collision normal.
If not, we can proceed as though the ray had started somewhere beside the cylinder, rather than above/below:
Case 2: Beside
Next we’ll check if the ray hits the circle. For that we want to know the earliest positive time $t$ at which…
$$||vec {text{origin}} + t cdot vec {text{direction}}|| = r\
(vec {text{origin}} + t cdot vec {text{direction}})^2 = r^2\
t^2 cdot text{direction}^2 + t cdot 2(vec {text{origin}} cdot vec {text{direction}}) + text{origin}^2 – r^2 = 0$$
You can solve this with quadratic formula. Choose the smallest positive solution, and call it $t_text{circle}$. If there is no positive real solution, return no hit.
Then, as we did with case 1, find the position at that time: $vec p_2= vec {text{origin}} + t_text{circle} cdot vec {text{direction}}$
Now we check whether that’s on our cylinder, or if it missed:
$$| {p_2}_y| < frac {text{height}} 2 ? $$
If the hit is within this height range, then we have a hit, with collision normal parallel to $vec p_2$. Otherwise, we miss the cylinder and return no hit.
If you do find a hit, add the cylinder center back to the hit position $vec p_1$ or $vec p_2$ to get the hit position in world space.
