(Riemannian geometry) differentiability of distance function at first conjugate point

Formulation of question: Consider a complete, simply connected Riemannian manifold $M$ with Riemannian metric $d$. For $x,y in M$ that are distinct but close enough to each other, there is a geodesic $gamma(t), 0 le t le 1$ such that $gamma(0)=x$ and $gamma(1)=y$. Assume that $gamma$ is the unique minimizing geodesic joining $x$ and $y$ and that $y$ is the first conjugate point of $x$ along $gamma$. Further, assume that $y$ has a suitable normal neighborhood that is contained in $M$. Is $d(x,y)$ differentiable at $y$ when $x$ is fixed?

Background: We know that cut loci carry important information on the topology of $M$ and information on how the distance function $d(x,cdot)$ behaves at a cut point also reveals important information on $M$. Further, a lot of techniques in Riemannian geometry utilize the differentiability of $d(x,cdot)$, and in statistics, uniqueness of Frechet mean, i.e., $$arg min_{y in M} int_{M} d^2(x,y) mu(dx)$$ for a probability measure $mu$ on $M$, depends critically on the cut loci of $d(x,cdot)$.

Related posts and results: there is a nice post here smoothness of $d(cdot,cdot)$ that discusses differentiability of $d$ on $M times M$, and a classic result on page 108, i.e., Proposition 4.8, in Takashi Sakai’s classic book “Riemannian Geometry”, that says “Suppose that there exist at least two normal minimal geodesics joining” $x$ and $y$, then $d(x,cdot)$ is not differentiable at $y$. These discussions confirm the differentiability of $d(x,y)$ when $y$ is not a cut point to $x$ when $x$ is fixed.

Recall that a cut point $z$ of $x=gamma(0)$ is either a first conjugate point to $x$ or there are at least two minimal geodesics joining $x$ and $z$. Recall also that from Theorem 2.1.12 on page 133 of W.A. Klingenberg’s book “Riemannian Geometry”, within any neighborhood of the first conjugate point $y$, there exists a point $w$ such that there are at least two distinct geodesics that connects $x$ and $w$. So, if $d(x,y)$ with fixed $x$ is indeed differentiable at $y$, then we also know that $d(x,cdot)$ is not differentiable in any neighborhood of $y$ except at $y$ itself.

However, none of the above discusses whether $d(x,y)$ is differentiable at $y$ in the setting of the question stated at the beginning. Is the question trivial that its answer is “No”? If the question is so easy, I must have missed something and would like to be enlightened by someone who knows the subject better. Thank you for your attention to my question.