kubernetes – istioctl kiali is not creating, how to fix that?

From the getting started link, I was able to create a gateway and the routing to bookinfo app provided in sample.

It got deployed and able to access the app from ingress gateway of istio.

The next step, deployed kiali using the addons.

kubectl apply -f samples/addons

 kubectl get pods -n istio-system
NAME                                    READY   STATUS              RESTARTS   AGE
grafana-56d978ff77-8hnwq                0/1     ContainerCreating   0          56s
istio-egressgateway-7d4f75956-d5qj2     1/1     Running             0          18m
istio-ingressgateway-5d57955454-l8n98   1/1     Running             0          18m
istiod-6f6c6bbbbd-wvwpf                 1/1     Running             0          18m
jaeger-5c7c5c8d87-7vcrp                 1/1     Running             0          55s
kiali-5bb9c9cf49-r7pp5                  0/1     ContainerCreating   0          54s
prometheus-8958b965-p4gcd               0/2     ContainerCreating   0          54s

But the kiali creation got stuck.

kubectl rollout status deployment/kiali -n istio-system
Waiting for deployment "kiali" rollout to finish: 0 of 1 updated replicas are available...
error: deployment "kiali" exceeded its progress deadline

kubectl get pods -n istio-system
NAME                                    READY   STATUS    RESTARTS   AGE
grafana-56d978ff77-8hnwq                0/1     Running   2          10m
istio-egressgateway-7d4f75956-d5qj2     1/1     Running   0          27m
istio-ingressgateway-5d57955454-l8n98   1/1     Running   0          27m
istiod-6f6c6bbbbd-wvwpf                 0/1     Running   0          28m
jaeger-5c7c5c8d87-7vcrp                 1/1     Running   0          10m
kiali-5bb9c9cf49-r7pp5                  0/1     Running   4          10m
prometheus-8958b965-p4gcd               1/2     Running   0          10m

Kiali is showing zero containers.

 kubectl describe pod kiali-5bb9c9cf49-r7pp5 -n istio-system
Name:         kiali-5bb9c9cf49-r7pp5
Namespace:    istio-system
Priority:     0
Node:         kworker2/172.16.16.102
Start Time:   Thu, 15 Jul 2021 00:05:31 +0530
Labels:       app=kiali
              app.kubernetes.io/instance=kiali-server
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=kiali
              app.kubernetes.io/part-of=kiali
              app.kubernetes.io/version=v1.34.0
              helm.sh/chart=kiali-server-1.34.0
              pod-template-hash=5bb9c9cf49
              version=v1.34.0
Annotations:  cni.projectcalico.org/podIP: 192.168.77.138/32
              cni.projectcalico.org/podIPs: 192.168.77.138/32
              kiali.io/runtimes: go,kiali
              prometheus.io/port: 9090
              prometheus.io/scrape: true
              sidecar.istio.io/inject: false
Status:       Running
IP:           192.168.77.138
IPs:
  IP:           192.168.77.138
Controlled By:  ReplicaSet/kiali-5bb9c9cf49
Containers:
  kiali:
    Container ID:  containerd://427a3b836007d8411f5cde3be22a371fab8e4e576e005e27a5c169f22658e4e0
    Image:         quay.io/kiali/kiali:v1.34
    Image ID:      quay.io/kiali/kiali@sha256:31286129a1c6f25275517ae4c243c1cde8c39b022c2c7e0b328648c639bcfb03
    Ports:         20001/TCP, 9090/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /opt/kiali/kiali
      -config
      /kiali-configuration/config.yaml
    State:          Running
      Started:      Thu, 15 Jul 2021 00:13:05 +0530
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Thu, 15 Jul 2021 00:11:52 +0530
      Finished:     Thu, 15 Jul 2021 00:13:01 +0530
    Ready:          False
    Restart Count:  4
    Liveness:       http-get http://:api-port/kiali/healthz delay=5s timeout=1s period=30s #success=1 #failure=3
    Readiness:      http-get http://:api-port/kiali/healthz delay=5s timeout=1s period=30s #success=1 #failure=3
    Environment:
      ACTIVE_NAMESPACE:       istio-system (v1:metadata.namespace)
      LOG_LEVEL:              info
      LOG_FORMAT:             text
      LOG_TIME_FIELD_FORMAT:  2006-01-02T15:04:05Z07:00
      LOG_SAMPLER_RATE:       1
    Mounts:
      /kiali-cert from kiali-cert (rw)
      /kiali-configuration from kiali-configuration (rw)
      /kiali-secret from kiali-secret (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jgd7w (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kiali-configuration:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kiali
    Optional:  false
  kiali-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio.kiali-service-account
    Optional:    true
  kiali-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kiali
    Optional:    true
  kube-api-access-jgd7w:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  11m                    default-scheduler  Successfully assigned istio-system/kiali-5bb9c9cf49-r7pp5 to kworker2
  Normal   Pulled     9m56s                  kubelet            Successfully pulled image "quay.io/kiali/kiali:v1.34" in 1m30.127787668s
  Normal   Pulled     8m28s                  kubelet            Successfully pulled image "quay.io/kiali/kiali:v1.34" in 1.528856212s
  Normal   Created    8m27s (x2 over 9m55s)  kubelet            Created container kiali
  Normal   Started    8m22s (x2 over 9m54s)  kubelet            Started container kiali
  Normal   Killing    7m3s (x2 over 8m32s)   kubelet            Container kiali failed liveness probe, will be restarted
  Warning  Unhealthy  7m3s (x6 over 9m33s)   kubelet            Liveness probe failed: Get "http://192.168.77.138:20001/kiali/healthz": dial tcp 192.168.77.138:20001: connect: connection refused
  Normal   Pulling    7m1s (x3 over 11m)     kubelet            Pulling image "quay.io/kiali/kiali:v1.34"
  Normal   Pulled     6m59s                  kubelet            Successfully pulled image "quay.io/kiali/kiali:v1.34" in 2.166659225s
  Warning  Unhealthy  6m3s (x10 over 9m33s)  kubelet            Readiness probe failed: Get "http://192.168.77.138:20001/kiali/healthz": dial tcp 192.168.77.138:20001: connect: connection refused