google cloud platform – Kubernetes managing many distinct UDP servers on GKE

I’m trying to set up a system that can automatically spin up and down video game servers as docker images. In this case, factoriotools/factorio-docker. Each game is a different, distinct single-pod deployment of that container, and therefore (in the simplified case) needs its own IP address that can listen on a specific UDP port. Load balancers are redundant and irrelevant, and Cloud NAT doesn’t appear to allow ingress traffic easily.

There’s a couple ways I know of to get this to work, both with pretty major compromises:

  • I can use a NodePort service, and lose control over which port the client needs to connect to. That’s an issue because the server registers itself with a server listing.
  • I can use host networking. If my information is correct, that requires privileged containers, which is Definitely Not Good.
  • I could maybe use a UDP load balancer, but even if that exists and works, it’s expensive.

There are probably ways to work around the limitations of either approach (for the second, keep the hosts short-lived and keep the firewall strict, and it should be mostly OK?), but I can’t help but think there’s a better option that I can’t find described in the official kubernetes docs. Does traefik have some trick I don’t know about? Is there some way to get a variant of MetalLB that can dynamically allocate public IP addresses as I need them?

How do I get each server-container to listen on a different public IP address with a specific UDP port, without making security impossible in the process?

kubernetes – What do you call a namespace that has multiple deployments?

I have a Kubernetes namespace that has multiple applications deployed in it.
I’m wondering what to call that. It’s not a namespace because it has an entire system deployed (multiple deployments with replicasets with pods).

So the terms for this discussion include:

  • Pod – is an app (can be other stuff too)
  • Deployment is multiple instances of an app with replicset property to define # of instances.
  • Namespace – a collection of Kubernetes resources. Multiple uses exist for namespaces. In my case, the namespace is used to manage resources that are deployed together as a system.

I’ve been informally referring to this different ways.
I’ve been through some training but can’t remember what the instructors called this.

Searching for this answer is difficult because I’m asking for a concept that has a name but not using the correct name (hence the question).

kubernetes – RShiny App Deployed in EKS on AWS – Pod reaches CPU utilization limit even when nobody is using it

I deployed a RShiny app in EKS via Amazon Web Services. It has its own load balancer so anybody with the link can access it. About every 2 days or so, the link to the app times out. I checked the CPU utilization through AWS’s Cloud Watch service and this is what I found:

Shiny App CPU Usage

The rshiny app is the red line. The usage steadily climbs until it goes over its limit, which explains why I can’t access the app every couple days. The only solution I have for now is to delete the pod running the shiny app and have EKS recreate it (indicated by the sudden drop in CPU usage). The app is used maybe a couple times throughout the day. Any suggestions for why this might be happening?

kubernetes – Kafka logs get reset when all brokers restart at the same time

I have a kafka cluster running on an integration Kubernetes cluster with 3 Kafka brokers and 3 Zookeepers, each component running in a statefulset.
The helm chart I use to deploy the cluster is a custom one as I needed external access to the cluster through a BigIp F5 and did not find a helm chart doing this.

The Kafka image is confluentinc/cp-kafka:5.4.0 and the zookeeper one is confluentinc/cp-zookeeper:5.4.0

/var/lib/zookeeper/data and /var/lib/zookeeper/log for Zookeeper are mapped to persistent volumes.
The same for /var/lib/kafka on Kafka

I use hlebalbau/kafka-manager:stable to watch the state of the cluster.

I set-up three partitions per topic, and replication factor is also equal to three.

Recently I realized that if I restarted all three kafka brokers at the same time (with kubectl delete pod) all the topics contents was lost :

  • the log size drops to zero for each topic
  • the list of topics stays the same
  • the consumers list stays the same, but each consumer current offset drops to negative value (if the consumer was at offset 10000 for a 10000 message topic, then the topic size drops to zero and the consumer offset to -10000 )

I never encountered any issue when restarting one kafka broker at a time and waiting for it to be started before restarting another one.
I know that a kafka cluster is not meant to be stopeed or restarted like this. But I did not expect this kind of behaviour.

Is it expected behaviour ? Or did I miss something obvious ?

Here is my Yaml template for a kafka broker. As indicated by its name, the wait-zookeeper.sh script “just” waits for the zookeepers to be started.

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka-controller-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-1
  serviceName: kafka1-headless
  template:
    metadata:
      labels:
        app: kafka-1
        cluster: kafka
    spec:
      initContainers:
        - name: init-wait-zookeeper
          image: bash:latest
          command: ("/usr/local/bin/bash","-c","cp /wait-zookeeper-configmap/wait-zookeeper.sh /wait-zookeeper-emptydir/ && chmod 755 /wait-zookeeper-emptydir/wait-zookeeper.sh && /wait-zookeeper-emptydir/wait-zookeeper.sh")
          volumeMounts:
            - name: wait-zookeeper-configmap
              mountPath: /wait-zookeeper-configmap
            - name: wait-zookeeper-emptydir
              mountPath: /wait-zookeeper-emptydir
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - kafka-2
                  - kafka-3
                  {{- if gt .Values.workerNodesNumber 5.0 }}
                  - zookeeper-1
                  - zookeeper-2
                  - zookeeper-3
                  {{- end }}
              topologyKey: "kubernetes.io/hostname"
      containers:
        - name: kafka1
          image: confluentinc/cp-kafka:5.4.0
          resources:
            requests:
              memory: "512Mi"
            limits:
              memory: "{{.Values.jvmMaxHeapSizeGb}}Gi"
          ports:
          - containerPort: 9092
          env:
          - name: HOST_IP
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
          - name: KAFKA_LISTENERS
            value: "PLAINTEXT://0.0.0.0:9092"
          - name: KAFKA_ADVERTISED_LISTENERS
            value: "PLAINTEXT://$(HOST_IP):{{ add .Values.startingNodePort 0 }}"
          - name: KAFKA_BROKER_ID
            value: "10"
          - name: KAFKA_ZOOKEEPER_CONNECT
            value: "zookeeper-controller-1-0.zoo1-headless:2181,zookeeper-controller-2-0.zoo2-headless:2181,zookeeper-controller-3-0.zoo3-headless:2181"
          - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
            value: "3"
          - name: KAFKA_DELETE_TOPIC_ENABLE
            value: "true"
          - name: KAFKA_DEFAULT_REPLICATION_FACTOR
            value: "3"
          - name: KAFKA_NUM_PARTITIONS
            value: "{{.Values.defaultPartitionsNumber}}"
          - name: KAFKA_LOG_RETENTION_HOURS
            value: "{{.Values.retentionTimeHours}}"
          - name: KAFKA_OFFSETS_RETENTION_MINUTES
            value: "{{.Values.retentionTimeHours | mul 60 }}"
          - name: JMX_PORT
            value: "{{ add .Values.startingNodePort 3 }}"
          - name: KAFKA_JMX_HOSTNAME
            value: "kafka-service-1"
          - name: KAFKA_HEAP_OPTS
            value: "-Xms512m -Xmx{{.Values.jvmMaxHeapSizeGb}}G"
          livenessProbe:
            exec:
              command:
              - /bin/bash
              - -c
              - "unset JMX_PORT && kafka-broker-api-versions --bootstrap-server=localhost:9092"
            initialDelaySeconds: 60
            periodSeconds: 20
          volumeMounts:
            - name: "kafka-logs"
              mountPath: "/var/lib/kafka"
      volumes:
        - name: "wait-zookeeper-configmap"
          configMap:
            name: "kafka-initializer"
            items:
              - key: "wait-zookeeper.sh"
                path: "wait-zookeeper.sh"
        - name: "wait-zookeeper-emptydir"
          emptyDir: {}
  volumeClaimTemplates:
  - metadata:
      name: kafka-logs
    spec:
      storageClassName: {{.Values.storageClassName}}
      accessModes: ("ReadWriteOnce")
      resources:
        requests:
          storage: {{.Values.storageSizeGb}}Gi

docker – Securing a Kubernetes application – SSL on Kubernetes or container?

I have a gRPC server written in golang and containerized with Docker. I would like to deploy this application to Kubernetes with TLS (Let’s Encrypt).

What is the best way to secure the application? I’ve read that Kubernetes can use a Let’s Encrypt ingress controller to handle TLS and securing the cluster. However, my gRPC web server can also load certificates to enable TLS. This is less convenient though, because I have to restart the container when certificates renew, bind them to a volume, etc.

Is there anything wrong with leaving the container insecure (serving HTTP) and have the Kubernetes cluster proxy take care of securing the connection?

Double port forwarding kubernetes + docker

Summary:

I have a docker container which is running kubectl port-forward, forwarding the port (5432) of a postgres service running as a k8s service to a local port (2223).
In the Dockerfile, I have exposed the relevant port 2223. Then I ran the container by publishing the said port (-p 2223:2223)

Now when I am trying to access the postgres through psql -h localhost -p 2223, I am getting the following error:

psql: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

However, when I do docker exec -ti to the said container and run the above psql command, I am able to connect to postgres.

Dockerfile CMD:

EXPOSE 2223
CMD ("bash", "-c", "kubectl -n namespace_test port-forward service/postgres-11-2 2223:5432")

Docker Run command:

docker run -it --name=k8s-conn-12 -p 2223:2223 my_image_name:latest

Output of the docker run command:

Forwarding from 127.0.0.1:2223 -> 5432

So the port forwarding is successful, and I am able to connect to the postgres instance from inside the docker container. What I am not able to do is to connect from outside the container with the exposed and published port

Kubernetes network polices are not enforced unless the network-plugin daemon-sets are restarted. Why?

I have only one network policy in my cluster in prod namespace that allows only ingress rules. The network plugin is weave-net. No rules are configured for Egress so I am expecting egress traffic will be blocked. But until I restart the network daemon-set pods the rule has no effect. I know by best practices I should have default ingress and egress rules. But I want to understand the reason of this behavior. Is this step always required to restart the network-plugin pods?

1. Network Policy Definition

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-network-policy
  namespace: prod
spec:
  ingress:
  - {}
  podSelector:
    matchLabels:
      run: prod-nginx
  policyTypes:
  - Ingress
  - Egress

2. Checking the netpol object

Name:         test-network-policy
Namespace:    prod
Created on:   2021-06-06 10:16:50 +0000 UTC
Labels:       <none>
Annotations:  <none>
Spec:
  PodSelector:     run=prod-nginx
  Allowing ingress traffic:
    To Port: <any> (traffic allowed to all ports)
    From: <any> (traffic not restricted by source)
  Allowing egress traffic:
    <none> (Selected pods are isolated for egress connectivity)
  Policy Types: Ingress, Egress

3. Testing egress traffic to nginx server (This is unexpected to my understanding)

Command : kubectl -n prod exec -it prod-nginx -- curl http://10.39.0.5 | grep successfully #egress
Response: <p>If you see this page, the nginx web server is successfully installed and

4. Restarted the weave-net pods

5. Retesting egress connection to same nginx server (expected)

Command: kubectl -n prod exec -it prod-nginx -- curl http://10.39.0.5 | grep successfully #egress**
Response: No connection

aws – Kubernetes auto scaling

We are currently in the process of doing an infrastructure overhaul.

A bit of background of what our business model currently is:

We are an aggregator of bills and payment services for businesses. Utilities, mobiles and much more. We are making it easier for companies to connect to these services by just connecting to our APIs. Each of these bills and payment services has their custom script with different APIs. We have a software acting as a middleman, it processes the incoming request, triggers the correct script, waits for the response, and then format and return the response.

Initially, our middleware and scripts are on two different servers. We use elastic beanstalk to host our scripts. However, we face a lot of issues. Since elastic beanstalk auto scale depending on the traffic coming in. Here is an example of what did go wrong.

  1. There are three instances running
  2. low traffic detected, shutting down one instance
  3. Suddenly, 100 request coming in, 20 requests are routed to the instance that is shutting down
  4. the middleware that is waiting for the 20 requests receives empty responses.

We then transition to a single large EC2 that host our middleware and our scripts together. The requests come in, and the middleware will point to the API (our PHP scripts) that is stored locally, runs it and return it to our users.

This worked out great. However, we encountered a hardware failure after two years of using this approach which brought down our whole server for 2 hours. Now we are trying to mitigate this and make sure the scripts have a 100% uptime.

It is also not scalable since edits to the code are made by RDP into the EC2 and changing the code locally.

Currently, we are trying to use EKS / Kubernetes instead.

Since Kubernetes also auto-scale nodes, I’m afraid the problem we faced when using elastic beanstalk might happen again.

TLDR:
To summarize, will kubernetes auto-scale affect responses that are in the midst of processing? Some of our scripts might take more than 5 minutes since our scripts are synchronous.

Any recommendation on how to improve this architecture are also welcomed.

I have deleted all the Azure AKS Kubernetes Nodes, how to restore back the Cluster to it’s original state?

I am new to the Azure AKS Cluster world, and while messing with a test cluster i have deleted all its Nodes with kubectl delete node xxxx, thinking that the cluster will heal itself. Boy, was i wrong.

Now, let me explain the issue, so, when i run kubectl get nodes, i get No resources found.
In the “Node Pools” in the portal, i can see that there are 3 Nodes,
i have scaled the Pool up and down, but in kubectl shows no nodes – No resources found.
When i run kubectl get pods, all the pods are shown in pending state.

Extra Info:

  • The AKS Cluster was created manually, no ARM template or script was saved.
  • The AKS Cluster is using Availability Set (not Scale Set) for the Pool, so i can not add new
    Pool, and move the Pods there.

My question(s) to you is:

  1. How to get the Nodes to be shown in kubectl again? (The Pool has 3 Nodes there sitting)
  2. Can i somehow restore the Cluster to be working again? Move the Pods somehow, somewhere?
  3. What would you do in this case?

EDIT:

  • after some time showing “No Resources found” when i ran “kubectl get nodes”, now 2 nodes came back online, but one is still missing. The Pool has count of 3. The 2 Nodes which are shown are in Ready State. But all the Pods are still in Pending state. No errors in Events.

New Question:

  • Is there a way to start populating the 2 Ready Nodes with the Pending Pods?

Thanks again folks.

I have deleted all the Azure AKS Kubernetes Nodes, how to restore back the Cluster to it’s original state?

I am new to the Azure AKS Cluster world, and while messing with a test cluster i have deleted all its Nodes with kubectl delete node xxxx, thinking that the cluster will heal itself. Boy, was i wrong.

Now, let me explain the issue, so, when i run kubectl get nodes, i get No resources found.
In the “Node Pools” in the portal, i can see that there are 3 Nodes,
i have scaled the Pool up and down, but in kubectl shows no nodes – No resources found.
When i run kubectl get pods, all the pods are shown in pending state.

Extra Info:

  • The AKS Cluster was created manually, no ARM template or script was saved.
  • The AKS Cluster is using Availability Set (not Scale Set) for the Pool, so i can not add new
    Pool, and move the Pods there.

My question(s) to you is:

  1. How to get the Nodes to be shown in kubectl again? (The Pool has 3 Nodes there sitting)
  2. Can i somehow restore the Cluster to be working again? Move the Pods somehow, somewhere?
  3. What would you do in this case?

EDIT:

  • after some time showing “No Resources found” when i ran “kubectl get nodes”, now 2 nodes came back online, but one is still missing. The Pool has count of 3. The 2 Nodes which are shown are in Ready State. But all the Pods are still in Pending state. No errors in Events.

New Question:

  • Is there a way to start populating the 2 Ready Nodes with the Pending Pods?

Thanks again folks.