Nginx 502 bad gateway google cloud run the site takes approximately 50 minutes then starts working

The docker container works just fine when tested locally, upon deployment on cloud run I get 502 bad gateway. It takes approximately 50 – 60 min then for no reason starts working….causing a site down time. Have been onto this for approximately a week now no success on figuring out why this is happenning.

History:

I was deploying a Vuejs build static build before everything was working fine… i recently re-wrote my front end to Nuxtjs that’s when the deployment problems started.

How am testing dockerfile build:

As recommended by google am using : PORT=8080 && docker run -p 9090:${PORT} -e PORT=${PORT} myImage:latest check here

Dockfile:

# Alpine Deployment Server
FROM nginx:stable-alpine as alpine-server
# Dependencies for backend
RUN apk update && apk add --no-cache python3 && 
    python3 -m ensurepip && 
    rm -r /usr/lib/python*/ensurepip && 
    pip3 install --upgrade pip setuptools wheel && 
    if ( ! -e /usr/bin/pip ); then ln -s pip3 /usr/bin/pip ; fi && 
    if (( ! -e /usr/bin/python )); then ln -sf /usr/bin/python3 /usr/bin/python; fi && 
    rm -r /root/.cache
RUN apk add --no-cache gcc libxml2-dev libxslt-dev libressl-dev libffi-dev musl-dev postgresql-dev openssl-dev python3-dev cargo chromium-chromedriver chromium coreutils
RUN pip install cryptography==3.1.1 cffi scrapy selenium flask flask-cors flask-caching fuzzywuzzy 
    python-levenshtein requests aiohttp gunicorn scrapy-useragents scrapyrt
ENV DISPLAY=:99
ENV PORT=8080
WORKDIR /app
# Building frontend
RUN apk add --update nodejs nodejs-npm
RUN npm install pm2@latest -g
COPY ./client .
RUN npm ci && npm run build
COPY ./api .
COPY ./server .
COPY ./server/Extras .
COPY .htpasswd /etc/nginx/.htpasswd
RUN mkdir -p /data/nginx/cache
COPY ./nginx/default.conf /etc/nginx/conf.d/default.conf
CMD pm2 start > pm2.log && 
    gunicorn -b 0.0.0.0:5000 --workers 1 --threads 8 --timeout 0 api:app --daemon && 
    gunicorn -b 0.0.0.0:5001 --workers 1 --threads 1 --timeout 0 extras:app --daemon && 
    nohup sh -c "scrapyrt -p 7000 -i 0.0.0.0" > /dev/null 2>&1 & 
    nginx -g 'daemon off;'

NB: Nuxt is also configured to start at 0.0.0.0:3000 am using pm2 to start Nuxt

NGINX CONFIG

proxy_cache_path  /data/nginx/cache  levels=1:2    keys_zone=STATIC:10m
    inactive=24h  max_size=3g;
# Expires map
map $sent_http_content_type $expires {
    default off;
    text/html epoch;
    "text/html; charset=utf-8"  epoch;
    font/woff2 max;
    font/woff max;
    text/css max;
    application/javascript max;
    ~image/ max;
}
server {
  listen 8080;
  server_name chekiprice.co.ke;

  charset utf-8;

   location / {
    expires $expires;

    proxy_redirect                      off;
    proxy_set_header Host               $host;
    proxy_set_header X-Real-IP          $remote_addr;
    proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_http_version                  1.1;
    proxy_set_header                    Upgrade $http_upgrade;
    proxy_set_header                    Connection "upgrade";
    proxy_set_header X-Forwarded-Host   $server_name;
    proxy_buffering                     on;
    proxy_read_timeout                  1m;
    proxy_connect_timeout               1m;
    proxy_pass                          http://127.0.0.1:3000;
    proxy_cache                         STATIC;
    proxy_cache_valid                   200  1d;
    proxy_cache_use_stale               error timeout invalid_header updating
                                        http_500 http_502 http_503 http_504; 
  }

  location ~^/api/v1/(.+)/search/(.+)$ {
    set $allowspace2 $2;
    proxy_pass          http://127.0.0.1:5000/api/v1/$1/search/$allowspace2;
    proxy_http_version  1.1;
    proxy_redirect      ~^/api/v1/(.+)/search/(.+)$ http://127.0.0.1:5000/api/v1/$1/search/$allowspace2;
    proxy_set_header    Upgrade $http_upgrade;
    proxy_set_header    Connection "upgrade";
    proxy_set_header    Host $host;
    proxy_set_header    X-Real-IP $remote_addr;
    proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header    X-Forwarded-Host $server_name;
    proxy_buffering     on;
    proxy_cache         STATIC;
    proxy_cache_valid   200  1d;
    proxy_cache_use_stale  error timeout invalid_header updating
                           http_500 http_502 http_503 http_504;
    auth_basic "Restricted Content";
    auth_basic_user_file /etc/nginx/.htpasswd;
  }

  location ~^/api/v1/(.+)/view/(.+)$ {
    set $allowspace2 $2;
    proxy_pass          http://127.0.0.1:5000/api/v1/$1/view/$allowspace2;
    proxy_http_version  1.1;
    proxy_redirect      ~^/api/v1/(.+)/view/(.+)$ http://127.0.0.1:5000/api/v1/$1/view/$allowspace2;
    proxy_set_header    Upgrade $http_upgrade;
    proxy_set_header    Connection "upgrade";
    proxy_set_header    Host $host;
    proxy_set_header    X-Real-IP $remote_addr;
    proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header    X-Forwarded-Host $server_name;
    proxy_buffering     on;
    proxy_cache         STATIC;
    proxy_cache_valid   200  1d;
    proxy_cache_use_stale  error timeout invalid_header updating
                           http_500 http_502 http_503 http_504;
    auth_basic "Restricted Content";
    auth_basic_user_file /etc/nginx/.htpasswd;
  }
  location ~^/extras/v1/(.+)/(.+)$ {
    set $allowspace2 $2;
    proxy_pass          http://127.0.0.1:5001/extras/v1/$1/$allowspace2;
    proxy_http_version  1.1;
    proxy_redirect      ~^/extras/v1/(.+)/(.+)$ http://127.0.0.1:5001/extras/v1/$1/$allowspace2;
    proxy_set_header    Upgrade $http_upgrade;
    proxy_set_header    Connection "upgrade";
    proxy_set_header    Host $host;
    proxy_set_header    X-Real-IP $remote_addr;
    proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header    X-Forwarded-Host $server_name;
    proxy_buffering     on;
    proxy_cache         STATIC;
    proxy_cache_valid   200  1d;
    proxy_cache_use_stale  error timeout invalid_header updating
                           http_500 http_502 http_503 http_504;
  }
    location /crawl {
    proxy_pass          http://127.0.0.1:7000/crawl.json;
    proxy_http_version  1.1;
    proxy_redirect      default;
    proxy_set_header    Upgrade $http_upgrade;
    proxy_set_header    Connection "upgrade";
    proxy_set_header    Host $host;
    proxy_set_header    X-Real-IP $remote_addr;
    proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header    X-Forwarded-Host $server_name;
  }
    gzip on;
    gzip_types      text/plain application/xml text/css application/javascript;
    gzip_comp_level 6;
    gzip_min_length 1000;
    gzip_proxied any;
    gzip_vary on;
}

What ERROR is logged cloud run when trying to navigate to url during 502 bad Gateway ?

*13 connect() failed (111: Connection refused) while connecting to upstream,....

What bums me out is why ??

  1. Why the upstream are offline, all services are started first then
    NGINX is started.

  2. Why does it take long and eventually start.

  3. Why it works just locally on deployment to cloud run its just hocus-pocus.

How do I resolve this ??