swh/ingresses: Fix 502 when services are redeployed

The nginx is not notified as soon a service endpoints are updated so it
sends a couple of requests to a terminated pod.
It's not a problem for idempotent request as nginx retry on another pod
but it's not the case by default for POST requests.

The option is activated by default for all the ingresses because
POST requests are used everywhere

Related to swh/infra/sysadm-environment#5210

swh/templates/_helper_ingress.tpl

@@ -21,6 +21,17 @@ metadata:
   namespace: {{ $.Values.namespace }}
   name: {{ $serviceType }}-ingress-{{ $endpoint_definition }}
   annotations:
+  {{- if or (not (hasKey $configuration.ingress "useEndpointsAsUpstream")) (eq $configuration.ingress.useEndpointsAsUpstream false) -}}
+  {{- /* undocumented swh's ingress option to configure the upstreams to use the service ip.
+        By default, ips of endpoints are directly used by nginx to load balance the requests, but it's
+        not working well for not "idempotent" requests (POST).
+        So by default, the ingresses are configured to use the service as upstream.
+        https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#service-upstream
+        Using the default behavior (endpoints ips) should not be necessary accorfding the swh services architecture,
+        but allowing it just in case
+   */}}
+    nginx.ingress.kubernetes.io/service-upstream: "true"
+  {{- end }}
   {{- if $whitelistSourceRange }}
     nginx.ingress.kubernetes.io/whitelist-source-range: {{ $whitelistSourceRange }}
   {{- end }}