[QUESTION/HELP] Exposing multiple services through ingress requires the use of a host for layer seven routing. How to configure it in the k3d environment? Are there any relevant examples #1435

On macOS for local testing, with Docker, I'm using a k3d config like:

apiVersion: k3d.io/v1alpha5
kind: Simple
metadata:
  name: my-cluster
servers: 1
agents: 1
ports:
  - port: 8080:80
    nodeFilters:
      - loadbalancer
  - port: 8443:443
    nodeFilters:
      - loadbalancer
registries:
  create:
    name: my-cluster.registry
    host: "0.0.0.0"
    hostPort: "8500"
options:
  k3d:
    wait: true
  k3s:
    extraArgs:
      - arg: --disable=traefik
        nodeFilters:
          - server:*

For exposing multiple services I'm using Contour with Envoy as a Deployment with HTTPProxy Inclusion. Depending if you also want to use TLS or not, you may need to look into Contour's disablePermitInsecure: false configuration option. For me that seemed easier to support services in multiple namespaces than trying out the Gateway API for the first time.

I update my /etc/hosts and generate certs with minica so I can access different services with URLs like https://kubernetes-dashboard.my-cluster.localdomain:8443. There isn't a complete example for me to share as my setup is code generated with some internal tools, e.g., scripts used by kustomize --enable-alpha-plugins --enable-exec for things like config values and secrets.

Scripting the cluster creation was helpful as I think I encountered k3d loadbalancing/networking to get in a bad state, so I've had to recreate the cluster a few times.

Finally, to save my laptop's CPU I run database outside of Docker directly in macOS. For workloads to access it directly through the host.k3d.internal hostname via ExternalName service, the enableExternalNameService: true contour configuration also has to be added.