Troubleshooting Gitpod Self-Hosted

This section should solve all errors that might come up during installation of Gitpod.

1. ws-daemon is stuck in Init: 0/1

kubectl describe pod ws-daemon-... gives: MountVolume.SetUp failed for volume "node-fs1" : hostPath type check failed: /run/containerd/io.containerd.runtime.v1.linux/k8s.io is not a directory

Solution

1. ssh onto the node, mount | grep rootfs and find the directory where your containers are stored. Common paths are:

   • /run/containerd/io.containerd.runtime.v1.linux/k8s.io
   • /run/containerd/io.containerd.runtime.v1.linux/moby
   • /run/containerd/io.containerd.runtime.v2.task/k8s.io

2. Merge the following into your values.custom.yaml:

   components:
     wsDaemon:
       containerRuntime:
         nodeRoots:
         - <your path here>

3. Do an helm upgrade --install -f values.custom.yaml gitpod gitpod.io/gitpod --version=0.10.0 to apply the changes.

Example: For k3s a common configuration looks like this:

components:
  wsDaemon:
    containerRuntime:
      containerd:
        socket: /var/run/k3s/containerd/containerd.sock
      nodeRoots:
        - /var/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io

2. helm install fails with: “minio access key is required, please add a value to your values.yaml”

Since 0.7.0 minio requires custom credentials to be configured.

Solution

1. Follow the Upgrade Guide.

3. After upgrade, the minio Pod is stuck in ContainerCreating

This is caused by a bug in the minio Helm chart which blocks itself on updates.

Solution

1. kubectl scale deployments/minio --replicas=0

2. kubectl scale deployments/minio --replicas=1

3. Wait until the pod comes up.

3. agent-smith daemonset fails in deployment

In the v0.10.0 release agent-smith is incorrectly enabled in Gitpod Self-Hosted.

Solution

Add the following to your values.yaml file to disable agent-smith:

components:
  agentSmith:
    disabled: true

4. Workspaces stopping once container image downloaded

This may be caused by your host operating system not supporting shiftfs. Check the ws-daemon logs for an error message like "error","message":"cannot mount shiftfs mark" to confirm.

Further proof can be found by checking if the shiftfs kernel module is present on your host operating system:

find /lib/modules/ -name '*shiftfs*'

If this returns nothing, shiftfs is not supported by your instance and you will need to use fuse instead.

Solution

Add the following to your values.yaml file to use fuse-overlayfs:

components:
  wsDaemon:
    userNamespaces:
      fsShift: fuse

5. Create workspace results in 7 PERMISSION_DENIED: cannot resolve workspace image error

If you are using the internal Docker registry, you will need to specify the username and password so your Gitpod instance can access it.

Solution

Add the following to your values.yaml file to authorize the Docker registry:

components:
  docker-registry:
    authentication:
      username: gitpod
      password: gitpod

Replace these with your own values.