MicroK8s join error: The joining node (X.X.X.X) is already known to dqlite (504).

While adding a new node to a microk8s cluster, it may happen that somehow the dqlite datastore already holds a reference to the node you are triyng to add (if a node with the same ip node has been removed earlier and something went wrong during the microk8s remove-node or the microk8s leave procedures).

The join error appears as follows:

sudo microk8s join X.X.X.X:25000/yyyy/zzzz
Contacting cluster at Y.Y.Y.Y
Connection failed. The joining node (X.X.X.X) is already known to dqlite (504).

In my case, the solution showed in this comment from ktsakalozos did the magic.

Looking into the file /var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml of the master node, I actually found the ip X.X.X.X of the joining node (as the error message said).

Then, I just had to run the following command (still on the master node):

/snap/microk8s/current/bin/dqlite -s file:///var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml -c /var/snap/microk8s/current/var/kubernetes/backend/cluster.crt -k /var/snap/microk8s/current/var/kubernetes/backend/cluster.key -f json k8s ".remove X.X.X.X:19001"

The X.X.X.X record actually disappeared from the datastore /var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml and the new join attempt succeeded.