Lakehouse/docs/ssh-mesh.md

SSH mesh — passwordless access

All ATC lab VMs share a cluster SSH key so hosts can hop between each other without passwords.

Design

Item	Path / value
Shared private key	/root/.ssh/atc_cluster (on each VM, not in git)
Shared public key	/root/.ssh/atc_cluster.pub
SSH client snippet	/root/.ssh/config.d/99-atc-lab.conf
Host aliases	/etc/hosts block tagged ATC Lakehouse lab

Proxmox (pve01) also keeps its admin key in each VM's authorized_keys for break-glass access.

Fleet (mesh enabled)

Hostname	IP	SSH user
atc-docker01	10.0.21.45	root
atc-docker02	10.0.21.47	root
atc-mgt01	10.0.20.104	root
atc-kafka01	10.0.21.36	root
atc-lake01	10.0.21.50	root
atc-elastic01	10.0.21.46	root
atc-db01	10.0.20.112	root
atc-db02	10.0.21.51	root
atc-grafana	10.0.20.103	root
atc-airflow01	10.0.21.55	root
atc-portal01	10.0.21.49	root
atc-lama01	10.0.21.39	root
atc-objectscale	10.0.20.111	admin (root via key after bootstrap)

ObjectScale (special case)

Dell ObjectScale runs Luna OS (CentOS 7-based). Out of the box:

• root SSH login is disabled — only admin accepts SSH (appliance default).
• Default lab password was used once to bootstrap keys; after bootstrap, use the cluster key only.

# From any mesh host (root works after bootstrap):
ssh -i /root/.ssh/atc_cluster root@atc-objectscale hostname   # → luna.local
ssh -i /root/.ssh/atc_cluster admin@atc-objectscale hostname    # also works

# Web UI
https://10.0.20.111/   or   https://atc-objectscale/

Bootstrap script: scripts/setup/setup-objectscale-ssh.sh (run from Proxmox if keys are missing).

Deploy / refresh

From Proxmox (or any host with root SSH to the fleet):

cd /root/lakehouse
./scripts/setup/setup-ssh-mesh.sh
./scripts/setup/setup-objectscale-ssh.sh   # if ObjectScale was reinstalled

Test

ssh -i ~/.ssh/atc_cluster root@atc-docker01
ssh -i ~/.ssh/atc_cluster root@atc-lake01 hostname
ssh -i ~/.ssh/atc_cluster root@atc-objectscale hostname -f

for h in atc-db01 atc-kafka01 atc-objectscale; do
  ssh -i /root/.ssh/atc_cluster root@$h hostname -f
done

Hosts not yet in the mesh

These VMs still reject the hypervisor key (different credentials or SSH policy):

• 10.0.21.52, 10.0.21.37, 10.0.21.38, 10.0.21.41
• 10.0.20.31, 10.0.21.44

Add the cluster public key manually after fixing access.

Security note

The cluster private key is powerful. Restrict Forgejo repo access and rotate keys if the lab is exposed outside your network. Do not commit ObjectScale admin passwords to git.