Lakehouse/docs/ssh-mesh.md

# SSH mesh — passwordless access

All ATC lab VMs share a cluster SSH key so hosts can hop between each other without passwords.

## Design

| Item | Path / value |
|------|----------------|
| Shared private key | `/root/.ssh/atc_cluster` (on each VM, **not** in git) |
| Shared public key | `/root/.ssh/atc_cluster.pub` |
| SSH client snippet | `/root/.ssh/config.d/99-atc-lab.conf` |
| Host aliases | `/etc/hosts` block tagged `ATC Lakehouse lab` |

Proxmox (`pve01`) also keeps its admin key in each VM's `authorized_keys` for break-glass access.

## Fleet (mesh enabled)

| Hostname | IP | SSH user |
|----------|-----|----------|
| atc-docker01 | 10.0.21.45 | root |
| atc-docker02 | 10.0.21.47 | root |
| atc-mgt01 | 10.0.20.104 | root |
| atc-kafka01 | 10.0.21.36 | root |
| atc-lake01 | 10.0.21.50 | root |
| atc-elastic01 | 10.0.21.46 | root |
| atc-db01 | 10.0.20.112 | root |
| atc-db02 | 10.0.21.51 | root |
| atc-grafana | 10.0.20.103 | root |
| atc-airflow01 | 10.0.21.55 | root |
| atc-portal01 | 10.0.21.49 | root |
| atc-lama01 | 10.0.21.39 | root |
| **atc-objectscale** | **10.0.20.111** | **admin** (root via key after bootstrap) |

## ObjectScale (special case)

Dell ObjectScale runs **Luna OS** (CentOS 7-based). Out of the box:

- **`root` SSH login is disabled** — only `admin` accepts SSH (appliance default).
- Default lab password was used once to bootstrap keys; after bootstrap, use the cluster key only.

```bash
# From any mesh host (root works after bootstrap):
ssh -i /root/.ssh/atc_cluster root@atc-objectscale hostname   # → luna.local
ssh -i /root/.ssh/atc_cluster admin@atc-objectscale hostname    # also works

# Web UI
https://10.0.20.111/   or   https://atc-objectscale/
```

Bootstrap script: `scripts/setup/setup-objectscale-ssh.sh` (run from Proxmox if keys are missing).

## Deploy / refresh

From Proxmox (or any host with root SSH to the fleet):

```bash
cd /root/lakehouse
./scripts/setup/setup-ssh-mesh.sh
./scripts/setup/setup-objectscale-ssh.sh   # if ObjectScale was reinstalled
```

## Test

```bash
ssh -i ~/.ssh/atc_cluster root@atc-docker01
ssh -i ~/.ssh/atc_cluster root@atc-lake01 hostname
ssh -i ~/.ssh/atc_cluster root@atc-objectscale hostname -f

for h in atc-db01 atc-kafka01 atc-objectscale; do
  ssh -i /root/.ssh/atc_cluster root@$h hostname -f
done
```

## Hosts not yet in the mesh

These VMs still reject the hypervisor key (different credentials or SSH policy):

- `10.0.21.52`, `10.0.21.37`, `10.0.21.38`, `10.0.21.41`
- `10.0.20.31`, `10.0.21.44`

Add the cluster public key manually after fixing access.

## Security note

The cluster private key is powerful. Restrict Forgejo repo access and rotate keys if the lab is exposed outside your network. Do not commit ObjectScale `admin` passwords to git.