Lakehouse/docs/disaster-recovery.md

# Disaster recovery

Goal: fully restore the **atc-docker01** dashboard stack and configuration from git.

## Requirements

- VM `atc-docker01` with Docker and git
- Network to internal `.dell-atc.lan` hosts
- Secrets: `.env` or existing `config/homepage/proxmox.yaml`
- SSH mesh deployed ([ssh-mesh.md](ssh-mesh.md)) for multi-host config pull

## Step 1 — Repository

```bash
dnf install -y git docker-compose-plugin
git clone http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse.git /root/lakehouse
cd /root/lakehouse
cp .env.example .env
# Edit .env: SUPERSET_SECRET_KEY, etc.
```

## Step 2 — Homepage + RSS + icons

```bash
docker compose -f deploy/docker-compose.homepage.yml up -d --build
docker exec homepage wget -qO- http://127.0.0.1:3000/api/revalidate
```

## Step 3 — Superset

```bash
docker compose -f compose/superset/docker-compose.yaml up -d --build
```

## Step 4 — Optional: Forgejo, LAM, cAdvisor

```bash
docker compose -f compose/forgejo/compose.yaml up -d
docker compose -f compose/lam/compose.yml up -d
```

Or all at once:

```bash
./scripts/deploy/deploy-atc-docker01.sh
```

## Step 5 — Other hosts (manual)

This repo holds **config references**, not full VM images:

| Host | Restore from git |
|------|------------------|
| Kafka connectors | `config/kafka/*.json` → Connect REST API |
| Trino | `config/trino/config.properties` → lake host |
| Spark | `config/spark/spark-defaults.conf` |
| Airflow | `config/airflow/` |
| Data scripts | `config/scripts/data-generation/` |

See `inventory/containers-atc-docker01.json` for last exported container state.

## Volumes (not in git)

Back up separately:

- `forgejo_forgejo` — Git data
- `superset_superset_home` — Superset metadata
- `lam_lam` — LAM configuration

```bash
docker run --rm -v forgejo_forgejo:/data -v /backup:/backup alpine \
  tar czf /backup/forgejo_forgejo.tgz -C /data .
```

## Verification

1. http://atc-docker01.dell-atc.lan/ — homepage loads, 4 tabs
2. http://atc-docker01.dell-atc.lan:8090/health — RSS proxy
3. http://atc-docker01.dell-atc.lan:8088/health — Superset
4. FEEDS tab shows RSS widgets

## Proxmox widget

Token in `config/homepage/proxmox.yaml`. After restore, verify in Proxmox → Permissions → API Tokens → `root@pam!homepage`.