Lakehouse Admin
f5d30824fc
- Add root README and docs/ (architecture, hosts, disaster-recovery, operations) - Add compose/ for superset, forgejo, lam, cadvisor; unified deploy compose - Add scripts/deploy (deploy-atc-docker01.sh, container inventory export) - Add inventory/containers-atc-docker01.json snapshot - Add config READMEs for kafka and airflow; .env.example; expand .gitignore - Sync atclab helper scripts; update homepage/superset compose with env support
85 lines
2.3 KiB
Markdown
85 lines
2.3 KiB
Markdown
# Disaster recovery
|
|
|
|
Doel: volledige **atc-docker01** dashboard-stack en configuratie terugzetten vanuit git.
|
|
|
|
## Vereisten
|
|
|
|
- VM `atc-docker01` met Docker en git
|
|
- Netwerk naar interne `.dell-atc.lan` hosts
|
|
- Secrets: `.env` of bestaande `config/homepage/proxmox.yaml`
|
|
|
|
## Stap 1 — Repository
|
|
|
|
```bash
|
|
dnf install -y git docker-compose-plugin # of equivalent
|
|
git clone http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse.git /root/lakehouse
|
|
cd /root/lakehouse
|
|
cp .env.example .env
|
|
# Bewerk .env: SUPERSET_SECRET_KEY, eventueel Proxmox token
|
|
```
|
|
|
|
## Stap 2 — Homepage + RSS + icons
|
|
|
|
```bash
|
|
docker compose -f deploy/docker-compose.homepage.yml up -d --build
|
|
docker exec homepage wget -qO- http://127.0.0.1:3000/api/revalidate
|
|
```
|
|
|
|
## Stap 3 — Superset
|
|
|
|
```bash
|
|
docker compose -f compose/superset/docker-compose.yaml up -d --build
|
|
```
|
|
|
|
## Stap 4 — Optioneel: Forgejo, LAM, cAdvisor
|
|
|
|
```bash
|
|
docker compose -f compose/forgejo/compose.yaml up -d
|
|
docker compose -f compose/lam/compose.yml up -d
|
|
# docker compose -f compose/cadvisor/compose.yaml up -d
|
|
```
|
|
|
|
Of alles tegelijk:
|
|
|
|
```bash
|
|
./scripts/deploy/deploy-atc-docker01.sh
|
|
```
|
|
|
|
## Stap 5 — Andere hosts (handmatig)
|
|
|
|
Deze repo bevat **config-referenties**, geen volledige VM-images:
|
|
|
|
| Host | Herstel uit git |
|
|
|------|-----------------|
|
|
| Kafka connectors | `config/kafka/*.json` → deploy op Connect API |
|
|
| Trino | `config/trino/config.properties` → `/etc/trino/` op lake01 |
|
|
| Spark | `config/spark/spark-defaults.conf` |
|
|
| Airflow | `config/airflow/` |
|
|
| Data scripts | `config/scripts/data-generation/` |
|
|
|
|
Zie `inventory/containers-atc-docker01.json` voor laatst geëxporteerde containerstate.
|
|
|
|
## Volumes (niet in git)
|
|
|
|
Back-up apart:
|
|
|
|
- `forgejo_forgejo` — Git data
|
|
- `superset_superset_home` — Superset metadata
|
|
- `lam_lam` — LAM configuratie
|
|
|
|
```bash
|
|
docker run --rm -v forgejo_forgejo:/data -v /backup:/backup alpine \
|
|
tar czf /backup/forgejo_forgejo.tgz -C /data .
|
|
```
|
|
|
|
## Verificatie
|
|
|
|
1. http://atc-docker01.dell-atc.lan/ — homepage laadt, 4 tabs
|
|
2. http://atc-docker01.dell-atc.lan:8090/health — RSS proxy
|
|
3. http://atc-docker01.dell-atc.lan:8088/health — Superset
|
|
4. FEEDS-tab toont RSS-widgets
|
|
|
|
## Proxmox-widget
|
|
|
|
Token staat in `config/homepage/proxmox.yaml`. Na restore: Proxmox → Permissions → API Tokens → controleer `root@pam!homepage`.
|