ATC/Lakehouse
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

English docs and passwordless SSH mesh for lab fleet

Browse Source 
- Translate all documentation to English
- Add docs/ssh-mesh.md and scripts/setup/setup-ssh-mesh.sh
- Add config/hosts/atc-lab.hosts for /etc/hosts on all VMs
- Ignore atc_cluster private key in git; mesh deployed on 12 hosts
This commit is contained in:
Lakehouse Admin
2026-05-19 22:46:40 +02:00
parent f5d30824fc
commit f3d24634e4
12 changed files with 330 additions and 118 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+26 -22
View File
@@ -1,10 +1,10 @@
# Architectuur — ATC Lakehouse
# Architecture — ATC Lakehouse
## Dataflow (conceptueel)
## Data flow (conceptual)
```mermaid
flowchart LR
subgraph sources [Bronnen]
subgraph sources [Sources]
DB1[(atc-db01)]
DB2[(atc-db02)]
end
@@ -20,7 +20,7 @@ flowchart LR
ICE[Iceberg tables]
end
subgraph consume [Consumptie]
subgraph consume [Consumption]
SS[Superset]
KB[Kibana]
GF[Grafana]
@@ -36,32 +36,36 @@ flowchart LR
KAFKA --> SS
```
## Lagen
## Layers
| Laag | Componenten | Doel |
|------|-------------|------|
| **Bron** | PostgreSQL, MySQL, MongoDB, Cassandra, Neo4j | OLTP / events / graph demo data |
| **Ingest** | Debezium, Kafka Connect | CDC naar Kafka topics |
| **Processing** | Spark, Airflow | ETL, batch, orchestratie |
| **Query** | Trino | Federated SQL over lake + bronnen |
| **Storage** | ObjectScale / MinIO (indien actief) | Object storage |
| **Analytics** | Superset, Kibana | BI en search UI |
| **Platform** | Proxmox, Docker hosts | VMs en containers |
| **Ops** | Homepage dashboard, Grafana | Zichtbaarheid en links |
| Layer | Components | Purpose |
|-------|------------|---------|
| **Source** | PostgreSQL, MySQL, MongoDB, Cassandra, Neo4j | OLTP / events / graph demo data |
| **Ingest** | Debezium, Kafka Connect | CDC to Kafka topics |
| **Processing** | Spark, Airflow | ETL, batch, orchestration |
| **Query** | Trino | Federated SQL over lake + sources |
| **Storage** | ObjectScale / MinIO (when active) | Object storage |
| **Analytics** | Superset, Kibana | BI and search UI |
| **Platform** | Proxmox, Docker hosts | VMs and containers |
| **Ops** | Homepage dashboard, Grafana | Visibility and links |
## atc-docker01 — centrale dashboard-host
## atc-docker01 — central dashboard host
Draait de **gethomepage**-dashboard (poort 80), **RSS-proxy** (8090), **icon server** (8080), **Apache Superset** (8088), en optioneel lokale **Forgejo** (4002) / **LAM** (4001).
Runs **gethomepage** (port 80), **RSS-proxy** (8090), **icon server** (8080), **Apache Superset** (8088), and optional local **Forgejo** (4002) / **LAM** (4001).
Configuratiepad op de host: `/root/lakehouse/config/homepage` → gemount in container `/app/config`.
Config path on host: `/root/lakehouse/config/homepage` → mounted at `/app/config` in the container.
## Configuratie in git
## SSH mesh
| Pad | Beschrijving |
All lab VMs use a shared cluster key for root-to-root access. See [ssh-mesh.md](ssh-mesh.md).
## Configuration in git
| Path | Description |
|-----|--------------|
| `config/kafka/*.json` | Debezium/Kafka Connect connector definities |
| `config/kafka/*.json` | Debezium/Kafka Connect connector definitions |
| `config/trino/config.properties` | Trino coordinator (single-node) |
| `config/spark/spark-defaults.conf` | Spark defaults |
| `config/airflow/` | Airflow DAG + `airflow.cfg` |
| `config/superset/` | Superset Python config + compose |
| `config/homepage/` | Volledige dashboardconfiguratie |
| `config/homepage/` | Full dashboard configuration |
docs/disaster-recovery.md
+28 -28
View File
@@ -1,84 +1,84 @@
# Disaster recovery
Doel: volledige **atc-docker01** dashboard-stack en configuratie terugzetten vanuit git.
Goal: fully restore the **atc-docker01** dashboard stack and configuration from git.
## Vereisten
## Requirements
- VM `atc-docker01` met Docker en git
- Netwerk naar interne `.dell-atc.lan` hosts
- Secrets: `.env` of bestaande `config/homepage/proxmox.yaml`
- VM `atc-docker01` with Docker and git
- Network to internal `.dell-atc.lan` hosts
- Secrets: `.env` or existing `config/homepage/proxmox.yaml`
- SSH mesh deployed ([ssh-mesh.md](ssh-mesh.md)) for multi-host config pull
## Stap 1 — Repository
## Step 1 — Repository
```bash
dnf install -y git docker-compose-plugin # of equivalent
dnf install -y git docker-compose-plugin
git clone http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse.git /root/lakehouse
cd /root/lakehouse
cp .env.example .env
# Bewerk .env: SUPERSET_SECRET_KEY, eventueel Proxmox token
# Edit .env: SUPERSET_SECRET_KEY, etc.
```
## Stap 2 — Homepage + RSS + icons
## Step 2 — Homepage + RSS + icons
```bash
docker compose -f deploy/docker-compose.homepage.yml up -d --build
docker exec homepage wget -qO- http://127.0.0.1:3000/api/revalidate
```
## Stap 3 — Superset
## Step 3 — Superset
```bash
docker compose -f compose/superset/docker-compose.yaml up -d --build
```
## Stap 4 — Optioneel: Forgejo, LAM, cAdvisor
## Step 4 — Optional: Forgejo, LAM, cAdvisor
```bash
docker compose -f compose/forgejo/compose.yaml up -d
docker compose -f compose/lam/compose.yml up -d
# docker compose -f compose/cadvisor/compose.yaml up -d
```
Of alles tegelijk:
Or all at once:
```bash
./scripts/deploy/deploy-atc-docker01.sh
```
## Stap 5 — Andere hosts (handmatig)
## Step 5 — Other hosts (manual)
Deze repo bevat **config-referenties**, geen volledige VM-images:
This repo holds **config references**, not full VM images:
| Host | Herstel uit git |
|------|-----------------|
| Kafka connectors | `config/kafka/*.json` → deploy op Connect API |
| Trino | `config/trino/config.properties` → `/etc/trino/` op lake01 |
| Host | Restore from git |
|------|------------------|
| Kafka connectors | `config/kafka/*.json` → Connect REST API |
| Trino | `config/trino/config.properties` → lake host |
| Spark | `config/spark/spark-defaults.conf` |
| Airflow | `config/airflow/` |
| Data scripts | `config/scripts/data-generation/` |
Zie `inventory/containers-atc-docker01.json` voor laatst geëxporteerde containerstate.
See `inventory/containers-atc-docker01.json` for last exported container state.
## Volumes (niet in git)
## Volumes (not in git)
Back-up apart:
Back up separately:
- `forgejo_forgejo` — Git data
- `superset_superset_home` — Superset metadata
- `lam_lam` — LAM configuratie
- `lam_lam` — LAM configuration
```bash
docker run --rm -v forgejo_forgejo:/data -v /backup:/backup alpine \
tar czf /backup/forgejo_forgejo.tgz -C /data .
```
## Verificatie
## Verification
1. http://atc-docker01.dell-atc.lan/ — homepage laadt, 4 tabs
1. http://atc-docker01.dell-atc.lan/ — homepage loads, 4 tabs
2. http://atc-docker01.dell-atc.lan:8090/health — RSS proxy
3. http://atc-docker01.dell-atc.lan:8088/health — Superset
4. FEEDS-tab toont RSS-widgets
4. FEEDS tab shows RSS widgets
## Proxmox-widget
## Proxmox widget
Token staat in `config/homepage/proxmox.yaml`. Na restore: Proxmox → Permissions → API Tokens → controleer `root@pam!homepage`.
Token in `config/homepage/proxmox.yaml`. After restore, verify in Proxmox → Permissions → API Tokens → `root@pam!homepage`.
docs/hosts.md
+26 -21
View File
@@ -1,38 +1,43 @@
# Hosts & poorten
# Hosts & ports
## VM-overzicht
## VM overview
| Hostname | IP (indien bekend) | Rol |
|----------|-------------------|-----|
| Hostname | IP | Role |
|----------|-----|------|
| `atc-docker01.dell-atc.lan` | `10.0.21.45` | Docker: homepage, superset, rss-proxy, icons |
| `atc-mgt01.dell-atc.lan` | — | Forgejo Git `:3001`, LDAP/LAM |
| `atc-kafka01.dell-atc.lan` | — | Kafka UI `:9000` |
| `atc-lake01.dell-atc.lan` | — | Spark `:8080`, Trino `:8089`, Debezium `:8083` |
| `atc-elastic01.dell-atc.lan` | — | ES `:9200`, Kibana `:5601` |
| `atc-docker02.dell-atc.lan` | `10.0.21.47` | Secondary Docker host |
| `atc-mgt01.dell-atc.lan` | `10.0.20.104` | Forgejo Git `:3001`, LDAP/LAM |
| `atc-kafka01.dell-atc.lan` | `10.0.21.36` | Kafka UI `:9000` |
| `atc-lake01.dell-atc.lan` | `10.0.21.50` | Spark `:8080`, Trino `:8089`, Debezium `:8083` |
| `atc-elastic01.dell-atc.lan` | `10.0.21.46` | ES `:9200`, Kibana `:5601` |
| `atc-db01.dell-atc.lan` | `10.0.20.112` | PostgreSQL `5432`, MySQL `3306` |
| `atc-db02.dell-atc.lan` | `10.0.21.51` | PG, MySQL, Mongo `27017`, Cassandra `9042`, Neo4j `7474`/`7687` |
| `atc-grafana.dell-atc.lan` | `10.0.20.103` | Grafana `:3000` |
| `atc-airflow01` | `10.0.21.55` | Airflow `:8080` |
| `atc-portal01.dell-atc.lan` | `10.0.21.49` | Portal |
| `atc-lama01.dell-atc.lan` | `10.0.21.39` | LAMA |
| `pve01` / Proxmox | `10.0.10.65` | Hypervisor `:8006` |
| iDRAC | `10.0.41.102` | Out-of-band management |
| ObjectScale | `10.0.20.111` | Object storage HTTPS |
| iDRAC | `10.0.41.102` | Out-of-band management |
SSH mesh: see [ssh-mesh.md](ssh-mesh.md).
## atc-docker01 — Docker services
| Service | Poort | URL |
|---------|-------|-----|
| Service | Port | URL |
|---------|------|-----|
| Homepage | 80 | http://atc-docker01.dell-atc.lan/ |
| Icons (nginx) | 8080 | http://atc-docker01.dell-atc.lan:8080/ |
| RSS proxy | 8090 | http://atc-docker01.dell-atc.lan:8090/health |
| Superset | 8088 | http://atc-docker01.dell-atc.lan:8088/ |
| Superset Redis | 6379 | intern |
| Forgejo (lokaal) | 4002, 222 | http://atc-docker01.dell-atc.lan:4002/ |
| LAM (lokaal) | 4001 | http://atc-docker01.dell-atc.lan:4001/ |
| cAdvisor | — | optioneel, niet standaard gestart |
| Superset Redis | 6379 | internal |
| Forgejo (local) | 4002, 222 | http://atc-docker01.dell-atc.lan:4002/ |
| LAM (local) | 4001 | http://atc-docker01.dell-atc.lan:4001/ |
| cAdvisor | 8081 | optional |
## Paden op atc-docker01
## Paths on atc-docker01
| Pad | Inhoud |
|-----|--------|
| `/root/lakehouse` | Deze git clone |
| `/root/compose/superset` | Legacy symlink-werkdir (sync met `compose/superset`) |
| `/var/lib/docker/volumes/` | Persistente data (niet in git) |
| Path | Content |
|------|---------|
| `/root/lakehouse` | Git clone |
| `/root/compose/` | Legacy compose dirs (mirror `compose/` in git) |
docs/operations.md
+19 -11
View File
@@ -1,16 +1,16 @@
# Operaties
# Operations
## Config wijzigen en deployen
## Change config and deploy
```bash
cd /root/lakehouse
# Bewerk config/homepage/*.yaml of rss-proxy/server.py
git add -A && git commit -m "beschrijving" && git push origin master
# Edit config/homepage/*.yaml or rss-proxy/server.py
git add -A && git commit -m "description" && git push origin master
# Homepage herladen
# Reload homepage
docker exec homepage wget -qO- http://127.0.0.1:3000/api/revalidate
# RSS proxy rebuild
# Rebuild RSS proxy
docker compose -f deploy/docker-compose.homepage.yml build rss-proxy
docker compose -f deploy/docker-compose.homepage.yml up -d rss-proxy
```
@@ -23,21 +23,29 @@ docker logs -f rss-proxy
docker logs -f superset
```
Homepage applicatielog staat buiten git: `config/homepage/logs/` (genegeerd door `.gitignore`).
Application logs are outside git: `config/homepage/logs/` (see `.gitignore`).
## Container inventory bijwerken
## Refresh container inventory
```bash
./scripts/deploy/export-inventory.sh
git add inventory/ && git commit -m "chore: refresh container inventory"
```
## Stack herstarten
## Restart full docker01 stack
```bash
./scripts/deploy/deploy-atc-docker01.sh
```
## Git op de server
## SSH between hosts
Remote staat op `atc-docker01` in `/root/lakehouse`. Gebruik je eigen credentials voor push naar Forgejo op `atc-mgt01`.
```bash
ssh -i /root/.ssh/atc_cluster root@atc-lake01
```
See [ssh-mesh.md](ssh-mesh.md).
## Git on the server
Clone lives at `/root/lakehouse` on `atc-docker01`. Use your Forgejo credentials to push to `atc-mgt01`.
docs/ssh-mesh.md
+69
View File
@@ -0,0 +1,69 @@
# SSH mesh — passwordless access
All ATC lab VMs share a cluster SSH key so root can hop between hosts without passwords.
## Design
| Item | Path / value |
|------|----------------|
| Shared private key | `/root/.ssh/atc_cluster` (on each VM, **not** in git) |
| Shared public key | `/root/.ssh/atc_cluster.pub` |
| SSH client snippet | `/root/.ssh/config.d/99-atc-lab.conf` |
| Host aliases | `/etc/hosts` block tagged `ATC Lakehouse lab` |
Proxmox (`pve01`) also keeps its admin key in each VM's `authorized_keys` for break-glass access.
## Fleet (mesh enabled)
| Hostname | IP |
|----------|-----|
| atc-docker01 | 10.0.21.45 |
| atc-docker02 | 10.0.21.47 |
| atc-mgt01 | 10.0.20.104 |
| atc-kafka01 | 10.0.21.36 |
| atc-lake01 | 10.0.21.50 |
| atc-elastic01 | 10.0.21.46 |
| atc-db01 | 10.0.20.112 |
| atc-db02 | 10.0.21.51 |
| atc-grafana | 10.0.20.103 |
| atc-airflow01 | 10.0.21.55 |
| atc-portal01 | 10.0.21.49 |
| atc-lama01 | 10.0.21.39 |
## Deploy / refresh
From Proxmox (or any host with root SSH to the fleet):
```bash
cd /root/lakehouse
./scripts/setup/setup-ssh-mesh.sh
```
## Test
```bash
ssh -i ~/.ssh/atc_cluster root@atc-docker01
ssh -i ~/.ssh/atc_cluster root@atc-lake01 hostname
# or from docker01:
for h in atc-db01 atc-kafka01 atc-mgt01; do
ssh -i /root/.ssh/atc_cluster root@$h hostname -f
done
```
## Hosts not yet in the mesh
These VMs did not accept the hypervisor key (different credentials or SSH policy):
- `10.0.21.52`, `10.0.21.37`, `10.0.21.38`, `10.0.21.41`
- `10.0.20.111` (ObjectScale), `10.0.20.31`, `10.0.21.44`
Add the cluster public key manually after fixing root access:
```bash
cat /root/.ssh/atc_cluster.pub # from any mesh host
# paste into target:/root/.ssh/authorized_keys
```
## Security note
The cluster private key is powerful. Restrict Forgejo repo access and rotate keys if the lab is exposed outside your network.