Lakehouse Admin
df5ec93dc3
- Trino catalogs, Grafana, Spark jobs, LDAP LDIF, NPM compose - Airflow DAG scripts, Proxmox VM inventory, network docs - Ansible playbook, Gitea CI validate workflow - Backup and health-check scripts, cron documentation - Homepage DOCS tab with links to all documentation - Extended collect-fleet-config.sh and populate-repo.py
65 lines
1.6 KiB
Markdown
65 lines
1.6 KiB
Markdown
# Network — ATC Lakehouse lab
|
|
|
|
## Subnets
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
subgraph mgmt [10.0.10.0/24 — Management]
|
|
PVE[pve01 · 10.0.10.65]
|
|
IDRAC[iDRAC · 10.0.41.102]
|
|
end
|
|
|
|
subgraph data20 [10.0.20.0/24 — Storage & legacy]
|
|
OS[ObjectScale · 10.0.20.111]
|
|
DB1[atc-db01 · 10.0.20.112]
|
|
MGT[atc-mgt01 · 10.0.20.104]
|
|
GF[atc-grafana · 10.0.20.103]
|
|
end
|
|
|
|
subgraph lake21 [10.0.21.0/24 — Lakehouse compute]
|
|
D1[atc-docker01 · 10.0.21.45]
|
|
D2[atc-docker02 · 10.0.21.47]
|
|
KF[atc-kafka01 · 10.0.21.36]
|
|
LK[atc-lake01 · 10.0.21.50]
|
|
DB2[atc-db02 · 10.0.21.51]
|
|
AF[atc-airflow01 · 10.0.21.55]
|
|
EL[atc-elastic01 · 10.0.21.46]
|
|
end
|
|
|
|
PVE --> D1 & D2 & KF & LK & DB2 & OS & DB1 & MGT
|
|
```
|
|
|
|
## DNS
|
|
|
|
Internal names: `*.dell-atc.lan` (see `config/hosts/atc-lab.hosts`).
|
|
|
|
| Pattern | Example |
|
|
|---------|---------|
|
|
| Short hostname | `atc-lake01` |
|
|
| FQDN | `atc-lake01.dell-atc.lan` |
|
|
|
|
## Key ports (east-west)
|
|
|
|
| Service | Port | Protocol |
|
|
|---------|------|----------|
|
|
| Kafka | 9092 | TCP |
|
|
| Debezium Connect | 8083 | HTTP |
|
|
| Spark UI | 8080 | HTTP |
|
|
| Trino | 8089 | HTTP |
|
|
| PostgreSQL | 5432 | TCP |
|
|
| ObjectScale S3 | 9020 | HTTP |
|
|
| ObjectScale UI | 443 | HTTPS |
|
|
| Homepage | 80 | HTTP |
|
|
|
|
## SSH mesh
|
|
|
|
All lab VMs: `/root/.ssh/atc_cluster` — see [ssh-mesh.md](ssh-mesh.md).
|
|
|
|
ObjectScale uses `admin@` for initial bootstrap; `root@` after key install.
|
|
|
|
## Firewall notes
|
|
|
|
- Lab assumes flat L2/L3 trust within `10.0.10/20/21.x`.
|
|
- ObjectScale `management_clients: 0.0.0.0/0` in deploy.yml — tighten for production.
|
|
- Proxmox API `:8006` reachable from docker01 for homepage widget.
|