diff --git a/README.md b/README.md index 51b3f2a..a806a9d 100644 --- a/README.md +++ b/README.md @@ -1,60 +1,38 @@ # ATC Lakehouse -Infrastructure-as-code and documentation for the Dell ATC Lakehouse lab — Forward Deployed Engineering (Bart & Mo). +Infrastructure-as-code and documentation for the Dell ATC Forward Deployed Engineering lab. + +## Start here + +| Resource | Link | +|----------|------| +| **Live architecture diagram** | http://atc-docker01.dell-atc.lan:8080/docs/architecture.html | +| **Dashboard** | http://atc-docker01.dell-atc.lan/ | +| **Application landscape (doc)** | [docs/landscape.md](docs/landscape.md) | +| **Docker inventory** | [docs/docker-inventory.md](docs/docker-inventory.md) | ## Repository layout ``` -├── config/ # Application config (homepage, kafka, trino, spark, airflow, superset) -├── compose/ # Docker Compose per stack (source of truth for containers) -├── deploy/ # Combined compose files per host +├── config/ # App config: homepage, kafka, docker per host, objectscale, elastic +├── compose/ # Docker Compose stacks +├── deploy/ # Combined deploy compose files ├── docs/ # Documentation as code -├── inventory/ # Exported container state (reference) -├── rss-proxy/ # RSS feed proxy for homepage widgets -└── scripts/ # Deploy and setup scripts +├── inventory/ # Container snapshots +├── rss-proxy/ # RSS proxy for homepage +└── scripts/ # deploy/, setup/, collect/ ``` -## Hosts (summary) +See [docs/landscape.md](docs/landscape.md) for the full application map. -| Host | Role | Key services | -|------|------|----------------| -| `atc-docker01` | Docker hub | Homepage, Superset, Forgejo (local), LAM, RSS-proxy, icons | -| `atc-mgt01` | Management | Forgejo (Git), LDAP/LAM | -| `atc-kafka01` | Streaming | Kafka, Kafka UI | -| `atc-lake01` | Lakehouse | Spark, Trino, Debezium, Airflow | -| `atc-elastic01` | Search | Elasticsearch, Kibana | -| `atc-db01` / `atc-db02` | Databases | PostgreSQL, MySQL, MongoDB, Cassandra, Neo4j | -| `atc-grafana` | Monitoring | Grafana | -| `pve01` (`10.0.10.65`) | Hypervisor | Proxmox VE | - -See [docs/hosts.md](docs/hosts.md) for ports and IPs. -See [docs/ssh-mesh.md](docs/ssh-mesh.md) for passwordless SSH between VMs. - -## Quick start — atc-docker01 +## Quick deploy (atc-docker01) ```bash cd /root/lakehouse -cp .env.example .env # fill in secrets +cp .env.example .env ./scripts/deploy/deploy-atc-docker01.sh ``` -## Documentation - -- [Architecture](docs/architecture.md) -- [Hosts & ports](docs/hosts.md) -- [SSH mesh (passwordless fleet)](docs/ssh-mesh.md) -- [Disaster recovery](docs/disaster-recovery.md) -- [Day-to-day operations](docs/operations.md) -- [Homepage dashboard](config/homepage/README.md) - ## Git remote -Forgejo: `http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse` - -```bash -git clone http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse.git /root/lakehouse -``` - -## Secrets - -Files with credentials (`proxmox.yaml`, `.env`, kafka connectors) are in git for **lab disaster recovery**. For production, use `.env` + `*.example` and keep secrets off public remotes. +http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse diff --git a/config/docker/README.md b/config/docker/README.md new file mode 100644 index 0000000..da65555 --- /dev/null +++ b/config/docker/README.md @@ -0,0 +1,17 @@ +# Docker configurations + +Per-host Compose files and READMEs. These mirror (or document) live deployments. + +| Host | Path | Live location | +|------|------|---------------| +| atc-docker01 | `../compose/`, `../../deploy/` | `/root/lakehouse/compose/` | +| atc-db02 | `atc-db02/docker-compose.yml` | `/opt/sources/docker-compose.yml` | +| atc-lake01 | `atc-lake01/docker-compose.yml` | Docker CLI (no single compose file on host) | +| atc-kafka01 | `atc-kafka01/docker-compose.yml` | Docker CLI | +| atc-mgt01 | `../../compose/forgejo/` | Docker volumes | + +Refresh from fleet: + +```bash +./scripts/collect/collect-fleet-config.sh +``` diff --git a/config/docker/atc-db02/docker-compose.yml b/config/docker/atc-db02/docker-compose.yml new file mode 100644 index 0000000..022c45c --- /dev/null +++ b/config/docker/atc-db02/docker-compose.yml @@ -0,0 +1,65 @@ +version: "3.9" + +services: + + postgres-sales: + image: postgres:15 + container_name: postgres_sales + environment: + POSTGRES_DB: sales + POSTGRES_USER: sales_user + POSTGRES_PASSWORD: sales_pwd + ports: + - "5432:5432" + volumes: + - postgres_sales_data:/var/lib/postgresql/data + + mysql-hr: + image: mysql:8.0 + container_name: mysql_hr + environment: + MYSQL_DATABASE: hr + MYSQL_USER: hr_user + MYSQL_PASSWORD: hr_pwd + MYSQL_ROOT_PASSWORD: root_pwd + ports: + - "3306:3306" + volumes: + - mysql_hr_data:/var/lib/mysql + + mongodb-supplychain: + image: mongo:4.4 + container_name: mongodb_supplychain + command: ["mongod", "--auth"] + ports: + - "27017:27017" + volumes: + - mongodb_supplychain_data:/data/db + + cassandra-telemetry: + image: cassandra:4.1 + container_name: cassandra-telemetry + ports: + - "9042:9042" + volumes: + - cassandra_telemetry_data:/var/lib/cassandra + environment: + CASSANDRA_CLUSTER_NAME: TelemetryCluster + + neo4j-graph: + image: neo4j:4.4 + container_name: neo4j_graph + environment: + NEO4J_AUTH: neo4j/testpwd + ports: + - "7474:7474" + - "7687:7687" + volumes: + - neo4j_graph_data:/data + +volumes: + postgres_sales_data: + mysql_hr_data: + mongodb_supplychain_data: + cassandra_telemetry_data: + neo4j_graph_data: diff --git a/config/docker/atc-kafka01/README.md b/config/docker/atc-kafka01/README.md new file mode 100644 index 0000000..b1e151f --- /dev/null +++ b/config/docker/atc-kafka01/README.md @@ -0,0 +1,7 @@ +# atc-kafka01 + +- **Kafka broker:** `10.0.21.36:9092` (native install under `/opt/kafka`) +- **Kafka UI:** Docker — port `9000` +- **Kafka Connect plugins:** `/opt/kafka-connect` on host + +Debezium Connect runs on **atc-lake01:8083**, not on this host. diff --git a/config/docker/atc-kafka01/docker-compose.yml b/config/docker/atc-kafka01/docker-compose.yml new file mode 100644 index 0000000..26bfba7 --- /dev/null +++ b/config/docker/atc-kafka01/docker-compose.yml @@ -0,0 +1,14 @@ +# Kafka UI — atc-kafka01 (10.0.21.36) +# Kafka broker runs natively on :9092 (see /opt/kafka on host). +version: "3.9" + +services: + kafka-ui: + image: provectuslabs/kafka-ui:latest + container_name: kafka-ui + restart: unless-stopped + ports: + - "9000:8080" + environment: + KAFKA_CLUSTERS_0_NAME: atc-kafka + KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: localhost:9092 diff --git a/config/docker/atc-lake01/docker-compose.yml b/config/docker/atc-lake01/docker-compose.yml new file mode 100644 index 0000000..00c5666 --- /dev/null +++ b/config/docker/atc-lake01/docker-compose.yml @@ -0,0 +1,53 @@ +# Lakehouse compute — atc-lake01 (10.0.21.50) +# Reconstructed from running containers; use docker network for spark worker DNS. +version: "3.9" + +services: + kafka-connect: + image: debezium/connect:2.5.4.Final + container_name: kafka-connect + restart: unless-stopped + ports: + - "8083:8083" + environment: + BOOTSTRAP_SERVERS: 10.0.21.36:9092 + GROUP_ID: "1" + CONFIG_STORAGE_TOPIC: connect-configs + OFFSET_STORAGE_TOPIC: connect-offsets + STATUS_STORAGE_TOPIC: connect-status + + spark-master: + image: bitnamilegacy/spark:latest + container_name: spark-master + restart: unless-stopped + environment: + SPARK_MODE: master + SPARK_RPC_AUTHENTICATION_ENABLED: "no" + ports: + - "7077:7077" + - "8080:8080" + + spark-worker: + image: bitnamilegacy/spark:latest + container_name: spark-worker + restart: unless-stopped + depends_on: + - spark-master + environment: + SPARK_MODE: worker + SPARK_MASTER_URL: spark://spark-master:7077 + SPARK_WORKER_MEMORY: 10G + SPARK_RPC_AUTHENTICATION_ENABLED: "no" + + trino: + image: trinodb/trino:405 + container_name: trino + restart: unless-stopped + ports: + - "8089:8080" + + # Ad-hoc Spark shell image (optional) + spark-temp: + image: apache/spark:3.4.0-scala2.12-java11-python3-ubuntu + container_name: spark-temp + profiles: ["tools"] diff --git a/config/elastic/elasticsearch.yml b/config/elastic/elasticsearch.yml new file mode 100644 index 0000000..afd108c --- /dev/null +++ b/config/elastic/elasticsearch.yml @@ -0,0 +1,21 @@ +cluster.name: atc-lakehouse +node.name: atc-elastic +node.roles: [master, data, ingest, ml, transform] +path.data: /home/elasticsearch/data +path.logs: /home/elasticsearch/logs +network.host: 0.0.0.0 +http.port: 9200 +discovery.type: single-node +bootstrap.memory_lock: false +xpack.security.enabled: true +xpack.security.enrollment.enabled: true +xpack.security.http.ssl.enabled: true +xpack.security.http.ssl.keystore.path: certs/http.p12 +xpack.security.transport.ssl.enabled: true +xpack.security.transport.ssl.keystore.path: certs/transport.p12 +indices.recovery.max_bytes_per_sec: 100mb +indices.queries.cache.size: 10% +indices.fielddata.cache.size: 20% +cluster.routing.allocation.disk.watermark.low: 85% +cluster.routing.allocation.disk.watermark.high: 90% +cluster.routing.allocation.disk.watermark.flood_stage: 95% diff --git a/config/homepage/icons/docs/architecture.html b/config/homepage/icons/docs/architecture.html new file mode 100644 index 0000000..3a21ca9 --- /dev/null +++ b/config/homepage/icons/docs/architecture.html @@ -0,0 +1,195 @@ + + + + + + ATC Lakehouse — Architecture + + + + +
+
+

ATC Lakehouse

+

Forward Deployed Engineering — Dell Technologies · Bart & Mo

+
+ ← Dashboard +
+
+
+ 
+flowchart TB
+  subgraph ops["Operations"]
+    HP["Homepage :80"]
+    GF["Grafana :3000"]
+    PVE["Proxmox :8006"]
+  end
+  subgraph git["Git"]
+    FG["Forgejo :3001"]
+  end
+  subgraph stream["Streaming"]
+    KF["Kafka :9092"]
+    KUI["Kafka UI :9000"]
+    DEB["Debezium :8083"]
+  end
+  subgraph compute["Compute"]
+    SP["Spark :8080"]
+    TR["Trino :8089"]
+    AF["Airflow :8080"]
+  end
+  subgraph data["Sources atc-db02"]
+    PG[(PostgreSQL)]
+    MY[(MySQL)]
+    MG[(MongoDB)]
+    CA[(Cassandra)]
+    N4[(Neo4j)]
+  end
+  subgraph store["Object Storage"]
+    OS["ObjectScale 10.0.20.111"]
+  end
+  subgraph bi["Analytics"]
+    SS["Superset :8088"]
+    ES["Elasticsearch :9200"]
+    KB["Kibana :5601"]
+  end
+  PG & MY & MG & CA --> DEB
+  DEB --> KF
+  KF --> SP
+  SP --> OS
+  TR --> PG
+  TR --> KF
+  HP --> KUI & DEB & SP & TR & OS & FG
+  KF --> SS
+  ES --> KB
+
+
+ +

Service map

+
+
+

Operations

+ +
+
+

Data pipeline

+ +
+
+

Storage

+
    +
  • ObjectScale
    • +
  • atc-db01 / atc-db02 — source databases
    • +
+
+
+

Analytics

+ +
+
+

Git & docs

+ +
+
+
+ + + + diff --git a/config/homepage/services.yaml b/config/homepage/services.yaml index 638f932..81c6ac9 100644 --- a/config/homepage/services.yaml +++ b/config/homepage/services.yaml @@ -1,6 +1,19 @@ --- # ATC Lakehouse — Dell Technologies FDE Dashboard (Bart & Mo) + +- Lakehouse · Architecture: + - Environment Map: + icon: mdi-sitemap + href: http://atc-docker01.dell-atc.lan:8080/docs/architecture.html + description: High-level diagram — data flow, hosts, and service map + color: "#007DB8" + - Git Docs: + icon: mdi-book-open-page-variant + href: http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse/src/branch/master/docs/landscape.md + description: Application landscape (Markdown in Forgejo) + color: "#E8752A" + - Data Pipeline: - Kafka UI: icon: http://atc-docker01.dell-atc.lan:8080/apachekafka.svg diff --git a/config/homepage/settings.yaml b/config/homepage/settings.yaml index 780f943..7070df6 100644 --- a/config/homepage/settings.yaml +++ b/config/homepage/settings.yaml @@ -15,6 +15,10 @@ background: brightness: 0 saturate: 0 layout: + Lakehouse · Architecture: + tab: OPS + style: row + columns: 2 Data Pipeline: tab: OPS style: row diff --git a/config/objectscale/README.md b/config/objectscale/README.md new file mode 100644 index 0000000..c98a2a0 --- /dev/null +++ b/config/objectscale/README.md @@ -0,0 +1,29 @@ +# Dell ObjectScale (ECS) + +| Item | Value | +|------|-------| +| Hostname | `luna.local` / `atc-objectscale` | +| IP | `10.0.20.111` | +| SSH user | `admin` (root enabled after key bootstrap) | +| Web UI | https://10.0.20.111/ | +| S3 API | ECS default endpoints (see ECS admin UI) | + +## Software + +- **Luna OS** — CentOS 7-based appliance image +- **ECS container:** `emccorp/ecs-software:latest` (`ecs-storageos`) +- **Install config:** `/opt/emc/ecs-install/deploy.yml` on the appliance + +## Git copy + +`deploy.yml` in this folder is a reference copy. Passwords are redacted — use vault or host file for live secrets. + +## Bootstrap SSH + +```bash +./scripts/setup/setup-objectscale-ssh.sh +``` + +## Storage + +Single-node lab deployment; block device `/dev/sdb` in storage pool `sp1` per deploy.yml. diff --git a/config/objectscale/deploy.yml b/config/objectscale/deploy.yml new file mode 100644 index 0000000..9e7a742 --- /dev/null +++ b/config/objectscale/deploy.yml @@ -0,0 +1,225 @@ +# deploy.yml reference implementation v2.8.0 + +# [Optional] +# By changing the license_accepted boolean value to "true" you are +# declaring your agreement to the terms of the license agreement +# contained in the license.txt file included with this software +# distribution. +licensing: + license_accepted: true + +#autonames: +# custom: +# - ecs01 +# - ecs02 +# - ecs03 +# - ecs04 +# - ecs05 +# - ecs06 + +# [Required] +# Deployment facts reference +facts: + + # [Required] + # Node IP or resolvable hostname from which installations will be launched + # The only supported configuration is to install from the same node as the + # bootstrap.sh script is run. + install_node: 10.0.20.111 + + # [Required] + # IPs of machines that will be whitelisted in the firewall and allowed + # to access management ports of all nodes. If this is set to the + # wildcard (0.0.0.0/0) then anyone can access management ports. + management_clients: + - 0.0.0.0/0 + + # [Required] + # These credentials must be the same across all nodes. Ansible uses these credentials to + # gain initial access to each node in the deployment and set up ssh public key authentication. + # If these are not correct, the deployment will fail. + ssh_defaults: + # [Required] + # Username to use when logging in to nodes + ssh_username: admin + # [Required] + # Password to use with SSH login + # *** Set to same value as ssh_username to enable SSH public key authentication *** + ssh_password: "REDACTED" + # [Required when enabling SSH public key authentication] + # Password to give to sudo when gaining root access. + ansible_become_pass: "REDACTED" + # [Required] + # Select the type of crypto to use when dealing with ssh public key + # authentication. Valid values here are: + # - "rsa" (Default) + # - "ed25519" + ssh_crypto: rsa + + # [Required] + # Environment configuration for this deployment. + node_defaults: + dns_domain: local + dns_servers: + - 10.0.20.1 + ntp_servers: + - 10.0.20.1 + # + # [Optional] + # VFS path to source of randomness + entropy_source: /dev/urandom + # + # [Optional] + # autonaming: custom + + # + # [Optional] + # If your ECS comes with differing default credentials, you can specify those here + # ecs_root_user: root + # ecs_root_pass: ChangeMe + + # [Optional] + # Storage pool defaults. Configure to your liking. + # All block devices that will be consumed by ECS on ALL nodes must be listed under the + # ecs_block_devices option. This can be overridden by the storage pool configuration. + # At least ONE (1) block device is REQUIRED for a successful install. More is better. + storage_pool_defaults: + is_cold_storage_enabled: false + is_protected: false + description: Default storage pool description + ecs_block_devices: + - /dev/sdb + + # [Required] + # Storage pool layout. You MUST have at least ONE (1) storage pool for a successful install. + storage_pools: + - name: sp1 + members: + - 10.0.20.111 + options: + is_protected: false + is_cold_storage_enabled: false + description: My First SP + ecs_block_devices: + - /dev/sdb + + # [Optional] + # VDC defaults. Configure to your liking. + virtual_data_center_defaults: + description: Default virtual data center description + + # [Required] + # Virtual data center layout. You MUST have at least ONE (1) VDC for a successful install. + # Multi-VDC deployments are not yet implemented + virtual_data_centers: + - name: vdc1 + members: + - sp1 + options: + description: My First VDC + + # [Optional] + # Replication group defaults. Configure to your liking. + replication_group_defaults: + description: Default replication group description + enable_rebalancing: true + allow_all_namespaces: true + is_full_rep: false + + # [Optional, required for namespaces] + # Replication group layout. You MUST have at least ONE (1) RG to provision namespaces. + replication_groups: + - name: rg1 + members: + - vdc1 + options: + description: My First RG + enable_rebalancing: true + allow_all_namespaces: true + is_full_rep: false + + # [Optional] + # Management User defaults + management_user_defaults: + is_system_admin: false + is_system_monitor: false + + # [Optional] + # Management Users + management_users: + - username: admin1 + password: ChangeMe + options: + is_system_admin: true + - username: monitor1 + password: ChangeMe + options: + is_system_monitor: true + + # [Optional] + # Namespace defaults + namespace_defaults: + is_stale_allowed: false + is_compliance_enabled: false + + # [Optional] + # Namespace layout + namespaces: + - name: ns1 + replication_group: rg1 + administrators: + - root + options: + is_stale_allowed: false + is_compliance_enabled: false + + # [Optional] + # Object User defaults + object_user_defaults: + # Comma-separated list of Swift authorization groups + swift_groups_list: + - users + # Lifetime of S3 secret key in minutes + s3_expiry_time: 2592000 + + # [Optional] + # Object Users + object_users: + - username: object_admin1 + namespace: ns1 + options: + swift_password: ChangeMe + swift_groups_list: + - admin + - users + s3_secret_key: ChangeMeChangeMeChangeMeChangeMeChangeMe + s3_expiry_time: 2592000 + - username: object_user1 + namespace: ns1 + options: + swift_password: ChangeMe + s3_secret_key: ChangeMeChangeMeChangeMeChangeMeChangeMe + + # [Optional] + # Bucket defaults + bucket_defaults: + namespace: ns1 + replication_group: rg1 + head_type: s3 + filesystem_enabled: False + stale_allowed: False + encryption_enabled: False + owner: object_admin1 + + # [Optional] + # Bucket layout (optional) + buckets: + - name: bucket1 + options: + namespace: ns1 + replication_group: rg1 + owner: object_admin1 + head_type: s3 + filesystem_enabled: False + stale_allowed: False + encryption_enabled: False \ No newline at end of file diff --git a/docs/docker-inventory.md b/docs/docker-inventory.md new file mode 100644 index 0000000..c8625bc --- /dev/null +++ b/docs/docker-inventory.md @@ -0,0 +1,65 @@ +# Docker inventory + +Containers per host (reference). Regenerate with `./scripts/collect/collect-fleet-config.sh`. + +## atc-docker01 (10.0.21.45) + +| Container | Image | Ports | +|-----------|-------|-------| +| homepage | gethomepage/homepage | 80→3000 | +| rss-proxy | rss-proxy:latest | 8090 | +| icons-server | nginx:alpine | 8080→80 | +| superset | superset-superset | 8088 | +| superset_redis | redis:7 | 6379 | +| forgejo | forgejo:14 | 4002, 222 | +| lam-app-1 | ldapaccountmanager/lam | 4001 | + +Compose: `compose/`, `deploy/docker-compose.homepage.yml` + +## atc-db02 (10.0.21.51) + +| Container | Image | Ports | +|-----------|-------|-------| +| postgres_sales | postgres:15 | 5432 | +| mysql_hr | mysql:8.0 | 3306 | +| mongodb_supplychain | mongo:4.4 | 27017 | +| cassandra_telemetry | cassandra:4.1 | 9042 | +| neo4j_graph | neo4j:4.4 | 7474, 7687 | +| mongo_express | mongo-express | 8081 | + +Compose: `config/docker/atc-db02/docker-compose.yml` + +## atc-lake01 (10.0.21.50) + +| Container | Image | Ports | +|-----------|-------|-------| +| kafka-connect | debezium/connect:2.5.4 | 8083 | +| spark-master | bitnami/spark | 7077, 8080 | +| spark-worker | bitnami/spark | (internal) | +| trino | trinodb/trino:405 | 8089→8080 | +| spark-temp | apache/spark:3.4.0 | — | + +Compose: `config/docker/atc-lake01/docker-compose.yml` + +## atc-kafka01 (10.0.21.36) + +| Container | Image | Ports | +|-----------|-------|-------| +| kafka-ui | provectuslabs/kafka-ui | 9000→8080 | + +Kafka broker: native on `:9092` (not containerized in current lab). + +## atc-mgt01 (10.0.20.104) + +| Container | Image | Ports | +|-----------|-------|-------| +| forgejo | forgejo:14 | 3001, 222 | +| npm-app-1 | nginx-proxy-manager | 80, 443, 81 | + +## atc-objectscale (10.0.20.111) + +| Container | Image | Notes | +|-----------|-------|-------| +| ecs-storageos | emccorp/ecs-software | Dell ECS / ObjectScale | + +Config: `config/objectscale/deploy.yml` diff --git a/docs/landscape.md b/docs/landscape.md new file mode 100644 index 0000000..ce46687 --- /dev/null +++ b/docs/landscape.md @@ -0,0 +1,129 @@ +# ATC Lakehouse — Application Landscape + +High-level view of the Dell ATC Forward Deployed Engineering lab. +**Live diagram:** http://atc-docker01.dell-atc.lan:8080/docs/architecture.html + +## Architecture diagram + +```mermaid +flowchart TB + subgraph users [Users] + FDE[FDE Engineers] + end + + subgraph ops [Operations Layer] + HP[Homepage Dashboard
atc-docker01:80] + GF[Grafana
atc-grafana:3000] + PVE[Proxmox pve01
10.0.10.65:8006] + IDRAC[iDRAC
10.0.41.102] + end + + subgraph git [Source Control] + FG[Forgejo Git
atc-mgt01:3001] + end + + subgraph ingest [Ingest & Streaming] + KF[Kafka
atc-kafka01:9092] + KUI[Kafka UI
:9000] + DEB[Debezium Connect
atc-lake01:8083] + end + + subgraph process [Processing] + SP[Spark Master/Worker
atc-lake01:8080] + AF[Airflow
atc-airflow01:8080] + TR[Trino
atc-lake01:8089] + end + + subgraph storage [Storage] + OS[ObjectScale ECS
atc-objectscale 10.0.20.111] + end + + subgraph sources [Source Databases — atc-db02] + PG[(PostgreSQL)] + MY[(MySQL)] + MG[(MongoDB)] + CA[(Cassandra)] + N4[(Neo4j)] + end + + subgraph analytics [Analytics & Search] + SS[Superset
atc-docker01:8088] + ES[Elasticsearch
atc-elastic01:9200] + KB[Kibana
:5601] + end + + FDE --> HP + FDE --> SS + FDE --> KB + HP --> KUI + HP --> DEB + HP --> SP + HP --> TR + HP --> OS + HP --> FG + + PG & MY & MG & CA --> DEB + DEB --> KF + KF --> SP + SP --> OS + TR --> PG + TR --> KF + KF --> SS + ES --> KB + AF --> SP +``` + +## Host roles + +| Host | IP | Role | Key services | +|------|-----|------|--------------| +| **atc-docker01** | 10.0.21.45 | Ops & BI hub | Homepage, Superset, RSS-proxy, icons, Forgejo (local), LAM | +| **atc-docker02** | 10.0.21.47 | Secondary Docker | Standby / experiments | +| **atc-mgt01** | 10.0.20.104 | Management | Forgejo (:3001), Nginx Proxy Manager, LDAP | +| **atc-kafka01** | 10.0.21.36 | Streaming | Kafka broker (:9092), Kafka UI (:9000) | +| **atc-lake01** | 10.0.21.50 | Lakehouse compute | Spark, Trino, Debezium Connect | +| **atc-airflow01** | 10.0.21.55 | Orchestration | Apache Airflow (systemd) | +| **atc-elastic01** | 10.0.21.46 | Search & observability | Elasticsearch, Kibana | +| **atc-db01** | 10.0.20.112 | Source DBs | PostgreSQL, MySQL (native or container) | +| **atc-db02** | 10.0.21.51 | Polyglot sources | PG, MySQL, Mongo, Cassandra, Neo4j (Docker) | +| **atc-grafana** | 10.0.20.103 | Monitoring | Grafana | +| **atc-objectscale** | 10.0.20.111 | Object storage | Dell ECS / S3 API (Luna OS) | +| **pve01** | 10.0.10.65 | Hypervisor | Proxmox VE | + +## Data flow + +1. **Operational data** lives on **atc-db02** (and db01) — sales, HR, supply chain, telemetry, graph. +2. **Debezium** on lake01 reads CDC from PostgreSQL → **Kafka** on kafka01. +3. **Spark** processes streams/batch; results can land in **ObjectScale** (S3-compatible). +4. **Trino** federates queries across catalogs (Postgres, Kafka, Hive, etc.). +5. **Superset** on docker01 is the primary BI UI; **Kibana** for search/logs. +6. **Airflow** orchestrates pipelines on airflow01. +7. **Homepage** aggregates links, health, RSS intel, and Proxmox status. + +## Docker vs native + +| Component | Deployment | Config in git | +|-----------|------------|---------------| +| Homepage, Superset, RSS | Docker (docker01) | `compose/`, `deploy/` | +| Spark, Trino, Debezium | Docker (lake01) | `config/docker/atc-lake01/` | +| Source DBs | Docker (db02) | `config/docker/atc-db02/` | +| Kafka UI | Docker (kafka01) | `config/docker/atc-kafka01/` | +| Kafka broker | Native/systemd (kafka01) | `config/kafka/` | +| Elasticsearch | Native (elastic01) | `config/elastic/` | +| Airflow | Native (airflow01) | `config/airflow/` | +| ObjectScale | ECS appliance | `config/objectscale/` | +| Forgejo | Docker (mgt01) | `compose/forgejo/` | + +## Network zones + +- **Management:** 10.0.10.x — Proxmox, iDRAC +- **Storage / legacy:** 10.0.20.x — ObjectScale, db01, mgt01, grafana, elastic +- **Lakehouse compute:** 10.0.21.x — docker, kafka, lake, db02, airflow + +## Related docs + +- [Hosts & ports](hosts.md) +- [Architecture layers](architecture.md) +- [SSH mesh](ssh-mesh.md) +- [Disaster recovery](disaster-recovery.md) +- [Docker inventory](docker-inventory.md) diff --git a/scripts/collect/collect-fleet-config.sh b/scripts/collect/collect-fleet-config.sh new file mode 100755 index 0000000..4106145 --- /dev/null +++ b/scripts/collect/collect-fleet-config.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# Pull live Docker/ObjectScale configs into the Lakehouse git repo. +# Run on atc-docker01 as root (requires SSH mesh). +set -euo pipefail + +REPO="${REPO:-/root/lakehouse}" +KEY="${KEY:-/root/.ssh/atc_cluster}" +SSH=(ssh -i "$KEY" -o StrictHostKeyChecking=no -o ConnectTimeout=10) + +echo "==> db02 docker-compose" +"${SSH[@]}" root@atc-db02 "cat /opt/sources/docker-compose.yml" > "$REPO/config/docker/atc-db02/docker-compose.yml" + +echo "==> objectscale deploy.yml (redacted)" +"${SSH[@]}" root@atc-objectscale "cat /opt/emc/ecs-install/deploy.yml" | \ + sed -E 's/(ssh_password|ansible_become_pass):.*/\1: "REDACTED"/' \ + > "$REPO/config/objectscale/deploy.yml" + +echo "==> elasticsearch.yml" +"${SSH[@]}" root@atc-elastic01 "cat /etc/elasticsearch/elasticsearch.yml" > "$REPO/config/elastic/elasticsearch.yml" + +echo "==> container inventory" +"$REPO/scripts/deploy/export-inventory.py" "$REPO/inventory/containers-atc-docker01.json" + +echo "Done. Review and commit: cd $REPO && git diff"