Document full application landscape and collect Docker configs

- Add docs/landscape.md and docker-inventory.md with Mermaid architecture - Add config/docker per host (db02, lake01, kafka01) and objectscale/elastic refs - Add live architecture page at icons/docs/architecture.html - Homepage OPS tab: Lakehouse Architecture group with diagram link - Add scripts/collect/collect-fleet-config.sh to refresh configs from fleet
2026-05-19 23:01:47 +02:00
parent e1180e14c9
commit 007f6cedc5
15 changed files with 881 additions and 42 deletions
@@ -1,60 +1,38 @@
 # ATC Lakehouse

-Infrastructure-as-code and documentation for the Dell ATC Lakehouse lab — Forward Deployed Engineering (Bart & Mo).
+Infrastructure-as-code and documentation for the Dell ATC Forward Deployed Engineering lab.
+
+## Start here
+
+| Resource | Link |
+|----------|------|
+| **Live architecture diagram** | http://atc-docker01.dell-atc.lan:8080/docs/architecture.html |
+| **Dashboard** | http://atc-docker01.dell-atc.lan/ |
+| **Application landscape (doc)** | [docs/landscape.md](docs/landscape.md) |
+| **Docker inventory** | [docs/docker-inventory.md](docs/docker-inventory.md) |

 ## Repository layout

 ```
-├── config/           # Application config (homepage, kafka, trino, spark, airflow, superset)
-├── compose/          # Docker Compose per stack (source of truth for containers)
-├── deploy/           # Combined compose files per host
+├── config/           # App config: homepage, kafka, docker per host, objectscale, elastic
+├── compose/          # Docker Compose stacks
+├── deploy/           # Combined deploy compose files
 ├── docs/             # Documentation as code
-├── inventory/        # Exported container state (reference)
-├── rss-proxy/        # RSS feed proxy for homepage widgets
-└── scripts/          # Deploy and setup scripts
+├── inventory/        # Container snapshots
+├── rss-proxy/        # RSS proxy for homepage
+└── scripts/          # deploy/, setup/, collect/
 ```

-## Hosts (summary)
+See [docs/landscape.md](docs/landscape.md) for the full application map.

-| Host | Role | Key services |
-|------|------|----------------|
-| `atc-docker01` | Docker hub | Homepage, Superset, Forgejo (local), LAM, RSS-proxy, icons |
-| `atc-mgt01` | Management | Forgejo (Git), LDAP/LAM |
-| `atc-kafka01` | Streaming | Kafka, Kafka UI |
-| `atc-lake01` | Lakehouse | Spark, Trino, Debezium, Airflow |
-| `atc-elastic01` | Search | Elasticsearch, Kibana |
-| `atc-db01` / `atc-db02` | Databases | PostgreSQL, MySQL, MongoDB, Cassandra, Neo4j |
-| `atc-grafana` | Monitoring | Grafana |
-| `pve01` (`10.0.10.65`) | Hypervisor | Proxmox VE |
-
-See [docs/hosts.md](docs/hosts.md) for ports and IPs.  
-See [docs/ssh-mesh.md](docs/ssh-mesh.md) for passwordless SSH between VMs.
-
-## Quick start — atc-docker01
+## Quick deploy (atc-docker01)

 ```bash
 cd /root/lakehouse
-cp .env.example .env   # fill in secrets
+cp .env.example .env
 ./scripts/deploy/deploy-atc-docker01.sh
 ```

-## Documentation
-
- [Architecture](docs/architecture.md)
- [Hosts & ports](docs/hosts.md)
- [SSH mesh (passwordless fleet)](docs/ssh-mesh.md)
- [Disaster recovery](docs/disaster-recovery.md)
- [Day-to-day operations](docs/operations.md)
- [Homepage dashboard](config/homepage/README.md)
-
 ## Git remote

-Forgejo: `http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse`
-
-```bash
-git clone http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse.git /root/lakehouse
-```
-
-## Secrets
-
-Files with credentials (`proxmox.yaml`, `.env`, kafka connectors) are in git for **lab disaster recovery**. For production, use `.env` + `*.example` and keep secrets off public remotes.
+http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse
@@ -0,0 +1,17 @@
+# Docker configurations
+
+Per-host Compose files and READMEs. These mirror (or document) live deployments.
+
+| Host | Path | Live location |
+|------|------|---------------|
+| atc-docker01 | `../compose/`, `../../deploy/` | `/root/lakehouse/compose/` |
+| atc-db02 | `atc-db02/docker-compose.yml` | `/opt/sources/docker-compose.yml` |
+| atc-lake01 | `atc-lake01/docker-compose.yml` | Docker CLI (no single compose file on host) |
+| atc-kafka01 | `atc-kafka01/docker-compose.yml` | Docker CLI |
+| atc-mgt01 | `../../compose/forgejo/` | Docker volumes |
+
+Refresh from fleet:
+
+```bash
+./scripts/collect/collect-fleet-config.sh
+```
@@ -0,0 +1,65 @@
+version: "3.9"
+
+services:
+
+  postgres-sales:
+    image: postgres:15
+    container_name: postgres_sales
+    environment:
+      POSTGRES_DB: sales
+      POSTGRES_USER: sales_user
+      POSTGRES_PASSWORD: sales_pwd
+    ports:
+      - "5432:5432"
+    volumes:
+      - postgres_sales_data:/var/lib/postgresql/data
+
+  mysql-hr:
+    image: mysql:8.0
+    container_name: mysql_hr
+    environment:
+      MYSQL_DATABASE: hr
+      MYSQL_USER: hr_user
+      MYSQL_PASSWORD: hr_pwd
+      MYSQL_ROOT_PASSWORD: root_pwd
+    ports:
+      - "3306:3306"
+    volumes:
+      - mysql_hr_data:/var/lib/mysql
+
+  mongodb-supplychain:
+    image: mongo:4.4
+    container_name: mongodb_supplychain
+    command: ["mongod", "--auth"]
+    ports:
+      - "27017:27017"
+    volumes:
+      - mongodb_supplychain_data:/data/db
+
+  cassandra-telemetry:
+    image: cassandra:4.1
+    container_name: cassandra-telemetry
+    ports:
+      - "9042:9042"
+    volumes:
+      - cassandra_telemetry_data:/var/lib/cassandra
+    environment:
+      CASSANDRA_CLUSTER_NAME: TelemetryCluster
+
+  neo4j-graph:
+    image: neo4j:4.4
+    container_name: neo4j_graph
+    environment:
+      NEO4J_AUTH: neo4j/testpwd
+    ports:
+      - "7474:7474"
+      - "7687:7687"
+    volumes:
+      - neo4j_graph_data:/data
+
+volumes:
+  postgres_sales_data:
+  mysql_hr_data:
+  mongodb_supplychain_data:
+  cassandra_telemetry_data:
+  neo4j_graph_data:
@@ -0,0 +1,7 @@
+# atc-kafka01
+
+- **Kafka broker:** `10.0.21.36:9092` (native install under `/opt/kafka`)
+- **Kafka UI:** Docker — port `9000`
+- **Kafka Connect plugins:** `/opt/kafka-connect` on host
+
+Debezium Connect runs on **atc-lake01:8083**, not on this host.
@@ -0,0 +1,14 @@
+# Kafka UI — atc-kafka01 (10.0.21.36)
+# Kafka broker runs natively on :9092 (see /opt/kafka on host).
+version: "3.9"
+
+services:
+  kafka-ui:
+    image: provectuslabs/kafka-ui:latest
+    container_name: kafka-ui
+    restart: unless-stopped
+    ports:
+      - "9000:8080"
+    environment:
+      KAFKA_CLUSTERS_0_NAME: atc-kafka
+      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: localhost:9092
@@ -0,0 +1,53 @@
+# Lakehouse compute — atc-lake01 (10.0.21.50)
+# Reconstructed from running containers; use docker network for spark worker DNS.
+version: "3.9"
+
+services:
+  kafka-connect:
+    image: debezium/connect:2.5.4.Final
+    container_name: kafka-connect
+    restart: unless-stopped
+    ports:
+      - "8083:8083"
+    environment:
+      BOOTSTRAP_SERVERS: 10.0.21.36:9092
+      GROUP_ID: "1"
+      CONFIG_STORAGE_TOPIC: connect-configs
+      OFFSET_STORAGE_TOPIC: connect-offsets
+      STATUS_STORAGE_TOPIC: connect-status
+
+  spark-master:
+    image: bitnamilegacy/spark:latest
+    container_name: spark-master
+    restart: unless-stopped
+    environment:
+      SPARK_MODE: master
+      SPARK_RPC_AUTHENTICATION_ENABLED: "no"
+    ports:
+      - "7077:7077"
+      - "8080:8080"
+
+  spark-worker:
+    image: bitnamilegacy/spark:latest
+    container_name: spark-worker
+    restart: unless-stopped
+    depends_on:
+      - spark-master
+    environment:
+      SPARK_MODE: worker
+      SPARK_MASTER_URL: spark://spark-master:7077
+      SPARK_WORKER_MEMORY: 10G
+      SPARK_RPC_AUTHENTICATION_ENABLED: "no"
+
+  trino:
+    image: trinodb/trino:405
+    container_name: trino
+    restart: unless-stopped
+    ports:
+      - "8089:8080"
+
+  # Ad-hoc Spark shell image (optional)
+  spark-temp:
+    image: apache/spark:3.4.0-scala2.12-java11-python3-ubuntu
+    container_name: spark-temp
+    profiles: ["tools"]
@@ -0,0 +1,21 @@
+cluster.name: atc-lakehouse
+node.name: atc-elastic
+node.roles: [master, data, ingest, ml, transform]
+path.data: /home/elasticsearch/data
+path.logs: /home/elasticsearch/logs
+network.host: 0.0.0.0
+http.port: 9200
+discovery.type: single-node
+bootstrap.memory_lock: false
+xpack.security.enabled: true
+xpack.security.enrollment.enabled: true
+xpack.security.http.ssl.enabled: true
+xpack.security.http.ssl.keystore.path: certs/http.p12
+xpack.security.transport.ssl.enabled: true
+xpack.security.transport.ssl.keystore.path: certs/transport.p12
+indices.recovery.max_bytes_per_sec: 100mb
+indices.queries.cache.size: 10%
+indices.fielddata.cache.size: 20%
+cluster.routing.allocation.disk.watermark.low: 85%
+cluster.routing.allocation.disk.watermark.high: 90%
+cluster.routing.allocation.disk.watermark.flood_stage: 95%
@@ -0,0 +1,195 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="utf-8" />
+  <meta name="viewport" content="width=device-width, initial-scale=1" />
+  <title>ATC Lakehouse — Architecture</title>
+  <script src="https://cdn.jsdelivr.net/npm/mermaid@10/dist/mermaid.min.js"></script>
+  <style>
+    :root {
+      --bg: #0a0e17;
+      --panel: #111827;
+      --border: #1e3a5f;
+      --accent: #007db8;
+      --orange: #e8752a;
+      --text: #e2e8f0;
+      --muted: #94a3b8;
+    }
+    * { box-sizing: border-box; }
+    body {
+      margin: 0;
+      font-family: "DM Sans", system-ui, sans-serif;
+      background: var(--bg);
+      color: var(--text);
+      min-height: 100vh;
+      background-image:
+        linear-gradient(rgba(0,125,184,0.04) 1px, transparent 1px),
+        linear-gradient(90deg, rgba(0,125,184,0.04) 1px, transparent 1px);
+      background-size: 48px 48px;
+    }
+    header {
+      border-bottom: 1px solid var(--border);
+      padding: 1.25rem 2rem;
+      display: flex;
+      align-items: center;
+      gap: 1rem;
+      background: rgba(10,14,23,0.95);
+      position: sticky;
+      top: 0;
+      z-index: 10;
+    }
+    header h1 {
+      margin: 0;
+      font-size: 1.35rem;
+      font-weight: 600;
+      letter-spacing: 0.02em;
+    }
+    header h1 span { color: var(--orange); }
+    header p { margin: 0.2rem 0 0; font-size: 0.85rem; color: var(--muted); }
+    .back {
+      margin-left: auto;
+      color: var(--accent);
+      text-decoration: none;
+      font-size: 0.9rem;
+      padding: 0.5rem 1rem;
+      border: 1px solid var(--border);
+      border-radius: 6px;
+    }
+    .back:hover { background: var(--panel); }
+    main { max-width: 1400px; margin: 0 auto; padding: 2rem; }
+    .diagram {
+      background: var(--panel);
+      border: 1px solid var(--border);
+      border-radius: 12px;
+      padding: 1.5rem;
+      margin-bottom: 2rem;
+      overflow-x: auto;
+    }
+    .grid {
+      display: grid;
+      grid-template-columns: repeat(auto-fill, minmax(280px, 1fr));
+      gap: 1rem;
+    }
+    .card {
+      background: var(--panel);
+      border: 1px solid var(--border);
+      border-left: 3px solid var(--accent);
+      border-radius: 8px;
+      padding: 1rem 1.25rem;
+    }
+    .card h3 { margin: 0 0 0.5rem; font-size: 1rem; color: var(--orange); }
+    .card ul { margin: 0; padding-left: 1.1rem; font-size: 0.85rem; color: var(--muted); }
+    .card a { color: #38bdf8; text-decoration: none; }
+    .card a:hover { text-decoration: underline; }
+    footer { text-align: center; padding: 2rem; color: var(--muted); font-size: 0.8rem; }
+    .mermaid { background: transparent !important; }
+  </style>
+</head>
+<body>
+  <header>
+    <div>
+      <h1>ATC <span>Lakehouse</span></h1>
+      <p>Forward Deployed Engineering — Dell Technologies · Bart &amp; Mo</p>
+    </div>
+    <a class="back" href="http://atc-docker01.dell-atc.lan/">← Dashboard</a>
+  </header>
+  <main>
+    <section class="diagram">
+      <pre class="mermaid">
+flowchart TB
+  subgraph ops["Operations"]
+    HP["Homepage :80"]
+    GF["Grafana :3000"]
+    PVE["Proxmox :8006"]
+  end
+  subgraph git["Git"]
+    FG["Forgejo :3001"]
+  end
+  subgraph stream["Streaming"]
+    KF["Kafka :9092"]
+    KUI["Kafka UI :9000"]
+    DEB["Debezium :8083"]
+  end
+  subgraph compute["Compute"]
+    SP["Spark :8080"]
+    TR["Trino :8089"]
+    AF["Airflow :8080"]
+  end
+  subgraph data["Sources atc-db02"]
+    PG[(PostgreSQL)]
+    MY[(MySQL)]
+    MG[(MongoDB)]
+    CA[(Cassandra)]
+    N4[(Neo4j)]
+  end
+  subgraph store["Object Storage"]
+    OS["ObjectScale 10.0.20.111"]
+  end
+  subgraph bi["Analytics"]
+    SS["Superset :8088"]
+    ES["Elasticsearch :9200"]
+    KB["Kibana :5601"]
+  end
+  PG & MY & MG & CA --> DEB
+  DEB --> KF
+  KF --> SP
+  SP --> OS
+  TR --> PG
+  TR --> KF
+  HP --> KUI & DEB & SP & TR & OS & FG
+  KF --> SS
+  ES --> KB
+      </pre>
+    </section>
+
+    <h2 style="font-size:1.1rem;margin-bottom:1rem;color:var(--muted);">Service map</h2>
+    <div class="grid">
+      <div class="card">
+        <h3>Operations</h3>
+        <ul>
+          <li><a href="http://atc-docker01.dell-atc.lan/">Homepage</a> — docker01</li>
+          <li><a href="http://atc-grafana.dell-atc.lan:3000/">Grafana</a></li>
+          <li><a href="https://10.0.10.65:8006/">Proxmox</a></li>
+        </ul>
+      </div>
+      <div class="card">
+        <h3>Data pipeline</h3>
+        <ul>
+          <li><a href="http://atc-kafka01.dell-atc.lan:9000/">Kafka UI</a></li>
+          <li><a href="http://atc-lake01.dell-atc.lan:8083/">Debezium Connect</a></li>
+          <li><a href="http://atc-lake01.dell-atc.lan:8080/">Spark</a></li>
+          <li><a href="http://atc-lake01.dell-atc.lan:8089/ui/">Trino</a></li>
+          <li><a href="http://10.0.21.55:8080/">Airflow</a></li>
+        </ul>
+      </div>
+      <div class="card">
+        <h3>Storage</h3>
+        <ul>
+          <li><a href="https://10.0.20.111/">ObjectScale</a></li>
+          <li>atc-db01 / atc-db02 — source databases</li>
+        </ul>
+      </div>
+      <div class="card">
+        <h3>Analytics</h3>
+        <ul>
+          <li><a href="http://atc-docker01.dell-atc.lan:8088/">Superset</a></li>
+          <li><a href="http://atc-elastic01.dell-atc.lan:5601/">Kibana</a></li>
+        </ul>
+      </div>
+      <div class="card">
+        <h3>Git &amp; docs</h3>
+        <ul>
+          <li><a href="http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse">Forgejo — Lakehouse repo</a></li>
+          <li>Docs: <code>docs/landscape.md</code> in git</li>
+        </ul>
+      </div>
+    </div>
+  </main>
+  <footer>
+    ATC Lakehouse lab · Config as code in Forgejo · Diagram auto-rendered with Mermaid
+  </footer>
+  <script>
+    mermaid.initialize({ startOnLoad: true, theme: 'dark', securityLevel: 'loose' });
+  </script>
+</body>
+</html>
@@ -1,6 +1,19 @@
 ---
 # ATC Lakehouse — Dell Technologies FDE Dashboard (Bart & Mo)

+
+- Lakehouse · Architecture:
+    - Environment Map:
+        icon: mdi-sitemap
+        href: http://atc-docker01.dell-atc.lan:8080/docs/architecture.html
+        description: High-level diagram — data flow, hosts, and service map
+        color: "#007DB8"
+    - Git Docs:
+        icon: mdi-book-open-page-variant
+        href: http://atc-mgt01.dell-atc.lan:3001/mo/Lakehouse/src/branch/master/docs/landscape.md
+        description: Application landscape (Markdown in Forgejo)
+        color: "#E8752A"
+
 - Data Pipeline:
    - Kafka UI:
        icon: http://atc-docker01.dell-atc.lan:8080/apachekafka.svg
@@ -15,6 +15,10 @@ background:
  brightness: 0
  saturate: 0
 layout:
+  Lakehouse · Architecture:
+    tab: OPS
+    style: row
+    columns: 2
  Data Pipeline:
    tab: OPS
    style: row
@@ -0,0 +1,29 @@
+# Dell ObjectScale (ECS)
+
+| Item | Value |
+|------|-------|
+| Hostname | `luna.local` / `atc-objectscale` |
+| IP | `10.0.20.111` |
+| SSH user | `admin` (root enabled after key bootstrap) |
+| Web UI | https://10.0.20.111/ |
+| S3 API | ECS default endpoints (see ECS admin UI) |
+
+## Software
+
+- **Luna OS** — CentOS 7-based appliance image
+- **ECS container:** `emccorp/ecs-software:latest` (`ecs-storageos`)
+- **Install config:** `/opt/emc/ecs-install/deploy.yml` on the appliance
+
+## Git copy
+
+`deploy.yml` in this folder is a reference copy. Passwords are redacted — use vault or host file for live secrets.
+
+## Bootstrap SSH
+
+```bash
+./scripts/setup/setup-objectscale-ssh.sh
+```
+
+## Storage
+
+Single-node lab deployment; block device `/dev/sdb` in storage pool `sp1` per deploy.yml.
@@ -0,0 +1,225 @@
+# deploy.yml reference implementation v2.8.0
+
+# [Optional]
+# By changing the license_accepted boolean value to "true" you are
+# declaring your agreement to the terms of the license agreement
+# contained in the license.txt file included with this software
+# distribution.
+licensing:
+  license_accepted: true
+
+#autonames:
+#  custom:
+#    - ecs01
+#    - ecs02
+#    - ecs03
+#    - ecs04
+#    - ecs05
+#    - ecs06
+
+# [Required]
+# Deployment facts reference
+facts:
+
+  # [Required]
+  # Node IP or resolvable hostname from which installations will be launched
+  # The only supported configuration is to install from the same node as the
+  # bootstrap.sh script is run.
+  install_node: 10.0.20.111
+
+  # [Required]
+  # IPs of machines that will be whitelisted in the firewall and allowed
+  # to access management ports of all nodes. If this is set to the
+  # wildcard (0.0.0.0/0) then anyone can access management ports.
+  management_clients:
+    - 0.0.0.0/0
+
+  # [Required]
+  # These credentials must be the same across all nodes. Ansible uses these credentials to
+  # gain initial access to each node in the deployment and set up ssh public key authentication.
+  # If these are not correct, the deployment will fail.
+  ssh_defaults:
+    # [Required]
+    # Username to use when logging in to nodes
+    ssh_username: admin
+    # [Required]
+    # Password to use with SSH login
+    # *** Set to same value as ssh_username to enable SSH public key authentication ***
+    ssh_password: "REDACTED"
+    # [Required when enabling SSH public key authentication]
+    # Password to give to sudo when gaining root access.
+    ansible_become_pass: "REDACTED"
+    # [Required]
+    # Select the type of crypto to use when dealing with ssh public key
+    # authentication. Valid values here are:
+    #  - "rsa" (Default)
+    #  - "ed25519"
+    ssh_crypto: rsa
+
+  # [Required]
+  # Environment configuration for this deployment.
+  node_defaults:
+    dns_domain: local
+    dns_servers:
+      - 10.0.20.1
+    ntp_servers:
+      - 10.0.20.1
+    #
+    # [Optional]
+    # VFS path to source of randomness
+    entropy_source: /dev/urandom
+    #
+    # [Optional]
+    # autonaming: custom
+
+    #
+    # [Optional]
+    # If your ECS comes with differing default credentials, you can specify those here
+    # ecs_root_user: root
+    # ecs_root_pass: ChangeMe
+
+  # [Optional]
+  # Storage pool defaults. Configure to your liking.
+  # All block devices that will be consumed by ECS on ALL nodes must be listed under the
+  # ecs_block_devices option. This can be overridden by the storage pool configuration.
+  # At least ONE (1) block device is REQUIRED for a successful install. More is better.
+  storage_pool_defaults:
+    is_cold_storage_enabled: false
+    is_protected: false
+    description: Default storage pool description
+    ecs_block_devices:
+      - /dev/sdb
+
+  # [Required]
+  # Storage pool layout. You MUST have at least ONE (1) storage pool for a successful install.
+  storage_pools:
+    - name: sp1
+      members:
+        - 10.0.20.111
+      options:
+        is_protected: false
+        is_cold_storage_enabled: false
+        description: My First SP
+        ecs_block_devices:
+          - /dev/sdb
+
+  # [Optional]
+  # VDC defaults. Configure to your liking.
+  virtual_data_center_defaults:
+    description: Default virtual data center description
+
+  # [Required]
+  # Virtual data center layout. You MUST have at least ONE (1) VDC for a successful install.
+  # Multi-VDC deployments are not yet implemented
+  virtual_data_centers:
+    - name: vdc1
+      members:
+        - sp1
+      options:
+        description: My First VDC
+
+  # [Optional]
+  # Replication group defaults. Configure to your liking.
+  replication_group_defaults:
+    description: Default replication group description
+    enable_rebalancing: true
+    allow_all_namespaces: true
+    is_full_rep: false
+
+  # [Optional, required for namespaces]
+  # Replication group layout. You MUST have at least ONE (1) RG to provision namespaces.
+  replication_groups:
+    - name: rg1
+      members:
+        - vdc1
+      options:
+        description: My First RG
+        enable_rebalancing: true
+        allow_all_namespaces: true
+        is_full_rep: false
+
+  # [Optional]
+  # Management User defaults
+  management_user_defaults:
+    is_system_admin: false
+    is_system_monitor: false
+
+  # [Optional]
+  # Management Users
+  management_users:
+    - username: admin1
+      password: ChangeMe
+      options:
+        is_system_admin: true
+    - username: monitor1
+      password: ChangeMe
+      options:
+        is_system_monitor: true
+
+  # [Optional]
+  # Namespace defaults
+  namespace_defaults:
+    is_stale_allowed: false
+    is_compliance_enabled: false
+
+  # [Optional]
+  # Namespace layout
+  namespaces:
+    - name: ns1
+      replication_group: rg1
+      administrators:
+        - root
+      options:
+        is_stale_allowed: false
+        is_compliance_enabled: false
+
+  # [Optional]
+  # Object User defaults
+  object_user_defaults:
+    # Comma-separated list of Swift authorization groups
+    swift_groups_list:
+      - users
+    # Lifetime of S3 secret key in minutes
+    s3_expiry_time: 2592000
+
+  # [Optional]
+  # Object Users
+  object_users:
+    - username: object_admin1
+      namespace: ns1
+      options:
+        swift_password: ChangeMe
+        swift_groups_list:
+          - admin
+          - users
+        s3_secret_key: ChangeMeChangeMeChangeMeChangeMeChangeMe
+        s3_expiry_time: 2592000
+    - username: object_user1
+      namespace: ns1
+      options:
+        swift_password: ChangeMe
+        s3_secret_key: ChangeMeChangeMeChangeMeChangeMeChangeMe
+
+  # [Optional]
+  # Bucket defaults
+  bucket_defaults:
+    namespace: ns1
+    replication_group: rg1
+    head_type: s3
+    filesystem_enabled: False
+    stale_allowed: False
+    encryption_enabled: False
+    owner: object_admin1
+
+  # [Optional]
+  # Bucket layout (optional)
+  buckets:
+    - name: bucket1
+      options:
+        namespace: ns1
+        replication_group: rg1
+        owner: object_admin1
+        head_type: s3
+        filesystem_enabled: False
+        stale_allowed: False
+        encryption_enabled: False
@@ -0,0 +1,65 @@
+# Docker inventory
+
+Containers per host (reference). Regenerate with `./scripts/collect/collect-fleet-config.sh`.
+
+## atc-docker01 (10.0.21.45)
+
+| Container | Image | Ports |
+|-----------|-------|-------|
+| homepage | gethomepage/homepage | 80→3000 |
+| rss-proxy | rss-proxy:latest | 8090 |
+| icons-server | nginx:alpine | 8080→80 |
+| superset | superset-superset | 8088 |
+| superset_redis | redis:7 | 6379 |
+| forgejo | forgejo:14 | 4002, 222 |
+| lam-app-1 | ldapaccountmanager/lam | 4001 |
+
+Compose: `compose/`, `deploy/docker-compose.homepage.yml`
+
+## atc-db02 (10.0.21.51)
+
+| Container | Image | Ports |
+|-----------|-------|-------|
+| postgres_sales | postgres:15 | 5432 |
+| mysql_hr | mysql:8.0 | 3306 |
+| mongodb_supplychain | mongo:4.4 | 27017 |
+| cassandra_telemetry | cassandra:4.1 | 9042 |
+| neo4j_graph | neo4j:4.4 | 7474, 7687 |
+| mongo_express | mongo-express | 8081 |
+
+Compose: `config/docker/atc-db02/docker-compose.yml`
+
+## atc-lake01 (10.0.21.50)
+
+| Container | Image | Ports |
+|-----------|-------|-------|
+| kafka-connect | debezium/connect:2.5.4 | 8083 |
+| spark-master | bitnami/spark | 7077, 8080 |
+| spark-worker | bitnami/spark | (internal) |
+| trino | trinodb/trino:405 | 8089→8080 |
+| spark-temp | apache/spark:3.4.0 | — |
+
+Compose: `config/docker/atc-lake01/docker-compose.yml`
+
+## atc-kafka01 (10.0.21.36)
+
+| Container | Image | Ports |
+|-----------|-------|-------|
+| kafka-ui | provectuslabs/kafka-ui | 9000→8080 |
+
+Kafka broker: native on `:9092` (not containerized in current lab).
+
+## atc-mgt01 (10.0.20.104)
+
+| Container | Image | Ports |
+|-----------|-------|-------|
+| forgejo | forgejo:14 | 3001, 222 |
+| npm-app-1 | nginx-proxy-manager | 80, 443, 81 |
+
+## atc-objectscale (10.0.20.111)
+
+| Container | Image | Notes |
+|-----------|-------|-------|
+| ecs-storageos | emccorp/ecs-software | Dell ECS / ObjectScale |
+
+Config: `config/objectscale/deploy.yml`
@@ -0,0 +1,129 @@
+# ATC Lakehouse — Application Landscape
+
+High-level view of the Dell ATC Forward Deployed Engineering lab.  
+**Live diagram:** http://atc-docker01.dell-atc.lan:8080/docs/architecture.html
+
+## Architecture diagram
+
+```mermaid
+flowchart TB
+  subgraph users [Users]
+    FDE[FDE Engineers]
+  end
+
+  subgraph ops [Operations Layer]
+    HP[Homepage Dashboard<br/>atc-docker01:80]
+    GF[Grafana<br/>atc-grafana:3000]
+    PVE[Proxmox pve01<br/>10.0.10.65:8006]
+    IDRAC[iDRAC<br/>10.0.41.102]
+  end
+
+  subgraph git [Source Control]
+    FG[Forgejo Git<br/>atc-mgt01:3001]
+  end
+
+  subgraph ingest [Ingest & Streaming]
+    KF[Kafka<br/>atc-kafka01:9092]
+    KUI[Kafka UI<br/>:9000]
+    DEB[Debezium Connect<br/>atc-lake01:8083]
+  end
+
+  subgraph process [Processing]
+    SP[Spark Master/Worker<br/>atc-lake01:8080]
+    AF[Airflow<br/>atc-airflow01:8080]
+    TR[Trino<br/>atc-lake01:8089]
+  end
+
+  subgraph storage [Storage]
+  OS[ObjectScale ECS<br/>atc-objectscale 10.0.20.111]
+  end
+
+  subgraph sources [Source Databases — atc-db02]
+    PG[(PostgreSQL)]
+    MY[(MySQL)]
+    MG[(MongoDB)]
+    CA[(Cassandra)]
+    N4[(Neo4j)]
+  end
+
+  subgraph analytics [Analytics & Search]
+    SS[Superset<br/>atc-docker01:8088]
+    ES[Elasticsearch<br/>atc-elastic01:9200]
+    KB[Kibana<br/>:5601]
+  end
+
+  FDE --> HP
+  FDE --> SS
+  FDE --> KB
+  HP --> KUI
+  HP --> DEB
+  HP --> SP
+  HP --> TR
+  HP --> OS
+  HP --> FG
+
+  PG & MY & MG & CA --> DEB
+  DEB --> KF
+  KF --> SP
+  SP --> OS
+  TR --> PG
+  TR --> KF
+  KF --> SS
+  ES --> KB
+  AF --> SP
+```
+
+## Host roles
+
+| Host | IP | Role | Key services |
+|------|-----|------|--------------|
+| **atc-docker01** | 10.0.21.45 | Ops & BI hub | Homepage, Superset, RSS-proxy, icons, Forgejo (local), LAM |
+| **atc-docker02** | 10.0.21.47 | Secondary Docker | Standby / experiments |
+| **atc-mgt01** | 10.0.20.104 | Management | Forgejo (:3001), Nginx Proxy Manager, LDAP |
+| **atc-kafka01** | 10.0.21.36 | Streaming | Kafka broker (:9092), Kafka UI (:9000) |
+| **atc-lake01** | 10.0.21.50 | Lakehouse compute | Spark, Trino, Debezium Connect |
+| **atc-airflow01** | 10.0.21.55 | Orchestration | Apache Airflow (systemd) |
+| **atc-elastic01** | 10.0.21.46 | Search & observability | Elasticsearch, Kibana |
+| **atc-db01** | 10.0.20.112 | Source DBs | PostgreSQL, MySQL (native or container) |
+| **atc-db02** | 10.0.21.51 | Polyglot sources | PG, MySQL, Mongo, Cassandra, Neo4j (Docker) |
+| **atc-grafana** | 10.0.20.103 | Monitoring | Grafana |
+| **atc-objectscale** | 10.0.20.111 | Object storage | Dell ECS / S3 API (Luna OS) |
+| **pve01** | 10.0.10.65 | Hypervisor | Proxmox VE |
+
+## Data flow
+
+1. **Operational data** lives on **atc-db02** (and db01) — sales, HR, supply chain, telemetry, graph.
+2. **Debezium** on lake01 reads CDC from PostgreSQL → **Kafka** on kafka01.
+3. **Spark** processes streams/batch; results can land in **ObjectScale** (S3-compatible).
+4. **Trino** federates queries across catalogs (Postgres, Kafka, Hive, etc.).
+5. **Superset** on docker01 is the primary BI UI; **Kibana** for search/logs.
+6. **Airflow** orchestrates pipelines on airflow01.
+7. **Homepage** aggregates links, health, RSS intel, and Proxmox status.
+
+## Docker vs native
+
+| Component | Deployment | Config in git |
+|-----------|------------|---------------|
+| Homepage, Superset, RSS | Docker (docker01) | `compose/`, `deploy/` |
+| Spark, Trino, Debezium | Docker (lake01) | `config/docker/atc-lake01/` |
+| Source DBs | Docker (db02) | `config/docker/atc-db02/` |
+| Kafka UI | Docker (kafka01) | `config/docker/atc-kafka01/` |
+| Kafka broker | Native/systemd (kafka01) | `config/kafka/` |
+| Elasticsearch | Native (elastic01) | `config/elastic/` |
+| Airflow | Native (airflow01) | `config/airflow/` |
+| ObjectScale | ECS appliance | `config/objectscale/` |
+| Forgejo | Docker (mgt01) | `compose/forgejo/` |
+
+## Network zones
+
+- **Management:** 10.0.10.x — Proxmox, iDRAC
+- **Storage / legacy:** 10.0.20.x — ObjectScale, db01, mgt01, grafana, elastic
+- **Lakehouse compute:** 10.0.21.x — docker, kafka, lake, db02, airflow
+
+## Related docs
+
+- [Hosts & ports](hosts.md)
+- [Architecture layers](architecture.md)
+- [SSH mesh](ssh-mesh.md)
+- [Disaster recovery](disaster-recovery.md)
+- [Docker inventory](docker-inventory.md)
@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+# Pull live Docker/ObjectScale configs into the Lakehouse git repo.
+# Run on atc-docker01 as root (requires SSH mesh).
+set -euo pipefail
+
+REPO="${REPO:-/root/lakehouse}"
+KEY="${KEY:-/root/.ssh/atc_cluster}"
+SSH=(ssh -i "$KEY" -o StrictHostKeyChecking=no -o ConnectTimeout=10)
+
+echo "==> db02 docker-compose"
+"${SSH[@]}" root@atc-db02 "cat /opt/sources/docker-compose.yml" > "$REPO/config/docker/atc-db02/docker-compose.yml"
+
+echo "==> objectscale deploy.yml (redacted)"
+"${SSH[@]}" root@atc-objectscale "cat /opt/emc/ecs-install/deploy.yml" | \
+  sed -E 's/(ssh_password|ansible_become_pass):.*/\1: "REDACTED"/' \
+  > "$REPO/config/objectscale/deploy.yml"
+
+echo "==> elasticsearch.yml"
+"${SSH[@]}" root@atc-elastic01 "cat /etc/elasticsearch/elasticsearch.yml" > "$REPO/config/elastic/elasticsearch.yml"
+
+echo "==> container inventory"
+"$REPO/scripts/deploy/export-inventory.py" "$REPO/inventory/containers-atc-docker01.json"
+
+echo "Done. Review and commit: cd $REPO && git diff"