Index: /branches/amp_4_0/platform/tools/container/DEPLOYMENT.md =================================================================== --- /branches/amp_4_0/platform/tools/container/DEPLOYMENT.md (revision 2948) +++ /branches/amp_4_0/platform/tools/container/DEPLOYMENT.md (working copy) @@ -93,9 +93,11 @@ **On Node 1 (Manager):** ```bash -docker swarm init --advertise-addr +./manage_amp.sh init ``` +> **Note**: Always use `./manage_amp.sh init` instead of `docker swarm init` because the script also configures the local registry and applies necessary Swarm node labels for storage. + Copy the join command output. **On Node 2 and Node 3:** @@ -251,12 +253,23 @@ * Applies system tuning (firewall, sysctl) * Loads Docker images and pushes to local registry -#### Step 4: Docker Swarm Setup +#### Step 4: Configure Secrets + +On the **Manager Node**, initialize the passwords before deploying: + +```bash +cp .secrets.example .secrets +vi .secrets +``` + +*Note: You only need to do this on the node where you will run the deploy command.* +#### Step 5: Docker Swarm Setup + **On Node 1 (Manager):** ```bash -docker swarm init --advertise-addr +./manage_amp.sh init ``` **On Node 2 and Node 3:** @@ -272,27 +285,26 @@ docker node promote ``` -#### Step 5: Configure Virtual IP (Optional) +#### Step 6: Configure Virtual IP (Optional) ```bash ./manage_amp.sh vip --vip --priority 101 # Node 1 ./manage_amp.sh vip --vip --priority 100 # Node 2 ``` -#### Step 6: Deploy the Stack +#### Step 7: Deploy the Stack ```bash ./manage_amp.sh deploy --auto ``` -#### Step 7: Post-Deployment Configuration +#### Step 8: Post-Deployment Configuration ```bash -```bash ./manage_amp.sh post_deploy ``` -#### Step 8: Verify Deployment +#### Step 9: Verify Deployment ```bash docker service ls Index: /branches/amp_4_0/platform/tools/container/OFFLINE_GUIDE.md =================================================================== --- /branches/amp_4_0/platform/tools/container/OFFLINE_GUIDE.md (revision 2948) +++ /branches/amp_4_0/platform/tools/container/OFFLINE_GUIDE.md (working copy) @@ -58,7 +58,7 @@ | File | Purpose | |------|---------| | `amp_offline_bundle.tar.gz` | Main bundle containing RPMs, Docker images, scripts, and configs | - | `tar-bootstrap.rpm` | Standalone `tar` package for minimal installs that lack `tar` | + | `tar-bootstrap.rpm` | Standalone `tar` package for minimal installs that lack `tar` | > [!TIP] > The bundle also includes a copy of `.env` from your build machine. You may need to update it for the target environment after extraction. @@ -99,6 +99,13 @@ * `AMP_DOMAIN_OR_IP` - Should match target node IP * `POSTGRES_PASSWORD` / `OPENSEARCH_INITIAL_ADMIN_PASSWORD` - Credentials + Similarly, initialize and configure your secrets: + + ```bash + cp .secrets.example .secrets + vi .secrets + ``` + 4. **Load Dependencies**: This step installs system prerequisites (using the downloaded RPMs) and imports the Docker images into the local daemon. @@ -110,10 +117,12 @@ > This may take several minutes as it extracts and loads large Docker images (~5-10GB). 5. **Deploy AMP Cluster**: - Once the dependencies are loaded, deploy the cluster using the standard deployment command. + Once the dependencies are loaded, initialize Swarm (if not running) and deploy the cluster using the standard deployment command. Then configure post-deployment options. ```bash + ./manage_amp.sh init ./manage_amp.sh deploy --auto + ./manage_amp.sh post_deploy ``` ## 5. Multi-Node Offline Deployment Index: /branches/amp_4_0/platform/tools/container/manage_amp.sh =================================================================== --- /branches/amp_4_0/platform/tools/container/manage_amp.sh (revision 2948) +++ /branches/amp_4_0/platform/tools/container/manage_amp.sh (working copy) @@ -422,8 +422,8 @@ # Create amp-core required directories (required for bind mounts) echo "" - echo "--- Creating amp-core directories ---" - for DIR in /ca/conf /ca/etc /ca/package "${AMP_LOG_ROOT:-/var/log/amp}/amp-core"; do + echo "--- Creating bind-mount directories ---" + for DIR in /ca/conf /ca/etc /ca/package "${AMP_LOG_ROOT:-/var/log/amp}/amp-core" "${AMP_LOG_ROOT:-/var/log/amp}/nginx"; do if [ ! -d "$DIR" ]; then echo "Creating directory: $DIR" sudo mkdir -p "$DIR" @@ -436,23 +436,41 @@ # Create Telegraf config directory for amp-api modifications echo "" echo "--- Initializing Telegraf Config Directory ---" + + TELEGRAF_MAIN_CONF="/opt/amp/telegraf.conf" + if [ ! -f "$TELEGRAF_MAIN_CONF" ]; then + echo "Creating default $TELEGRAF_MAIN_CONF" + if [ -f "$SERVICES_DIR/telegraf/telegraf.conf" ]; then + sudo cp "$SERVICES_DIR/telegraf/telegraf.conf" "$TELEGRAF_MAIN_CONF" 2>/dev/null || true + sudo chmod 644 "$TELEGRAF_MAIN_CONF" 2>/dev/null || true + else + echo "⚠️ Source config not found: $SERVICES_DIR/telegraf/telegraf.conf" + fi + else + echo "✅ Telegraf main config already exists: $TELEGRAF_MAIN_CONF" + fi + TELEGRAF_CONFIG_DIR="/opt/amp/telegraf.d" if [ ! -d "$TELEGRAF_CONFIG_DIR" ]; then echo "Creating directory: $TELEGRAF_CONFIG_DIR" sudo mkdir -p "$TELEGRAF_CONFIG_DIR" - - # Copy default SNMP configs from services directory - if [ -d "$SERVICES_DIR/telegraf/telegraf.d" ]; then - echo "Copying default Telegraf configs..." - sudo cp "$SERVICES_DIR/telegraf/telegraf.d"/*.toml "$TELEGRAF_CONFIG_DIR/" 2>/dev/null || true - sudo chmod 644 "$TELEGRAF_CONFIG_DIR"/*.toml 2>/dev/null || true - echo "✅ Copied default configs to $TELEGRAF_CONFIG_DIR" - else - echo "⚠️ Source config directory not found: $SERVICES_DIR/telegraf/telegraf.d" - fi - echo "✅ Created $TELEGRAF_CONFIG_DIR" else echo "✅ Telegraf config directory already exists: $TELEGRAF_CONFIG_DIR" + fi + + # Copy default SNMP configs from services directory (only if target files don't exist) + if [ -d "$SERVICES_DIR/telegraf/telegraf.d" ]; then + echo "Ensuring default Telegraf configs exist..." + for conf in "$SERVICES_DIR/telegraf/telegraf.d"/*.toml; do + if [ -f "$conf" ]; then + filename=$(basename "$conf") + if [ ! -f "$TELEGRAF_CONFIG_DIR/$filename" ]; then + sudo cp "$conf" "$TELEGRAF_CONFIG_DIR/" 2>/dev/null || true + sudo chmod 644 "$TELEGRAF_CONFIG_DIR/$filename" 2>/dev/null || true + fi + fi + done + echo "✅ Default configs synced to $TELEGRAF_CONFIG_DIR" fi # Configure Firewall Rules (if firewalld is available) @@ -1355,10 +1373,12 @@ fi done - # 1b. Create directories required for amp-core bind mounts - echo "Creating required host directories for amp-core..." - sudo mkdir -p /ca/conf /ca/etc /ca/package "${AMP_LOG_ROOT:-/var/log/amp}/amp-core" - echo "✅ Created /ca/conf, /ca/etc, /ca/package, ${AMP_LOG_ROOT:-/var/log/amp}/amp-core" + # 1b. Create directories required for bind mounts + echo "Creating required host directories for bind mounts..." + sudo mkdir -p /ca/conf /ca/etc /ca/package \ + "${AMP_LOG_ROOT:-/var/log/amp}/amp-core" \ + "${AMP_LOG_ROOT:-/var/log/amp}/nginx" + echo "✅ Created /ca/conf, /ca/etc, /ca/package, ${AMP_LOG_ROOT:-/var/log/amp}/amp-core, ${AMP_LOG_ROOT:-/var/log/amp}/nginx" # 2. Check if Certificates Exist echo "Checking for existing certificates..." @@ -1623,11 +1643,23 @@ create_grafana_db() { echo "--- Initializing Grafana Database in HA Postgres ---" - # Find any running TimescaleDB container to use as a psql client - TS_CONTAINER=$(docker ps -q -f name=amp_timescaledb | head -n 1) + echo "Waiting for TimescaleDB container to start..." + local ts_retries=30 + local ts_count=0 + local TS_CONTAINER="" + + until [ -n "$TS_CONTAINER" ] || [ $ts_count -ge $ts_retries ]; do + TS_CONTAINER=$(docker ps -q -f name=amp_timescaledb | head -n 1) + if [ -n "$TS_CONTAINER" ]; then + break + fi + echo " [$ts_count/$ts_retries] Waiting for container..." + sleep 5 + ts_count=$((ts_count + 1)) + done if [ -z "$TS_CONTAINER" ]; then - echo "❌ No running TimescaleDB container found. Is the stack deployed?" + echo "❌ No running TimescaleDB container found after waiting. Is the stack deployed?" exit 1 fi @@ -1857,20 +1889,17 @@ cp manage_amp.sh "$BUNDLE_DIR/" cp install_prerequisites.sh "$BUNDLE_DIR/" cp stack.yml.template "$BUNDLE_DIR/" + cp .secrets.example "$BUNDLE_DIR/" 2>/dev/null || echo "⚠️ .secrets.example not found" + cp .env.example "$BUNDLE_DIR/" 2>/dev/null || echo "⚠️ .env.example not found" cp -r services "$BUNDLE_DIR/" cp -r scripts "$BUNDLE_DIR/" 2>/dev/null || echo "⚠️ No scripts directory found" # Copy .env if it exists + # Copy optional configuration files if they exist if [ -f ".env" ]; then cp .env "$BUNDLE_DIR/.env" echo "✅ Copied .env (non-secret configuration)" fi - - # Copy secrets template and optionally the secrets file - if [ -f ".secrets.example" ]; then - cp .secrets.example "$BUNDLE_DIR/.secrets.example" - echo "✅ Copied .secrets.example (password template)" - fi if [ -f ".secrets" ]; then cp .secrets "$BUNDLE_DIR/.secrets" chmod 600 "$BUNDLE_DIR/.secrets" @@ -1901,7 +1930,10 @@ echo " tar -xf $FINAL_ARCHIVE" echo " cd $BUNDLE_DIR" echo " ./manage_amp.sh load_offline" + echo " cp .secrets.example .secrets # Configuration" + echo " ./manage_amp.sh init # If not already initialized" echo " ./manage_amp.sh deploy --auto" + echo " ./manage_amp.sh post_deploy" } # Function: Load Offline Bundle @@ -2048,8 +2080,10 @@ echo "✅ Offline bundle loaded successfully!" echo "" echo "Next steps:" - echo " 1. Initialize Swarm (if not already): ./manage_amp.sh init" - echo " 2. Deploy with auto-config: ./manage_amp.sh deploy --auto" + echo " 1. Configure secrets : cp .secrets.example .secrets && vi .secrets" + echo " 2. Initialize Swarm : ./manage_amp.sh init (if not initialized)" + echo " 3. Deploy stack : ./manage_amp.sh deploy --auto" + echo " 4. Post-deploy config : ./manage_amp.sh post_deploy" fi } Index: /branches/amp_4_0/platform/tools/container/services/pgbouncer/pgbouncer.ini.template =================================================================== --- /branches/amp_4_0/platform/tools/container/services/pgbouncer/pgbouncer.ini.template (revision 2948) +++ /branches/amp_4_0/platform/tools/container/services/pgbouncer/pgbouncer.ini.template (working copy) @@ -1,6 +1,6 @@ [databases] -cm = host=timescaledb port=5432 dbname=cm user=amp_admin password=__AMP_ADMIN_PASSWORD__ -amp_ts = host=timescaledb port=5432 dbname=__AMP_DB_NAME__ user=__AMP_DB_USER__ password=__AMP_DB_PASSWORD__ +cm = host=127.0.0.1 port=5432 dbname=cm user=amp_admin password=__AMP_ADMIN_PASSWORD__ +amp_ts = host=127.0.0.1 port=5432 dbname=__AMP_DB_NAME__ user=__AMP_DB_USER__ password=__AMP_DB_PASSWORD__ [pgbouncer] listen_port = 6432 Index: /branches/amp_4_0/platform/tools/container/services/telegraf/telegraf.conf =================================================================== --- /branches/amp_4_0/platform/tools/container/services/telegraf/telegraf.conf (revision 2952) +++ /branches/amp_4_0/platform/tools/container/services/telegraf/telegraf.conf (working copy) @@ -17,7 +17,7 @@ [[outputs.postgresql]] connection = "host=${TSDB_HOST:-127.0.0.1} port=${TSDB_PORT:-5432} user=${TSDB_USER:-amp_ts_user} password=${TSDB_PASSWORD:-Array@123$} dbname=${TSDB_NAME:-amp_ts} sslmode=disable" schema = "public" - method = "insert" + # method = "insert" # Unsupported in this version # Route metrics to Timescale, configured to drop logs namedrop = ["docker_log"] @@ -45,7 +45,7 @@ # Using Docker input for container stats (Metrics -> TimescaleDB) [[inputs.docker]] endpoint = "unix:///var/run/docker.sock" - docker_api_version = "" + # docker_api_version = "" # Unsupported in this version gather_services = false timeout = "5s" @@ -55,7 +55,7 @@ # Using Docker logs (Logs -> Logstash) [[inputs.docker_log]] endpoint = "unix:///var/run/docker.sock" - docker_api_version = "" + # docker_api_version = "" # Unsupported in this version container_name_include = [] # All containers timeout = "5s" @@ -151,23 +151,20 @@ val = metric.fields[f] if val == None: continue - elif type(val) == int: + elif type(val) == "int": metric.fields[f] = val - elif type(val) == float: + elif type(val) == "float": metric.fields[f] = int(val) - elif type(val) == str: + elif type(val) == "string": # remove non-digit characters like 'kbps' or 'ms' digits = "" - for c in val: + for c in val.elems(): if c in "0123456789.": digits += c if digits == "": metric.fields[f] = 0 else: - try: - metric.fields[f] = int(float(digits)) - except: - metric.fields[f] = 0 + metric.fields[f] = int(float(digits)) else: metric.fields[f] = 0 Index: /branches/amp_4_0/platform/tools/container/services/timescaledb/post_bootstrap.sh =================================================================== --- /branches/amp_4_0/platform/tools/container/services/timescaledb/post_bootstrap.sh (revision 2948) +++ /branches/amp_4_0/platform/tools/container/services/timescaledb/post_bootstrap.sh (working copy) @@ -17,10 +17,12 @@ # 1. Create Grafana User and Database GRAFANA_USER="grafana" -# Read from Docker secret (preferred) or environment variable if [ -f /run/secrets/grafana_admin_password ]; then GRAFANA_PASSWORD="$(cat /run/secrets/grafana_admin_password)" else + GRAFANA_PASSWORD="${GF_SECURITY_ADMIN_PASSWORD}" +fi + if [ -z "$GRAFANA_PASSWORD" ]; then echo "❌ GRAFANA_PASSWORD not set (check secrets)." exit 1 @@ -51,6 +53,9 @@ if [ -f /run/secrets/amp_db_password ]; then AMP_PASS="$(cat /run/secrets/amp_db_password)" else + AMP_PASS="${AMP_DB_PASSWORD}" +fi + if [ -z "$AMP_PASS" ]; then echo "❌ AMP_DB_PASSWORD not set (check secrets)." exit 1 Index: /branches/amp_4_0/platform/tools/container/stack.yml =================================================================== --- /branches/amp_4_0/platform/tools/container/stack.yml (revision 2948) +++ /branches/amp_4_0/platform/tools/container/stack.yml (working copy) @@ -333,6 +333,9 @@ volumes: - certs-vol:/etc/nginx/certs:ro - type: bind + source: ${AMP_LOG_ROOT:-/var/log/amp}/nginx + target: /var/log/nginx + - type: bind source: /dev/null target: /etc/nginx/conf.d/default.conf networks: Index: /branches/amp_4_0/platform/tools/container/stack.yml.template =================================================================== --- /branches/amp_4_0/platform/tools/container/stack.yml.template (revision 2952) +++ /branches/amp_4_0/platform/tools/container/stack.yml.template (working copy) @@ -11,8 +11,8 @@ configs: nginx_app_conf: file: services/nginx/conf.d/app.conf - telegraf_main_conf: - file: services/telegraf/telegraf.conf + # telegraf_main_conf: + # file: services/telegraf/telegraf.conf logstash_pipeline_conf: file: services/logstash/pipeline/syslog.conf logstash_config_yml: @@ -278,6 +278,9 @@ target: /etc/nginx/certs/root-ca.pem volumes: - type: bind + source: ${AMP_LOG_ROOT:-/var/log/amp}/nginx + target: /var/log/nginx + - type: bind source: /dev/null target: /etc/nginx/conf.d/default.conf networks: @@ -406,10 +409,8 @@ HOST_MOUNT_PREFIX: /rootfs secrets: - pg_password - configs: - - source: telegraf_main_conf - target: /etc/telegraf/telegraf.conf volumes: + - /opt/amp/telegraf.conf:/etc/telegraf/telegraf.conf:ro - /var/run/docker.sock:/var/run/docker.sock - /dev:/dev:ro - /:/rootfs:ro @@ -480,7 +481,7 @@ retries: 3 start_period: 30s environment: - - DB_HOST=host.docker.internal + - DB_HOST=127.0.0.1 - DB_PORT=5432 - DB_NAME=${POSTGRES_DB_CM:-cm} - DB_USER=${POSTGRES_USER:-postgres}