Container Tools
This directory contains the Docker container configurations and tools for the AMP platform.
Directory Structure
The environment is modularized to allow for easier management and verification of individual services.

compose/: Contains individual Docker Compose files for each service definition (e.g., opensearch.yml, timescaledb.yml) and a base.yml for shared networks and volumes.
services/: Contains configuration files and build contexts for specific services (e.g., services/opensearch/, services/nginx/).
manage_service.sh: The primary script to manage the lifecycle and health of these services.
install_prerequisites.sh: Script to install host dependencies (rsync, docker, python3, java) and configure system limits.

File Locations & Persistence
The AMP platform is designed to be stateless in its container definitions, with configuration and persistent data decoupled from the images.
1. Configuration Files
All service configurations (e.g., nginx.conf, telegraf.conf, certificates) are stored in the host repository and mounted into the containers as read-only volumes.

Path: platform/tools/container/services/<service_name>/
Updates: Changes made to files in these directories are typically picked up by the services after a restart via ./manage_service.sh <service> up.

2. Persistent Data (Volumes)
Stateful data (databases, logs, indices) is stored in Named Docker Volumes. This ensures data survives container updates and accidental removals.



Volume Name
Description




opensearch-data
OpenSearch indices and cluster state.


timescaledb-data
TimescaleDB (PostgreSQL) relational data and logs.


grafana-data
Grafana sqlite database (dashboards, users).


certs-vol
Shared SSL certificates generated during setup.


security-config-vol
OpenSearch Security plugin configurations.


*-logs
Log volumes for each service (e.g., opensearch-logs, nginx-logs).



3. Accessing Persistent Data & Logs
Since data and logs are stored in Docker volumes (not directly on the host filesystem), you cannot browse them with a standard file explorer. Instead, use Docker tools or helper containers.
Accessing Logs
Logs are stored in named volumes (e.g., opensearch-logs). To view or export them:
Quick View (Last 100 lines):

docker run --rm -v opensearch-logs:/logs alpine tail -n 100 /logs/opensearch.log


Interactive Shell:

docker run --rm -it -v nginx-logs:/logs alpine sh
# cd /logs && ls -l


Accessing Data
Database files are similarly protected. To inspect raw data files (debug only):

docker run --rm -v timescaledb-data:/data alpine ls -lR /data


Accessing Configuration
Configuration files are mounted from the host and are directly editable:

Go to: platform/tools/container/services/<service_name>/
Edit the files (e.g., nginx/conf.d/app.conf).
Apply changes: ./manage_service.sh <service_name> up




[!TIP]

To find the physical location of a volume on your host system (Linux only), use:
bash
docker volume inspect <volume_name>
On macOS/Windows (Docker Desktop), the physical files are inside the Docker VM and not directly accessible on the host OS without mounting them as shown above.

Physical Locations (Linux / Docker VM)
If running on a standard Linux Docker host (or inside the Docker VM), the logs are located at /var/lib/docker/volumes/<project>_<volume_name>/_data.

Assuming the project name is compose (default), the locations are:



Service
Host Path (Linux)




OpenSearch
/var/lib/docker/volumes/compose_opensearch-logs/_data/


OpenSearch Dashboards
/var/lib/docker/volumes/compose_opensearch-dashboards-logs/_data/


Nginx
/var/lib/docker/volumes/compose_nginx-logs/_data/


Grafana
/var/lib/docker/volumes/compose_grafana-logs/_data/


Telegraf
/var/lib/docker/volumes/compose_telegraf-logs/_data/


Logstash
/var/lib/docker/volumes/compose_logstash-logs/_data/


PgBouncer
/var/lib/docker/volumes/compose_pgbouncer-logs/_data/


TimescaleDB
/var/lib/docker/volumes/compose_timescaledb-data/_data/pg_log/



Setup & Prerequisites
Before running services, ensure your system has the necessary tools. OpenSearch requires vm.max_map_count >= 262144.
Run the helper script to attempt automatic host setup:

./install_prerequisites.sh


Usage
Use manage_service.sh to manage and verify services.
1. Basic Usage

./manage_service.sh [service_name] [action]



service_name: Name of the service (filename in compose/ without .yml).
action (optional):
up (default): Start service and run health checks.
down: Stop and remove the service.
purge: Stop service and remove its volumes (fresh start).
validate: Run health checks on a running service.

2. Recommended Startup Order

Setup & Core Storage

bash
   ./manage_service.sh setup
   ./manage_service.sh opensearch
   ./manage_service.sh timescaledb

Backend & Middleware

bash
   ./manage_service.sh configurator
   ./manage_service.sh pgbouncer

Frontend & Observability

bash
   ./manage_service.sh opensearch-dashboards
   ./manage_service.sh grafana
   ./manage_service.sh telegraf
   ./manage_service.sh logstash
   ./manage_service.sh nginx
Development Notes

Modifying Configuration: Service configurations are located in services/<service_name>.
Modifying Compose Definitions: Docker compose definitions are in compose/<service_name>.yml.
Volumes in compose/*.yml reference ../services/<service_name>/....

The changes has been tested locally.