AE-1952 : Configure OpenSearch stack to parser the Array product logs

Review Request #1029 — Created Aug. 22, 2025 and submitted

Information
pmurugaiyan
AMP
amp_4_0
AE-1952
Reviewers
apoorva.sn, mmiriam, ngurunathan, shuinvy 
Configure OpenSearch stack to parser the Array product logs




sh install_logstash_oss.sh


[INFO] Starting Logstash 8.15.0 installation and configuration for OpenSearch 3.x...


[INFO] Checking for Java installation (OpenJDK 17 or higher required)...


[SUCCESS] Java 21 found. Compatible with Logstash 8.x.


[INFO] Installing Logstash OSS 8.15.0...

Rocky Linux 9 - BaseOS                                                                                                                                                         1.9 kB/s | 4.1 kB     00:02

Rocky Linux 9 - BaseOS                                                                                                                                                         843 kB/s | 2.5 MB     00:03

Rocky Linux 9 - AppStream                                                                                                                                                      3.2 kB/s | 4.5 kB     00:01

Rocky Linux 9 - AppStream                                                                                                                                                      2.4 MB/s | 9.5 MB     00:03

Rocky Linux 9 - Extras                                                                                                                                                         1.5 kB/s | 2.9 kB     00:01

logstash-oss-8.15.0-x86_64.rpm                                                                                                                                                  13 MB/s | 352 MB     00:27

Dependencies resolved.

===============================================================================================================================================================================================================

 Package                                             Architecture                                  Version                                           Repository                                           Size

===============================================================================================================================================================================================================

Installing:

 logstash-oss                                        x86_64                                        1:8.15.0-1                                        @commandline                                        352 M


Transaction Summary


Install  1 Package


Total size: 352 M

Installed size: 610 M

Downloading Packages:

Running transaction check

Transaction check succeeded.

Running transaction test

Transaction test succeeded.

Running transaction

  Preparing        :                                                                                                                                                                                       1/1

  Running scriptlet: logstash-oss-1:8.15.0-1.x86_64                                                                                                                                                        1/1

  Installing       : logstash-oss-1:8.15.0-1.x86_64                                                                                                                                                        1/1

  Running scriptlet: logstash-oss-1:8.15.0-1.x86_64                                                                                                                                                        1/1

  Verifying        : logstash-oss-1:8.15.0-1.x86_64                                                                                                                                                        1/1


Installed:

  logstash-oss-1:8.15.0-1.x86_64


Complete!


[SUCCESS] Logstash installed.


[INFO] Installing logstash-output-opensearch plugin...


[INFO] Attempting to install logstash-output-opensearch with 2g heap...

Using bundled JDK: /usr/share/logstash/jdk

Validating logstash-output-opensearch

Resolving mixin dependencies

Updating mixin dependencies logstash-mixin-ecs_compatibility_support

Bundler attempted to update logstash-mixin-ecs_compatibility_support but its version stayed the same

Installing logstash-output-opensearch


Installation successful


[SUCCESS] OpenSearch output plugin installed.


[INFO] Checking for logstash-integration-jdbc plugin...

Errno::EPIPE: Broken pipe - <STDOUT>

            write at org/jruby/RubyIO.java:1590

            write at org/jruby/RubyIO.java:2870

             puts at org/jruby/RubyIO.java:2713

             puts at org/jruby/RubyKernel.java:723

          execute at /usr/share/logstash/lib/pluginmanager/list.rb:53

             each at org/jruby/RubyArray.java:1981

  each_with_index at org/jruby/RubyEnumerable.java:1214

  each_with_index at org/jruby/RubyEnumerable.java:1204

          execute at /usr/share/logstash/lib/pluginmanager/list.rb:49

             each at org/jruby/RubyArray.java:1981

          execute at /usr/share/logstash/lib/pluginmanager/list.rb:39

              run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:68

          execute at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/subcommand/execution.rb:11

              run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:68

              run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:133

           <main> at /usr/share/logstash/lib/pluginmanager/main.rb:64


[SUCCESS] logstash-integration-jdbc plugin already installed, providing jdbc_streaming filter.


[INFO] Checking for logstash-filter-geoip plugin...

Errno::EPIPE: Broken pipe - <STDOUT>

    write at org/jruby/RubyIO.java:1590

    write at org/jruby/RubyIO.java:2870

     puts at org/jruby/RubyIO.java:2713

     puts at org/jruby/RubyKernel.java:723

  execute at /usr/share/logstash/lib/pluginmanager/list.rb:42

     each at org/jruby/RubyArray.java:1981

  execute at /usr/share/logstash/lib/pluginmanager/list.rb:39

      run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:68

  execute at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/subcommand/execution.rb:11

      run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:68

      run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:133

   <main> at /usr/share/logstash/lib/pluginmanager/main.rb:64


[SUCCESS] logstash-filter-geoip plugin already installed.


[INFO] Creating Logstash certificate directory: /etc/logstash/certs...


[INFO] Copying CA certificate to Logstash and set permissions...


[SUCCESS] CA certificate copied and permissions set.


[INFO] Installing PostgreSQL JDBC driver...

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100 1062k  100 1062k    0     0   166k      0  0:00:06  0:00:06 --:--:--  229k


[SUCCESS] PostgreSQL JDBC driver installed.


[INFO] Checking CA certificate file...


[SUCCESS] Required certificate files exist.


[INFO] Checking for security_admin role...


[INFO] security_admin role not found. Creating role...


[SUCCESS] security_admin role created.


[INFO] Checking admin certificate permissions...


[INFO] Mapping admin DN to security_admin role...


[SUCCESS] Admin DN mapped to security_admin role.


[INFO] Configuring OpenSearch security for Logstash authentication...


[INFO] Configuring logstash_custom role in OpenSearch for acm-* indices...


[SUCCESS] logstash_custom role configured with acm-* index permissions.


[INFO] Creating Logstash user in OpenSearch...


[SUCCESS] Logstash user created.


[INFO] Mapping Logstash user to 'logstash_custom' role...


[SUCCESS] Logstash user mapped to logstash_custom role.


[INFO] Verifying Logstash role mapping...


[SUCCESS] Logstash user mapping to logstash_custom verified.


[INFO] Checking if OpenSearch Dashboards is running on https://127.0.0.1:5601...


[SUCCESS] OpenSearch Dashboards is running.


[INFO] Verifying admin user authentication for OpenSearch Dashboards...


[SUCCESS] Admin user authentication verified.


[INFO] Ensuring admin user has kibana_admin role for saved objects...


[SUCCESS] Admin user or DN already mapped to kibana_admin role.


[INFO] Creating index pattern for acm-* with @timestamp as time field (ignoring SSL verification)...


[SUCCESS] Index pattern 'acm-*' created with @timestamp as time field.


[INFO] Configuring syslog pipeline for OpenSearch output to acm-%{+YYYY.MM.dd} indices...


[SUCCESS] Logstash syslog pipeline configured to write to acm-%{+YYYY.MM.dd} indices. Customize /etc/logstash/conf.d/syslog.conf for production (e.g., update jdbc_connection_string, jdbc_user, jdbc_password).


[INFO] Configuring Logstash data directory...


[SUCCESS] Logstash data directory configured.


[INFO] Configuring Logstash systemd service...


[SUCCESS] Logstash systemd service configured.


[INFO] Enabling and starting Logstash service...

Created symlink /etc/systemd/system/multi-user.target.wants/logstash.service → /usr/lib/systemd/system/logstash.service.


[INFO] Waiting for Logstash to start...


[INFO] Attempt 1/60: Logstash not ready. Waiting 10 seconds...


[INFO] Attempt 2/60: Logstash not ready. Waiting 10 seconds...


[SUCCESS] Logstash service is up!


[INFO] Configuring firewall for Logstash syslog input in zone public...


[WARNING] Firewall zone public is not active. Ensure it is assigned to an interface.

success


[SUCCESS] IP masquerading enabled in zone public.

success


[SUCCESS] Opened port 5514/udp in zone public.

success


[SUCCESS] Opened port 5514/tcp in zone public.

success


[SUCCESS] Added port forwarding from 514/udp to 5514 in zone public.

success


[SUCCESS] Added port forwarding from 514/tcp to 5514 in zone public.

success


[SUCCESS] Firewall reloaded successfully.


[SUCCESS] Logstash 8.15.0 installation and configuration for OpenSearch 3.x complete!


[INFO] Logstash is configured to process syslog messages on port 5514 (with forwarding from 514) and write to OpenSearch acm-%{+YYYY.MM.dd} indices at https://127.0.0.1:9200.


[INFO] Index pattern 'acm-*' is configured with @timestamp as the time field for OpenSearch Dashboards.
The changes has been tested locally.

shuinvy
shuinvy 
  1. Ship It!
  2. 
  
    
 
    

 
    
  


    
 
    
 
      

  

 
    


    

 
    

    3. 




  

 
















 





 

  

   




apoorva.sn




   

   







  

 



 

  


   

   

    
 
apoorva.sn


    
   


  

  

  



    




  1. 
 
    
  
    
   
    Ship It!
    
  
    
 
    

 
    
  


    
 
    
 
      

  

 
    


    

 
    

    2. 















  2. 
 
    
  
    
   
    
  
    
 
    

 
    
  


    
 
    
 
      

  

 
    


    

 
    

    3. 




  

 
















 



 

  

   




pmurugaiyan




   

   

  

 



 

  


   

   

    
 Review request changed

    
   


  

  

  






 

  Status:
  

  Closed (submitted)

  
 







 


 










  

 









     

    

   

  

  

   
   Loading...