Troubleshooting
Topics
Diagnostic tools, common issues, and resolution procedures.
Recovery default base indexes
Only applies to versions 6.1.5 and older. From version 6.1.6 and later, default indexes are created automatically
If you lost or damage following index:
|Index name | Index ID |
|-----------------|-----------------------|
| .security |Pfq6nNXOSSmGhqd2fcxFNg |
| .taskmanagement |E2Pwp4xxTkSc0gDhsE-vvQ |
| alert_status |fkqks4J1QnuqiqYmOFLpsQ |
| audit |cSQkDUdiSACo9WlTpc1zrw |
| alert_error |9jGh2ZNDRumU0NsB3jtDhA |
| alert_past |1UyTN1CPTpqm8eDgG9AYnw |
| .trustedhost |AKKfcpsATj6M4B_4VD5vIA |
| .logserver-gui |cmN5W7ovQpW5kfaQ1xqf2g |
| .scheduler_job |9G6EEX9CSEWYfoekNcOEMQ |
| .authconfig |2M01Phg2T-q-rEb2rbfoVg |
| .auth |ypPGuDrFRu-_ep-iYkgepQ |
| .reportscheduler|mGroDs-bQyaucfY3-smDpg |
| .authuser |zXotLpfeRnuzOYkTJpsTaw |
| alert_silence |ARTo7ZwdRL67Khw_HAIkmw |
| .elastfilter |TtpZrPnrRGWQlWGkTOETzw |
| alert |RE6EM4FfR2WTn-JsZIvm5Q |
| .alertrules |SzV22qrORHyY9E4kGPQOtg |
You may to recover it from default installation folder with following steps:
Stop Logserver-Probe instances which load data into cluster
systemctl stop logserver-probe
Disable shard allocation
PUT _cluster/settings { "persistent": { "cluster.routing.allocation.enable": "none" } }
Stop indexing and perform a synced flush
POST _flush/synced
Shutdown all nodes:
systemctl stop logserver.service
Copy appropriate index folder from installation folder to Energylogserver cluster data node folder (example of .auth folder)
cp -rf ypPGuDrFRu-_ep-iYkgepQ /var/lib/logserver/nodes/0/indices/
Set appropriate permission
chown -R logserver:logserver /var/lib/logserver/
Start all Energylogserver instance
systemctl start logserver
Wait for yellow state of Energylogserver cluster and then enable shard allocation
PUT _cluster/settings { "persistent": { "cluster.routing.allocation.enable": "all" } }
Wait for green state of Energylogserver cluster and then start the Logserver-Probe instances
systemctl start logserver-probe
Too many open files
If you have a problem with too many open files by the Energylogserver process, modify the values in the following configuration files:
/etc/sysconfig/logserver
/etc/security/limits.d/30-logserver.conf
/usr/lib/systemd/system/logserver.service
Check these three files for:
LimitNOFILE=65536
logserver nofile 65537
MAX_OPEN_FILES=65537
Changes to service file require:
systemctl daemon-reload
And changes to limits.d require:
sysctl -p /etc/sysctl.d/90-logserver.conf
The Logserver-GUI status code 500
If the login page is displayed in Logserver-GUI, but after the attempt to login, the browser displays “error: 500”, and the logs will show entries:
Error: Failed to encode cookie (sid-auth) value: Password string too short (min 32 characters required).
Generate a new server.ironsecret with the following command:
echo "server.ironsecret: \"$(/urandom tr -dc _A-Z-a-z-0-9 | head -c32)\"" >> /etc/logserver-gui/logserver-gui.yml
Diagnostic tool
Energylogserver includes a diagnostic tool that helps solve your problem by collecting system data necessary for problem analysis by the support team.
The diagnostic tool is located in the installation directory: /usr/share/logserver/utils/diagnostic-tool.sh
Diagnostic tool collect the following information:
configuration files for Logserver-GUI, Energylogserver, Alert
logs file for Logserver-GUI, Alert, Cerebro, Energylogserver
Cluster information from Energylogserver API
When the diagnostic tool collects data passwords and IP address are removed from the content of files.
Running the diagnostic tool
To run the diagnostic tool, you must provide three parameters: - user assigned admin role, default ‘logserver’ - user password; - URL of cluster API, default: http://localhost:9200
Example of a command:
./diagnostic-tool.sh $user $password http://localhost:9200
The diagnostic tool saves the results to .tar file located in the user’s home directory.
Verification steps and logs
Verification of Energylogserver service
To verify of Logserver service you can use following command:
Control of the Energylogserver system service via systemd:
sysetmctl status logserver
output:
● logserver.service - Logserver
Loaded: loaded (/etc/systemd/system/logserver.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2023-11-14 15:17:16 CET; 5 days ago
Main PID: 58816 (java)
CGroup: /system.slice/logserver.service
└─58816 /etc/alternatives/jre/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.n.
Control of Logserver instance via tcp port:
curl -XGET '127.0.0.1:9200/'
output:
{
"name" : "node-1",
"cluster_name" : "logserver",
"cluster_uuid" : "B5SDCaaKQU2JdJpsKy6quQ",
"version" : {
"distribution" : "opensearch",
"number" : "2.8.0",
"build_type" : "tar",
"build_hash" : "db90a415ff2fd428b4f7b3f800a51dc229287cb4",
"build_date" : "2023-07-28T09:54:26.952266Z",
"build_snapshot" : false,
"lucene_version" : "9.6.0",
"minimum_wire_compatibility_version" : "7.10.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}
Control of Logserver instance via log file:
tail -f /var/log/logserver/logserver.log
other control commands via *curl* application:
curl -xGET "http://localhost:9200/_cat/health?v" curl -XGET "http://localhost:9200/_cat/nodes?v" curl -XGET "http://localhost:9200/_cat/indicates"
Verification of Logserver-Probe service
To verify of Logserver-Probe service you can use following command:
control Logserver-Probe service via systemd:
systemctl status logserver-probe
output:
logserver-probe.service - logserver-probe
Loaded: loaded (/etc/systemd/system/logserver-probe.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2017-07-12 10:30:55 CEST; 1 months 23 days ago
Main PID: 87818 (java)
CGroup: /system.slice/logserver-probe.service
└─87818 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
control Logserver-Probe service via port tcp:
curl -XGET '127.0.0.1:9600'
output:
{ "host": "skywalker", "version": "4.5.3", "http_address": "127.0.0.1:9600" }
control Logserver-Probe service via log file:
tail -f /var/log/logserver-probe/logserver-probe-plain.log
Debugging
dynamically update logging levels through the logging API (service restart not needed):
curl -XPUT 'localhost:9600/_node/logging?pretty' -H 'Content-Type: application/json' -d' { "logger.logserver-probe.outputs.logserver" : "DEBUG" } '
permanent change of logging level (service need to be restarted):
edit file /etc/logserver-probe/logserver-probe.yml and set the following parameter:
*log.level: debug*
restart logserver-probe service:
*systemctl restart logserver-probe*
checking correct syntax of configuration files:
*/usr/share/logserver-probe/bin/logserver-probe -tf /etc/logserver-probe/conf.d*
get information about load of the Logserver-Probe:
curl -XGET '127.0.0.1:9600/_node/jvm?pretty=true'
output:
{
"host" : "logserver-test",
"version" : "5.6.2",
"http_address" : "0.0.0.0:9600",
"id" : "5a440edc-1298-4205-a524-68d0d212cd55",
"name" : "logserver-test",
"jvm" : {
"pid" : 14705,
"version" : "1.8.0_161",
"vm_version" : "1.8.0_161",
"vm_vendor" : "Oracle Corporation",
"vm_name" : "Java HotSpot(TM) 64-Bit Server VM",
"start_time_in_millis" : 1536146549243,
"mem" : {
"heap_init_in_bytes" : 268435456,
"heap_max_in_bytes" : 1056309248,
"non_heap_init_in_bytes" : 2555904,
"non_heap_max_in_bytes" : 0
},
"gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ]
}
}
Verification of Energylogserver GUI service
To verify of Energylogserver GUI service you can use following command:
control the Energylogserver GUI service via systemd:
systemctl status logserver-gui
output:
● logserver-gui.service - Logserver-GUI
Loaded: loaded (/etc/systemd/system/logserver-gui.service; disabled; vendor preset: disabled)
Active: active (running) since Mon 2018-09-10 13:13:19 CEST; 23h ago
Main PID: 1330 (node)
CGroup: /system.slice/logserver-gui.service
└─1330 /usr/share/logserver-gui/bin/../node/bin/node --no-warnings /usr/share/logserver-gui/bin/../src/cli -c /etc/logserver-gui/logserver-gui.yml
control the Energylogserver GUI via port tcp/http:
curl -XGET '127.0.0.1:5601/'
output:
<script>var hashRoute = '/app/logserver-gui';
var defaultRoute = '/app/logserver-gui';
var hash = window.location.hash;
if (hash.length) {
window.location = hashRoute + hash;
} else {
window.location = defaultRoute;
}script>
Control the Energylogserver GUI via log file:
tail -f /var/log/messages
SIEM PLAN - Windows CP1250 decoding problem
If Siem Agent works on operation system which works using non Latin-script alphabet, the encoding of latter could cause dropping documents by logserver-probe. In logserver-probe log you can notice lines like the one below.
[2023-06-01T15:36:02,091][WARN ][logserver-probe.codecs.json ] Received an event that has a different character encoding than you configured. {:text=>"{\\\"timestamp\\\":\\\"2023-06-01T15:36:01.214+0000\\\",\\\"agent\\\":{\\\"id\\\":\\\"002\\\",\\\"name\\\":\\\"win10_laptop\\\"},\\\"manager\\\":{\\\"name\\\":\\\"SiemPlan.local\\\"},\\\"id\\\":\\\"1549035361.0\\\",\\\"full_log\\\":\\\"{\\\\\\\"type\\\\\\\":\\\\\\\"program\\\\\\\",\\\\\\\"ID\\\\\\\":78741874,\\\\\\\"timestamp\\\\\\\":\\\\\\\"2023/06/01 15:36:00\\\\\\\",\\\\\\\"program\\\\\\\":{\\\\\\\"format\\\\\\\":\\\\\\\"win\\\\\\\",\\\\\\\"name\\\\\\\":\\\\\\\"Skype\\x99 7.34\\\\\\\",\\\\\\\"architecture\\\\\\\":\\\\\\\"i686\\\\\\\",\\\\\\\"version\\\\\\\":\\\\\\\"7.34.102\\\\\\\",\\\\\\\"vendor\\\\\\\":\\\\\\\"Skype Technologies S.A.\\\\\\\",\\\\\\\"install_time\\\\\\\":\\\\\\\"20180212\\\\\\\",\\\\\\\"location\\\\\\\":\\\\\\\"C:\\\\\\\\\\\\\\\\Program Files (x86)\\\\\\\\\\\\\\\\Skype\\\\\\\\\\\\\\\\\\\\\\\"}}\\\",\\\"decoder\\\":{\\\"name\\\":\\\"syscollector\\\"},\\\"location\\\":\\\"syscollector\\\"}", :expected_charset=>"UTF-8"}
This is caused by default Windows encoding CP1250. You can change default encoding to UTF-8 by following this steps:
Go to Language settings …/media/14_System_Administration/08-00-01-language.png](…/media/14_System_Administration/08-00-01-language.png)
Open Administrative language settings …/media/14_System_Administration/08-00-02-admin-language.png](…/media/14_System_Administration/08-00-02-admin-language.png)
Click on
Change system locale...button …/media/14_System_Administration/08-00-03-region.png](…/media/14_System_Administration/08-00-03-region.png)Tick the checkbox
Use Unicode UTF-8..…/media/14_System_Administration/08-00-04-utf8.png](…/media/14_System_Administration/08-00-04-utf8.png)To make this change active you have to reboot system.