• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

ELK Stack crashes - Specifically with Kibana

GregTheHun

Member


Messages: 25

#1
Hello all,

Well, I have a problem, it seems that when installing elasticsearch I've had no problems. The on screen commands were pretty easy to do as well as getting the entries in fstab that were required were easy enough to get elastic running.

However, when I tried to start Kibana after installing, after changing the one setting just to have a vanilla installation at least for starting off it keeps crashing every time I try to do "sudo service kibana start". This was after an initial onestart of it.

Any tips?

Thanks all.
 

GregTheHun

Member


Messages: 25

#3
This is the tail of kibana.log under /var/log:
Code:
{"type":"log","@timestamp":"2017-09-01T15:49:32Z","tags":["info","optimize"],"pid":86564,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2017-09-01T15:49:46Z","tags":["info","optimize"],"pid":98726,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2017-09-01T16:39:57Z","tags":["info","optimize"],"pid":76159,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2017-09-01T16:41:17Z","tags":["info","optimize"],"pid":95221,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2017-09-01T16:43:29Z","tags":["info","optimize"],"pid":26735,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2017-09-01T16:48:26Z","tags":["info","optimize"],"pid":86116,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2017-09-02T02:24:54Z","tags":["info","optimize"],"pid":58516,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2017-09-03T03:47:20Z","tags":["info","optimize"],"pid":27960,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2017-09-03T03:48:45Z","tags":["info","optimize"],"pid":40984,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2017-09-03T03:56:35Z","tags":["info","optimize"],"pid":76993,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
This is the messages log after restarting:
Code:
Sep  3 16:23:56 freebsd kernel: pid 5665 (node), uid 80: exited on signal 6
If there are any other logs needed that I'm not thinking of I will post, just shoot me the full path.

Thanks all! :D
 
Last edited by a moderator:

GregTheHun

Member


Messages: 25

#4
Maybe if I took this from a different route.

If anybody has a kibana.yml configuration that works on FreeBSD that they are willing to share here, I can take a look at it and try one change at a time to see if it works out for me.

Thanks all! :D
 

GregTheHun

Member


Messages: 25

#5
There is also the possibility that I can erase this system and start a fresh install of it to allow any configuration settings that anyone else has gotten to work.
 

GregTheHun

Member


Messages: 25

#6
OK, so, now I've reset it. Now I have some Wifi issues that weren't there before.

However, anyone who has a successful setup I would love to see their kibana.yml and any other configuration tips you guys have.

Thanks.
 

GregTheHun

Member


Messages: 25

#7
After resetting the system, these are my current configurations for how I setup elasticsearch and kibana (haven't installed logstash yet, just trying to get one piece at a time up)

elasticsearch.yml:

cluster.name: ELK-Lab
node.name: master
path.data: /var/db/elasticsearch
path.logs: /var/log/elasticsearch
path.scripts: /usr/local/libexec/elasticsearch

And kibana.yml:

server.port: 5601
server.host: "localhost"
server.basepath: ""
elasticsearch.url: "http://localhost:9200"
elasticsearch.preserveHost: true

Also a list of pkgs installed (most are from the install of elasticsearch):

alsa-lib-1.1.2 ALSA compatibility library
bash-4.4.12_2 GNU Project's Bourne Again SHell
c-ares-1.12.0_2 Asynchronous DNS resolver library
ca_root_nss-3.32.1 Root certificate bundle from the Mozilla Project
curl-7.55.1 Command line tool and library for transferring data with URLs
dejavu-2.37 Bitstream Vera Fonts clone with a wider range of characters
elasticsearch5-5.3.0 Full-text search engine for Java
expat-2.2.1 XML 1.0 parser written in C
fixesproto-5.0 Fixes extension headers
fontconfig-2.12.1,1 XML-based font configuration API for X Windows
freetype2-2.8 Free and portable TrueType font rendering engine
gettext-runtime-0.19.8.1_1 GNU gettext runtime libraries and programs
giflib-5.1.4 Tools and library routines for working with GIF images
htop-2.0.2 Better top(1) - interactive process viewer
icu-59.1,1 International Components for Unicode (from IBM)
indexinfo-0.2.6 Utility to regenerate the GNU info page index
inputproto-2.3.2 Input extension headers
java-zoneinfo-2017.b Updated Java timezone definitions
javavmwrapper-2.5_2 Wrapper script for various Java Virtual Machines
kbproto-1.0.7 KB extension headers
kibana5-5.3.0_1 Browser based analytics and search interface to ElasticSearch
libICE-1.0.9_1,1 Inter Client Exchange library for X11
libSM-1.2.2_3,1 Session Management library for X11
libX11-1.6.5,1 X11 library
libXau-1.0.8_3 Authentication Protocol library for X11
libXdmcp-1.1.2 X Display Manager Control Protocol library
libXext-1.3.3_1,1 X11 Extension library
libXfixes-5.0.3 X Fixes extension library
libXi-1.7.9,1 X Input extension library
libXrender-0.9.10 X Render extension library
libXt-1.1.5,1 X Toolkit library
libXtst-1.2.3 X Test extension
libfontenc-1.1.3_1 The fontenc Library
libnghttp2-1.26.0 HTTP/2.0 C Library
libpthread-stubs-0.4 This library provides weak aliases for pthread functions
libuv-1.14.1 Multi-platform support library with a focus on asynchronous I/O
libxcb-1.12_2 The X protocol C-language Binding (XCB) library
libxml2-2.9.4 XML parser library for GNOME
lsof-4.90.m,8 Lists information about open files (similar to fstat(1))
mkfontdir-1.0.7 Create an index of X font files in a directory
mkfontscale-1.1.2 Creates an index of scalable font files for X
node-8.6.0 V8 JavaScript for client and server
openjdk8-8.144.1 Java Development Kit 8
pftop-0.7_8 Utility for real-time display of statistics for pf
pkg-1.10.1 Package manager
recordproto-1.14.2 RECORD extension headers
renderproto-0.11.1 RenderProto protocol headers
sshguard-pf-1.7.1 Protect hosts from brute force attacks against ssh and other services using pf
sudo-1.8.20p2_3 Allow others to run commands as root
xextproto-7.3.0 XExt extension headers
xproto-7.0.31 X11 protocol headers



If anything else is needed, please let me know.
 

lamia

New Member

Thanks: 4
Messages: 17

#8
Have you got the ELK working?
Installing it can be a pain in the a**. Try installing Kibana via pkg. If some thing breaks, try installing it again via the ports. You might need swing between the two installation methods before getting ELK to work. Ultimately, you should only install the right version of Kibana for your Elasticsearch. e.g. Kibana5 with Elasticsearch5.5.
 

GregTheHun

Member


Messages: 25

#9
Thank you,

Actually, I have installed it (all on fresh installs) with pkg and ports and somehow it magically made itself work. However, now I cannot curl the default info you get when you "curl -XGET http://localhost:9200" even with the firewall disabled.

Seems to be a bit of whack-a-mole with the setup.
 

lamia

New Member

Thanks: 4
Messages: 17

#10
Got it working?
You may first need to access it via its UI. After that, you can try the curl command in the dev page of the UI.

If you get it working and accessible via its UI, you will be fine with the running of commands at the terminal.
 

Alberth

New Member


Messages: 7

#11
Hello All

Good day

To provide some help maybe, i got it working over FReeBSD 11.0, these are the versions

elasticsearch2-2.4.6
logstash-2.4.1
kibana43-4.3.3
nginx-1.12.2_1,2

I'm using logstash-forwarder as log collector (haven't quite use yet filebeat)

i went through pkg installation for all of them and had no issues, now have it as reverse proxy and also working together with ModSecurity CRS 3.0

I can share all my procedure and specs, if you still need help on it.

What i have some problems right now is upgrading of trying to get working ELK in latest versions as Kibana 5.6 or at least 5.3 requires ES5, and that one is giving and error at initialization, that i can't solve yet

Let me know

Best Regards

Alberth
 

lamia

New Member

Thanks: 4
Messages: 17

#12
What kind of error? Please paste it here.
And did you try installing the ES5 from the pkg manager and ports one after the other and then try using each?
Again, it took some time for me to get it working. I jumped from ports to pkg, most notably when I needed to get Kibana to run.
I also would advise you remove or disable the additional security - modsecurity CRS3.0 at least for now.
 

Alberth

New Member


Messages: 7

#13
What kind of error? Please paste it here.
And did you try installing the ES5 from the pkg manager and ports one after the other and then try using each?
Again, it took some time for me to get it working. I jumped from ports to pkg, most notably when I needed to get Kibana to run.
I also would advise you remove or disable the additional security - modsecurity CRS3.0 at least for now.
Hi Lamia

Good day

Thanks for the reply; here's the thing:

>> And did you try installing the ES5 from the pkg manager and ports one after the other and then try using each?

This is correct, i tried pkg and then from ports one by one and configuring one by one and enabling one by one, but in either way ES5 (5.3 in specific, haven't found a way to install the 5.6 yet for FreeBSD) is giving a different result. And just by testing i tried ES2 with Kibana 5 but at the initial page of Kibana said that requires at least ES5.

>> I also would advise you remove or disable the additional security - modsecurity CRS3.0 at least for now

I separated both servers, also for performance reasons, now all goes first through the WAF, then ELK does its job in a separate server.

Here is the error in specific from ES5.3 when i tried to initialize it:

java.lang.IllegalStateException: Unable to access 'path.scripts' (/usr/local/etc/elasticsearch/scripts)

I've been looking and researching about it with no luck so far, in some forums suggest to create it manually and link it but that was for version ES2.

Thanks for any help suggestion that you may have.

Best Regards

Alberth.
 

lamia

New Member

Thanks: 4
Messages: 17

#14
Hi Lamia

Good day

Thanks for the reply; here's the thing:

>> And did you try installing the ES5 from the pkg manager and ports one after the other and then try using each?

This is correct, i tried pkg and then from ports one by one and configuring one by one and enabling one by one, but in either way ES5 (5.3 in specific, haven't found a way to install the 5.6 yet for FreeBSD) is giving a different result. And just by testing i tried ES2 with Kibana 5 but at the initial page of Kibana said that requires at least ES5.

>> I also would advise you remove or disable the additional security - modsecurity CRS3.0 at least for now

I separated both servers, also for performance reasons, now all goes first through the WAF, then ELK does its job in a separate server.

Here is the error in specific from ES5.3 when i tried to initialize it:

java.lang.IllegalStateException: Unable to access 'path.scripts' (/usr/local/etc/elasticsearch/scripts)

I've been looking and researching about it with no luck so far, in some forums suggest to create it manually and link it but that was for version ES2.

Thanks for any help suggestion that you may have.

Best Regards

Alberth.
For the error:
Here is the error in specific from ES5.3 when i tried to initialize it:

java.lang.IllegalStateException: Unable to access 'path.scripts' (/usr/local/etc/elasticsearch/scripts)

  1. >>>> Yes, I can confirm that elasticsearch throws that error. You should insert a path to the script in the elasticsearch.yml like the below:

    path.scripts: /usr/local/etc/elasticsearch/scripts
 

Alberth

New Member


Messages: 7

#15
For the error:
Here is the error in specific from ES5.3 when i tried to initialize it:

java.lang.IllegalStateException: Unable to access 'path.scripts' (/usr/local/etc/elasticsearch/scripts)

  1. >>>> Yes, I can confirm that elasticsearch throws that error. You should insert a path to the script in the elasticsearch.yml like the below:

    path.scripts: /usr/local/etc/elasticsearch/scripts
Hi Lamia

Good day

Thanks for that suggestion

I'll try this between today and tomorrow and share the results.

Did you try it?

Best Regards
 

Alberth

New Member


Messages: 7

#16
Hi Lamia
For the error:
Here is the error in specific from ES5.3 when i tried to initialize it:

java.lang.IllegalStateException: Unable to access 'path.scripts' (/usr/local/etc/elasticsearch/scripts)

  1. >>>> Yes, I can confirm that elasticsearch throws that error. You should insert a path to the script in the elasticsearch.yml like the below:

    path.scripts: /usr/local/etc/elasticsearch/scripts
Hi Lamia

To give you an update about it;

First at all, thanks for the suggestion, worked very well.

In a staging env, i create a new server and installed ES5.3, logstash2 and Kibana5.3 Nginx (latest) and made the little change in the /usr/local/etc/elasticsearch/elasticsearch.yml adding the path.

Also created the dir and changed owner and group to elasticsearch

Had to disabled a couple of paths that were not able to start: path.work/path.plugins, that i had in the older version.

After this went through the rest of the installation and configuration for the ELK stack and now all services are working fine and starting up fine, i was able to access the new Kibana GUI and as well to feed it with logs (still using logstash-forwarder).

So i will make the change in production shortly.

Working great so far!!.


I will like to ask you if you play a little with logstash-modsecurity so far, i'm trying to make it work to display over Kibana but no luck yet.

Thanks again.

Best Regards

Alberth.
 

lamia

New Member

Thanks: 4
Messages: 17

#17
Hi Lamia


Hi Lamia

To give you an update about it;

First at all, thanks for the suggestion, worked very well.

In a staging env, i create a new server and installed ES5.3, logstash2 and Kibana5.3 Nginx (latest) and made the little change in the /usr/local/etc/elasticsearch/elasticsearch.yml adding the path.

Also created the dir and changed owner and group to elasticsearch

Had to disabled a couple of paths that were not able to start: path.work/path.plugins, that i had in the older version.

After this went through the rest of the installation and configuration for the ELK stack and now all services are working fine and starting up fine, i was able to access the new Kibana GUI and as well to feed it with logs (still using logstash-forwarder).

So i will make the change in production shortly.

Working great so far!!.


I will like to ask you if you play a little with logstash-modsecurity so far, i'm trying to make it work to display over Kibana but no luck yet.

Thanks again.

Best Regards

Alberth.

Hello Alberth,
I am delighted that my suggestion worked. Please always use the thanks button below a contribution to express your gratitude, perhaps, in addition to your textual expression. You could that for me for all my suggestions above and this new one.

W.r.t. logstash-modsecurity, I have not tried using it. A quick look at it showed that it is a set of filters for IDS/IPS. One advice I would give you is that you do not force install apps/packages that are not available in the freebsd freshports unless you know what you are doing. And for these filters, you should rather add them one-by-one. You will easily find out if one of the files is breaking your ELK than just adding all the filters at a go. That would save your hundreds of hours of troubleshooting :):).

You may find out that you need install a package or two for the modsecurity filters to work and such packages might not be available in the freshports. If my memory serves me right, one of such libraries/apps is the logstash-translate plugin. There is not a version of if that would work with the ELK ver. 5. You may need to downgrade your ELK to get it installed. Would you want to downgrade your ELK for a filter/plugin? There are a bunch of other libraries and apps like that. This situation is not only perculiar to the ELK.
 

Alberth

New Member


Messages: 7

#18
Hello Alberth,
I am delighted that my suggestion worked. Please always use the thanks button below a contribution to express your gratitude, perhaps, in addition to your textual expression. You could that for me for all my suggestions above and this new one.

W.r.t. logstash-modsecurity, I have not tried using it. A quick look at it showed that it is a set of filters for IDS/IPS. One advice I would give you is that you do not force install apps/packages that are not available in the freebsd freshports unless you know what you are doing. And for these filters, you should rather add them one-by-one. You will easily find out if one of the files is breaking your ELK than just adding all the filters at a go. That would save your hundreds of hours of troubleshooting :):).

You may find out that you need install a package or two for the modsecurity filters to work and such packages might not be available in the freshports. If my memory serves me right, one of such libraries/apps is the logstash-translate plugin. There is not a version of if that would work with the ELK ver. 5. You may need to downgrade your ELK to get it installed. Would you want to downgrade your ELK for a filter/plugin? There are a bunch of other libraries and apps like that. This situation is not only perculiar to the ELK.

Hi Lamia

I did not know the mechanics of the forum, apologize for that. Will keep it in mind.

I'll keep working on the logstash-modsecurity part, i think that is a good one as ModSecurity does not have its own GUI and is very helpful for the SOC guys. The good of Unix is that you have several ways to make the things happen.

I'll make a new post when get it working with all the details about it.

Thanks for all your help.

Best Regards

Heriberto.