ELK Stack crashes - Specifically with Kibana

GregTheHun · Aug 30, 2017

Hello all,

Well, I have a problem, it seems that when installing elasticsearch I've had no problems. The on screen commands were pretty easy to do as well as getting the entries in fstab that were required were easy enough to get elastic running.

However, when I tried to start Kibana after installing, after changing the one setting just to have a vanilla installation at least for starting off it keeps crashing every time I try to do "sudo service kibana start". This was after an initial onestart of it.

Any tips?

Thanks all.

GregTheHun · Aug 30, 2017

More specifically it was the elasticsearch.url setting it to "http://localhost:9200", which I can curl and get a response.

GregTheHun · Sep 3, 2017

This is the tail of kibana.log under /var/log:

Code:

{"type":"log","@timestamp":"2017-09-01T15:49:32Z","tags":["info","optimize"],"pid":86564,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2017-09-01T15:49:46Z","tags":["info","optimize"],"pid":98726,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2017-09-01T16:39:57Z","tags":["info","optimize"],"pid":76159,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2017-09-01T16:41:17Z","tags":["info","optimize"],"pid":95221,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2017-09-01T16:43:29Z","tags":["info","optimize"],"pid":26735,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2017-09-01T16:48:26Z","tags":["info","optimize"],"pid":86116,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2017-09-02T02:24:54Z","tags":["info","optimize"],"pid":58516,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2017-09-03T03:47:20Z","tags":["info","optimize"],"pid":27960,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2017-09-03T03:48:45Z","tags":["info","optimize"],"pid":40984,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}
{"type":"log","@timestamp":"2017-09-03T03:56:35Z","tags":["info","optimize"],"pid":76993,"message":"Optimizing and caching bundles for kibana, timelion and status_page. This may take a few minutes"}

This is the messages log after restarting:

Code:

Sep  3 16:23:56 freebsd kernel: pid 5665 (node), uid 80: exited on signal 6

If there are any other logs needed that I'm not thinking of I will post, just shoot me the full path.

Thanks all!

GregTheHun · Sep 14, 2017

Maybe if I took this from a different route.

If anybody has a kibana.yml configuration that works on FreeBSD that they are willing to share here, I can take a look at it and try one change at a time to see if it works out for me.

Thanks all!

GregTheHun · Sep 29, 2017

There is also the possibility that I can erase this system and start a fresh install of it to allow any configuration settings that anyone else has gotten to work.

GregTheHun · Oct 3, 2017

OK, so, now I've reset it. Now I have some Wifi issues that weren't there before.

However, anyone who has a successful setup I would love to see their kibana.yml and any other configuration tips you guys have.

Thanks.

GregTheHun · Oct 5, 2017

After resetting the system, these are my current configurations for how I setup elasticsearch and kibana (haven't installed logstash yet, just trying to get one piece at a time up)

elasticsearch.yml:

cluster.name: ELK-Lab
node.name: master
path.data: /var/db/elasticsearch
path.logs: /var/log/elasticsearch
path.scripts: /usr/local/libexec/elasticsearch

And kibana.yml:

server.port: 5601
server.host: "localhost"
server.basepath: ""
elasticsearch.url: "http://localhost:9200"
elasticsearch.preserveHost: true

Also a list of pkgs installed (most are from the install of elasticsearch):

alsa-lib-1.1.2 ALSA compatibility library
bash-4.4.12_2 GNU Project's Bourne Again SHell
c-ares-1.12.0_2 Asynchronous DNS resolver library
ca_root_nss-3.32.1 Root certificate bundle from the Mozilla Project
curl-7.55.1 Command line tool and library for transferring data with URLs
dejavu-2.37 Bitstream Vera Fonts clone with a wider range of characters
elasticsearch5-5.3.0 Full-text search engine for Java
expat-2.2.1 XML 1.0 parser written in C
fixesproto-5.0 Fixes extension headers
fontconfig-2.12.1,1 XML-based font configuration API for X Windows
freetype2-2.8 Free and portable TrueType font rendering engine
gettext-runtime-0.19.8.1_1 GNU gettext runtime libraries and programs
giflib-5.1.4 Tools and library routines for working with GIF images
htop-2.0.2 Better top(1) - interactive process viewer
icu-59.1,1 International Components for Unicode (from IBM)
indexinfo-0.2.6 Utility to regenerate the GNU info page index
inputproto-2.3.2 Input extension headers
java-zoneinfo-2017.b Updated Java timezone definitions
javavmwrapper-2.5_2 Wrapper script for various Java Virtual Machines
kbproto-1.0.7 KB extension headers
kibana5-5.3.0_1 Browser based analytics and search interface to ElasticSearch
libICE-1.0.9_1,1 Inter Client Exchange library for X11
libSM-1.2.2_3,1 Session Management library for X11
libX11-1.6.5,1 X11 library
libXau-1.0.8_3 Authentication Protocol library for X11
libXdmcp-1.1.2 X Display Manager Control Protocol library
libXext-1.3.3_1,1 X11 Extension library
libXfixes-5.0.3 X Fixes extension library
libXi-1.7.9,1 X Input extension library
libXrender-0.9.10 X Render extension library
libXt-1.1.5,1 X Toolkit library
libXtst-1.2.3 X Test extension
libfontenc-1.1.3_1 The fontenc Library
libnghttp2-1.26.0 HTTP/2.0 C Library
libpthread-stubs-0.4 This library provides weak aliases for pthread functions
libuv-1.14.1 Multi-platform support library with a focus on asynchronous I/O
libxcb-1.12_2 The X protocol C-language Binding (XCB) library
libxml2-2.9.4 XML parser library for GNOME
lsof-4.90.m,8 Lists information about open files (similar to fstat(1))
mkfontdir-1.0.7 Create an index of X font files in a directory
mkfontscale-1.1.2 Creates an index of scalable font files for X
node-8.6.0 V8 JavaScript for client and server
openjdk8-8.144.1 Java Development Kit 8
pftop-0.7_8 Utility for real-time display of statistics for pf
pkg-1.10.1 Package manager
recordproto-1.14.2 RECORD extension headers
renderproto-0.11.1 RenderProto protocol headers
sshguard-pf-1.7.1 Protect hosts from brute force attacks against ssh and other services using pf
sudo-1.8.20p2_3 Allow others to run commands as root
xextproto-7.3.0 XExt extension headers
xproto-7.0.31 X11 protocol headers

If anything else is needed, please let me know.

Lamia · Oct 22, 2017

Have you got the ELK working?
Installing it can be a pain in the a**. Try installing Kibana via pkg. If some thing breaks, try installing it again via the ports. You might need swing between the two installation methods before getting ELK to work. Ultimately, you should only install the right version of Kibana for your Elasticsearch. e.g. Kibana5 with Elasticsearch5.5.

GregTheHun · Oct 26, 2017

Thank you,

Actually, I have installed it (all on fresh installs) with pkg and ports and somehow it magically made itself work. However, now I cannot curl the default info you get when you "curl -XGET http://localhost:9200" even with the firewall disabled.

Seems to be a bit of whack-a-mole with the setup.

Lamia · Nov 4, 2017

Got it working?
You may first need to access it via its UI. After that, you can try the curl command in the dev page of the UI.

If you get it working and accessible via its UI, you will be fine with the running of commands at the terminal.

Alberth · Nov 7, 2017

Hello All

Good day

To provide some help maybe, i got it working over FReeBSD 11.0, these are the versions

elasticsearch2-2.4.6
logstash-2.4.1
kibana43-4.3.3
nginx-1.12.2_1,2

I'm using logstash-forwarder as log collector (haven't quite use yet filebeat)

i went through pkg installation for all of them and had no issues, now have it as reverse proxy and also working together with ModSecurity CRS 3.0

I can share all my procedure and specs, if you still need help on it.

What i have some problems right now is upgrading of trying to get working ELK in latest versions as Kibana 5.6 or at least 5.3 requires ES5, and that one is giving and error at initialization, that i can't solve yet

Let me know

Best Regards

Alberth

Lamia · Nov 10, 2017

What kind of error? Please paste it here.
And did you try installing the ES5 from the pkg manager and ports one after the other and then try using each?
Again, it took some time for me to get it working. I jumped from ports to pkg, most notably when I needed to get Kibana to run.
I also would advise you remove or disable the additional security - modsecurity CRS3.0 at least for now.

Alberth · Nov 13, 2017

lamia said:
What kind of error? Please paste it here.
And did you try installing the ES5 from the pkg manager and ports one after the other and then try using each?
Again, it took some time for me to get it working. I jumped from ports to pkg, most notably when I needed to get Kibana to run.
I also would advise you remove or disable the additional security - modsecurity CRS3.0 at least for now.

Hi Lamia

Good day

Thanks for the reply; here's the thing:

>> And did you try installing the ES5 from the pkg manager and ports one after the other and then try using each?

This is correct, i tried pkg and then from ports one by one and configuring one by one and enabling one by one, but in either way ES5 (5.3 in specific, haven't found a way to install the 5.6 yet for FreeBSD) is giving a different result. And just by testing i tried ES2 with Kibana 5 but at the initial page of Kibana said that requires at least ES5.

>> I also would advise you remove or disable the additional security - modsecurity CRS3.0 at least for now

I separated both servers, also for performance reasons, now all goes first through the WAF, then ELK does its job in a separate server.

Here is the error in specific from ES5.3 when i tried to initialize it:

java.lang.IllegalStateException: Unable to access 'path.scripts' (/usr/local/etc/elasticsearch/scripts)

I've been looking and researching about it with no luck so far, in some forums suggest to create it manually and link it but that was for version ES2.

Thanks for any help suggestion that you may have.

Best Regards

Alberth.

Lamia · Nov 14, 2017

Alberth said:
Hi Lamia

Good day

Thanks for the reply; here's the thing:

>> And did you try installing the ES5 from the pkg manager and ports one after the other and then try using each?

This is correct, i tried pkg and then from ports one by one and configuring one by one and enabling one by one, but in either way ES5 (5.3 in specific, haven't found a way to install the 5.6 yet for FreeBSD) is giving a different result. And just by testing i tried ES2 with Kibana 5 but at the initial page of Kibana said that requires at least ES5.

>> I also would advise you remove or disable the additional security - modsecurity CRS3.0 at least for now

I separated both servers, also for performance reasons, now all goes first through the WAF, then ELK does its job in a separate server.

Here is the error in specific from ES5.3 when i tried to initialize it:

java.lang.IllegalStateException: Unable to access 'path.scripts' (/usr/local/etc/elasticsearch/scripts)

I've been looking and researching about it with no luck so far, in some forums suggest to create it manually and link it but that was for version ES2.

Thanks for any help suggestion that you may have.

Best Regards

Alberth.

For the error:
Here is the error in specific from ES5.3 when i tried to initialize it:

java.lang.IllegalStateException: Unable to access 'path.scripts' (/usr/local/etc/elasticsearch/scripts)

>>>> Yes, I can confirm that elasticsearch throws that error. You should insert a path to the script in the elasticsearch.yml like the below:

path.scripts: /usr/local/etc/elasticsearch/scripts

Alberth · Nov 15, 2017

lamia said:
For the error:
Here is the error in specific from ES5.3 when i tried to initialize it:

java.lang.IllegalStateException: Unable to access 'path.scripts' (/usr/local/etc/elasticsearch/scripts)

>>>> Yes, I can confirm that elasticsearch throws that error. You should insert a path to the script in the elasticsearch.yml like the below:

path.scripts: /usr/local/etc/elasticsearch/scripts

Hi Lamia

Good day

Thanks for that suggestion

I'll try this between today and tomorrow and share the results.

Did you try it?

Best Regards

Alberth · Nov 16, 2017

Hi Lamia

lamia said:
For the error:
Here is the error in specific from ES5.3 when i tried to initialize it:

java.lang.IllegalStateException: Unable to access 'path.scripts' (/usr/local/etc/elasticsearch/scripts)

>>>> Yes, I can confirm that elasticsearch throws that error. You should insert a path to the script in the elasticsearch.yml like the below:

path.scripts: /usr/local/etc/elasticsearch/scripts

Hi Lamia

To give you an update about it;

First at all, thanks for the suggestion, worked very well.

In a staging env, i create a new server and installed ES5.3, logstash2 and Kibana5.3 Nginx (latest) and made the little change in the /usr/local/etc/elasticsearch/elasticsearch.yml adding the path.

Also created the dir and changed owner and group to elasticsearch

Had to disabled a couple of paths that were not able to start: path.work/path.plugins, that i had in the older version.

After this went through the rest of the installation and configuration for the ELK stack and now all services are working fine and starting up fine, i was able to access the new Kibana GUI and as well to feed it with logs (still using logstash-forwarder).

So i will make the change in production shortly.

Working great so far!!.

I will like to ask you if you play a little with logstash-modsecurity so far, i'm trying to make it work to display over Kibana but no luck yet.

Thanks again.

Best Regards

Alberth.

Lamia · Nov 18, 2017

Alberth said:
Hi Lamia

Hi Lamia

To give you an update about it;

First at all, thanks for the suggestion, worked very well.

In a staging env, i create a new server and installed ES5.3, logstash2 and Kibana5.3 Nginx (latest) and made the little change in the /usr/local/etc/elasticsearch/elasticsearch.yml adding the path.

Also created the dir and changed owner and group to elasticsearch

Had to disabled a couple of paths that were not able to start: path.work/path.plugins, that i had in the older version.

After this went through the rest of the installation and configuration for the ELK stack and now all services are working fine and starting up fine, i was able to access the new Kibana GUI and as well to feed it with logs (still using logstash-forwarder).

So i will make the change in production shortly.

Working great so far!!.

I will like to ask you if you play a little with logstash-modsecurity so far, i'm trying to make it work to display over Kibana but no luck yet.

Thanks again.

Best Regards

Alberth.

Hello Alberth,
I am delighted that my suggestion worked. Please always use the thanks button below a contribution to express your gratitude, perhaps, in addition to your textual expression. You could that for me for all my suggestions above and this new one.

W.r.t. logstash-modsecurity, I have not tried using it. A quick look at it showed that it is a set of filters for IDS/IPS. One advice I would give you is that you do not force install apps/packages that are not available in the freebsd freshports unless you know what you are doing. And for these filters, you should rather add them one-by-one. You will easily find out if one of the files is breaking your ELK than just adding all the filters at a go. That would save your hundreds of hours of troubleshooting

.

You may find out that you need install a package or two for the modsecurity filters to work and such packages might not be available in the freshports. If my memory serves me right, one of such libraries/apps is the logstash-translate plugin. There is not a version of if that would work with the ELK ver. 5. You may need to downgrade your ELK to get it installed. Would you want to downgrade your ELK for a filter/plugin? There are a bunch of other libraries and apps like that. This situation is not only perculiar to the ELK.

Alberth · Nov 21, 2017

lamia said:
Hello Alberth,
I am delighted that my suggestion worked. Please always use the thanks button below a contribution to express your gratitude, perhaps, in addition to your textual expression. You could that for me for all my suggestions above and this new one.

W.r.t. logstash-modsecurity, I have not tried using it. A quick look at it showed that it is a set of filters for IDS/IPS. One advice I would give you is that you do not force install apps/packages that are not available in the freebsd freshports unless you know what you are doing. And for these filters, you should rather add them one-by-one. You will easily find out if one of the files is breaking your ELK than just adding all the filters at a go. That would save your hundreds of hours of troubleshooting .

You may find out that you need install a package or two for the modsecurity filters to work and such packages might not be available in the freshports. If my memory serves me right, one of such libraries/apps is the logstash-translate plugin. There is not a version of if that would work with the ELK ver. 5. You may need to downgrade your ELK to get it installed. Would you want to downgrade your ELK for a filter/plugin? There are a bunch of other libraries and apps like that. This situation is not only perculiar to the ELK.

Hi Lamia

I did not know the mechanics of the forum, apologize for that. Will keep it in mind.

I'll keep working on the logstash-modsecurity part, i think that is a good one as ModSecurity does not have its own GUI and is very helpful for the SOC guys. The good of Unix is that you have several ways to make the things happen.

I'll make a new post when get it working with all the details about it.

Thanks for all your help.

Best Regards

Heriberto.