Logstash6 issue

suntzu00

suntzu00

Hey,

I'm having some problems with sysutils/logstash6 .

pkg info logstash6

Code: 
root@nemesis:~ # pkg info logstash6
logstash6-6.8.6
Name           : logstash6
Version        : 6.8.6
Installed on   : Thu Mar  5 14:25:21 2020 GMT
Origin         : sysutils/logstash6
Architecture   : FreeBSD:12:amd64
Prefix         : /usr/local
Categories     : sysutils java
Licenses       : APACHE20
Maintainer     : elastic@FreeBSD.org
WWW            : https://www.elastic.co/products/logstash
Comment        : Server-side data ingest and transfomation tool
Shared Libs provided:
        libjffi-1.2.so
Annotations    :
        FreeBSD_version: 1201000
        cpe            : cpe:2.3:a:elasticsearch:logstash:6.8.6:::::freebsd12:x64
        repo_type      : binary
        repository     : poudriere
Flat size      : 292MiB
Description    :
Logstash is an open source, server-side data processing pipeline that
ingests data from a multitude of sources simultaneously, transforms it,
and then sends it to your favorite “stash.” (Ours is Elasticsearch,
naturally.)

WWW: https://www.elastic.co/products/logstash

mount

Code: 
linprocfs on /usr/compat/linux/proc (linprocfs, local)
fdescfs on /dev/fd (fdescfs)
procfs on /proc (procfs, local)

Has anyone managed to get it to run as the logstash user?
I'm using the default values for the logstash config.

After running /usr/local/etc/rc.d/logstash start the process starts briefly then it stops.

The output for ps axuww before dying is:
Code: 
logstash 71399   63.0  1.4 2793124 472768  -  S    14:36         0:06.47 /usr/local/openjdk8/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/local/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/local/logstash/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/local/logstash/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/local/logstash/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/local/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/local/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/local/logstash/logstash-core/lib/jars/guava-22.0.jar:/usr/local/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/local/logstash/logstash-core/lib/jars/jackson-annotations-2.9.9.jar:/usr/local/logstash/logstash-core/lib/jars/jackson-core-2.9.9.jar:/usr/local/logstash/logstash-core/lib/jars/jackson-databind-2.9.9.3.jar:/usr/local/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.9.jar:/usr/local/logstash/logstash-core/lib/jars/janino-3.0.8.jar:/usr/local/logstash/logstash-core/lib/jars/javassist-3.22.0-GA.jar:/usr/local/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar:/usr/local/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/local/logstash/logstash-core/lib/jars/log4j-api-2.9.1.jar:/usr/local/logstash/logstash-core/lib/jars/log4j-core-2.9.1.jar:/usr/local/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/usr/local/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/local/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/local/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/local/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/local/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100

There's absolutely nothing in the logs.

Running su -m logstash -c '/usr/local/logstash/bin/logstash -f /usr/local/logstash/config/logstash.conf' provides this output:

Code: 
[ERROR] 2020-03-05 14:43:51.194 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (ArgumentError) could not find a temporary directory

Running the logstash daemon as root 'solves' the problem.

Thanks very much!
 
Code: 
dice@kibana:~ % ps -aux | grep logstash
logstash      73124   4.9 13.7 2744896  858428  -  I    00:38      90:49.85 /usr/local/openjdk8/bin/java -Xms1g -Xmx1g -XX:+UseConc
logstash      73123   0.0  0.0   10848    1608  -  Is   00:38       0:00.00 daemon: /usr/local/logstash/bin/logstash[73124] (daemon
dice          32595   0.0  0.0     524     220  3  D+   16:11       0:00.00 grep logstash

suntzu00 said:
Running the logstash daemon as root 'solves' the problem.
Click to expand...
That's your problem. Because you ran it as root certain files are now root owned and the logstash user doesn't have permissions on them any more.
 
I wish that was the case but it isn't. I've recreated the environment like 10 times(jails and bare metal) and I've only switched to root as the last resort.
I used the packages provided by FreeBSD and my own(poudriere).

SirDice Did you upgrade from a previous version or was it a fresh install?
Cheers!
 
Note that /usr/local/logstash/config/logstash.conf isn't the default config, it's /usr/local/logstash/logstash.conf.
 
According to /usr/local/etc/rc.d/logstash

Code: 
: ${logstash_home="/usr/local/logstash"}
: ${logstash_config="/usr/local/etc/logstash"}

and my /etc/rc.conf

Code: 
logstash_home="/usr/local/logstash"
logstash_config="/usr/local/etc/logstash"

so I don't think that's it.

in the logstash home folder /usr/local/logstash there's a symlink

Code: 
lrwxr-xr-x   1 logstash  logstash      15 Jan 22 17:58 config@ -> ../etc/logstash
to /usr/local/etc/logstash
 
What are the permissions and ownership on /usr/local/logstash/data and below? Also check /var/db/logstash. They should all be owned by logstash:logstash.

Also keep in mind that it takes some time for logstash to start and produce logging, I fell for that too and thought it didn't work. But it took a few minutes for it to initialize.
 
I've just checked and the permissions are correct. When I run it on foreground it takes between 5 and 10 seconds for the daemon to start. I've waited a minute and the daemon still doesn't start cause there's no logstash process running.
Now I'm installing the packages from https://pkg.freebsd.org/FreeBSD:12:amd64/release_1 to see if a previous version works.


LATER EDIT:
I've installed sysutils/logstash from https://pkg.freebsd.org/FreeBSD:12:amd64/release_1 and it works with no issues. So something happened between version logstash6-6.5.4 and logstash6-6.8.6.

LATER LATER EDIT:
After that I've switched the repo to http://pkg.FreeBSD.org/${ABI}/quarterly and performed an upgrade of all the packages. Quarterly has logstash6-6.8.5.
It no longer starts so back to square one.

The tests were performed in a clean environment with the default config files.
 
Can you try sysutils/logstash7? I see I upgraded some time ago:
Code: 
Dec 29 22:02:58 kibana pkg[3090]: logstash6-6.8.5 deinstalled
Dec 29 22:04:37 kibana pkg[3122]: logstash7-7.4.2 installed
 
It's my home test/play server anyway:
Code: 
root@kibana:~ # service logstash stop
Stopping logstash.
Waiting for PIDS: 33217.
root@kibana:~ # pkg delete logstash7
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
        logstash7: 7.6.0

Number of packages to be removed: 1

The operation will free 283 MiB.

Proceed with deinstalling packages? [y/N]: y
[1/1] Deinstalling logstash7-7.6.0...
You may need to manually remove /usr/local/etc/logstash/logstash.conf if it is no longer needed.
[1/1] Deleting files for logstash7-7.6.0: 100%
==> You should manually remove the "logstash" user.
==> You should manually remove the "logstash" group
root@kibana:~ # pkg install logstash6
Updating dicelan repository catalogue...
dicelan repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        logstash6: 6.8.6

Number of packages to be installed: 1

The process will require 292 MiB more space.
144 MiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching logstash6-6.8.6.txz: 100%  144 MiB  21.6MB/s    00:07
Checking integrity... done (0 conflicting)
[1/1] Installing logstash6-6.8.6...
===> Creating groups.
Using existing group 'logstash'.
===> Creating users
Using existing user 'logstash'.
[1/1] Extracting logstash6-6.8.6: 100%
=====
Message from logstash6-6.8.6:

--
To start logstash as an agent during startup, add

    logstash_enable="YES"

to your /etc/rc.conf.

Extra options can be found in startup script.
root@kibana:~ # service logstash start
Starting logstash.

After waiting a couple of seconds:
Code: 
root@kibana:~ # ps -aux | grep logstash
root          40873   0.0  0.0   11352    2256  3  S+   18:19       0:00.00 grep logstash

Bugger. Seems to be the same issue.

Back to 7 then:
Code: 
root@kibana:~ # pkg install logstash7
Updating dicelan repository catalogue...
dicelan repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        logstash7: 7.6.0

Number of packages to be installed: 1

The process will require 283 MiB more space.
138 MiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching logstash7-7.6.0.txz: 100%  138 MiB  20.7MB/s    00:07
Checking integrity... done (1 conflicting)
  - logstash7-7.6.0 conflicts with logstash6-6.8.6 on /usr/local/etc/logstash/logstash.conf.sample
Checking integrity... done (0 conflicting)
Conflicts with the existing packages have been found.
One more solver iteration is needed to resolve them.
The following 2 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
        logstash6: 6.8.6

New packages to be INSTALLED:
        logstash7: 7.6.0

Number of packages to be removed: 1
Number of packages to be installed: 1

The operation will free 8 MiB.

Proceed with this action? [y/N]: y
[1/2] Deinstalling logstash6-6.8.6...
You may need to manually remove /usr/local/etc/logstash/logstash.conf if it is no longer needed.
[1/2] Deleting files for logstash6-6.8.6: 100%
==> You should manually remove the "logstash" user.
==> You should manually remove the "logstash" group
[2/2] Installing logstash7-7.6.0...
===> Creating groups.
Using existing group 'logstash'.
===> Creating users
Using existing user 'logstash'.
[2/2] Extracting logstash7-7.6.0: 100%
=====
Message from logstash7-7.6.0:

--
To start logstash as an agent during startup, add

    logstash_enable="YES"

to your /etc/rc.conf.

Extra options can be found in startup script.
root@kibana:~ # service logstash start
Starting logstash.
Code: 
[2020-03-05T18:22:05,887][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
 
Yeah just tested sysutils/logstash7 and it works beautifully. Although now I'm a having a different problem since there's no beats-7 port yet to push the logs and I've no idea how beats-6.8.6 plays with elasticsearch-7.6.0 and logstash-7.6.0. By any chance have you tested that(I've no idea if you use beats or not)?

Thanks a lot!
 
I haven't done the switch for elasticsearch yet, that's still 6.8.6. I'm running a couple of filebeats (also 6.8.6), those are configured to connect to elasticsearch directly, that seems to be the recommended way nowadays. I have logstash running to accept plain syslog data. Logstash 7 doesn't appear to have a problem with an elasticsearch still at 6. I was actually planning on trying elasticsearch7 some time tonight.

The only thing I'm having issues with is the fact that both filebeat and metricbeat seem to insist on using linprocfs(5). I don't want to enable that everywhere. So I may ditch them altogether.
 
Thanks for the answers!

for the people looking how to install beats7(filebeat - that's the one that I use) on FreeBSD since there's no port for it:

The shell is shells/bash

pkg install go git gmake
mkdir ~/go
export GOPATH=~/go
mkdir -p $GOPATH/src/github.com/elastic
cd $GOPATH/src/github.com/elastic
git clone https://github.com/elastic/beats.git
cd beats/
git fetch --all --tags --prune
git checkout tags/v7.6.0 ### it matches the version of logstash7 and elasticsearch7 from the ports
git branch -a
cd filebeat/
go get
gmake


the file that you need is now in $GOPATH/bin/filebeat that need copying to /usr/local/sbin/filebeat

cp $GOPATH/bin/filebeat /usr/local/sbin/filebeat
chmod 555 /usr/local/sbin/filebeat


then the config file

cp $GOPATH/src/github.com/elastic/beats/filebeat/filebeat.yml /usr/local/etc/filebeat.yml

A startup script can be grabbed from here https://svnweb.freebsd.org/ports/head/sysutils/beats/files/filebeat.in?revision=470949&view=markup
 
SirDice said:
The only thing I'm having issues with is the fact that both filebeat and metricbeat seem to insist on using linprocfs(5). I don't want to enable that everywhere. So I may ditch them altogether.
Click to expand...

I think only metricbeat insists about the linux compat layer. For gathering system metrics I use(influxdb, telegraf, etc) way more reliable.
 
Well, I updated elasticsearch last night and managed to break just about everything else in the process. Elasticsearch itself seems to work just fine. I'm going to have another go during the weekend. Maybe I'll just start with a fresh system and redo everything from the ground up.
 
That upgrade never goes well in my experience. Beats, elasticsearch or logstash stopped working even after minor upgrades. I've replaced logstash6/kibana6 with graylog in my setup and it's working alright( it adds the mongo dependency but that's not too bad)
 
I've played around with logstash6 last weekend but ran into the same issue. It loops endlessly claiming it can't find a temporary directory. This error should only happen if /tmp doesn't have the sticky bit set, which it obviously has.

As I was having problems with Puppetserver and the latest OpenJDK version I thought this might be the case for logstash too. But unfortunately downgrading OpenJDK didn't work for Logstash (it did help with the Puppetserver issue).
 
that's not an OpenJDK problem but a Ruby one. the bundled version of Ruby that comes with logstash6 has changed and that's causing all the trouble


Code: 
/usr/local/logstash/bin/ruby -rtmpdir -e "p ENV['TMPDIR']; p Dir.tmpdir"
nil
ArgumentError: could not find a temporary directory
  tmpdir at /usr/local/logstash/vendor/jruby/lib/ruby/stdlib/tmpdir.rb:35
  <main> at -e:1
 
Hi there,

Sorry to re-float this topic but I am trying to get a working ELK stack on FreeBSD and at this moment I'm not able to. I've tried version 6 and 7 and each has some issues. Clean installs, simple configs.

Version 6 of the stack runs but logstash goes South after a few seconds. It leaves no logs and I've set the logging directive as follows:

Code: 
# ------------ Debugging Settings --------------
#
# Options for log.level:
#   * fatal
#   * error
#   * warn
#   * info (default)
#   * debug
#   * trace
#
 log.level: debug
 path.logs: /var/log/logstash/logstash.log
#

Similar behaviour on version 7, but this time Kibana is the one not working. Tries to start up but it takes two seconds for it to drop down.

Simple question: Is there any way to get any log from those crashes? Java plays a role here and I have no idea where to look.

My simplest setup configuration can be read here below, just in case anyone wants to look what the config changes are.

github.com

ELK/elk-stack-v6_freebsd-12.sh at master · Adminbyaccident/ELK

ELK stack install and config scripts. Contribute to Adminbyaccident/ELK development by creating an account on GitHub.
github.com github.com
 
SnowCowboy said:
Simple question: Is there any way to get any log from those crashes? Java plays a role here and I have no idea where to look.
Click to expand...
Logstash is written in Ruby and Kibana is a node.js app. Don't look for Java logs. The only part of the ELK stack that's written in Java is Elasticsearch.

I do wish a Kafka-Elasticsearch-Grafana stack (KEG?) would become more popular.
 
Jose said:
Logstash is written in Ruby and Kibana is a node.js app. Don't look for Java logs. The only part of the ELK stack that's written in Java is Elasticsearch.

I do wish a Kafka-Elasticsearch-Grafana stack (KEG?) would become more popular.
Click to expand...

Hola Jose,

I mentioned Java because it seems logstash is using it. See this following log of the same stack on a CentOS box.

Code: 
[albert@centollo8 ~]$ ps -ef | grep logstash
logstash  4071     1 45 13:20 ?        00:00:00 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.13.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-22.0.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.10.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.1.0.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.26.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.9.0.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.12.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.12.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.12.1.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/reflections-0.9.11.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings /etc/logstash
albert    4097  1908  0 13:20 pts/0    00:00:00 grep --color=auto logstash
[albert@centollo8 ~]$

Plus for logstash6 on FreeBSD 12.1, latest packages, openjdk8 is a direct dependency.

Anyhow, I've digged a bit but I am afraid I should do some more. Hopefully someone reads this and can shed some light. Maybe I am just missing some extra configuration bit.
 
You must log in or register to reply here.
Back
Top