Solved logstash will not run as a service, but runs on cmdline

T

tOsYZYny

I am unable to run logstash as a service, but if I run the cmd and grab the cmdline from what the service is running, it runs fine. I have mounted procfs @ /proc for it to run. The cmdline that works for me is:

Code: 
LS_HOME=/usr/local/logstash /usr/local/openjdk11/bin/java -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=${ES_TMPDIR} -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -cp /usr/local/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/local/logstash/logstash-core/lib/jars/checker-compat-qual-2.0.0.jar:/usr/local/logstash/logstash-core/lib/jars/commons-codec-1.14.jar:/usr/local/logstash/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/usr/local/logstash/logstash-core/lib/jars/commons-logging-1.2.jar:/usr/local/logstash/logstash-core/lib/jars/error_prone_annotations-2.1.3.jar:/usr/local/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/local/logstash/logstash-core/lib/jars/guava-24.1.1-jre.jar:/usr/local/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/local/logstash/logstash-core/lib/jars/jackson-annotations-2.9.10.jar:/usr/local/logstash/logstash-core/lib/jars/jackson-core-2.9.10.jar:/usr/local/logstash/logstash-core/lib/jars/jackson-databind-2.9.10.8.jar:/usr/local/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.10.jar:/usr/local/logstash/logstash-core/lib/jars/jackson-dataformat-yaml-2.9.10.jar:/usr/local/logstash/logstash-core/lib/jars/janino-3.1.0.jar:/usr/local/logstash/logstash-core/lib/jars/javassist-3.26.0-GA.jar:/usr/local/logstash/logstash-core/lib/jars/jruby-complete-9.2.20.1.jar:/usr/local/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/local/logstash/logstash-core/lib/jars/log4j-1.2-api-2.17.1.jar:/usr/local/logstash/logstash-core/lib/jars/log4j-api-2.17.1.jar:/usr/local/logstash/logstash-core/lib/jars/log4j-core-2.17.1.jar:/usr/local/logstash/logstash-core/lib/jars/log4j-jcl-2.17.1.jar:/usr/local/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.17.1.jar:/usr/local/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/local/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/local/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/local/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/local/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/local/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/local/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/local/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/local/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/local/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/local/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/local/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/local/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/local/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/local/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/local/logstash/logstash-core/lib/jars/reflections-0.9.11.jar:/usr/local/logstash/logstash-core/lib/jars/slf4j-api-1.7.30.jar:/usr/local/logstash/logstash-core/lib/jars/snakeyaml-1.23.jar org.logstash.Logstash --path.settings /usr/local/etc/logstash -l /var/log/logstash

I manually set LS_HOME, one would think that would be automatically set by the init script. Even after exporting it there, it still does not run. Logstash dies before any logs are written to /var/log/logstash or LS_HOME/logs so debugging is difficult.

I uninstalled and removed all configuration files and now have a plain vanilla installation and the same thing happens.
 
tOsYZYny said:
I am unable to run logstash as a service, but if I run the cmd and grab the cmdline from what the service is running, it runs fine.
Click to expand...
Doing this probably caused several files to be root owned now. The /usr/local/etc/rc.d/logstash rc(8) script runs everything on the logstash user and group. And that may now not be able to open certain files.

Code: 
dice@kibana:/usr/local/etc/logstash % ps -ax | grep logstash
  603  -  Is       0:00.16 daemon: /usr/local/logstash/bin/logstash[604] (daemon)
38333  0  S+       0:00.00 grep logstash
dice@kibana:/usr/local/etc/logstash %

So, check the permissions in /var/log/logstash/ (directory and the files within it). Also verify /var/db/logstash/ and /var/run/logstash/.

It should start just fine with the default logstash.conf (obviously configure the correct output) and logstash.yml.
 
Ah, yes, good points, /var/log/logstash was one that needed fixed earlier and I caught that, but there must be others. I'm getting closer, I'm running it as logstash now and able to get an error message whereas before I was shooting in the dark.
 
I'm working through this:

Code: 
/usr/local/logstash/logstash-core-plugin-api/lib/logstash-core-plugin-api/version.rb:22: warning: The Psych YAML extension failed to load.
Check your env for conflicting versions of SnakeYAML
See [URL]https://github.com/jruby/jruby/wiki/FAQs#why-does-the-psych-yaml-extension-fail-to-load-in-my-environment[/URL]
[FATAL] 2022-12-29 09:07:20.479 [main] Logstash - Logstash stopped processing because of an error: (GemspecError)
[!] There was an error while loading `logstash-core-plugin-api.gemspec`: load error: psych -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory. Bundler cannot continue.
Fox]","{audio/vnd.nuera.ecelp7470=[URL]http://www.iana.org/assig[/URL]
 #  from /usr/local/logstash/logstash-core-plugin-api/logstash-core-plugin-api.gemspec:23
 #  -------------------------------------------
 #
 >  require "logstash-core-plugin-api/version"
 #
 #  -------------------------------------------
fs":{"person":["Michael_Fox"],"template":["audio/vnd.nuera.
org.jruby.exceptions.StandardError: (GemspecError)
[!] There was an error while loading `logstash-core-plugin-api.gemspec`: load error: psych -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory. Bundler cannot continue.
gistered":true},{"content-type":"audio/vnd.qcelp","encoding
 #  from /usr/local/logstash/logstash-core-plugin-api/logstash-core-plugin-api.gemspec:23
 #  -------------------------------------------
 #
 >  require "logstash-core-plugin-api/version"
 #
 #  -------------------------------------------
reuil]","{audio/vnd.rhetorex.32kadpcm=[URL]http://www.iana.org/a[/URL]
    at RUBY.eval_gemspec(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler.rb:680) ~[?:?]
    at RUBY.load_gemspec_uncached(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler.rb:597) ~[?:?]
    at org.jruby.RubyDir.chdir(org/jruby/RubyDir.java:422) ~[jruby-complete-9.2.20.1.jar:?]
    at RUBY.chdir(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/shared_helpers.rb:52) ~[?:?]
    at RUBY.mon_synchronize(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/monitor.rb:237) ~[?:?]
    at org.jruby.RubyThread.handle_interrupt(org/jruby/RubyThread.java:759) ~[jruby-complete-9.2.20.1.jar:?]
    at RUBY.mon_synchronize(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/monitor.rb:236) ~[?:?]
    at org.jruby.RubyThread.handle_interrupt(org/jruby/RubyThread.java:759) ~[jruby-complete-9.2.20.1.jar:?]
    at RUBY.mon_synchronize(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/monitor.rb:233) ~[?:?]
    at RUBY.chdir(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/shared_helpers.rb:51) ~[?:?]
    at RUBY.load_gemspec_uncached(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler.rb:596) ~[?:?]
    at RUBY.load_gemspec(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler.rb:582) ~[?:?]
    at RUBY.load_gemspec(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/source/path. (AAC)"},"encoding":"base64","extensions":["aac"],"registerb:162) ~[?:?]
    at RUBY.load_spec_files(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/source/path.rb:177) ~[?:?]
    at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1821) ~[jruby-complete-9.2.20.1.jar:?]
    at RUBY.load_spec_files(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/source/path.rb:176) ~[?:?]
    at RUBY.local_specs(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/source/path.rb:107) ~[?:?]
    at RUBY.specs(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/source/path.rb:115) ~[?:?]
    at RUBY.specs_for_source_changed?(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/definition.rb:576) ~[?:?]
    at RUBY.specs_changed?(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/definition.rb:561) ~[?:?]
    at RUBY.converge_paths(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/definition.rb:605) ~[?:?]
    at org.jruby.RubyArray.any?(org/jruby/RubyArray.java:4553) ~[jruby-complete-9.2.20.1.jar:?]
    at RUBY.converge_paths(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/definition.rb:604) ~[?:?]
    at RUBY.initialize(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/definition.rb:134) ~[?:?]
    at RUBY.to_definition(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/dsl.rb:219) ~[?:?]
    at RUBY.evaluate(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/dsl.rb:13) ~[?:?]
    at RUBY.build(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/definition.rb:38) ~[?:?]
    at RUBY.definition(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler.rb:208) ~[?:?]
    at RUBY.setup(/usr/local/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler.rb:156) ~[?:?]
    at RUBY.setup!(/usr/local/logstash/lib/bootstrap/bundler.rb:87) ~[?:?]
    at usr.local.logstash.lib.bootstrap.environment.<main>(/usr/local/logstash/lib/bootstrap/environment.rb:89) ~[?:?]

I would think it is a permissions error, but haven't found exactly what yet.

/tmp has the sticky bit set along with ${ES_TMPDIR}.

The other strange thing is that ES_TMPDIR is not resolving, it is literally ${ES_TMPDIR}. I just changed it to something more logical like /tmp/logstash-tmp (to avoid conflicting with /tmp/logstash) and that gives me a different error which I think I'm getting somewhere:

Code: 
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2022-12-29T09:22:00,971][INFO ][logstash.runner          ] Log4j configuration path used is: /usr/local/etc/logstash/log4j2.properties
[2022-12-29T09:22:00,981][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.17.7", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.16+8-1 on 11.0.16+8-1 +jit [freebsd-x86_64]"}
[2022-12-29T09:22:00,983][INFO ][logstash.runner          ] JVM bootstrap flags: [-XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/logstash-tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log]
Your settings are invalid. Reason: Path "/usr/local/logstash/data/dead_letter_queue" must be a writable directory. It is not writable.
[2022-12-29T09:22:00,993][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
    at usr.local.logstash.lib.bootstrap.environment.<main>(/usr/local/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

So, very clearly, /usr/local/logstash/data/dead_letter_queue must be writable.
 
Thanks, it is working now with the vanilla configuration.

I think the culprit is that I am using a bunch of scripts to setup my system and part of those scripts copy configuration files over. It would be using tar to copy the files and it must be changing ownership of files and directories causing this issue. Instead, I think I need to just copy the files and use existing permissions for the directories already there.
 
I usually set path.data, I don't like services (or anything else for that matter) having write access to anything in /usr/local/. On various Linux distributions path.data is often set to /var/lib/logstash. On FreeBSD I think /var/db/logstash is a good location for it.
 
Good point, yes, that makes sense, then you know exactly what is read-only and what is used for data / writes.
 
tOsYZYny said:
I think the culprit is that I am using a bunch of scripts to setup my system and part of those scripts copy configuration files over. It would be using tar to copy the files and it must be changing ownership of files and directories causing this issue. Instead, I think I need to just copy the files and use existing permissions for the directories already there.
Click to expand...
The -p option to tar(1) causes it to preserve ownership and permissions.
 
You must log in or register to reply here.
Back
Top