How to configure filebeat, logstash-forwarder n lumberjack???

Hi Everyone! Plz Please, can anyone guide me about how to install and configure filebeat, lumberjack or logstash-forwarder on FreeBSD? Or any other way to ship (nginx) logs to logstash server from FreeBSD. I shall be very thankful to you.
 
Hello

Good day

In case that you still need this, i managed to make it work with logstash-forwarder, here 's what ia did to make it work:

>> Once you have ELK working (GUI working with its own traffic); make sure logstash is listening (lumberjack: 4433; you can change it also)

I use "logstash-forwarder" manually configure, not installed from ports/pkg

First create a directory for the certs and as well logstash-forwarder

mkdir /usr/local/logstash-forwarder/


Download the .zip

wget --no-check-certificate https://github.com/didfet/logstash-...d/0.2.4/logstash-forwarder-java-0.2.4-bin.zip (latest version)

unzip logstash-forwarder-java-0.2.4-bin.zip

create a dir for /lib and move things to the new /lib

mkdir /usr/local/logstash-forwarder/lib

move all of the main dir as well to the parent folder (just for simplicity at the command call)

create the .json file for its configuration

nano/vi/touch/cat /usr/local/logstash-forwarder/logstash-forwarder.json

configuration of the "logstash-forwarder.json"

-------------------------------------------------------------

{
"network": {
"servers": [ "127.0.0.1:4433" ], # > in the remote servers can be used the fqdn as in the cert was created as it, port has to eb the same as defined in logstash configuration file
"ssl ca": "/usr/local/etc/logstash/keystore.jks", # >> has to be the exact location in the server, in remote servers export it to the server location that you want
"timeout": 15
},
"files": [
{
"paths": [ "/var/log/nginx/access.log", "/var/log/nginx/error.log" ], # > here you define the logs to monitor/send, remember to create the proper filters in logstash for the data be recognize by kibana
"fields": { "type": "nginx" } # > put the type (nginx, syslog, etc)
}
]
}

------------------------------------------------------------

In case that you need here are the commands that i use for the certs that worked fine for me:

openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout /usr/local/logstash-forwarder/logstash-forwarder.key -out /usr/local/logstash-forwarder/logstash-forwarder.crt -subj /CN=name_of_the_server

keytool -importcert -trustcacerts -file /usr/local/logstash-forwarder/logstash-forwarder.crt -alias ca -keystore /usr/local/logstash-forwarder/keystore.jks # >> use the same path for the logstash-forwarder if you want for simplicity, just remember to call the correct path in the config files

In the part that prompts if you trust the cert, say "yes"



If all is fine at ELK level you should be able to initialize the .json at this point

/usr/local/bin/java -jar /usr/local/logstash-forwarder/logstash-forwarder-java-0.2.4.jar -config /usr/local/logstash-forwarder/logstash-forwarder.json

should see something similar to this:

root@client:/usr/local/logstash-forwarder # /usr/local/bin/java -jar logstash-forwarder-java-0.2.4.jar -config l ogstash-forwarder.json
2017-10-24 14:31:36,876 WARN FileWatcher - Could not load saved states : .logstash-forwarder-java (No such file or directory)
java.io.FileNotFoundException: .logstash-forwarder-java (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at com.fasterxml.jackson.core.JsonFactory.createJsonParser(JsonFactory.java:768)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1897)
at info.fetter.logstashforwarder.Registrar.readStateFromJson(Registrar.java:35)
at info.fetter.logstashforwarder.Registrar.readStateFromJson(Registrar.java:40)
at info.fetter.logstashforwarder.FileWatcher.setSincedb(FileWatcher.java:375)
at info.fetter.logstashforwarder.Forwarder.main(Forwarder.java:78)
2017-10-24 14:31:37,091 INFO FileWatcher - Watching file : /var/log/nginx/access.log
2017-10-24 14:31:37,134 INFO FileWatcher - Watching file : /var/log/nginx/error.log
2017-10-24 14:31:37,401 INFO Forwarder - Trying to connect to syslog:4433
2017-10-24 14:31:37,985 INFO LumberjackClient - Connected to syslog:4433
2017-10-24 14:31:37,996 INFO LumberjackClient - Sending 3 events
2017-10-24 14:34:49,365 INFO LumberjackClient - Sending 12 events
2017-10-24 14:38:15,764 INFO LumberjackClient - Sending 20 events


in case you need help with your nginx filter i can share mine

hope this help you
 
There's absolutely no reason to do this. Use the port or package.
Didi not work for me, never connected and had me errors, for now is working fine with no issues manually, i'm exploring filebeats, so eventually when get it working properly for our infrastructure possibly will switch.
 
Back
Top