RageLtManhow does one go about installing a plugin from source these days?
RageLtManSeems it only wants to install from contrib, and trying to build the gem results in errors looking for logstash-core
RageLtManprobably missing something from ENV, but the docs are... scant
quei have a logstash plugin, but it needs additional ruby gem ( not logstash plugin ) where should i install it ?
quei have a logstash plugin, but it needs additional ruby gem ( not logstash plugin ) where should i install it ?
queis someone here ?
queor is it dead channel at this time
quewarkolm: maybe You can help me ?
BlackCrypt0in the logstash conf.d directory does it read every conf file in there. So I can have a seperate conf file for Syslog and one for winlog beats or is that wrong.
rastroBlackCrypt0: it merges them together. for filters, they're combined in the alphabetical sort order of the file name (which is why you'll see "01-foo.conf", etc).
darkmoonvtIf I have multiple output stanzas and say filter1 { ...} output1 {...} filter2 {...} output2 {...} it's not going to do what I want.
darkmoonvtI'll get something closer to filter1 { } filter2 { } output1 { } output2 { } ?
rastrodarkmoonvt: it's really input{} -> filter{} -> output{} (all the filters run before the outputs)
darkmoonvtThanks. I was afraid of that.
rastrodarkmoonvt: what are you trying to accomplish?
darkmoonvtTransaction logs, but keeping the original logs (at least for now).
darkmoonvtOutput 1 wrote to the normal index, the output2 did an update to the transaction index.
darkmoonvtBut, when I trimmed out extra fields in filter2 (between the outputs), things disappeared from both indicies.
darkmoonvtExperimenting with the clone filter now. Had to wrap both outputs in conditionals.
darkmoonvt(sendmail logs, trying to get the to,from, and subject in the same document.)
rastrodarkmoonvt: clone and prune, me thinks.
darkmoonvtI'd love to use prune. The plugin tool can not find it.
darkmoonvt(for testing, I'm doing the prune by hand, but it would be useful elsewhere.)
smeadHey all, I'm trying logstash now. I've got it working... kind of. If I use filebeat, it outputs to both stdout and elasticsearch properly. If I use syslog, it outputs to stdout, but not elasticsearch
rastrosmead: check your LS and ES logs for errors.
darkmoonvtrastro: it's working with clone. Needs some polish, but it's even dealing with the timestamps well.
rastrodarkmoonvt: cool.
smeadSo, stash seems to get the message fine and parses it, but: Rejecting mapping update to [logstash-2017.11.30] as the final mapping would have more than 1 type: [doc, syslog]
smeadES is grumpy
rastrosmead: gave you a good error message though!
darkmoonvtDo you have an add_field => { "type" => "syslog" } somewhere?
smeaddarkmoonvt: No, that wasn't in the example I saw. That just goes in my grok ?
smeadrastro: sure did!
darkmoonvtPossibly. It depends.
darkmoonvtTry switching from the syslog input to a udp input listening on the same port.
darkmoonvt(the built in syslog parser doesn't catch everything, and can conflict with anything you write. It's easier to build your own.)
smeadI currently have udp{ port => 5000 type => syslog} I should just remove 'type => syslog' ?
darkmoonvtNo, that should work.
darkmoonvtTry a filter { if ([type] == "doc" and [type] == "syslog") { mutate { replace => [ "[type]","syslog" ] } } } maybe?
darkmoonvt(you might double check my syntax)
rastrosmead: do you have a second input, where you have `doc`?
smeadI do have a second input, a beats input
rastrosmead: does it set type = doc?
smeadstrangely, using the stdout plugin of logstash shows only one type
rastrosmead: : Unless you set document_type, the event type will be used if it exists otherwise the document type will be assigned the value of doc.
rastrosmead: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
smeadThanks for the help, I'll keep tryin'
torrancewrastro: I think 6.x had some weird changes to "type" on the ES side
rastrotorrancew: clearly one per index...
torrancew(not caught up on my releases yet)
darkmoonvtbeats 6.0 will tell you to use fields.type rather than document_type (which gets indexed as type).
rastrodarkmoonvt: that's the event's type, which is will use also (see doc, above).
gebbionehi, usually in my logs i get an entry for an application name as a word
gebbionelike [Application]
gebbioneso my grok filter specifies \[%{WORD:app}\]
gebbioneit looks like it is not matching for [exception.Exception]
gebbionehow can i make sure i does