Hey everyone ! I finally decided to monitor my applications more closely with Grafana. However I’m having issues building dashboards their logs.

Their logs are currently sent over syslog (in RFC3164 format) into telegraf. But it simply puts the whole message into the message field, so I can’t use specific fields (eg. URL for httpd, source IP for DNS requests, username for SSH, …) to build graphs.

I’ve read about grok patterns, but I have no idea how to use them.

Would someone have any pointer on how I could make sense out of these logs for later use ?

  • vegetaaaaaaa@lemmy.worldEnglish
    4·
    1 year ago

    I have a similar setup (all hosts sending logs through syslog protocol to a central collector), but the collector is graylog. A few years back it used to use Grok expressions, but now it has its own filter syntax. My notes on extractors/grok patterns are still there (unfold details). Can’t help you much more than that, sorry!

    • z3bra@lemmy.sdf.orgOPEnglish
      1·
      1 year ago

      It does help thank you ;)

      I’ve found that you can use custom grok patterns to parse logs just as grayling extractors do. I’m still trying to figure it out, but so far I could start parsing logs using a [[processor.parser]] block. I’ll document my findings when I get it working as I want it.

  • nrlulz@lemmy.worldEnglish
    1·
    1 year ago

    You said you’re using telegraf, I assume to collect them - where are you storing/querying them? Have you looked into using Loki/Promtail for this?

    • z3bra@lemmy.sdf.orgOPEnglish
      1·
      1 year ago

      I store and query them using influxdb. I checked Loki but apparently it’s main feature is that it store the message as a single field, this not parsing the log at all. I didn’t know about Promtail. Is it better suited than influxdb for my usecase ?

      • nrlulz@lemmy.worldEnglish
        1·
        1 year ago

        I don’t think Loki itself parses logs on ingestion at all. I’m not sure if Promtail can ship logs to influx, I’ve only ever used it to ship to Loki. Promtail can be configured to add or parse or labels from the logs it sends, or you can just parse them at query time using builtin parsers like logfmt, json or regex. The hard part here will be figuring out the query to pull out the metrics you want to graph, which sounds like where you’re stuck already. So it’s hard to say which is actually better suited here.

  • z3bra@lemmy.sdf.orgOPEnglish
    1·
    1 year ago

    I found how to parse and tokenize logs withing telegraf. One must use grok patterns to parse the logs. Here is the config sample I use:

    # bind locally to ingest syslog messages
[[inputs.syslog]]
   server = "udp://<ipaddress>:6514"
   syslog_standard = "RFC3164"

[[processors.parser]]
  parse_fields = ["message"]
  merge = "override"
  data_format = "grok"
  grok_patterns = ["%{HTTPD}", "%{GEMINI}"] # this must reference the name from grok_custom_patterns
  # format; PATTERN_NAME GROK_PATTERN…
  grok_custom_patterns = '''
HTTPD ^%{HOSTNAME:httphost} %{COMBINED_LOG_FORMAT} (?:%{IPORHOST:proxyip}|-) (?:%{NUMBER:proxyprot}|-)$
GEMINI ^(?:\"(?:gemini\:\/\/%{HOSTNAME:gmihost}(:%{NUMBER:gmiport})?%{NOTSPACE:request}|%{DATA:raw_request})\" %{NUMBER:response} %{NUMBER:bytes}|%{DATA})$
  '''

# send parsed logs to influxdb
[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "telegraf"

    Telegraf supports logstash core patterns, as well as its own custom patterns (like %{COMBINED_LOG_FORMAT}).

    You can then query your influxdb using the fields extracted from these patterns:

    > USE telegraf
> SELECT xff,httphost,request FROM syslog WHERE appname = 'httpd' AND verb = 'GET' ORDER BY time DESC