Configure Filebeat to send Logs to Kafka on Confluent Cloud

featured image

Our game servers are configured to emit their log messages to log files, where we send them to our log aggregator/visualizer, in this case logz.io.

However, we also wanted to perform analytics and calculated KPIs on the fly so we decided to try out Kafka and ksqldb, so we set up a trial with Confluent Cloud to get a managed Kafka up an running as easy as possible.

We set up our Filebeat config as this:

### Output ###
output.kafka:
  hosts: ["<redacted>.northeurope.azure.confluent.cloud:9092"]
  topic: "raw-log"
  sasl-mechanism: PLAIN
  username: <redacted>
  password: <redacted>
  compression: none

With this setup we didn’t get any messages sent to our topic. The filebeat log file (found in “C:\ProgramData\filebeat\logs\”) had these rows in them:

2021-05-07T06:27:04.520Z	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to kafka(<redacted>.northeurope.azure.confluent.cloud:9092)
2021-05-07T06:27:04.520Z	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to kafka(<redacted>.northeurope.azure.confluent.cloud:9092) established
2021-05-07T06:27:04.520Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2021-05-07T06:27:04.520Z	INFO	[publisher]	pipeline/retry.go:223	  done
2021-05-07T06:27:07.958Z	ERROR	[kafka]	kafka/client.go:317	Kafka (topic=raw-log): kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

So, apparently something went wrong.

It turned out we had to explicitly enable ssl for the kafka connection in filebeat.yml config file, so with this we got everything working:

### Output ###
output.kafka:
  hosts: ["<redacted>.northeurope.azure.confluent.cloud:9092"]
  topic: "raw-log"
  sasl-mechanism: PLAIN
  username: <redacted>
  password: <redacted>
  compression: none
  ssl.enabled: true # <-- problem solver!

Hope this can help anyone else with the same problem!

Senaste inlägg

February 1,2022   •   2 min läsning

This is a follow up post to our last one regarding the launch of our platform. Welcome (or welcome again)! We have always valued communication with you…

comment icon 0
March 27,2022   •   < 1 min läsning

A short interview with Unordinal’s founder and CEO Erik Åkerfeldt. We thought it would be great for you to know us a bit better. We hope you enjoy! (PS: Lighting is…

comment icon 0

SUBSCRIBE ON OUR NEWSLETTER