Configure Filebeat to send Logs to Kafka on Confluent Cloud

featured image

Our game servers are configured to emit their log messages to log files, where we send them to our log aggregator/visualizer, in this case logz.io.

However, we also wanted to perform analytics and calculated KPIs on the fly so we decided to try out Kafka and ksqldb, so we set up a trial with Confluent Cloud to get a managed Kafka up an running as easy as possible.

We set up our Filebeat config as this:

### Output ###
output.kafka:
  hosts: ["<redacted>.northeurope.azure.confluent.cloud:9092"]
  topic: "raw-log"
  sasl-mechanism: PLAIN
  username: <redacted>
  password: <redacted>
  compression: none

With this setup we didn’t get any messages sent to our topic. The filebeat log file (found in “C:\ProgramData\filebeat\logs\”) had these rows in them:

2021-05-07T06:27:04.520Z	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to kafka(<redacted>.northeurope.azure.confluent.cloud:9092)
2021-05-07T06:27:04.520Z	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to kafka(<redacted>.northeurope.azure.confluent.cloud:9092) established
2021-05-07T06:27:04.520Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2021-05-07T06:27:04.520Z	INFO	[publisher]	pipeline/retry.go:223	  done
2021-05-07T06:27:07.958Z	ERROR	[kafka]	kafka/client.go:317	Kafka (topic=raw-log): kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

So, apparently something went wrong.

It turned out we had to explicitly enable ssl for the kafka connection in filebeat.yml config file, so with this we got everything working:

### Output ###
output.kafka:
  hosts: ["<redacted>.northeurope.azure.confluent.cloud:9092"]
  topic: "raw-log"
  sasl-mechanism: PLAIN
  username: <redacted>
  password: <redacted>
  compression: none
  ssl.enabled: true # <-- problem solver!

Hope this can help anyone else with the same problem!

Senaste inlägg

March 15,2021   •   2 min läsning

We believe that the next generation of Video Games are Social and Cross platform. The games will bring entertainment over years…

comment icon 0
March 8,2021   •   2 min läsning

We’re building a couple of demo games to demonstrate our technology for multiplayer game servers. We use Github for hosting …

comment icon 0
January 27,2022   •   3 min läsning

So, let’s say I want to make a multiplayer game. As a developer, there are so many things to think about during development: designing and programming…

comment icon 0

SUBSCRIBE ON OUR NEWSLETTER