Our game servers are configured to emit their log messages to log files, where we send them to our log aggregator/visualizer, in this case logz.io.
However, we also wanted to perform analytics and calculated KPIs on the fly so we decided to try out Kafka and ksqldb, so we set up a trial with Confluent Cloud to get a managed Kafka up an running as easy as possible.
We set up our Filebeat config as this:
### Output ###
output.kafka:
hosts: ["<redacted>.northeurope.azure.confluent.cloud:9092"]
topic: "raw-log"
sasl-mechanism: PLAIN
username: <redacted>
password: <redacted>
compression: none
With this setup we didn’t get any messages sent to our topic. The filebeat log file (found in “C:\ProgramData\filebeat\logs\”) had these rows in them:
2021-05-07T06:27:04.520Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to kafka(<redacted>.northeurope.azure.confluent.cloud:9092)
2021-05-07T06:27:04.520Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to kafka(<redacted>.northeurope.azure.confluent.cloud:9092) established
2021-05-07T06:27:04.520Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-05-07T06:27:04.520Z INFO [publisher] pipeline/retry.go:223 done
2021-05-07T06:27:07.958Z ERROR [kafka] kafka/client.go:317 Kafka (topic=raw-log): kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
So, apparently something went wrong.
It turned out we had to explicitly enable ssl for the kafka connection in filebeat.yml config file, so with this we got everything working:
### Output ###
output.kafka:
hosts: ["<redacted>.northeurope.azure.confluent.cloud:9092"]
topic: "raw-log"
sasl-mechanism: PLAIN
username: <redacted>
password: <redacted>
compression: none
ssl.enabled: true # <-- problem solver!
Hope this can help anyone else with the same problem!
0 