kafka101

1. zookeeper 无法启动


[2023-07-25 11:09:36,769] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2023-07-25 11:09:36,769] INFO zookeeper.commitLogCount=500 (org.apache.zookeeper.server.ZKDatabase)
[2023-07-25 11:09:36,773] INFO zookeeper.snapshot.compression.method = CHECKED (org.apache.zookeeper.server.persistence.SnapStream)
[2023-07-25 11:09:36,790] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
java.io.IOException: No snapshot found, but there are log entries. Something is broken!
	at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:290)
	at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:285)
	at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:505)
	at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:680)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:758)
	at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:130)
	at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:159)
	at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:112)
	at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:67)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:140)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:90)
[2023-07-25 11:09:36,793] INFO ZooKeeper audit is disabled. (org.apache.zookeeper.audit.ZKAuditProvider)
[2023-07-25 11:09:36,795] ERROR Exiting JVM with code 1 (org.apache.zookeeper.util.ServiceUtils)

查看zookeeper 的配置文件 ./config/zookeeper.properties


# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
# 
#    http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# the directory where the snapshot is stored.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
# Disable the adminserver by default to avoid port conflicts.
# Set the port to something non-conflicting if choosing to enable this
admin.enableServer=false
# admin.serverPort=8080

删除zk snapshot，rm -rf /tmp/zookeeper


bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

2. kafka 和 zookeeper 通信失败


ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$)
org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /consumers
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:570)
        at kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1883)
        at kafka.zk.KafkaZkClient.makeSurePersistentPathExists(KafkaZkClient.scala:1781)
        at kafka.zk.KafkaZkClient.$anonfun$createTopLevelPaths$1(KafkaZkClient.scala:1773)
        at kafka.zk.KafkaZkClient.$anonfun$createTopLevelPaths$1$adapted(KafkaZkClient.scala:1773)
        at scala.collection.immutable.List.foreach(List.scala:333)
        at kafka.zk.KafkaZkClient.createTopLevelPaths(KafkaZkClient.scala:1773)
        at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:635)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:221)
        at kafka.Kafka$.main(Kafka.scala:113)
        at kafka.Kafka.main(Kafka.scala)
[2023-07-25 15:13:39,827] INFO shutting down (kafka.server.KafkaServer)



[2023-07-25 15:13:31,545] WARN User 'client' not found in list of DIGEST-MD5 authenticateable users. (org.apache.zookeeper.server.auth.SaslServerCallbackHandler)
[2023-07-25 15:13:31,545] WARN No password found for user: null (org.apache.zookeeper.server.auth.SaslServerCallbackHandler)
[2023-07-25 15:13:31,546] WARN Client /[0:0:0:0:0:0:0:1]:62020 failed to SASL authenticate: {} (org.apache.zookeeper.server.ZooKeeperServer)
javax.security.sasl.SaslException: DIGEST-MD5: cannot acquire password for client in realm : zk-sasl-md5
        at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:601)
        at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:247)
        at org.apache.zookeeper.server.ZooKeeperSaslServer.evaluateResponse(ZooKeeperSaslServer.java:49)
        at org.apache.zookeeper.server.ZooKeeperServer.processSasl(ZooKeeperServer.java:1692)
        at org.apache.zookeeper.server.ZooKeeperServer.processPacket(ZooKeeperServer.java:1641)
        at org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:387)
        at org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:182)
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:347)
        at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
        at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
        at java.base/java.lang.Thread.run(Thread.java:831)
[2023-07-25 15:13:39,695] WARN Closing client connection due to SASL authentication failure. (org.apache.zookeeper.server.ZooKeeperServer)
[2023-07-25 15:13:51,666] INFO Expiring session 0x1004374b5d30000, timeout of 18000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)

原因是 kafka配置的 通信账户不正确(kafka_server_jaas.conf), 该配置要和 zookeeper 的配置一致 (zookeeper_server_jaas.conf)


// kafka_server_jaas.conf
Client {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="zk"
    password="zkpassword";
};



// zookeeper_server_jaas.conf
Server {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="zkadmin"
    password="zkadminpassword"
    user_zk="zkpassword";
};

3. kafka 报错



[2023-07-25 15:43:07,364] ERROR [Controller id=0, targetBrokerId=0] Connection to node 0 (10.100.6
5.93/10.100.65.93:9092) failed authentication due to: Authentication failed: Invalid username or p
assword (org.apache.kafka.clients.NetworkClient)



INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /10.100.65.93 (channelId=10.100.65.93:9092-10.100.65.93:57012-29) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector)

原因是 listener 配置


listeners=SASL_PLAINTEXT://10.100.65.93:9092
advertised.listeners=SASL_PLAINTEXT://10.100.65.93:9092
authorizer.class.name=kafka.security.authorizer.AclAuthorizer
allow.everyone.if.no.acl.found=true
security.inter.broker.protocol= SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
super.users=User:admin
Creative Commons License Flag Counter