What's Changed

This release introduces a new implementation of Pop consumption based on RocksDB (in the alpha phase, see RIP-73 for details). It improves and provides a migration plan for the capability of using RocksDB to store consumption queue indexes, adds support for the ability to cancel scheduled messages, enhances graceful online and offline capabilities, and includes numerous regular detail fixes related to HA, ACL, Metrics and Pop consumption.

• [ISSUE #8784] PrintMessageSubCommand support lmq by @RongtongJin in #8785
• [ISSUE #8790] Add Utils for put header to Metadata to avoid duplicate data by @dingshuangxi888 in #8792
• [ISSUE #8802] Update controller design.md by @luozongle01 in #8803
• [ISSUE #8798] Fix typo by @luozongle01 in #8799
• [ISSUE #8804] Clean offset when remove group offset by @leizhiyuan in #8805
• [ISSUE #8806] Fix autoBatch bug when connecting multiple RocketMQ clusters. by @luozongle01 in #8807
• [ISSUE #8810] Fix independent execution of e2e and benchmark deployments by @chi3316 in #8812
• [ISSUE #8816] Fix log typo. by @luozongle01 in #8817
• [ISSUE #8764] Implement consume lag estimation in cq rocksdb store by @LetLetMe in #8800
• [ISSUE #8824] Fix IllegalStateException caused by logical errors by @yx9o in #8825
• [ISSUE #8780] Implement asynchronous storage of ack/ck messages in pop consume to enhance performance by @RongtongJin in #8727
• [ISSUE #8835] When ck is in the buffer, incomplete ack will lead to message duplication. by @GenerousMan in #8836
• [ISSUE #8820] Fix variables match annotation (@RocketMQResource) by @mawen12 in #8821
• [ISSUE #8830] Fix atomic flush incorrect use and clean up code by @lizhanhui in #8830
• [ISSUE #8829] Fix make ConsumeQueueStore bottom most compression type configurable by @lizhanhui in #8841
• [ISSUE #8839] Remove unnecessary synchronized to improve concurrency by @lizhanhui in #8840
• [ISSUE #8833] Update netty version to 4.1.114 to fix CVE-2023-34462 by @ShannonDing in #8832
• [ISSUE #8846] Fix registerProducer should not be affected by concurrent scanNotAct… by @lizhanhui in #8847
• [ISSUE #8829] Support LMQ dispatch in case if Consume Queue Store is RocksDB-based by @lizhanhui in #8842
• [ISSUE #8850] Use correct log method by @mawen12 in #8851
• [ISSUE #8772] Remove lock mq step in broadcasting mode rebalancing by @redlsz in #8774
• [ISSUE #8442] [RIP-70] Extract adaptive lock mechanism by @3424672656 in #8663
• [ISSUE #8829] Feat: provide ConfigManagerV2 to make best uses of RocksDB by @lizhanhui in #8856
• [ISSUE #8822] Double write cq, reduce unnecessary switches by @LetLetMe in #8823
• [ISSUE #8875] Fix HAConnection leak by @crazywen in #8876
• [ISSUE #8725] Clean DefaultMQPushConsumer after start fail by @yuz10 in #8726
• [ISSUE #8829] Keep data version while reload and XXXConfigManagerV2 turns off sync by @lizhanhui in #8891
• [ISSUE #8885] Resolve the issue of inaccurate CK number statistics by @RongtongJin in #8886
• [ISSUE #8808] Resolve unsupported 'UseBiasedLocking' VM Option for JDK21 by @taomaree in #8809
• [ISSUE #8882] Change the compare method for acl signature to improve the security. by @dingshuangxi888 in #8883
• [ISSUE #8889] handle namespace outside the loop by @mawen12 in #8890
• [ISSUE #8906] Handle string toUpperCase outside the loop by @mawen12 in #8907
• [ISSUE #8599] Throw exception when receiving GO_AWAY twice to close channel. by @qianye1001 in #8862
• [ISSUE #8917] Topic route return none permission message queues for gRPC client by @dingshuangxi888 in #8919
• [ISSUE #8921] Add isWakeCommitWhenPutMessage for AIO by @zk-drizzle in #8922
• [ISSUE #8925] Fix the bug of update user for ACL 2.0 by @dingshuangxi888 in #8926
• [ISSUE #8909] Move nullcheck ahead by @jjastan in #8910
• [ISSUE #8942] Add incGroupAckNums and incGroupCkNums to LmqBrokerStatsManager by @RongtongJin in #8943
• [ISSUE #8829] Support for Persisting LMQ Consumer Offsets in Config V1 Using RocksDB by @RongtongJin in #8939
• [ISSUE #8945] Remove unnecessary operations from the critical section by @3424672656 in #8946
• [ISSUE #8940] Fix incorrect path for exportMetadataInRocksDBCommand by @RongtongJin in #8941
• [ISSUE #8935] Fix behind metrics unit error in timer message store by @GenerousMan in #8936
• [ISSUE #8765] Fix low performance of delay message when enable rocksdb consume queue by @yuz10 in #8766
• [ISSUE #8953] Improve IO for asynchronous delivery processes by @3424672656 in #8954
• [ISSUE #8829] Fix avoid memory overhead when there is large number of LMQ CQ by @lizhanhui in #8956
• [ISSUE #8947] Notify pop request before calculate consumer lag by @lizhimins in #8949
• [ISSUE #8933] feat: DefaultPullConsumer add balance switch. by @humkum in #8934
• [ISSUE #8929] Proxy adds message body empty check when send in grpc protocol by @qianye1001 in #8930
• [ISSUE #8877] Refactor lock in ReceiptHandleGroup to make the lock can be properly released when future can not be completed. by @qianye1001 in #8916
• [ISSUE #8955] Fix message buffer not release and dispatch thread exit in tiered storage by @lizhimins in #8965
• [ISSUE #8972] Adding the EnableLmqStats option allows monitoring of LMQ statistics at runtime by @RongtongJin in #8973
• [ISSUE #8961] Automatic recognition of address scheme in Topic Route by host by @dingshuangxi888 in #8962
• [ISSUE #8460] Set default broker name when revive found ack without broker name field by @lizhimins in #8981
• [ISSUE #8982] Dynamically install latest Go version for e2e pipeline by @chi3316 in #8985
• [ISSUE #8976] Modify file segment construct method by @lizhimins in #8977
• [ISSUE #8963] Fix code36 request sent to ns by @yx9o in #8964
• [ISSUE #8968] Introduce the clearRetryTopicWhenDeleteTopic option to enable precise external deletion of topics by @RongtongJin in #8969
• [ISSUE #8991] PrepareHeartbeatData should not be set by default subscriptionDataSet data by @weihubeats in #8992
• [ISSUE #7199] GrpcClientChannel header add null judgement by @weihubeats in #7238
• [ISSUE #9009] Optimize message accumulation display and Metrics in DefaultPullConsumer by @qianye1001 in #9010
• [ISSUE #8984] Fix the broker switch enableMixedMessageType doesn't work by @redlsz in #8986
• [ISSUE #8950] Remove Redundant nullcheck of configPath by @jjastan in #8951
• [ISSUE #8966] Feat: add remote address information to acl perm error by @humkum in #8967
• [ISSUE #9007] Fix client connection local ip is null in RemotingClient by @weihubeats in #9008
• [ISSUE #9014] Fix clusterAclConfigVersion command execution failed by @yx9o in #9017
• [ISSUE #7480] Fix the offset in the timerCheckPoint will not be corrected when the commitlog and consumeQueue are truncated by @RongtongJin in #7488
• [ISSUE #9015] Sync SysFlag and message body inflation status; allow omit of message body by @lizhanhui in #9016
• [ISSUE #8979] Add configurable switch for timer message retry logic by @chi3316 in #8980
• [ISSUE #9034] Refactor(LmqBrokerStatsManager): Refactor Redundant Code in LmqBrokerStatsManager by @asapple in #9034
• [ISSUE #8974] Support recalling of delay message by @imzs in #8975
• [ISSUE #9042] Update createTimerMessageStore call with new parameter by @chi3316 i...

This release contains some important bug fix and improvement since version 5.3.0

What's Changed

• [ISSUE #8375] Add more test coverage for MqClientAdminImpl by @yx9o in #8376
• [ISSUE #8365] Add remoting client non-oneway updateConsumerOffset function by @qianye1001 in #8368
• [ISSUE #8384] Add more test coverage for ClientConfig by @yx9o in #8385
• [ISSUE #8350] Fix properties store error: crc32ReservedLength make undefine mem… by @DongyuanPan in #8351
• [ISSUE #8365] Add remoting client non-oneway updateConsumerOffset function by @qianye1001 in #8391
• [ISSUE #8372] Add more test coverage for AdminBrokerProcessor by @TanXiang7o in #8373
• [ISSUE #8396] Fix typo in TraceConstants by @yx9o in #8398
• [ISSUE #8392] Add tests for QueryMessageProcessor by @TanXiang7o in #8393
• [ISSUE #8366] Eliminate deadlocks during the client shutdown process. by @YanYunyang in #8367
• [ISSUE #8400] Add tests for ConsumerManageProcessor by @TanXiang7o in #8401
• [ISSUE #8345] Add some tests for nameserver by @TanXiang7o in #8349
• [ISSUE #8411] Add more test coverage for DefaultMQPushConsumerImpl by @yx9o in #8412
• [ISSUE #8413] Add some test cases for commom module by @yueranniu in #8414
• [ISSUE #8405] Add the ability to write ConsumeQueue using fileChannel to prevent JVM crashes in some situations by @RongtongJin in #8403
• [ISSUE #8402] Fix init retry topic offset incorrect when EscapeBridge enabled by @imzs in #8404
• [ISSUE #8421] Add more test coverage for SlaveSynchronize by @TanXiang7o in #8422
• [ISSUE #8417] Add some test cases for org.apache.rocketmq.common.AclConfig by @TeFuirnever in #8418
• [ISSUE #8409] Fix tiered storage roll file logic if committing the last part of a file failed by @bxfjb in #8410
• [ISSUE #8437] Add more test coverage for ClientRemotingProcessor by @yx9o in #8433
• [ISSUE #8438] Fix broker return two messages when query message and index service bug by @lizhimins in #8439
• [ISSUE #8434] Add test cases for org.apache.rocketmq.common.action by @TeFuirnever in #8435
• [ISSUE #8446] Add more test coverage for MQClientInstance by @yx9o in #8447
• [ISSUE #8458] Add more test coverage for ProcessQueue by @yx9o in #8459
• [ISSUE #8454] Active brokers number should be initailized to 1 in broker heartbeat manager. by @ShannonDing in #8453
• [ISSUE #8448] Commitlog class annotation optimize by @Qoozm in #8452
• [ISSUE #8429] Fix trace message loss when traffic is heavy by @LetLetMe in #8430
• [ISSUE #8261] Avoid unnecessary waiting when a response is successfully returned by @3424672656 in #8272
• [ISSUE #8332] Fix ack msg which has reached maxReconsumeTimes by @cserwen in #8333
• [ISSUE #7731] Fix runBroker.cmd cannot start on Windows (#7731) by @hackjus in #8338
• [ISSUE #8465] Add more test coverage for ConsumeMessagePopConcurrentlyService by @yx9o in #8466
• [ISSUE #8432] Make autoDeleteUnusedStats default to true by @RongtongJin in #8431
• [ISSUE #8463] Some statistical items should also be deleted to prevent memory leakage when a topic or group is deleted by @RongtongJin in #8464
• [ISSUE #8472] Fix pop message delay due to not notify message arriving after suspend by @lizhimins in #8473
• only initialize all the variables once to speed up test ConsumeMessageConcurrentlyServiceTest by @TestBoost in #8436
• [ISSUE #8460] Improve the pop revive process when reading biz messages from a remote broker by @imzs in #8475
• [ISSUE #8476] Add test cases for org.apache.rocketmq.common.attribute by @TeFuirnever in #8477
• [ISSUE #8490] Fix getMaxReconsumeTimes calculation error in concurrent consumption mode by @yx9o in #8491
• [ISSUE #8495] Add more test coverage for PeekMessageProcessor by @TanXiang7o in #8498
• [ISSUE #8481] Improve delete and rolling strategy for tiered storage modules by @lizhimins in #8493
• [ISSUE #8486] Add more test coverage for BrokerMetricsManager by @ziiyee in #8487
• [ISSUE #8500] Add more test coverage for RocksDBLmqConsumerOffsetManager by @StudentGu in #8502
• [ISSUE #8496] Add more test coverage for ConsumeMessagePopOrderlyService by @yx9o in #8497
• [ISSUE #8503] Add test cases for org.apache.rocketmq.common.chain/coldstr/compression/consumer by @TeFuirnever in #8504
• [ISSUE #8514] Fix bazel-compile (ubuntu-latest) ci run failure by @yx9o in #8515
• [ISSUE #8510] Fix CI Failure in Test E2E Golang Job of PUSH-CI and PR-E2E-TEST by @chi3316 in #8520
• [ISSUE #8499] Modify batch send delay time to 3000ms in unit test. by @ShannonDing in #8522
• [ISSUE #8517] Fix client send UNREGISTER_CLIENT request twice may cause proxy NPE by @qianye1001 in #8528
• [ISSUE #8517] Add more test coverage for PullAPIWrapper by @yx9o in #8518
• [ISSUE #8460] Improve the pop revive process when reading biz messages from a remote broker - part2 by @imzs in #8494
• [ISSUE #8532] Fix flush metadata when commit file because of full file by @bxfjb in #8533
• [ISSUE #8531] Update jaeger-thrift, exclude unnecessary tomcat-embed-core by @leizhiyuan in #8536
• [ISSUE #8289] Fixed network bugs and merged networkutil code by @zekai-li in #8290
• [ISSUE #8519] Add test case for rocketmq acl module. by @HardX8 in #8508
• [ISSUE #8517] Add more test coverage for PullMessageService by @yx9o in #8542
• [ISSUE #8544] Add a retry mechanism to the unit test pipeline by @chi3316 in #8545
• [ISSUE #8547] Add more ControllableOffset by @syhleo in #8548
• [ISSUE #8551] Add more test coverage for AuthMigrator by @yx9o in #8552
• [ISSUE #8534] Supports timer message queries by @leizhiyuan in #8535
• [ISSUE #8565] Set specific permissions to trigger the workflow retry mechanism by @chi3316 in #8566
• [ISSUE #8553] Add UnitTest of OffsetSerialize by @syhleo in #8554
• [ISSUE #8562] Add more test coverage for StatefulAuthorizationStrategy by @yx9o in #8563
• [ISSUE #8549] Ipv6 enabled in broker, pickupStoreTimestamp size should be 20 by @leizhiyuan in #8567
• [ISSUE #8555] Make ctx constructed in scheduleRenewTask by @qianye1001 in #8556
• [ISSUE #8058] Support for upgrading metadata in json to rocksdb (#8045) by @LetLetMe in #8571
• [ISSUE #8573] Correct mismatched comments by @yx9o in #8574
• fix document typo in SlaveActingMasterMode.md by @maclong1989 in #8575
• [ISSUE #8544] Restore retry mechanism in unit test pipeline by @chi3316 in #8581
• [ISSUE #8137] Support pop consumption for light message queue by @HScarb in #8138
• [ISSUE #8576] Support Creating or Updating Subscription Groups in Batch by @caigy in #8577
• [ISSUE #8586] Add more test coverage for SelectMessageQueueByRandom by @yx9o in #8587
• [ISSUE #8592] Not notify long polling request when pop orderly consume blocked by @lizhimins in #8593
• [ISSUE #8607] Exclude loopback addresses when iterating over local network interfaces by @chi3316 in #8608
• [ISSUE #8601 ]When isPopShouldStop hit,unlock queueLockManager by @leizhiyuan in #8602
• [ISSUE #8591] Preliminary support for key commands of LMQ by @RongtongJin in #8590
• [ISSUE #8483] Optimize unnecessary broker reverse notification (notifyConsumerIdsChanged) in broadcast mode by @yx9o in h...

Feature

• [ISSUE #7560] [RIP-68] Support RocketMQ ACL 2.0 by @dingshuangxi888 in #7725

Improvement

• [ISSUE #7831] Make rat-check successful by @caigy in #7832
• [ISSUE #7821] Add notifyLast flag for PopLongPollingService by @drpmma in #7835
• [ISSUE #7815] Use createChannelAsync for async invoke rpc by @drpmma in #7816
• [ISSUE #7840] Update the version in the README.md document to 5.2.0 by @mxsm in #7841
• Add statistics for the production of transaction messages. by @kingkh1995 in #7766
• [ISSUE #7868] Use entrySet to close channel by @ChineseTony in #7869
• [ISSUE #7845] Simplify the AbstractSendMessageProcessor#buildMsgContext code using Optional by @mxsm in #7846
• [ISSUE #7875] Add constructor for ProxyTopicRouteData by @drpmma in #7876
• [ISSUE #7881] Add parameter configuration explanations for jRaft controller by @RongtongJin in #7882
• fix document typo by @zjko in #7721
• [ISSUE #7907] Revise the description of transaction messages in Rocke… by @Gezi-lzq in #7908
• [ISSUE #7904] Use string builder to concat string by @ChineseTony in #7905
• [ISSUE #7926] Delete the unnecessary 'else' statement by @oopooa in #7920
• [ISSUE #7923] Exclude the master that are currently down. by @zhuyuemufeng in #7925
• [ISSUE #7932] Rectify the modifier order in namesrv by @oopooa in #7921
• Fix some ambiguous logs by @caizhaomin1 in #7934
• [ISSUE #7929] Add some request codes to the permission verification for the admin role by @RongtongJin in #7930
• [ISSUE #7945] Make HAProxyMessageForwarder Scalable by @dingshuangxi888 in #7946
• Fix notification integration test in pop consume mode by @drpmma in #7947
• [ISSUE #7943] Add bazel config for auth module and fix bazel test by @dingshuangxi888 in #7944
• [ISSUE #7955] Don't set default auth metadata provider by @dingshuangxi888 in #7956
• [ISSUE #7974] Add repeatedly read same offset log to find unexpected situations by @lizhimins in #7975
• [ISSUE #7961] Use BoundaryType in binarySearchInCQByTime by @Koado in #7968
• Consume request count threshold log fix by @iamgd67 in #7969
• [ISSUE #7970] remove unnecessary type cast by @ChineseTony in #7971
• [ISSUE #7983] Use java optional by @ChineseTony in #7984
• [ISSUE #7988] Refector client trace by @drpmma in #7989
• [ISSUE #7963] Check consumer group existence in updateConsumerOffset by @redlsz in #7964
• [ISSUE #7988] Set enableTrace default to false by @drpmma in #8015
• [ISSUE #8020] Fix document typo by @hakusai22 in #8021
• [ISSUE #8032] Set checkDupInfo value from config by @LittleBoy18 in #8033
• [ISSUE #6398] Remove duplicate code in TopicPublishInfo by @syshenyao in #8034
• [ISSUE #8023] Add expression filtering capability to the pullBlockIfNotFound method of pull consumer by @RongtongJin in #8024
• [ISSUE #8044] Add Override annotation for AllocateMappedFileService#run by @mxsm in #8045
• build: fix workflow and skip failed test for auth module on mac by @HScarb in #8068
• [ISSUE #8100] Expose print audit log function by @dingshuangxi888 in #8101
• [ISSUE #8105] Fix typo about udpate user by @zhiliatom in #8106
• [ISSUE #5923] Fix tiered store README.md error about Configuration by @yuz10 in #8110
• [ISSUE #8092] Fixed non-idempotent test by @kaiyaok2 in #8093
• [ISSUE #8090] Optimize isSetEqual for DefaultLitePullConsumerImpl by @Willhow-Gao in #8091
• [ISSUE #5838] Retry to send when broker returns SYSTEM_BUSY by @cserwen in #5845
• [ISSUE #5923] Fix tiered store README.md error by @yuz10 in #8115
• [ISSUE #8124] Avoid scheduled tasks exiting because of unknown exceptions by @colin719 in #8121
• [ISSUE #8079] Add unit test for MQClientAPIExtTest by @drpmma in #8080
• [ISSUE #8118] Remove redundant mod in client by @miles-ton in #8119
• [ISSUE #8136] Replace with createProcessQueue and remove createProce… by @miles-ton in #8139
• [ISSUE #8145] Optimize some code style in client module by @Willhow-Gao in #8146
• [ISSUE #8148] Fix variable typo by @oopooa in #8149
• [ISSUE #8155] Fix doc typo by @oopooa in #8157
• [ISSUE #8142] Show time of create topic and subScriptionGroup by @superdev42 in #8143
• [ISSUE #8164] Log more accurate for the MQClientInstance#doRebalance by @miles-ton in #8165
• [ISSUE #8162] Optimize the logging printout for the ConfigManager#loadBak method by @mxsm in #8163
• [ISSUE #8129] Support topic reserved time in tiered storage by @yuz10 in #8130
• [ISSUE #8166] Optimize make compression type configurable in producer clinet level by @humkum in #8167
• [ISSUE #8182] Modify variable names to enhance readability #8182 by @Stephanie0002 in #8183
• [ISSUE #6873] If dns resolve controller address exception will update controllerAddresses to null by @weihubeats in #8180
• [ISSUE #8222] Fix spelling errors in comments by @yx9o in #8224
• Revert "[ISSUE #7757] UseCompositeByteBufto prevent memory copy." by @yuz10 in #8209
• [ISSUE #8053] Return SYSTEM_BUSY if PutMessageStatus is OS_PAGE_CACHE_BUSY by @biningo in #8054
• [ISSUE #8211] Add two metrics rocketmq_topic_create_execution_time and rocketmq_consumer_group_create_execution_time by @Stephanie0002 in #8212
• [ISSUE #8223] Add two metrics rocketmq_topic_number and rocketmq_consumer_group_number by @Stephanie0002 in #8225
• [ISSUE #8235] Add @OverRide annotation for handleDiskFlush method by @mxsm in #8236
• [ISSUE #8241] Remove duplicate code by @3424672656 in #8242
• [ISSUE #8245] Fix typo in user_guide.md by @liuzc9 in #8244
• [ISSUE #8197] Support fast filter message by tag in tiered storage by @lizhimins in #8198
• [ISSUE #8269] Support pop consumption filter in long polling service by @lizhimins in #8271
• [ISSUE #8265] Implement Batch Creation of Topics in RocketMQ Admin by @guyinyou in #8267
• [ISSUE #8278] Fix fail test by @3424672656 in #8279
• [ISSUE #8285] Add more test coverage for BrokerPreOnlineService by @yx9o in #8286
• [ISSUE #8276] Merge duplicate code in DefaultMQProducer constructor by @yx9o in #8277
• Restrict some actions to be triggered only in the official repository by @totalo in #7695
• [ISSUE #7941] Increase annotation by @3424672656 in #7959
• [ISSUE #8227] Repeated merging in the constructor by @yx9o in #8228
• [ISSUE #8281] Optimize pop log level by @drpmma in #8282
• [ISSUE #8293] Remove the redundant code from the MessageDecoder#encodeMessage method by @mxsm in #8294
• [ISSUE #7466] Added fast failure in adminBrokerThreadPoolQueue (#7466) by @yp969803 in #7798
• [ISSUE #8300] Add more test coverage for DefaultMQProducer by @yx9o in #8301
• fix document typo in SlaveActingMasterMode.md by @maclong1989 in #8315
• [ISSUE #8274] Optimize some codestyles and fix some warnings by @Mrhorse99 in #8275
• [ISSUE #8291] Format proxy watermark output by @zhiliatom in #8292
• [ISSUE #8324] Add more test coverage for DefaultMQProducerI...

What's Changed

• [ISSUE #7579] Fix globalWhiteRemoteAddressStrategy concurrent bug by @leizhiyuan in #7580
• [ISSUE #7663] Bump snakeyaml version from 1.32 to 2.0 by @ShannonDing in #7663
• [ISSUE #7649] Add validation in broker/nameserver configure updating command by @RongtongJin in #7649
• [ISSUE #7653] Remove getBrokerClusterAclConfig from admin and server by @RongtongJin in #7653
• [ISSUE #7688] Bump guava version from 31.1-jre to 32.0.1-jre by @RongtongJin in #7688
• [ISSUE #7716] MQFaultStrategy check queue if writable by @leizhiyuan in #7716
• [ISSUE #7783] Fix the selectOneMessageQueue to avoid change messageQueue permanently by @Tcytw in #7783
• [ISSUE #7740] Optimize LocalFileOffsetStore by @redlsz in #7745
• [ISSUE #7538] Fix wrong cachedMsgSize if msg body is changed in consumer callback by @yuz10 in #7820
• [ISSUE #7849] Promptly return a response when the validation parameters cannot be modified by @RongtongJin #7849

Full Changelog: rocketmq-all-4.9.7...rocketmq-all-4.9.8

Feature

• [ISSUE #7545] [RIP-65] Support efficient random index for massive messages by @lizhimins in #7546
• [ISSUE #7064] [RIP-66] Support KV(RocksDB) Storage for ConsumeQueue by @fujian-zfj in #7120
• [ISSUE #7300] [RIP-67] jRaft-Controller Implementation by @yulangz in #7301

Improvement

• [ISSUE #7381] Fix the problem of inaccurate timer message metric by @GenerousMan in #7382
• [ISSUE #7389] Fix the problem that getLastMappedFile function affects performance by @guyinyou in #7390
• [ISSUE #7393] Add timeout configuration for grpc server by @drpmma in #7394
• [ISSUE #7396] Fix wrong word in BrokerController#doResterBrokerAll by @francisoliverlee in #7397
• [ISSUE #7313] Enhancement Optimization Method name by @shriVATSA54 in #7420
• [ISSUE #7321] Refactor NettyRemotingAbstract with unify future implementation by @drpmma in #7322
• [ISSUE #7296] Add ChannelEventListener for MQClientAPIImpl by @drpmma in #7324
• [ISSUE #7429] Clean channel map when CLIENT_UNREGISTER in proxy by @xdkxlk in #7426
• [ISSUE #7431] Fix flaky test of DLedgerControllerTest#testBrokerLifecycleListener by @RongtongJin in #7432
• [ISSUE #7433] Update the version in the README.md document to 5.1.4 by @mxsm in #7434
• [ISSUE #7441] Fix log "Init the confirmOffset" keep printing error in controller mode by @fujian-zfj in #7442
• [ISSUE #7444] Fix testCalculateFileSizeInPath test can not rerun in same environment by @RongtongJin in #7445
• [ISSUE #7425] Add RocketmqControllerConsole log to print config to console by @ingdex in #7458
• [ISSUE #7451] Override toString for TopicConfigAndQueueMapping by @LetLetMe in #7452
• [ISSUE #7265] Adding how to debug in Idea document by @joeCarf in #7266
• [ISSUE #7464] Polish the pop logger format by @joeCarf in #7465
• [ISSUE #7330] Fix channel connect issue for goaway by @drpmma in #7467
• [ISSUE #7472] AddBroker removes parsing configuration from body by @RongtongJin in #7472
• [ISSUE #7454] Utilizing cache to avoid duplicate parsing by @guyinyou in #7455
• [ISSUE #7475] Check the input yaml and the path in ACL by @GenerousMan in #7475
• [ISSUE #7231] Fix proxy client language error by @weihubeats in #7200
• [ISSUE #7489] Code comment enhancement in example by @joeCarf in #7490
• [ISSUE #7493] Introduce a new event NettyEventType.ACTIVE by @drpmma in #7494
• [ISSUE #7486] Remove getBrokerClusterAclConfig from admin and server by @RongtongJin in #7486
• [ISSUE #7497] Extract the frequency of calling updateNamesrvAddr into a configuration by @xdkxlk in #7498
• [ISSUE #7437] Add the CRC check of commitlog by @guyinyou in #7468
• [ISSUE #7505] Do not validate the length when deleting a topic by @RongtongJin in #7506
• [ISSUE #7501] The broker supports idempotence in creating topics by @fuyou001 in #7502
• [ISSUE #7523] Message will flush timeout when transientStorePoolEnable=true and flushDiskType=SYNC_FLUSH by @fujian-zfj in #7524
• [ISSUE #7531] Clear POP_CK when sending messages by @xdkxlk in #7532
• [ISSUE #7419] Fix unstable UtilAllTest#testCalculateFileSizeInPath on Windows by @mureinik in #7419
• [ISSUE #7547] Let consumer be aware of message queue assignment change by @lizhanhui in #7548
• [ISSUE #7551] Reuse helper methods from Netty to free direct byte buffer by @lizhanhui in #7550
• [ISSUE #7570] Add default value for lastPopTimestamp by @HScarb in #7571
• [ISSUE #7462] Remove deprecated LocalTransactionExecuter by @985492783 in #7463
• [ISSUE #7567] TransactionProducer get the topic route before sending the message by @panzhi33 in #7569
• [ISSUE #7543] Retry topic v2 in pop by @drpmma in #7544
• [ISSUE #7577] SlaveActingMaster Timer Message retry without escape logic by @AYue-94 in #7578
• [ISSUE #7584] Add validation in broker/namesrv configure updating command. by @ShannonDing in #7584
• [ISSUE #7587] Add validation in broker container configure updating command by @RongtongJin in #7587
• [ISSUE #7586] Add set method for config black list in broker/namesrv/controller config. by @ShannonDing in #7586
• [ISSUE #7592] testCleanBuffer unit test modifies, changed non-direct … by @yp969803 in #7593
• [ISSUE #7534] Use high performance concurrent set to replace copyonwriteset by @keranbingaa in #7583
• [ISSUE #7585] Support message filtering in rocketmq tiered storage by @lizhimins in #7594
• [ISSUE #7614] Fix flaky test RocksDBMessageStoreTest by @lizhanhui in #7625
• [ISSUE #7638] Bump com.squareup.okio:okio-jvm from 3.0.0 to 3.4.0 by @dependabot in #7638
• [ISSUE #7627] Fix flaky test testSemiSyncReplicaWhenAdaptiveDegradation by @RongtongJin in #7631
• [ISSUE #7636] Bump snakeyaml from 1.32 to 2.0 by @RongtongJin in #7632
• [ISSUE #7637] Bump grpc from 1.50.0 to 1.53.0 by @RongtongJin in #7633
• [ISSUE #7634] Introduce controllableOffset to prevent unnecessary suspension when OFFSET_ILLEGAL by @drpmma in #7635
• [ISSUE #7585] Always return duplicate buffer when filter message and fix log format by @lizhimins in #7654
• [ISSUE #7543] Use "+" as the new separator for retry topic by @drpmma in #7655
• [ISSUE #7543] Only call a single type of retry topic in pop by @drpmma in #7665
• [ISSUE #7646] Optimize pull onException logging by @drpmma in #7647
• [ISSUE #7659] Trim property by @DongyuanPan in #7660
• [ISSUE #7656] ConfirmOffset directly takes the max offset when allAckInSyncStateSet is false by @RongtongJin in #7657
• [ISSUE #7669] Map variable delayLevelTable changed to ConcurrentSkipL… by @yp969803 in #7675
• [ISSUE #7679] Optimize the serialization of RemotingCommand processTimer property by @mxsm in #7683
• [ISSUE #7676] Use clientDecode for consuming message directly by @cserwen in #7677
• [ISSUE #7680] Bump guava version from 31.1-jre to 32.0.1-jre by @RongtongJin in #7681
• [ISSUE #7330] Add goaway and reconnection mechanism by @drpmma in #7331
• [ISSUE #7686] The bornTime is not set when using the popMessage API in cluster mode. by @dingshuangxi888 in #7687
• [ISSUE #7642] Add return value for sendHeartbeat related method by @drpmma in #7643
• [ISSUE #7644] Optimize client rebalance by @drpmma in #7645
• [ISSUE #7710] Handle blank string for UtilAll#split to fix the bugs of ACL by @Qinglong-Lee in #7712
• [ISSUE #7719] Add more property check for dealy message when auto-ba… by @zhiliatom in #7720
• [ISSUE #7543] Add enableRetryTopicV2 brokerConfig by @drpmma in #7734
• [ISSUE #7699] Add namespace v2 in client by @drpmma in #7700
• [ISSUE #7747] Simplify code checks using Optional in ClientRequestProcessor#getRouteInfoByTopic method by @mxsm in #7748
• [ISSUE #7495] Support Higher Java version in Windows by @joeCarf in #7507
• [ISSUE #7757] Use CompositeByteBuf to prevent memory copy. by @dao-jun in #7694
• [ISSUE #7760] Make timerSkipUnknownError can be set by config file by @leizhiyuan in #7763
• [ISSUE #7500] Enhanced metrics for timing and transactional messages by @GenerousMan in #7500
• [ISSUE #7699] Refactor NamespaceRpcHook by @drpmma in https://github.com/apache/rocketmq/pull/...

What's Changed

• [maven-release-plugin] prepare for next development iteration by @RongtongJin in #6939
• [ISSUE #4612] Fix trace not complete by @panzhi33 in #6941
• [ISSUE #6943] fix docs typo in docs/cn/controller/design.md #6943 by @yao-wenbin in #6942
• [ISSUE #6940] change dataReadAheadEnable default to false by @yuz10 in #6944
• [ISSUE #6945] Add doc issue template by @mxsm in #6946
• [ISSUE #6933] Support delete expired or damaged file in tiered storage and optimize fetch code by @lizhimins in #6952
• [ISSUE #6754] Support reentrant orderly consumption for proxy by @xdkxlk in #6755
• add removeOne method for ReceiptHandleGroup by @xdkxlk in #6955
• [ISSUE #6964] use the correct context in telemetry; polish the code structure by @xdkxlk in #6965
• [ISSUE #6933] Optimize delete topic in tiered storage by @lizhimins in #6973
• [ISSUE #6982]Update the version in the README.md document to 5.1.3 by @mxsm in #6983
• [ISSUE #6974] Feature/refector receipt processor by @drpmma in #6975
• [ISSUE #6742]Support daily build tests by @cryptoya in #6744
• [ISSUE #6991] Delete rocketmq.client.logUseSlf4j=true in JAVA_OPT by @DongyuanPan in #6992
• [ISSUE #6957] Support Proxy Protocol for gRPC and Remoting server by @dingshuangxi888 in #6958
• [ISSUE #6960]Added Slot formatting sketch comments by @mxsm in #6961
• [ISSUE #7008] Fix the issue of protocol parsing failure when using haproxy and tls together by @RongtongJin in #7009
• [ISSUE #6999] Add interface ReceiptHandleManager by @drpmma in #7000
• [ISSUE #7005] The bug label relies on manual identification rather than automatic application by @RongtongJin in #7006
• fix receive message activity attempt id not correct by @lizhimins in #7012
• [ISSUE #7010] Fix the HandshakeHandler returns when detect haproxy version need more data by @dingshuangxi888 in #7011
• [ISSUE #7013]polish ColdDataCheckService's logic by @GenerousMan in #7014
• [ISSUE #6979] fix opaque will be duplicate in multi client scene by @leizhiyuan in #6985
• [ISSUE #7018] fix append in tiered storage when message offset incorrect by @lizhimins in #7019
• Fix LEAK: HAProxyMessage.release() was not called before it's garbage-collected by @dingshuangxi888 in #7025
• [ISSUE #6968] fix grpc acl bug by @lyx2000 in #6969
• [ISSUE #7031] fix PoP caused broker memory leak bug by @fuyou001 in #7032
• [ISSUE #7029] Add a config to determine whether pop response should return the actual retry topic or tamper with the original topic by @RongtongJin in #7030
• [ISSUE #6879] ConcurrentHashMapUtils fails to solve the loop bug in JDK8 by @leeyiyu in #6883
• passing the renew event type to create the correct context by @xdkxlk in #7045
• [#ISSUE 7035] Fix correct min offset behavior in tiered storage by @lizhimins in #7038
• [ISSUE #7047] NettyRemotingClient#invokeOneway throw Exception with address by @gaoyf in #7048
• [ISSUE ##7036] rename method: getWriteQueueIdByBroker to getWriteQueu… by @yao-wenbin in #7037
• [ISSUE #3717][RIP-27] Auto batching in producer by @guyinyou in #3718
• [ISSUE #7061] Support forward HAProxyMessage for Multi Protocol server. by @dingshuangxi888 in #7062
• [ISSUE #7063] doc: fix typo in user_guide.md by @maclong1989 in #7060
• fix: npe in lockBatchMQ and unlockBatchMQ by @wuyoudexiao in #7078
• [ISSUE #7052] Ensuring consistency between broker and nameserver data when deleting a topic by @RongtongJin in #7066
• [ISSUE #7068] Fix failed to create syncer topic when the proxy was just started by @gaoyf in #7076
• [ISSUE #7056]fix: avoid close success channel if invokeSync most time cost on get connection for channel by @absolute8511 in #7057
• [ISSUE #6810] Fix the bug of mistakenly deleting data in clientChannelTable when the channel expires by @zd46319 in #7073
• [ISSUE #7039] fix: fix retry message filter when subtype is TAG by @HScarb in #7040
• [ISSUE #7091] Update the cd command in README by @AayushSaini101 in #7096
• [Bug] schedule CQ offset invalid. offset=77, cqMinOffset=0, cqMaxOffset=74, queueId=1 by @weihubeats in #7084
• [#ISSUE 7093] Avoid dispatch tasks too much cause dispatch task failed by @lizhimins in #7094
• [#ISSUE 7104] Add ReceiptHandleGroupKey for RenewEvent by @drpmma in #7105
• [ISSUE #7102] Making perm equal to 0 is valid by @RongtongJin in #7103
• [ISSUE #7074] Allow a BoundaryType to be specified when retrieving offset based on the timestamp by @Koado in #7082
• [ISSUE #7109] support the mixed topic type by @dingshuangxi888 in #7110
• [ISSUE #6962]operation.md Format adjustment by @haolinkong in #7106
• [ISSUE #7064] [RIP-66-1] Support KV(RocksDB) Storage for Metadata by @fujian-zfj in #7092
• [ISSUE #7135] Temporarily ignoring plainAccessValidator test by @RongtongJin in #7135
• [ISSUE 7117] check message is in memory or not when init consumer offset for pop by @xdkxlk in #7118
• [ISSUE #7146] Log output error needs to be corrected by @RongtongJin in #7147
• [ISSUE #7144] Accelerate the recovery speed of the tiered storage module by @lizhimins in #7145
• [ISSUE #7115] Fix the response message NPE by @dingshuangxi888 in #7116
• [ISSUE #7129] fix resource collisions in acl tests by @caigy in #7130
• [ISSUE #7153] Add switch for MIXED message type by @dingshuangxi888 in #7154
• [Feature 7155] add errlog when cmd err by @shirenchuang in #7157
• Change metadataThreadPoolQueueCapacity to 100000 by @drpmma in #7170
• [ISSUE #7172] Unified Chinese for Name Server by @yx9o in #7173
• [ISSUE #7176] Correct mismatched logs by @yx9o in #7177
• [ISSUE #7183] Correct mismatched commandDesc by @yx9o in #7184
• [ISSUE #7001] Fix IndexOutOfBoundsException when process pop response by @redlsz in #7003
• [ISSUE #7192] Correct typos by @yx9o in #7193
• [ISSUE #7189] Optimize time complexity of loop in ConsumerGroupInfo#updateSubscription by @drpmma in #7190
• To config the cardinalityLimit for openTelemetry metrics exporting an… by @lollipopjin in #7196
• [ISSUE #7178] refresh metadata after broker startup by @Ziy1-Tan in #7197
• [ISSUE #7201] Remove the DefaultMessageStore.class dependency in TransientStorePool by @guyinyou in #7202
• Update mqbroker to use runbroker.sh instead of runserver.sh when enabling --enable-proxy by @iamgd67 in #7150
• [ISSUE #7042] throw UnsupportedOperationException after client using TLS Request but Server in Disabled Mode. by @yao-wenbin in #7179
• [ISSUE #7205] support batch ack for pop orderly by @xdkxlk in #7206
• [ISSUE #7203] Set table reference the same object for setSubscriptionGroupTable method by @RongtongJin in #7204
• [ISSUE #7166] Optimize the display format of admin by @yx9o in #7210
• [ISSUE #7142] Add command RocksDBConfigToJson to inspect rocksdb content by @Ziy1-Tan in #7180
• [ISSUE #7124] Fix the typos in the code comments by @mxsm in #7125
• [ISSUE #7222] Bug fix and refactoring of the Indexfile in tiered storage by @lizhimins in #7224
• [ISSUE #7228] Converge the use of some important variables for...

What's Changed

• [ISSUE #6576] Fix pop lmq message by @HScarb in #6577
• [ISSUE #5866] Fix client-side memory leak of inactive channel by @imzs in #5867
• [ISSUE #6889] Add attribute for SubscriptionGroupConfig by @drpmma in #6891
• [ISSUE #6720] [RIP-64] Heartbeat Optimization by @zk-drizzle in #6724
• [ISSUE #6884] Resolve proxy sending mentality to broker and unable to find ACL configuration related by @fengcharly in #6885
• [ISSUE #6900] Add attributesBuilderSupplier for BrokerMetricsManager by @drpmma in #6901
• [ISSUE #6908] Fix the problem that the proxy in the cluster mode obtains the wrong address of broker by @gaoyf in #6909
• [ISSUE #6863] SQL expression CONTAINS support by @f1amingo in #6864
• [ISSUE #6841] Pop batch ack for pushConsumer by @f1amingo in #6842
• [ISSUE #6929] Temporarily ignoring testQueryMessageAsync testing on the Linux platform by @RongtongJin in #6930
• [ISSUE #6904] Fix fail to start broker if mapped file size is 0 by @yuz10 in #6903
• [ISSUE #6926] Fixing the error in the initialization order of replicasManager caused the controller mode to fail to start. by @RongtongJin in #6927
• [ISSUE #6931] Set table reference the same object for setSubscriptionGroupTable method by @RongtongJin in #6932
• [ISSUE #6917] Fix cannot query all messages of the same msgId by @yuz10 in #6918
• [ISSUE #6910] Extract the interval of fetchNameServerAddr by @totalo in #6912
• [ISSUE #6933] Support recreate file if local cq and tiered storage offset not match by @lizhimins in #6934
• [ISSUE #6916] Support startsWith and endsWith in sql filter. by @yuz10 in #6915
• [ISSUE #6935] Prepare to release Apache RocketMQ 5.1.3 by @RongtongJin in #6936

New Contributors

• @imzs made their first contribution in #5867
• @fengcharly made their first contribution in #6885
• @f1amingo made their first contribution in #6864

Full Changelog: rocketmq-all-5.1.2...rocketmq-all-5.1.3

What's Changed

• [ISSUE #6747] Add INNER_MULTI_DISPATCH check, can not contains path separator by @humkum in #6746
• [ISSUE #6761] Support reentrant notification orderly for broker by @xdkxlk in #6762
• [ISSUE #6765] org.apache.rocketmq.util.cache doesn't look like it's being used by @socutes in #6766
• [ISSUE #6745] Remove redundant variables by @haiyanghan in #6740
• [ISSUE #6771] Merge some cases in PullMessageProcessor#composeResponseHeader method by @mxsm in #6772
• [ISSUE #6774] Update README.md rocketmq version to 5.1.1 by @mxsm in #6775
• [ISSUE #6789] Modify placeholder in DefaultMQProducerImpl by @DL1231 in #6790
• [ISSUE #6792] Fix the bug that send method blocked on log.warn for a long time by @DL1231 in #6793
• [ISSUE #6785] Isolate the remoteChannel by group by @xdkxlk in #6786
• [ISSUE #6797] Support batch ack when reput buffer ak to store in PopBufferMergeService by @DongyuanPan in #6798
• Fix typo in operation.md by @maclong1989 in #6799
• [ISSUE#6595] Fix NPE and repeat access key log by @miles-ton in #6596
• [ISSUE #6816] Polish the persistent_unique_broker_id document by @RongtongJin in #6817
• [ISSUE #6803] Benchmark support reportInterval option by @DL1231 in #6804
• [ISSUE #6819] Rename brokerHeartbeatExecutorService executorService name by @xiaochangbai in #6820
• [ISSUE #6828] Return the number of assignments equal to the number of messageQueues for order consumer by @xdkxlk in #6829
• [ISSUE #6777] Support metric in controller by @TheR1sing3un in #6778
• [ISSUE #6624] Support mark() & reset() for TieredFileSegmentInputStream by @TheR1sing3un in #6625
• [ISSUE #6845] Build controller with bazel by @TheR1sing3un in #6846
• Fix incorrect naming by @RongtongJin in #6843
• [ISSUE #6849] Fix the issue of increasing RT in three replicators by @GenerousMan in #6850
• [ISSUE #6779] Support scheduled message for LMQ by @HScarb in #6780
• [ISSUE #6336] [RIP-62] Cold Read Control by @zk-drizzle in #6507
• [ISSUE #6858] Passing through ProxyContext for future expansion by @xdkxlk in #6859
• [ISSUE #6154] Support Amazon S3 backend in TieredStorage by @TheR1sing3un in #6495
• [ISSUE #6861] Modify CURRENT_VERSION to 5.1.2 by @RongtongJin in #6862
• [ISSUE #6866] Move the judgment logic of grpc TLS mode to improve the scalability of ProtocolNegotiator by @dingshuangxi888 in #6867
• [ISSUE #6824] Add golang e2e test by @deepsola in #6825
• [ISSUE #6633] [RIP-65] Improving Tiered Storage Implementation by @lizhimins in #6781
• [ISSUE #6881] Fix scheduled messages are replayed bug by @gaoyf in #6882
• [ISSUE #6800] Change the config variable isEnableBatchPush to enableBatchPush by @absolute8511 in #6801

New Contributors

• @DL1231 made their first contribution in #6790
• @maclong1989 made their first contribution in #6799
• @xiaochangbai made their first contribution in #6820
• @zk-drizzle made their first contribution in #6507
• @gaoyf made their first contribution in #6882

Full Changelog: rocketmq-all-5.1.1...rocketmq-all-5.1.2

What's Changed

• Fix incorrect naming in update configuration protection by @RongtongJin

Full Changelog: rocketmq-all-4.9.6...rocketmq-all-4.9.7

What's Changed

• Make configPath unable to update at runtime by @RongtongJin
• Remove the filter server module by @RongtongJin
• [ISSUE #6758] Prepare to release version 4.9.6 by @RongtongJin in #6759

Full Changelog: rocketmq-all-4.9.5...rocketmq-all-4.9.6