RocketMQ-TODO
Last updated
Last updated
A broker contains a master node and a slave node
Broker 1 has topic 1 to 5
Broker 2 has topic 6 to 10
NameNode cluster contains the mapping from Topic=>Broker
Scenario 1. Consumer group tells name node cluster which topic it subscribe to 2. Broker pulls from name node cluster about the heartbeat message (whether I am alive / topic mapping on the broker) 3. Producer group pushes events to the broker 4. Broker push events to consumer group
The processing of message follows the producing order.
Order types
Global order
Partition order
How to guarantee
Not support any granularity. There are a couple granularity level such as 1s, 5s, 10s, 1 minute, 2 minute, ... 1 hour, 5 hour.
Example
Example: A user is purchasing items on an ecommerce website. There are two operations
Create an order in the database
Delete ordered items from the shopping cart. Since this step is not a necessary step to be completed within the order operation, the command could be processed asynchronously, e.g. putting into a message queue.
Concept
Half (prepare) message: Refers to a message that cannot be delivered temporarily. When a message is successfully sent to the MQ server, but the server did not receive the second acknowledgement of the message from the producer, then the message is marked as “temporarily undeliverable”. The message in this status is called a half message.
Message status check: Network disconnection or producer application restart may result in the loss of the second acknowledgement of a transactional message. When MQ server finds that a message remains a half message for a long time, it will send a request to the message producer, checking the final status of the message (Commit or Rollback).
Algorithm
Producer send half message to MQ server.
After send half message succeed, execute local transaction.
Send commit or rollback message to MQ Server based on local transaction results.
If commit/rollback message missed or producer pended during the execution of local transaction，MQ server will send check message to each producers in the same group to obtain transaction status.
Producer reply commit/rollback message based on local transaction status.
Committed message will be delivered to consumer but rolled back message will be discarded by MQ server.
Kafka at Netflix: https://netflixtechblog.com/kafka-inside-keystone-pipeline-dd5aeabaf6bb
