• 欢迎访问速搜资源吧,如果在网站上找不到你需要的资源,可以在留言板上留言,管理员会尽量满足你!

【速搜问答】RocketMQ是什么

问答 admin 3年前 (2020-07-13) 715次浏览 已收录 0个评论

汉英对照:
Chinese-English Translation:

RocketMQ是阿里巴巴在2012年开源的一款高性能、高吞吐量的分布式消息中间件,目前已经捐赠给Apache基金会,并于2016年11月成为 Apache 孵化项目。 RocketMQ 是阿里巴巴在 2012 年开源的分布式消息中间件,目前已经捐赠给 Apache 基金会,并于 2016 年 11 月成为 Apache 孵化项目。

Rocketmq is a high-performance, high-throughput distributed message middleware that Alibaba opened in 2012. At present, rocketmq has been donated to the Apache foundation and became an Apache incubation project in November 2016. Rocketmq is an open source distributed message middleware developed by Alibaba in 2012. It has been donated to the Apache foundation and became an Apache incubation project in November 2016.

中间件是一类连接软件组件和应用的计算机软件,它包括一组服务。以便于运行在一台或多台机器上的多个软件通过网络进行交互。中间件技术所提供的互操作性,推动了分布式体系架构的演进,该架构通常用于支持并简化那些复杂的分布式应用程序,它包括 web 服务器、事务监控器和消息队列软件。

Middleware is a kind of computer software that connects software components and applications. It includes a set of services. In order to facilitate the interaction of multiple software running on one or more machines through the network. The interoperability provided by middleware technology promotes the evolution of distributed architecture, which is usually used to support and simplify those complex distributed applications, including web server, transaction monitor and message queuing software.

中间件(middleware)是基础软件的一大类,属于可复用软件的范畴。顾名思义,中间件处于操作系统软件与用户的应用软件的中间。中间件在操作系统、网络和数据库之上,应用软件的下层,总的作用是为处于自己上层的应用软件提供运行与开发的环境,帮助用户灵活、高效地开发和集成复杂的应用软件。

Middleware is a large class of basic software, which belongs to the category of reusable software. As the name implies, middleware is in the middle of operating system software and user’s application software. Middleware is on the top of operating system, network and database, and the lower layer of application software. The overall function of middleware is to provide an environment for running and developing application software on its upper layer, and help users develop and integrate complex application software flexibly and efficiently.

中间件是位于平台(硬件和操作系统)和应用之间的通用服务,这些服务具有标准的程序接口和协议。针对不同的操作系统和硬件平台,中间件可以有符合接口和协议规范的多种实现。

Middleware is a common service between platform (hardware and operating system) and application, which has standard program interface and protocol. For different operating systems and hardware platforms, middleware can be implemented in accordance with interface and protocol specifications.

RocketMQ 就是一款分布式消息中间件。那么,RocketMQ 主要为了解决哪些问题呢?

Rocketmq is a distributed message middleware. So, what are the main problems rocketmq is trying to solve?

(1)Publish/Subscribe

(1)Publish/Subscribe

发布与订阅是消息中间件的最基本功能,也是相对于传统 RPC 通信而言。

Publish and subscribe is the most basic function of message oriented middleware, which is also relative to traditional RPC communication.

(2)Message Priority

(2)Message Priority

规范中描述的优先级是指在一个消息队列中,每条消息都有不同的优先级,一般用整数来描述,优先级高的消息先投递,如果消息完全在一个内存队列中,那么在投递前可以按照优先级排序,令优先级高的先投递。

The priority described in the specification refers to that in a message queue, each message has a different priority, which is generally described as an integer. The message with higher priority is delivered first. If the message is completely in a memory queue, the priority can be sorted before delivery, so that the higher priority message is delivered first.

由于 RocketMQ 所有消息都是持久化的,所以如果按照优先级来排序,开销会非常大,因此 RocketMQ 没有特意支持消息优先级,但是可以通过变通的方式实现类似功能,即单独配置一个优先级高的队列,和一个普通优先级的队列, 将不同优先级发送到不同队列即可。

Since all messages in rocketmq are persistent, it will be very expensive to sort them according to their priority. Therefore, rocketmq does not support message priority. However, it can implement similar functions through flexible ways, that is, configuring a queue with high priority and a queue with common priority, and sending different priorities to different queues.

(3)Message Order

(3)Message Order

消息有序指的是一类消息消费时,能按照发送的顺序来消费。例如:一个订单产生了 3 条消息,分别是订单创建,订单付款,订单完成。消费时,要按照这个顺序消费才能有意义。但是同时订单之间是可以并行消费的。

Message ordering refers to a kind of message consumption, which can be consumed according to the order of sending. For example, an order generates three messages: order creation, order payment and order completion. When consuming, it is meaningful to consume in this order. But at the same time, orders can be consumed in parallel.

RocketMQ 可以严格的保证消息有序。

Rocketmq can strictly guarantee the order of messages.

(4)Message Filter

(4)Message Filter

①Broker 端消息过滤

① Broker side message filtering

在 Broker 中,按照 Consumer 的要求做过滤,优点是减少了对于 Consumer 无用消息的网络传输。缺点是增加了 Broker 的负担,实现相对复杂。

In the broker, filtering according to the requirements of the consumer has the advantage of reducing the network transmission of useless messages to the consumer. The disadvantage is that it increases the burden of broker and the implementation is relatively complex.

②Consumer 端消息过滤

② Consumer side message filtering

这种过滤方式可由应用完全自定义实现,但是缺点是很多无用的消息要传输到 Consumer 端。

This filtering method can be completely customized by the application, but the disadvantage is that many useless messages have to be transmitted to the consumer side.

(5)Message Persistence

(5)Message Persistence

消息中间件通常采用的几种持久化方式:

Message middleware usually adopts several persistence methods

①持久化到数据库,例如 Mysql。

① Persist to a database, such as mysql.

②持久化到 KV 存储,例如 levelDB、伯克利 DB 等 KV 存储系统。

② Persistent to kV storage, such as leveldb, Berkeley DB and other kV storage systems.

③文件记录形式持久化,例如 Kafka,RocketMQ

③ File record form persistence, such as Kafka, rocketmq

④对内存数据做一个持久化镜像,例如 beanstalkd,VisiNotify

④ Make a persistent image of the memory data, such as beanstalkd, visinotify

⑤前三种持久化方式都具有将内存队列 Buffer 进行扩展的能力,第四种方式只是一个内存的镜像,作用是当 Broker 挂掉重启后仍然能将之前内存的数据恢复出来。

⑤ The first three persistence methods have the ability to expand the memory queue buffer. The fourth method is just a memory mirror, which can recover the data in the previous memory after the broker crashes and restarts.

RocketMQ 充分利用 Linux 文件系统内存 cache 来提高性能。

Rocketmq makes full use of Linux file system memory cache to improve performance.

(6)Message Reliablity

(6)Message Reliablity

影响消息可靠性的几种情况:

Several situations that affect message reliability are as follows:

①Broker 正常关闭;

① The broker is closed normally;

②Broker 异常 Crash;

② Broker exception crash;

③OS Crash;

③OS Crash;

④机器掉电,但是能立即恢复供电情况。

④ The machine loses power, but it can restore the power supply immediately.

⑤机器无法开机(可能是 cpu、主板、内存等关键设备损坏)

⑤ The machine fails to start up (it may be that the CPU, motherboard, memory and other key equipment are damaged)

⑥磁盘设备损坏。

⑥ The disk device is damaged.

前四种情况都属于硬件资源可立即恢复情况,RocketMQ 在这四种情况下能保证消息不丢,或者丢失少量数据(依赖刷盘方式是同步还是异步)。

The first four cases all belong to the situation that hardware resources can be recovered immediately. Rocketmq can ensure that messages are not lost or a small amount of data is lost (depending on whether the disk flushing mode is synchronous or asynchronous).

后两种情况属于单点故障,且无法恢复,一旦发生,在此单点上的消息全部丢失。RocketMQ 在这两种情况下,通过异步复制,可保证 99%的消息不丢,但是仍然会有极少量的消息可能丢失。通过同步双写技术可以完全避免单点,同步双写势必会影响性能,适合对消息可靠性要求极高的场合,例如与 Money 相关的应用。

The latter two cases belong to single point failure and cannot be recovered. Once it happens, all messages on this single point are lost. In these two cases, rocketmq can ensure that 99% of the messages are not lost through asynchronous replication, but there are still very few messages that may be lost. Single point can be completely avoided by using synchronous double writing technology. Synchronous double writing is bound to affect performance, and it is suitable for occasions with high requirements for message reliability, such as money related applications.

RocketMQ 从 3.0 版本开始支持同步双写。

Rocketmq supports synchronous double writing since version 3.0.

(7)Low Latency Messaging

(7)Low Latency Messaging

在消息不堆积情况下,消息到达 Broker 后,能立刻到达 Consumer。RocketMQ 使用长轮询 Pull 方式,可保证消息非常实时,消息实时性不低于 Push。

When messages do not accumulate, they can arrive at the consumer immediately after they arrive at the broker. Rocketmq uses long polling pull mode to ensure that the message is very real-time, and the message real-time performance is not lower than push.

(8)At least Once

(8)At least Once

是指每个消息必须投递一次。RocketMQ Consumer 先 pull 消息到本地,消费完成后,才向服务器返回 ack,如果没有消费一定不会 ack 消息,所以 RocketMQ 可以很好的支持此特性。

It means that every message must be delivered once. Rocketmq consumer first pulls the message to the local, and then returns the ack to the server after consumption. If there is no consumption, the ACK message will not be sent. Therefore, rocketmq can well support this feature.

(9)Exactly Only Once

(9)Exactly Only Once

①发送消息阶段,不允许发送重复的消息。

① In the send message phase, duplicate messages are not allowed to be sent.

②消费消息阶段,不允许消费重复的消息。

② In the consumption message phase, duplicate messages are not allowed to be consumed.

只有以上两个条件都满足情况下,才能认为消息是“Exactly Only Once”,而要实现以上两点,在分布式系统环境下,不可避免要产生巨大的开销。所以 RocketMQ 为了追求高性能,并不保证此特性,要求在业务上进行去重,也就是说消费消息要做到幂等性。RocketMQ 虽然不能严格保证不重复,但是正常情况下很少会出现重复发送、消费情况,只有网络异常,Consumer 启停等异常情况下会出现消息重复。

Only when the above two conditions are met can the message be considered as “exactly only once”. To achieve the above two points, it is inevitable to generate huge overhead in the distributed system environment. Therefore, in order to pursue high performance, rocketmq does not guarantee this feature. It requires business de duplication, which means that consumption messages should be idempotent. Although rocketmq cannot strictly guarantee that it is not repeated, it is rare for repeated sending and consumption under normal conditions. Only network exceptions, such as the start and stop of the consumer, will cause duplicate messages.

(10)Broker 的 Buffer 问题

(10) The buffer problem of broker

Broker 的 Buffer 通常指的是 Broker 中一个队列的内存 Buffer 大小,这类 Buffer 通常大小有限。

The buffer of a broker usually refers to the memory buffer size of a queue in the broker. This type of buffer is usually limited in size.

另外,RocketMQ 没有内存 Buffer 概念,RocketMQ 的队列都是持久化磁盘,数据定期清除。RocketMQ 同其他 MQ 有非常显著的区别,RocketMQ 的内存 Buffer 抽象成一个无限长度的队列,不管有多少数据进来都能装得下,这个无限是有前提的,Broker 会定期删除过期的数据,例如 Broker 只保存 3 天的消息,那么这个 Buffer 虽然长度无限,但是 3 天前的数据会被从队尾删除。

In addition, rocketmq does not have the concept of memory buffer. Rocketmq queues are persistent disks, and the data is cleared regularly. Rocketmq is very different from other MQS. Rocketmq’s memory buffer is abstracted into an infinite length queue, which can hold no matter how much data comes in. This infinity is based on the premise that the broker will delete expired data periodically. For example, the broker only saves messages for three days. Therefore, although the buffer is infinite in length, it is 3 Data from days ago will be deleted from the end of the team.

(11)回溯消费

(11) Retrospective consumption

回溯消费是指 Consumer 已经消费成功的消息,由于业务上的需求需要重新消费,要支持此功能,Broker 在向 Consumer 投递成功消息后,消息仍然需要保留。并且重新消费一般是按照时间维度,例如由于 Consumer 系统故障,恢复后需要重新消费 1 小时前的数据,那么 Broker 要提供一种机制,可以按照时间维度来回退消费进度。

Backtracking consumption refers to the message that the consumer has successfully consumed. Since the business needs to be re consumed, to support this function, the message still needs to be retained after the broker delivers the success message to the consumer. In addition, re consumption is generally based on the time dimension. For example, due to the consumer system failure, the data one hour ago needs to be re consumed after recovery. Then the broker should provide a mechanism to roll back the consumption progress according to the time dimension.

RocketMQ 支持按照时间回溯消费,时间维度精确到毫秒,可以向前回溯,也可以向后回溯。

Rocketmq supports backtracking consumption according to time. The time dimension is accurate to milliseconds. It can be backtracked or backtracked.

(12)消息堆积

(12) News pile up

消息中间件的主要功能是异步解耦,还有个重要功能是挡住前端的数据洪峰,保证后端系统的稳定性,这就要求消息中间件具有一定的消息堆积能力,消息堆积分以下两种情况:

The main function of message middleware is asynchronous decoupling, and another important function is to block the front-end data flood peak and ensure the stability of the back-end system. This requires the message middleware to have a certain ability of message accumulation. The message heap integral can be divided into the following two situations:

①消息堆积在内存 Buffer,一旦超过内存 Buffer,可以根据一定的丢弃策略来丢弃消息,如 CORBA Notification 规范中描述。适合能容忍丢弃消息的业务,这种情况消息的堆积能力主要在于内存 Buffer 大小,而且消息堆积后,性能下降不会太大,因为内存中数据多少对于对外提供的访问能力影响有限。

① Messages are stacked in the memory buffer. Once the memory buffer is exceeded, messages can be discarded according to certain discarding strategies, as described in CORBA notification specification. It is suitable for businesses that can tolerate dropping messages. In this case, the ability of message accumulation mainly depends on the size of the memory buffer. Moreover, after the messages are stacked, the performance degradation will not be too large, because the amount of data in memory has limited impact on the external access ability.

②消息堆积到持久化存储系统中,例如 DB,KV 存储,文件记录形式。 当消息不能在内存 Cache 命中时,要不可避免的访问磁盘,会产生大量读 IO,读 IO 的吞吐量直接决定了消息堆积后的访问能力。

② Messages are accumulated in persistent storage systems, such as dB, kV storage, and file record. When the message cannot hit the memory cache, it is inevitable to access the disk, which will produce a large number of read io. The throughput of read IO directly determines the access ability of messages after accumulation.

评估消息堆积能力主要有以下四点:

There are four main points to evaluate the message accumulation capability

消息能堆积多少条,多少字节?即消息的堆积容量。

How many pieces and bytes can messages stack? That is, the accumulation capacity of messages.

消息堆积后,发消息的吞吐量大小,是否会受堆积影响?

After message accumulation, will the throughput of sending messages be affected by the accumulation?

消息堆积后,正常消费的 Consumer 是否会受影响?

After the messages accumulate, will the normal consumers be affected?

消息堆积后,访问堆积在磁盘的消息时,吞吐量有多大?

After message accumulation, what is the throughput when accessing messages stacked on disk?

(13)分布式事务

(13) Distributed transaction

已知的几个分布式事务规范,如 XA,JTA 等。其中 XA 规范被各大数据库厂商广泛支持,如 Oracle,Mysql 等。其中 XA 的 TM 实现佼佼者如 Oracle Tuxedo,在金融、电信等领域被广泛应用。

Several known distributed transaction specifications, such as Xa, JTA, etc. Xa specification is widely supported by database manufacturers, such as Oracle, mysql, etc. Among them, the TM implementation of Xa, such as Oracle tuxedo, is widely used in finance, telecommunications and other fields.

分布式事务涉及到两阶段提交问题,在数据存储方面的方面必然需要 KV 存储的支持,因为第二阶段的提交回滚需要修改消息状态,一定涉及到根据 Key 去查找 Message 的动作。RocketMQ 在第二阶段绕过了根据 Key 去查找 Message 的问题,采用第一阶段发送 Prepared 消息时,拿到了消息的 Offset,第二阶段通过 Offset 去访问消息,并修改状态,Offset 就是数据的地址。

Distributed transaction involves two-phase commit problem. In terms of data storage, kV storage must be supported. Because in the second phase, the commit rollback needs to modify the message state, which must involve the action of finding message according to the key. Rocketmq bypasses the problem of finding messages according to the key in the second stage. When sending prepared messages in the first stage, rocketmq gets the offset of the message. In the second stage, rocketmq accesses the message and modifies the state through offset. Offset is the address of the data.

RocketMQ 这种实现事务的方式,没有通过 KV 存储做,而是通过 Offset 方式,存在一个显著缺陷,即通过 Offset 更改数据,会令系统的脏页过多,需要特别关注。

Rocketmq, which implements transactions through offset instead of kV storage, has a significant defect, that is, changing data through offset will cause too many dirty pages in the system, which requires special attention.

(14)定时消息

(14) Timing message

定时消息是指消息发到 Broker 后,不能立刻被 Consumer 消费,要到特定的时间点或者等待特定的时间后才能被消费。

Timing message refers to that after the message is sent to the broker, it cannot be consumed by the consumer immediately. It can only be consumed at a specific time point or after waiting for a specific time.

如果要支持任意的时间精度,在 Broker 层面,必须要做消息排序,如果再涉及到持久化,那么消息排序要不可避免的产生巨大性能开销。

If you want to support arbitrary time precision, message sorting must be done at the broker level. If persistence is involved, then message sorting will inevitably generate huge performance overhead.

RocketMQ 支持定时消息,但是不支持任意时间精度,支持特定的 level,例如定时 5s,10s,1m 等。

Rocketmq supports timing messages, but does not support any time precision. It supports specific levels, such as timing 5S, 10s, 1M, etc.

(15)消息重试

(15) Message retrying

Consumer 消费消息失败后,要提供一种重试机制,令消息再消费一次。Consumer 消费消息失败通常可以认为有以下几种情况:

After the consumer fails to consume the message, it needs to provide a retrial mechanism to make the message consume again. Consumer consumption message failure can be considered as follows:

由于消息本身的原因,例如反序列化失败,消息数据本身无法处理(例如话费充值,当前消息的手机号被注销,无法充值)等。这种错误通常需要跳过这条消息,再消费其他消息,而这条失败的消息即使立刻重试消费,99%也不成功,所以最好提供一种定时重试机制,即过 10s 秒后再重试。

Due to the reasons of the message itself, such as the failure of deserialization, the message data itself cannot be processed (for example, the phone number of the current message is cancelled and cannot be recharged), etc. This kind of error usually requires skipping this message and consuming other messages. Even if the failed message is immediately retried, 99% of the messages will not succeed. Therefore, it is better to provide a timed retrial mechanism, that is, try again after 10 seconds.

由于依赖的下游应用服务不可用,例如 db 连接不可用,外系统网络不可达等。遇到这种错误,即使跳过当前失败的消息,消费其他消

Because the dependent downstream application services are not available, such as the DB connection is not available, the external system network is not accessible, etc. When this kind of error occurs, even if the current failure message is skipped, other consumption will be consumed


速搜资源网 , 版权所有丨如未注明 , 均为原创丨转载请注明原文链接:【速搜问答】RocketMQ是什么
喜欢 (0)
[361009623@qq.com]
分享 (0)
发表我的评论
取消评论
表情 贴图 加粗 删除线 居中 斜体 签到

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址