• 欢迎访问速搜资源吧,如果在网站上找不到你需要的资源,可以在留言板上留言,管理员会尽量满足你!

【速搜问答】Apache Hadoop YARN是什么

问答 admin 1年前 (2020-09-04) 222次浏览 已收录 0个评论

汉英对照:
Chinese-English Translation:

Apache Hadoop YARN是一种新的 Hadoop 资源管理器,它是一个通用资源管理系统,可为上层应用提供统一的资源管理和调度,它的引入为集群在利用率、资源统一管理和数据共享等方面带来了巨大好处。 Apache Hadoop YARN (Yet Another Resource Negotiator,另一种资源协调者)是一种新的 Hadoop 资源管理器,它是一个通用资源管理系统,可为上层应用提供统一的资源管理和调度,它的引入为集群在利用率、资源统一管理和数据共享等方面带来了巨大好处。

Apache Hadoop yarn is a new kind of Hadoop resource manager. It is a general resource management system, which can provide unified resource management and scheduling for the upper application. Its introduction brings great benefits to the cluster in terms of utilization, unified resource management and data sharing. Apache Hadoop yarn (another Resource Coordinator) is a new kind of Hadoop resource manager. It is a general resource management system, which can provide unified resource management and scheduling for upper level applications. The introduction of Apache Hadoop yarn has brought great benefits to cluster in terms of utilization, unified resource management and data sharing.

YARN 是新一代 Hadoop 资源管理器,通过 YARN,用户可以运行和管理同一个物理集群机上的多种作业,例如 MapReduce 批处理和图形处理作业。这样不仅可以巩固一个组织管理的系统数目,而且可以对相同的数据进行不同类型的数据分析。某些情况下,整个数据流可以执行在同一个集群机上。

Yarn is a new generation of Hadoop resource manager. Through yarn, users can run and manage multiple jobs on the same physical cluster, such as MapReduce batch processing and graphics processing jobs. This not only consolidates the number of systems managed by an organization, but also performs different types of data analysis on the same data. In some cases, the entire data flow can be executed on the same cluster machine.

YARN 的基本思想是将 JobTracker 的两个主要功能(资源管理和作业调度/监控)分离,主要方法是创建一个全局的 ResourceManager(RM)和若干个针对应用程序的 ApplicationMaster(AM)。这里的应用程序是指传统的 MapReduce 作业或作业的 DAG(有向无环图)。

The basic idea of yarn is to separate the two main functions of job tracker (resource management and job scheduling / monitoring). The main method is to create a global resource manager (RM) and several application Masters (AM) for applications. The application here refers to the traditional MapReduce job or DAG (directed acyclic graph) of a job.

YARN 分层结构的本质是 ResourceManager。这个实体控制整个集群并管理应用程序向基础计算资源的分配。ResourceManager 将各个资源部分(计算、内存、带宽等)精心安排给基础 NodeManager(YARN 的每节点代理)。ResourceManager 还与 ApplicationMaster 一起分配资源,与 NodeManager 一起启动和监视它们的基础应用程序。在此上下文中,ApplicationMaster 承担了以前的 TaskTracker 的一些角色,ResourceManager 承担了 JobTracker 的角色。

The essence of yarn hierarchy is resource manager. This entity controls the entire cluster and manages the allocation of applications to the underlying computing resources. The resource manager orchestrates various resource parts (computation, memory, bandwidth, etc.) to the underlying nodemanager (yarn’s per node agent). The ResourceManager also allocates resources with the applicationmaster, starts and monitors their underlying applications with nodemanager. In this context, the applicationmaster takes on some of the roles of the previous tasktracker, and the ResourceManager assumes the role of jobtracker.

ApplicationMaster 管理一个在 YARN 内运行的应用程序的每个实例。ApplicationMaster 负责协调来自 ResourceManager 的资源,并通过 NodeManager 监视容器的执行和资源使用(CPU、内存等的资源分配)。请注意,尽管目前的资源更加传统(CPU 核心、内存),但未来会带来基于手头任务的新资源类型(比如图形处理单元或专用处理设备)。从 YARN 角度讲,ApplicationMaster 是用户代码,因此存在潜在的安全问题。YARN 假设 ApplicationMaster 存在错误或者甚至是恶意的,因此将它们当作无特权的代码对待。

The applicationmaster manages each instance of an application running within yarn. The application master is responsible for coordinating the resources from the resource manager, and monitors the execution of the container and resource utilization (CPU, memory, etc.) through nodemanager. Note that although the current resources are more traditional (CPU core, memory), there will be new types of resources based on the tasks at hand (such as graphics processing units or dedicated processing devices) in the future. From the yarn point of view, applicationmaster is user code, so there are potential security issues. Yarn assumes that the applicationmaster is faulty or even malicious, so it treats them as unprivileged code.

NodeManager 管理一个 YARN 集群中的每个节点。NodeManager 提供针对集群中每个节点的服务,从监督对一个容器的终生管理到监视资源和跟踪节点健康。MRv1 通过插槽管理 Map 和 Reduce 任务的执行,而 NodeManager 管理抽象容器,这些容器代表着可供一个特定应用程序使用的针对每个节点的资源。YARN 继续使用 HDFS 层。它的主要 NameNode 用于元数据服务,而 DataNode 用于分散在一个集群中的复制存储服务。

Nodemanager manages each node in a yarn cluster. Nodemanager provides services for each node in the cluster, from supervising lifetime management of a container to monitoring resources and tracking node health. Mrv1 manages the execution of map and reduce tasks through slots, while nodemanager manages Abstract containers, which represent resources for each node that can be used by a specific application. Yarn continues to use the HDFS layer. Its main namenode is used for metadata services, while datanode is used for replication storage services scattered in a cluster.

要使用一个 YARN 集群,首先需要来自包含一个应用程序的客户的请求。ResourceManager 协商一个容器的必要资源,启动一个 ApplicationMaster 来表示已提交的应用程序。通过使用一个资源请求协议,ApplicationMaster 协商每个节点上供应用程序使用的资源容器。执行应用程序时,ApplicationMaster 监视容器直到完成。当应用程序完成时,ApplicationMaster 从 ResourceManager 注销其容器,执行周期就完成了。

To use a yarn cluster, you first need a request from a client that contains an application. The Resource Manager negotiates the necessary resources for a container and starts an applicationmaster to represent the submitted application. By using a resource request protocol, the application master negotiates the resource containers on each node for the application to use. When the application is executed, the applicationmaster monitors the container until it is complete. When the application is complete, the application master logs off its container from the resource manager, and the execution cycle is complete.

基本缺陷

Basic defects

MapReduce 的第一个版本既有优点也有缺点。MRv1 是目前使用的标准的大数据处理系统。但是,这种架构存在不足,主要表现在大型集群上。当集群包含的节点超过 4,000 个时(其中每个节点可能是多核的),就会表现出一定的不可预测性。其中一个最大的问题是级联故障,由于要尝试复制数据和重载活动的节点,所以一个故障会通过网络泛洪形式导致整个集群严重恶化。

The first version of MapReduce has both advantages and disadvantages. Mrv1 is a standard big data processing system currently used. However, there are some shortcomings in this architecture, mainly in large clusters. When the cluster contains more than 4000 nodes (each node may be multi-core), it will show some unpredictability. One of the biggest problems is cascading failure. Because it is trying to copy data and overload active nodes, a fault will cause serious deterioration of the whole cluster in the form of network flooding.

但 MRv1 的最大问题是多租户。随着集群规模的增加,一种可取的方式是为这些集群采用各种不同的模型。MRv1 的节点专用于 Hadoop,所以可以改变它们的用途以用于其他应用程序和工作负载。当大数据和 Hadoop 成为云部署中一个更重要的使用模型时,这种能力也会增强,因为它允许在服务器上对 Hadoop 进行物理化,而无需虚拟化且不会增加管理、计算和输入/输出开销。

But the biggest problem with mrv1 is multi tenancy. With the increase of cluster size, it is advisable to adopt different models for these clusters. Mrv1 nodes are dedicated to Hadoop, so they can be reused for other applications and workloads. This capability will also be enhanced when big data and Hadoop become a more important usage model in cloud deployment, because it allows Hadoop to be physically materialized on the server without virtualization and without increasing management, computing, and I / O overhead.

主要优点

Main advantages

大大减小了 JobTracker(也就是现在的 ResourceManager)的资源消耗,并且让监测每一个 Job 子任务 (tasks) 状态的程序分布式化了,更安全、更优美。

It greatly reduces the resource consumption of jobtracker (i.e. the resource manager now), and makes the program monitoring the status of each job subtask distributed, safer and more beautiful.

在新的 Yarn 中,ApplicationMaster 是一个可变更的部分,用户可以对不同的编程模型写自己的 AppMst,让更多类型的编程模型能够跑在 Hadoop 集群中,可以参考 hadoop Yarn 官方配置模板中的 mapred-site.xml 配置。

In the new yarn, applicationmaster is a changeable part. Users can write their own appmst for different programming models, so that more types of programming models can run in the Hadoop cluster. You can refer to mapred in the Hadoop yarn official configuration template- site.xml to configure.

对于资源的表示以内存为单位 ( 在目前版本的 Yarn 中,没有考虑 cpu 的占用 ),比之前以剩余 slot 数目更合理。

It is more reasonable to represent resources in memory (in the current version of yarn, CPU consumption is not considered), which is more reasonable than the number of remaining slots before.

老的框架中,JobTracker 一个很大的负担就是监控 job 下的 tasks 的运行状况,现在,这个部分就扔给 ApplicationMaster 做了,而 ResourceManager 中有一个模块叫做 ApplicationsMasters( 注意不是 ApplicationMaster),它是监测 ApplicationMaster 的运行状况,如果出问题,会将其在其他机器上重启。

In the old framework, a big burden of job tracker is to monitor the running status of tasks under the job. Now, this part is left to the application master. There is a module in the resource manager called applicationmasters (note that it is not applicationmaster), which monitors the application master If there is a problem, it will be restarted on other machines.

Container 是 Yarn 为了将来作资源隔离而提出的一个框架。这一点应该借鉴了 Mesos 的工作,目前是一个框架,仅仅提供 java 虚拟机内存的隔离,hadoop 团队的设计思路应该后续能支持更多的资源调度和控制 , 既然资源表示成内存量,那就没有了之前的 map slot/reduce slot 分开造成集群资源闲置的尴尬情况。

Container is a framework proposed by yarn for resource isolation in the future. This point should draw lessons from the work of mesos. At present, it is a framework that only provides isolation of memory of Java virtual machine. The design idea of Hadoop team should support more resource scheduling and control in the future. Since the resources are expressed as memory, there is no embarrassing situation of idle cluster resources caused by the separation of map slot / reduce slot.


速搜资源网 , 版权所有丨如未注明 , 均为原创丨转载请注明原文链接:【速搜问答】Apache Hadoop YARN是什么
喜欢 (0)
[361009623@qq.com]
分享 (0)
发表我的评论
取消评论
表情 贴图 加粗 删除线 居中 斜体 签到

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址