HDInsight Cluster Configuration Options

HDInsight is a Microsoft-hosted Hadoop service that offers multiple native Hadoop deployment configurations optimized for different workloads. The illustration shows three panels from the Azure portal that are used to configure an HDInsight cluster.

A new cluster configuration begins with selecting the type of cluster and version (left). For each cluster type, users can select from at least two newest community editions of the package. For example, a Spark cluster can currently be deployed using Spark 1.6.3, 2.0.2, or 2.1.0. Other Apache Hadoop components (bottom left) that support the environment, such as metastores and cluster management, are automatically deployed with the cluster.

Approved third-party applications (right) that provide specialized functionality, such as data stream management, artificial intelligence, and data security, can be added to the cluster. Third-party applications incur additional cost.

The final step is selecting the number and size of Linux VMs, called nodes (middle), to meet performance requirements. Each cluster type defaults to a specific VM series that is appropriate for the functions it performs. For example, Hadoop clusters typically use general purpose A series VMs, whereas Spark clusters typically use high-performing D series with large memory levels and solid-state drives.