Comparing Azure-Hosted Hadoop Services

Three Microsoft hosted Hadoop-based data management services, Azure HDInsight, Azure Data Lake, and Azure Databricks, are based on Apache’s Hadoop software framework, a popular solution for Big Data storage and analysis.

The services provide many of the same features, and they all separate storage and computing resources to allow customers to maintain a data repository at a steady rate and deploy computing resources as needed to meet query demand. While they deliver many of the same capabilities, the services are designed for different workloads.

HDInsight is a multipurpose Hadoop service and the most like an on-premises environment. It provides multiple deployment configurations optimized for a range of workloads, such as general data mining, processing semi-structured data, and high-speed analysis of streaming data. It automates the deployment of a Hadoop platform, including VMs, storage, Hadoop core components, and add-on packages; however, the service is designed for customers with Hadoop skills and expertise, and the service accepts data only from a limited set of products and services.