SQL Server Big Data Cluster Architecture

SQL Server 2019 Big Data Clusters use a multitier configuration with groups of servers performing different functions, such as ingesting, querying, aggregating, and data management.

The first tier is a Kubernetes cluster (top). The Kubernetes cluster manages all SQL Server instances in Kubernetes pods, which are collections of Docker-style container instances that can be expanded or reduced in response to demand. The Kubernetes cluster is created and managed using a hosted Kubernetes service, such as Azure Kubernetes Services (AKS) or Amazon Elastic Container Service (ECS), or a Kubernetes service that can be deployed on-premises such as OpenShift.

The second tier is where the Master Node (top) resides. The Master Node is a traditional SQL Server instance with normal read-write online transaction processing capabilities and databases. It is also the head node of the Big Data Cluster, which brings new responsibilities: It receives T-SQL and Spark requests (top middle), distributes the queries to the appropriate servers in the cluster, gathers the results, and returns the data to the requesting application or user. It also works with the Hadoop Distributed File System (HDFS) to expose the underlying unstructured data as regular tables for users and developers to reference when creating queries. The Master Node is a single server container instance that can belong to an Always On Availability Group.