Hadoop Unstructured Data Accessible with PolyBase Technologies

The PolyBase technology in SQL Server 2012 Parallel Data Warehouse (PDW) simplifies access to data stored in Hadoop. PolyBase is a technology in PDW (center), which is a special edition of SQL Server designed to be a low-cost, easy-to-deploy appliance for data warehouses, which are special-purpose databases that store large volumes of structured data. The PDW data, such as online purchases and airline reservations, are typically manageable and consistent, making them easily available for access and analysis by line-of-business applications, business intelligence (BI) tools, reporting services, and other traditional database solutions. PolyBase enables a SQL query (left) to concurrently retrieve data from a PDW database and a Hadoop data cluster (right).

Hadoop is an Apache open-source software framework designed to speed processing of large unstructured data sets and improve reliability and cost by distributing processing across clusters of commodity computers. The data, such as Web server logs and social media posts (like Facebook and Twitter), are typically unstructured with significant variety and unpredictable changes in content and structure over time, which makes analyzing the data with traditional database solutions complex. Hadoop is able to organize unstructured data with two key components: the Hadoop Distributed File System (HDFS) and MapReduce. The HDFS is a core feature of Hadoop, which manages storing large, unstructured data across multiple computers and provides the first level of data retrieval in the form of file definitions. MapReduce is a programming module in Hadoop that allows developers to create jobs that map, filter, sort, and summarize complex unstructured data into a table format.