Azure Data Factory Components

Azure Data Factory is a Microsoft-hosted extract, transform, and load (ETL) service that helps developers automate data integration and transformation of data from on-premises and Azure-hosted sources.

The illustration shows an example using Azure Data Factory to extract, transform, and load data into final locations. A typical workflow includes the following steps:

Extract. In the example, data is extracted (left) from an on-premises enterprise resource planning (ERP) database in SQL Server (bottom left), and unstructured data is stored in Azure Data Lake Store (top left). The service can extract traditional relational data, such as SQL Server, SAP HANA, and Oracle, and nonrelational data (such as Web logs, clickstream results, and social network posts) stored in Azure Big Data offerings, including HDInsight, Blob Storage, Azure Data Lake Store, and Azure Cosmos DB.

Transform. In the example, data are transformed (middle) in pipelines, which are a sequence of activities, such as summarizing data and combining multiple tables into a single table. Data are also sent to Azure Machine Learning (top middle) for additional processing, prior to completing the overall transformation. Activities can be scheduled to execute in a specific order or based on the results of other activities. Pipelines can be scheduled to run at specific times and frequencies, and they can be chained to handle complex processing. The service simplifies the process of combining disparate data types, such as structured data with unstructured data, by exposing the underlying data schema (regardless of data type) in a standard format, which enables developers to write workflows that combine the disparate data sources. A unique part of the service is the ability to call Azure-based advanced data processing services (Azure HDInsight, Azure Data Lake Analytics, and Azure Machine Learning) as part of a workflow and include the results in further activities.