Data Capture and Integration

An important part of the Big Data landscape are the offerings that receive and process or receive and store data in a process known as data ingestion. For example, various offerings specialize in the ability to gather equipment sensor data, receive and store credit card transactions, process financial market updates, track weather forecasts, and filter and process social media posts.

Choosing the right product or service requires understanding the velocity and volume of data being captured and whether the data processing will occur during the capture process or is performed afterward. For low-end workloads, organizations may choose to build their own ingestion solutions to preprocess and clean the data, while high-end workloads will likely require Azure services or third-party solutions.

On-Premises and Hybrid

On-premises, organizations can use SQL Server, native Hadoop solutions, or third-party tools to capture and integrate data.

Data Capture

SQL Server includes StreamInsight, an event processing component that enables complex event processing (CEP) applications to scan input from sources such as equipment sensors or Web logs to detect patterns and respond automatically.