Integrate all types of data on distributed and mainframe platforms
IBM® InfoSphere® DataStage® integrates data across multiple systems using a high performance parallel framework, and it supports extended metadata management and enterprise connectivity. The scalable platform provides more flexible integration of all types of data, including big data at rest (Hadoop-based) or in motion (stream-based), on distributed and mainframe platforms.
InfoSphere DataStage provides these features and benefits:
- Powerful, scalable ETL platform—supports the collection, integration and transformation of large volumes of data, with data structures ranging from simple to complex.
- Support for big data and Hadoop—enables you to directly access big data on a distributed file system, and helps clients more efficiently leverage new data sources by providing JSON support and a new JDBC connector.
- Near real-time data integration—as well as connectivity between data sources and applications.
- Workload and business rules management—helps you optimize hardware utilization and prioritize mission-critical tasks.
- Ease of use—helps improve speed, flexibility and effectiveness to build, deploy, update and manage your data integration infrastructure.
- Rich support for DB2Z and DB2 for z/OS—including data load optimization for DB2Z and balanced optimization for DB2 on z/OS
Powerful, scalable ETL platform
- Manages data arriving in near real-time as well as data received on a periodic or scheduled basis.
- Provides high-performance processing of very large data volumes.
- Leverages the parallel processing capabilities of multiprocessor hardware platforms to help you manage growing data volumes and shrinking batch windows.
- Supports heterogeneous data sources and targets in a single job including text files, XML, ERP systems, most databases (including partitioned databases), web services, and business intelligence tools.
Support for big data and Hadoop
- Includes support for IBM InfoSphere BigInsights, Cloudera, Apache and Hortonworks Hadoop Distributed File System (HDFS).
- Offers Balanced Optimization for Hadoop capabilities to push processing to the data and improve efficiency.
- Supports big-data governance including features such as impact analysis and data lineage.
Workload and business rules management
- Helps enable policy-driven control of system resources and prioritization of different classes of workloads.
- Helps you optimize hardware utilization and prioritize tasks, control job activities where resources exceed specified thresholds, and assess and reassign the priority of jobs as they are submitted into the queue.
- Integrates with IBM Operational Decision Management (formerly ILOG JRules), allowing you to implement decision logic within IBM InfoSphere Information Server.
Near real-time data integration
- Captures messages from Message Oriented Middleware (MOM) queues using Java Message Services (JMS) or WebSphere MQ adapters, allowing you to combine data into conforming operational and historical analysis perspectives.
- Provides a service-oriented architecture (SOA) for publishing data integration logic as shared services that can be reused over the enterprise.
- Can simultaneously support high-speed, high reliability requirements of transactional processing and the large volume bulk data requirements of batch processing.
Ease of use
- Includes an operations console and interactive debugger for parallel jobs to help you enhance productivity and accelerate problem resolution.
- Helps reduce the development and maintenance cycle for data integration projects by simplifying administration and maximizing development resources.
- Offers operational intelligence capabilities, smart management of metadata and metadata imports, and parallel debugging capabilities to help enhance productivity when working with partitioned data.