New Teradata Unity Director/Loader 14.11

We are pleased to announce the General Customer Availability (GCA) of Unity Director/Loader 14.11. With release 14.11, Unity Director/Loader now conforms to the Viewpoint 15.00 portlet requirements. Unity Director/Loader 14.11 has added support for TPT Update and Multi-Load as ETL portocols. With the addition of these two protocols, Unity Director/Loader now supports all of the TPT and Legacy Bulk Load protocols.

Teradata Unity Director/Loader Overview

Unity Director/Loader is part of the Teradata Unity portfolio of products supporting multi-system Teradata environments. It provides user and query routing between systems, database synchronization, data synchronization, and a query auto resubmit capability all in one product. Unity Director/Loader also helps customers with high availability and disaster recovery and workload balancing across Teradata systems. The Loader feature of Unity Director/Loader is a powerful replacement for custom-built “dual load” solutions. It applies the underlying capabilities of Teradata’s patented SQL Multicast technology to Teradata Parallel Transporter (TPT) bulk load utilities for intelligent and selective routing of bulk loads to the appropriate Teradata system(s).  

Unity Director/Loader 14.11 Features

Query Routing

Queries are routed to systems in a dual active environment based routing rules created by the user and location of the data. Query routing enables high availability and workload balancing between systems.

Data Synchronization

If the data is on both systems in a dual active environment then reads are routed to a single system while writes are routed to both systems. Transactional updates are applied across all participating systems to maintain consistency of data.

Auto Resubmit

Aborted in-flight queries are automatically re-submitted to an alternate system that can satisfy the query

High Volume Loading

Unity Director/Loader is scalable to meet requirements of data storage volume, load window, and throughput requirements. Intelligently loads data to the appropriate system(s). It validates successful loading to eliminate DBA interventions.

Bulk Load Utility Support

Complete support of TPT protocols including Load, Export and Update. Complete support of legacy bulk load protocols including Fast Load, Fast Export and Multi-Load.

Dual Active Teradata Systems

Director Loader

Unity Director/Loader can be used to keep two Teradata systems in synch in either a dual active or active/DR configuration.

The boxes depicting the two Teradata systems A and B are purposely shown to be different sizes. That implies that the systems do not have to be identical. System A could be an EDW, system B could be an Appliance or a deep storage device.

In front of the two systems we will add the Unity Director/Loader server. The Unity Director/Loader server contains infrastructure that it uses to manage the two systems. It includes a parser that does a light parsing of the incoming SQL, a Data Dictionary that keeps track of database objects, and a sequencer that keeps all of the writes in the correct order and completes all of the locking tasks required within Unity Director/Loader.

Instead pointing the users directly to one of the Teradata systems, they are connected to the Unity Director/Loader server instead. All reads from the users are flowing through the Unity Director/Loader server which routes them to one of the Teradata systems. It is doing some light parsing of the SQL, figuring out what objects are being accessed. Then looking up those objects in its Data Dictionary to figure out where the objects reside, either on one or both of the systems. And then using the sequencer to apply the locks and go get the data. The users do not know and do not care from which system the data comes from if the table is kept in synch via Unity Director/Loader.

Data is kept in synch on both systems on writes and we‘ll see how that happens next.

In order to maintain synchronization on both systems Teradata uses a technology called SQL Multicast. SQL Multicast will take any SQL updates or DDL changes and apply them to both systems in the same sequence. No guarantee that the changes will be done at the same time as that will depend on the workloads on the individual Teradata systems, just that they will be done in the same order.  Unity Director/Loader knows which system has completed a particular update and knows to route subsequent reads to that Teradata system with the freshest data.

What if some ETL activity needs to be done in addition to SQL Writes? This is where the Loader feature comes in. Unity Loader runs on the same TMS as Unity Director and they work together. Unity Loader relies on the same infrastructure as Unity Director which is how things are kept in synch.

Instead of pointing the ETL streams at the individual Teradata systems, point the ETL stream at the Unity Director/Loader TMS. Unity Loader will figure out if one or both systems need to be loaded and then go execute. Simultaneously to loading, reads and writes can be done through Unity Director. The Sequencer will make sure that everything happens in the correct order.

Within the Unity Director/Loader TMS a recovery log is maintained. That means all writes to System A and System B are recorded within the Unity Director and Unity Loader TMS. If System A were to go off-line for either planned or unplanned downtime, Unity Director/Loader will remember all of the writes that happened while System A was off-line. When System A comes back up they are replayed in the same order until System A gets back in synch with System B and then workload can be distributed between the two systems.

Now System A can be taken down to do a database upgrade, brought back up, have Director/Loader get it back in synch with System B, and then operate normally again. Then System B can be taken down to do the same database upgrade, Unity Director/Loader will replay all of the writes and get it back in synch and now both systems have been upgraded without the users seeing any downtime for those objects under Unity Director/Loader management. This is the power of Unity Director/Loader.

More on Loader and the Dual Load Problem

In the previous section we talked about pointing a single ETL stream to the Unity Director/Loader server and having Director/Loader handle the load to multiple Teradata systems.

Loading two systems instead of one sounds easy on the surface; just run the ETL job twice. But it can become real complicated in a hurry. What if data is changing? What do I do if one of the jobs fails? How can I maintain consistency across systems?

Unity Director/Loader solves these problems and more without having to make changes to the ETL jobs. With the release of Director/Loader 14.11 all of the TPT and Legacy protocols are supported.

Compatibility Matrix

Unity Director/Loader 14.11 conforms to Viewpoint 15.00 portlet requirements. All product portlets need to be upgraded in sync when going from Viewpoint 14.10 to Viewpoint 15.00.  The listing below documents the minimal product versions necessary for Teradata Viewpoint 15.00 compatibility.

Teradata Database = 13.00 to 14.10

Aster Database = N/A

Hadoop = N/A