All Forums Connectivity
k3na 19 posts Joined 09/14
22 Jun 2015
Teradata JDBC fastload hangs after 50K records

Hello Teradata experts,

I'm now working on optimization of loading of data into Teradata.

Our application allows to load big csv files by means of jdbc connection and executeBatch.
I've run the load with a relatively big dataset (about 6M rows and 1.2 Gb of data). I've started with our default parameters:
1) batch size = 1000 records
2) a target table had a primary key constraint

The load was very slow and could take about 3-4 hours.

When I've increased the batch size to 50000, and made a source table "multitable" and NoPI, the loading time decreased dramatically to about 10-12 min, which is already acceptable for us.

I also wanted to evaluate the "FASTLOAD" tool and set the type=FASTLOAD in our jdbc connection string.
However, when I try to load data with the fastload, it seems to hang after 50K loaded records (and it doesn't matter whether the batch size = 1000 or 50000). I also found a similar topic here: http://forums.teradata.com/forum/connectivity/ teradata-jdbc-fastload-hangs-after-approximately-50- batch-insert

What could be the reason for such a behavior and how could we avoid it?
Many thanks for any hint!
Katja

k3na 19 posts Joined 09/14
22 Jun 2015

For information:
jdbc driver 14.00.00.37, Teradata DB version 15.0.1.06.

tomnolan 594 posts Joined 01/08
22 Jun 2015

If you are a customer, then please open an incident with Teradata Customer Service.

TDThrottle 51 posts Joined 11/11
22 Jun 2015

In my experience with 13.10 & 14.10 calling down FastLoad utility from JDBC connection string end-up in hung state with large record set. We had an Incident to Teradata GSC for 13.10 and the response was use of JDBC for large data set is not recommended in other words its not certified.
This should have been fixed in 15.0 release. Upgrade your jdbc driver to recent 15.x version and give a try. Else I would prefer switching to command line/shall revoke of fastload utility.
Thanks!!

tomnolan 594 posts Joined 01/08
22 Jun 2015

>>> We had an Incident to Teradata GSC for 13.10 and the response was use of JDBC for large data set is not recommended in other words its not certified.
 
That is not correct. JDBC FastLoad is fully supported for any size data set.
 
When loading large data sets, the primary caveat to be aware of is that JDBC FastLoad does not support the checkpoint feature of TPT and the standalone FastLoad utility; therefore, if a JDBC FastLoad operation is interrupted for some reason, it must be restarted from the beginning.

TDThrottle 51 posts Joined 11/11
22 Jun 2015

Thanks for clarification on checkpoint feature for JDBC FastLoad. Have Checkpoint enforement is fixed in 15.0
 
 

tomnolan 594 posts Joined 01/08
22 Jun 2015

The addition of checkpoint functionality to JDBC FastLoad would be a new feature, not something to be "fixed".
 
And No, that functionality is not present in Teradata JDBC Driver 15.0 or 15.10.

TDThrottle 51 posts Joined 11/11
22 Jun 2015

Thank you Tom

k3na 19 posts Joined 09/14
23 Jun 2015

Thanks you for clarification!

You must sign in to leave a comment.