All Forums Hadoop
iamRaj 1 post Joined 09/15
16 Sep 2015
Hive table export to Teradata using TDCH failing... due to connection reset
when exporting 2billion+ records into teradata from hadoop using TDCH (Teradata Connector for Hadoop) using the below command with "batch.insert",

hadoop jar teradata-connector-1.3.2-hadoop210.jar com.teradata.connector.common.tool.ConnectorExportTool \
-D mapreduce.job.queuename=<queuename> \
-libjars ${LIB_JARS} \
-classname com.teradata.jdbc.TeraDriver \
-url <jdbc_connection_string> \
-username <user_id> \
-password "********" \
-jobtype hive \
-sourcedatabase <hive_src_dbase> \
-sourcetable <hive_src_table> \
-fileformat orcfile \
-stagedatabase <stg_db_in_tdata> \
-stagetablename <stg_tbl_in_tdata> \
-targettable <target_tbl_in_tdata> \
-nummappers 25 \
-batchsize 13000 \
-method batch.insert \
-usexviews false \
-keepstagetable true \
-queryband '<queryband>'


Data is loading successfully into stage table but, then the export job fails before inserting the records in stage table into target table saying, "Connection Reset"

Can someone please help me identify the reason for this, and how to fix this. Thanks a lot in advance !!
-Raj
rvgn 1 post Joined 04/15
02 Nov 2015

Did you set the environment variables? Export these variables first and run your command.
export TDCH_JAR=/usr/lib/tdch/1.3/lib/teradata-connector-1.3.3.jar
export HADOOP_HOME=/usr/lib/hadoop
export HIVE_HOME=/usr/lib/hive
export HADOOP_CLASSPATH=$HIVE_HOME/conf:$HIVE_HOME/lib/antlr-runtime-3.4.jar:$HIVE_HOME/lib/commons-dbcp-1.4.jar:$HIVE_HOME/lib/commons-pool-1.5.4.jar:$HIVE_HOME/lib/datanucleus-api-jdo-3.2.6.jar:$HIVE_HOME/lib/datanucleus-core-3.2.10.jar:$HIVE_HOME/lib/datanucleus-rdbms-3.2.9.jar:$HIVE_HOME/lib/hive-cli-0.13.1.2.1.11.0-891.jar:$HIVE_HOME/lib/hive-exec-0.13.1.2.1.11.0-891.jar:$HIVE_HOME/lib/hive-metastore-0.13.1.2.1.11.0-891.jar:$HIVE_HOME/lib/jdo-api-3.0.1.jar:$HIVE_HOME/lib/libfb303-0.9.0.jar:$HIVE_HOME/lib/libthrift-0.9.0.jar:$HIVE_HOME/lib/slf4j-api-1.6.4.jar:$HIVE_HOME/lib/hive-common-0.13.1.2.1.11.0-891.jar:$HIVE_HOME/lib/hive-shims-common-0.13.1.2.1.11.0-891.jar:$HIVE_HOME/lib/hive-shims-0.23-0.13.1.2.1.11.0-891.jar:$HIVE_HOME/lib/hive-shims-common-secure-0.13.1.2.1.11.0-891.jar
 
export HIVE_LIB_JARS=$HIVE_HOME/lib/hive-cli-0.13.1.2.1.11.0-891.jar,$HIVE_HOME/lib/hive-exec-0.13.1.2.1.11.0-891.jar,$HIVE_HOME/lib/hive-metastore-0.13.1.2.1.11.0-891.jar,$HIVE_HOME/lib/libfb303-0.9.0.jar,$HIVE_HOME/lib/libthrift-0.9.0.jar,$HIVE_HOME/lib/jdo-api-3.0.1.jar,$HIVE_HOME/lib/slf4j-api-1.6.4.jar,$HIVE_HOME/lib/hive-common-0.13.1.2.1.11.0-891.jar,$HIVE_HOME/lib/hive-shims-common-0.13.1.2.1.11.0-891.jar,$HIVE_HOME/lib/hive-shims-0.23-0.13.1.2.1.11.0-891.jar,$HIVE_HOME/lib/hive-shims-common-secure-0.13.1.2.1.11.0-891.jar
 
Example of Export command:
hadoop jar $TDCH_JAR com.teradata.connector.common.tool.ConnectorExportTool -D convert.datetype.format="yyyy-MM-dd HH:mm:ss" -D convert.sourceTz="America/Chicago" -D convert.targetTz="America/Chicago" -libjars $HIVE_LIB_JARS  -classname com.teradata.jdbc.TeraDriver  -url jdbc:teradata://tdt/DATABASE=sandbox -username *****  -password ******  -jobtype hive -method internal.fastload  -sourcetable database.table_name  -sourcefieldnames "****(source filed names)  -targetfieldnames "****(your targe field names)" -hiveconf file:///etc/hive/conf.dist/hive-site.xml -fileformat ORCFile -nummappers 18 -sourcerecordschema "int,int,string,StringFMTTZToTimestampTZ(convert.datetype.format, convert.sourceTz, convert.targetTz),string,string,StringFMTTZToTimestampTZ(convert.datetype.format, convert.sourceTz, convert.targetTz)" &> log/query.outut &

You must sign in to leave a comment.