All Forums General
anoopjohn 2 posts Joined 09/14
07 Jun 2016
Teradata to AWS(Redshift)

Hi Team,
I have a table in Teradata with more than 9 Tb data.I need to copy the entire content of this table to AWS-Redshift DB.
I have identified below options ,
1.Fast export the Teradata table content into flat files ,then SCPs these files to AWS servers and then load Redshift
2.Instead of Transferring flat files,transfer the Teradata specific data files (the teradata files which actually store table data internally) to AWS server.(An additional logic need to be introduced in AWS server to read these files).By using ARC utility this file can be extracted at Teradata server.
(not sure whether the above logic will actually work or not :) )
3.Use AWS's snowbowl facility ,where the Tearadata table content will be moved into AWS's equipment (AWS claims that this is very fast) and then connect the equipmet to AWS and then by that way load Redshift
Could you please share your suggestions on each options.+ves and -ves regarding server space,time required etc.

dhg 2 posts Joined 04/10
01 Jul 2016

Redshift will want a delimited flat files per table.  The simipliest way is to use TPT export/fastexport to gz compressed delimited flat files to on prem storage.  If you look around you can probably find a TDC macro that will build the export job for you.   You will want to encrypt or pwd protect your cmpressed flat files.   Use AWS SDK or other S3 tool to multipart upload your files to S3 bucket in AWS.  If your files are multiple 5+TB/PB then you may want to look at snowball (network transfer speed MB/sec * total volume vs. ~2 weeks for snowball calculation).   Then run your Redshift copy commands using files in S3 buckets to load your data into Redshift.  
Arc at the end of the day may make an inital backup easier but you will need 1 flat file per table so it is more work than first option but for giggles you could spin up a TDC instance inside AWS restore you data there then generate the flat files to S3 / EBS.  This is alot of extra steps and cost to me since the compressed flat files should be about the same size as compressed arc file but you should test this to see if that is true for your data set.    Good luck! 

ToddAWalter 316 posts Joined 10/11
01 Jul 2016

Or better yet just use Teradata on AWS to run your work there!

You must sign in to leave a comment.