All Forums Tools
TryTeradata 3 posts Joined 07/10
03 Sep 2010
How-to Skip duplicates

Hi Guys,

I'm loading flat files into a staging table using the LOAD operator (no indexes). Once the load is done, i do a select * on that table and insert into a allready populated table with a unique ppi using a DDL Operator.
My problem is that if the insert encounters a duplicate record, the entire batch insert is skipped. So no records are inserted. Is there a way for me to just ignore duplicate records like i can do when using the load operator?

I have changed my script to use an EXPORT and the IMPORT instead of the select * insert into, but that results in slower performance.

Any suggestions would be greatly appreciated.

Tags:
VelvetElvis 4 posts Joined 09/10
03 Sep 2010

Rather than kicking of an insert/select *, write the sql with a group by around the PI field(s). It sounds like you're loading into a UPI table, rather than complete duplicate records. The load operator mimics fastload in that it discards duplicate records as it loads. You might also spend some time to understand your source data better, i.e. are those records valid? etc...

02 Nov 2010

Try using fast load. It ignores duplicate rows.

feinholz 1234 posts Joined 05/08
03 Nov 2010

The Load operator (FastLoad protocol) cannot be used because the target table is already populated. The FastLoad protocol requires an empty table.

Why not use the Update operator? That uses the MultiLoad protocol, might be faster than the "Load to staging table, followed by INS-SEL", unless you are loading a MULTISET table.

--SteveF

You must sign in to leave a comment.