All Forums UDA
SayeeRamPrasad 2 posts Joined 03/08
31 Mar 2008
Data Skew on a UPI table (with HIGH volume) - How?

We've a SET table defined with a UPI that has 9.3 Million rows on a 1400 AMP, V2R6 Teradata system. Please look at the numbers below. The AMP that has the MAX rows has 3.47 times more rows than the AMP with the MIN rows. This does not make sense to me given the fact that we've a UPI and a high volume of rows in the table for an even distribution.xTimesGreaterMaxAvg - 1.56xTimesGreaterMaxMin - 3.47Mx - 1393664Avrg - 890657.07Mn - 401920CurPerm - 1282546176Can someone please shed some light on this? Is there something we can look at to improve data distribution? Any help would be greatly appreciated.PS: The table also has a PPI defined on a date field which is part of the UPI (if this matters).Thanks,Sayee.

dnoeth 4628 posts Joined 11/04
01 Apr 2008

Hi Sayee,most of the skew is due to the high number of AMPs:There are 65536 entries within the hashmap, so the average number of entries per AMP is:SELECT 2**16/1400 -> 46,81Of course there's no .81 entry, so some AMPs will get 46 and some 47:SELECT 47/46.0000 -> 1,0217 -> 2.17 percent more data even if there's a UPI.Now the table is still small (for a 1400 AMP system), 6642 rows per AMP in average, this is probably not enough for a perfectly even distribution. Now add this small skew to the 2.17 percent...Dieter

Dieter

SayeeRamPrasad 2 posts Joined 03/08
08 Apr 2008

Thank you, Dieter! That was pretty good information. I never would have thought 9MM rows was not enough volume. :-)We've more than 1 day's data in those tables now and we're seeing much better numbers (distribution).

martproc 1 post Joined 04/12
09 Apr 2016

Hi,
You can get into troubles when values of your UPI are not distributed uniformly. Following example generates rows with such UPI values. All 100 rows of SKEWED_UPI_TABLE generated by SKEWED_UPI_VIEW belong to AMP number 1.
It is clear that this is very unlikely in the real world, but one cannot rule it out. ;-)
Martin
CREATE RECURSIVE VIEW SKEWED_UPI_VIEW (N, AMP_NO, AMP_1_CNT)
AS
(
    SELECT t.*
    FROM
        (
        SELECT
            CAST(0 AS BIGINT) AS n_
        ,   HASHAMP(HASHBUCKET(HASHROW(n_))) AS amp_no_
        ,   CASE amp_no_ WHEN 1 THEN 1 ELSE 0 END AS amp_1_cnt_
        ) AS t
UNION ALL
    SELECT
        X.N + 1 AS N_
    ,   HASHAMP(HASHBUCKET(HASHROW(N_))) AS AMP_NO_
    ,   X.AMP_1_CNT + CASE AMP_NO_ WHEN 1 THEN 1 ELSE 0 END
    FROM SKEWED_UPI_VIEW AS X
    WHERE X.AMP_1_CNT < 100
)
;
CREATE MULTISET TABLE SKEWED_UPI_TABLE (N BIGINT)
UNIQUE PRIMARY INDEX (N)
;
INSERT INTO SKEWED_UPI_TABLE (N)
SELECT N
FROM SKEWED_UPI_VIEW
WHERE AMP_NO = 1
;
SELECT n, HASHAMP(HASHBUCKET(HASHROW(n))) AS amp_no
FROM SKEWED_UPI_TABLE
;

You must sign in to leave a comment.