All Forums Database
Mrao2013 9 posts Joined 04/10
28 Jan 2014
Near Real Time Stored Procedure Architecture Framework

Hi,
Can someone pls explain what is Near Real Time stored procedure architecture framework?  How it works?  If you have implemented the stored procedure in your project, pls share how you implemented, challenges faced etc.  Can we replace it by bteq?  If yes, what are advantages and disadvanges with stored procedure and bteq approaches?
any info will be appreciated.  Thanks in Advance.
Mrao

Raja_KT 1246 posts Joined 07/09
29 Jan 2014

I think that tpump is a better option.
I do agree when we use stored procedure in ELT. 

Real time and Near real time is a challenge always in my past experience: Ab Initio and SAP DS--

Fresh data like stream from web, crm, fraud data in conjucntion with historical data is always 

a challenge. Load performance is controlled by TASM.

 

AEDW handles CDC,Queues from one source and ETL data, with robust transformation coming from DI server with no hiccups.Continuous streams and mini-batch come into play when customers want quick data and up-to-date.Latency is important. I do not know how people have opted stored procedure.

 

So I feel that tpump can consume data from messages , queues. 

 

Let us hear from other folks, who implement it.

Cheers,

 

 

 

Raja K Thaw
My wiki: http://en.wikipedia.org/wiki/User:Kt_raj1
Street Children suffer not by their fault. We can help them if we want.

Mrao2013 9 posts Joined 04/10
29 Jan 2014

Thanks Raja for your input.  Let me add more details on what we are trying to achieve.
There is a transactional system, which is a source for EDW.  The task is to create an enterprise summary from the transacational system on near real time.  Having said that, one question.
The stored procedure can talk to the transactional system directly? or have to stage the transactions on EDW before running the stored procedure?

Adeel Chaudhry 773 posts Joined 04/08
01 Feb 2014

Ideally, you need to bring in data in TD before you could get the benefits of SP. The name 'Near real-time architecture' suggests you should be using tpump to trickle the data from source to TD (stage) and then can process it using SPs/BTEQs/etc.
 
I would personally prefer SPs due to the control they give and its just easy to manage.

-- If you are stuck at something .... consider it an opportunity to think anew.

You must sign in to leave a comment.