Wednesday 6 May 2015

Full and incremental loads in ODI in Biapps 11g(11.1.1.8.1)



               Full or Incremental loads are implemented using the below function.

RUN_FULL_INCREMENTAL:
RUN_FULL_INCREMENTAL('IS_INCREMENTAL',1=1,RUN_REPLICATED_TRANSACTIONAL('#IS_SDS_DEPLOYED',(F4801.WAUPMJ >= JDEDATE(TO_DATE_VAR('#LAST_EXTRACT_DATE'))),(F4801.CDC$_SRC_LAST_UPDATE_DATE >= TO_DATE_VAR('#LAST_EXTRACT_DATE')))
The above function should place  in filter of each and every interface.
This function is implemented based on the is_incremental variable.

IS_INCREMENTAL Variable:
if (IS_INCREMENTAL).toString().equalsIgnoreCase("Y")) then run incremental else full load.
 IS_INCREMENTAL_EXPRESSION is based on the below query.
SELECT IIF(COUNT(*)>0,'Y','N') FROM      QUALIFY_DS(W_ETL_LOAD_DATES) WHERE      PACKAGE_NAME = 'GET_CONTAINER_NAME()'AND                (DATASOURCE_NUM_ID=#DATASOURCE_NUM_ID OR                DATASOURCE_NUM_ID=#WH_DATASOURCE_NUM_ID) AND      ETL_USAGE_CODE = '#ETL_USAGE_CODE' AND                COMMITTED='1'

RUN_REPLICATED_TRANSACTIONAL Function:
This function is based on the value of  IS_SDS_DEPLOYED variable and date conditions between source columns and LAST_EXTRACT_DATE.
If ((IS_SDS_DEPLOYED).toString().equalsIgnoreCase("Y"))
/* Only run when SDS is deployed */ $$>
(YES_EXPRESSION) 
Yes_Expression is between source change data capture lastupdate date of golden gate and Last_extract_date like below.
(F0411.CDC$_SRC_LAST_UPDATE_DATE >= TO_DATE_VAR('#LAST_EXTRACT_DATE') OR F0012_KGCO.CDC$_SRC_LAST_UPDATE_DATE >= TO_DATE_VAR('#LAST_EXTRACT_DATE') OR F0901_KGCO.CDC$_SRC_LAST_UPDATE_DATE >= TO_DATE_VAR('#LAST_EXTRACT_DATE') OR F0012_DFLT.CDC$_SRC_LAST_UPDATE_DATE >= TO_DATE_VAR('#LAST_EXTRACT_DATE') OR F0901_DFLT.CDC$_SRC_LAST_UPDATE_DATE >= TO_DATE_VAR('#LAST_EXTRACT_DATE') OR F0413_DATE.CDC$_SRC_LAST_UPDATE_DATE >= TO_DATE_VAR('#LAST_EXTRACT_DATE')))

No_expression is between JDE date column and LAST_EXTRACT_DATE like below
(F0411.RPUPMJ >= JDEDATE(TO_DATE_VAR('#LAST_EXTRACT_DATE')) OR F0012_KGCO.KGUPMJ >= JDEDATE(TO_DATE_VAR('#LAST_EXTRACT_DATE')) OR F0901_KGCO.GMUPMJ >= JDEDATE(TO_DATE_VAR('#LAST_EXTRACT_DATE')) OR F0012_DFLT.KGUPMJ >= JDEDATE(TO_DATE_VAR('#LAST_EXTRACT_DATE')) OR F0901_DFLT.GMUPMJ >= JDEDATE(TO_DATE_VAR('#LAST_EXTRACT_DATE')) OR F0413_DATE.RMUPMJ >= JDEDATE(TO_DATE_VAR('#LAST_EXTRACT_DATE')) OR F0413_DATE.RNUPMJ >= JDEDATE(TO_DATE_VAR('#LAST_EXTRACT_DATE')))
Else
 /* Only run when SDS is not deployed */ $$>
NO_EXPRESSION:

LAST_EXTRACT_DATE:
LASTEXTRACTDATE is implemented based on the below query.
SELECT  COALESCE(TO_CHAR_FORMAT(MIN( ADD_TO_DATE(ETL_LOAD_DATE,"DD",(-1)*#PRUNE_DAYS)), 'YYYY-MM-DD HH24:MI:SS'), '1970-01-01 00:00:00'  ) FROM   W_ETL_LOAD_DATES WHERE PACKAGE_NAME = 'GET_CONTAINER_NAME()' AND DATASOURCE_NUM_ID=#DATASOURCE_NUM_ID  AND  ETL_USAGE_CODE = '#ETL_USAGE_CODE'

W_ETL_LOAD_DATES is system generated table.

1 comment:

  1. What is Qualify_DS from the code #IS_INCREMENTAL .....QUALIFY_DS(W_ETL_LOAD_DATES) where....?

    ReplyDelete