I have been working with ADF in production for 5 years. Most irritating errors are not in Pipeline monitoring dashboards we built , not in production review calls , but raised by user . Something is wrong but no trace in the pipeline on what tripped the system to get this bug . Here are 3 of those irritating issue which i wish i caught before any user raises concern on integrity of the data we are showing.
The Decimal ID That Broke Your Fact Table Join
ID Column from the Source is coming with decimals i.e 1453.0 , 1238.0 etc because source view changed the formatting , The Same ID Column in other Dimension Data Load is still integer , Now your Fact Table Cannot Join with Dimension Table Causing Sudden Failure of the Dashboard numbers , No ADF Pipeline Notification , but now you need to answer to Management on why numbers are wrong.
ADF Does not Catch it as there is no issue in data type , because to handle nulls you have made all your landing table columns as varchar which accepts integer as well as…