Incremental Refresh Detect data changes have different schedules for [short]] period and all data
Say you have 5 years of data in your model
Currently you have new records and data changes so we set the model to process the last years data and detect changes Using LastUpdatedate in our data
This is all great but very occasionally, there may be a data change outside of the years worth of updates
the question is, how do you deal with this? I Dont want to be updating the entire models data set when updates to older data doesnt happen that often. BUT, if it does happen, we want to reflect it in the model
Could you Store the last 5 years worth of data
Refresh the last 5 years worth of data but only if changes have been detected?
Is that still efficient? It feels to me like its not because it then has to go and look across the entire 5 years of data set. has it changed? Yes or No.