Incremental Data Loads
Is it possible to add incremental load functionality to Power Query? Pull and load just new or changed rows from source system
We’re pleased to announce that in the May release of Power BI Desktop you can now define incremental refresh policies for your models.
At present these only apply once the model is published to a Premium workspace in the Power BI service; we’ll be bringing it to all Pro users in the future.
You can read more about it in our blog here: https://powerbi.microsoft.com/en-us/blog/power-bi-desktop-may-2018-feature-summary/#incrementalRefresh
We’re going to leave this idea open for now, so please give us your feedback while we’re in this preview phase!
Moneer Jlelaty commented
A must have feature, we are loading a lot of data which will be loaded again every time. If we can only load the newest data, that will save a lot of time, please start working on it.
imran khan commented
i need to refresh the whole data set everytime i refresh please get this feature
Imke Feldmann commented
Stronlgy support this feature.
In the meantime you can use this workaround: http://www.thebiccountant.com/2016/02/09/how-to-create-a-load-history-or-load-log-in-power-query-or-power-bi/
Dmitry Stramilov commented
Also, we need the ability to do incremental loads from Analysis Services.
I understand the difficult design complexities behind implementing this but doing so will allow average users to create some pretty powerful workflows!
Matt Horn commented
Absolute must have feature
A must have functionality specially in case of importing from Folders....
I'd also like there to be straight forward way of doing this please!
Daniel Wu commented
what we really need is the ability to incrementally load data into a power pivot data model
This post http://ms-olap.blogspot.com/2015/05/incremental-data-loads-in-microsoft.html says it works in Power Query for Excel. I tried doing the use same in Power BI Desktop but it still refreshes the existing data even thought refresh has been disable for that query.
Basically here is the use case. We have to load a fact table with say 1 million rows and daily 50K rows get added. I would want to the refresh operation to just add those new 50K rows or maybe only those with some update stamp.
could you explain a little bit more on how you've done this?
Giampaolo Pitzalis commented
I suggest to transform the checkbox "Enable refresh of this query" to a radio button with "Enable refresh..."+*"Disable Refresh..."* options, untied from the option "Enable load to report". If you check the "Disable" option you could refresh the query only manually and thus you can perform an incremental data load staging data in one or more disabled queries and combining them with which are enabled.
Yes it is. But you have to use advanced editor. The way I did this was to first load data from an input file into a table. Then I created a second query that loads data from that newly created table. Then I used advanced editor to change the first query in start with the steps from the first query and append the results. And you can avoid duplicate rows by creating a new column with some sort of compound key from your various columns (and later remove this column if you want, once dups are removed).
Yes really need this, as my data loads are way to large to pull the whole thing every refresh.