Incremental Data Loads
Is it possible to add incremental load functionality to Power Query? Pull and load just new or changed rows from source system
We’re pleased to announce that in the May release of Power BI Desktop you can now define incremental refresh policies for your models.
At present these only apply once the model is published to a Premium workspace in the Power BI service; we’ll be bringing it to all Pro users in the future.
You can read more about it in our blog here: https://powerbi.microsoft.com/en-us/blog/power-bi-desktop-may-2018-feature-summary/#incrementalRefresh
We’re going to leave this idea open for now, so please give us your feedback while we’re in this preview phase!
Kevin H commented
(a) There seems to be an issue where an incremental refresh policy disables ALL parameters in the Power BI Service:
(b) Sometimes I want my data model to do a full refresh, and ignore incremental refresh policies. E.g., when scheduling refreshes in the Service, for each scheduled time, could there please be an option for either incremental or full? And for dataset refreshes triggered by the Power BI API, could there please be an option to trigger an incremental or full refresh?
It will be great to have incremental data load feature added to Power BI desktop for pro accounts soon. I have a query with historical data and another query with new data. Both these queries are appended into a new query and it is painful to watch million of rows from historical data refresh every day. Hope you add it soon!!!
Todd Chittenden commented
Found a link to this idea from a forum post on strange behavior of "Include in Report Refresh".
I'm trying to have an Excel connection in the model that is static data, so does NOT need to be refreshed by the service. But the Gateway requires info on the source file and will NOT do any refreshes until that is resolved.
WTF!!! Disabling in Report Refresh should NOT be dependent on if you are using the Desktop or the Service. It is a BUG in my opinion. Or at least, poorly thought-out requirements.
Power Bi Premium is not an option for us right now. It would be good if incremental refresh feature for Odata is enabled in Powerbi Pro.
Still looking for pro support. Seems like the shared capacity scenario is the one which would most benefit from incremental.
Announced feature is different than incremental load, in incremental load we wanted to append new data to the original data. For example I connected to a website which updates on daily basis, I want to make database of that data to analyze it over time. And of course it should be automated and work in service.
Ryan R commented
I am thankful that you will be sharing this with all pro users. I agree with the others, it would be nice to know when you think this will be available for Pro users?
Are there any plans to add incremental refresh to the Power BI Report Server implementation?
What about on the Power BI Desktop?
This is a must! Top priority.
Michael Bristol commented
i am looking forward to this coming to Pro. Our organization is not big enough to justify Premium as of yet.
Por favor que esté disponible en power bi desktop.
Earnestly looking forward to have this feature available in Power BI Desktop.
It would be fantastic to have this also in Power BI Desktop. Currently we have to use some not ideal DAX workarounds how to load new data only :(
Have to agree with what Withold mentioned earlier. Also was my understanding that this idea was to solve the problem of taking snapshots of a database, that involves having to continually download excel files and dump them into a folder.
great work! Need for Power BI PRO, please...
Please introduce this for Power BI Desktop to connect to SharePoint Lists for ODataFeed and SharePoint List Connectors.
Joe Champagne commented
This is a great feature. Why can't you download a pbix file from the service once incremental refresh is on?
This is a great feature, however, for certain reference data sources that don't need a daily refresh or a refreshed schedule at all, it would be great to be able to designate a lesser frequency for the entire data source. An example might be - a refresh frequency setting - daily, monthly, quarterly?
Is there a scheduled date for PRO users?
When will we have other users?