Incremental Data Loads
Is it possible to add incremental load functionality to Power Query? Pull and load just new or changed rows from source system
We’re pleased to announce that in the May release of Power BI Desktop you can now define incremental refresh policies for your models.
At present these only apply once the model is published to a Premium workspace in the Power BI service; we’ll be bringing it to all Pro users in the future.
You can read more about it in our blog here: https://powerbi.microsoft.com/en-us/blog/power-bi-desktop-may-2018-feature-summary/#incrementalRefresh
We’re going to leave this idea open for now, so please give us your feedback while we’re in this preview phase!
Announced feature is different than incremental load, in incremental load we wanted to append new data to the original data. For example I connected to a website which updates on daily basis, I want to make database of that data to analyze it over time. And of course it should be automated and work in service.
Ryan R commented
I am thankful that you will be sharing this with all pro users. I agree with the others, it would be nice to know when you think this will be available for Pro users?
Are there any plans to add incremental refresh to the Power BI Report Server implementation?
What about on the Power BI Desktop?
This is a must! Top priority.
Michael Bristol commented
i am looking forward to this coming to Pro. Our organization is not big enough to justify Premium as of yet.
Por favor que esté disponible en power bi desktop.
Earnestly looking forward to have this feature available in Power BI Desktop.
It would be fantastic to have this also in Power BI Desktop. Currently we have to use some not ideal DAX workarounds how to load new data only :(
Have to agree with what Withold mentioned earlier. Also was my understanding that this idea was to solve the problem of taking snapshots of a database, that involves having to continually download excel files and dump them into a folder.
great work! Need for Power BI PRO, please...
Please introduce this for Power BI Desktop to connect to SharePoint Lists for ODataFeed and SharePoint List Connectors.
[Deleted User] commented
This is a great feature, however, for certain reference data sources that don't need a daily refresh or a refreshed schedule at all, it would be great to be able to designate a lesser frequency for the entire data source. An example might be - a refresh frequency setting - daily, monthly, quarterly?
Is there a scheduled date for PRO users?
When will we have other users?
Michael Hughes commented
Hi, coulf you make it possible to keep data for a rolling 24 hours and load new data every two hours (as the relevent old data is removed from the service)?
One of our API's makes a call to one of our internal systems on a ~2 hour basis, extracting the most relevant information for say 10% of our requests (eg the next 200 combinations of city pairs out of a total ranked 2,000) and places the results in a Oracle database (200 city pairs equals output of around 12m records).
Our Oracle database is emptied everyday at midnight meaning that we dont get the vast majority of the 'city pairs' until later in the day when everyone has gone home (because the API starts at city pair '1' and goes down the list).
I want to keep the data for say 22 hours, and load the new data as it comes in using Inremental loading, whilst dropping off the previous days 1-200 city pairs when todays pull of 1-200 city pairs is added.
Any News on this? It has been opened since 2015 . . .
Excited about this feature! One issue from initial testing: using incremental refresh seems to make a datasource ineligible for the UpdateDatasources api call--I get a BadRequest response if the report has an incremental refresh set. Will that be changed? Incremental refresh will drastically improve our ability to work with larger datasets, but only if we can still programmatically change the database connection being used.
When can we expect this for Pro users? TBH I think this is poor form to only allow the feature for Premium users only
hello, this seems to be a powerful update, but id doesn´t meet the need I originally understood from this post.
What we need is in Power BI desktop the possibility to do shots of full dataset everytime you refresh data that appends into the existing table, including the refresh date in an additional column.
we have a SharePoint database with changing content and want to monitor the evolution of the data periodically. Currently, to obtain this we need to download from SharePoint to Excel manually to a library twice a month, add refresh date from creation date of the excel file and import to PowerBI.
This is a highly manual process that could be very easily solved with an incremental refresh of the SharePoint list directly to PowerBi desktop, by appending current dataset and giving it a timestamp.
matthew Jin commented
this feature will allow appending new data to existing as far as I can tell. For example, you can set "store rows in the last" to 10 years, and "refresh rows in the last" to 1 day will continue to add last one day of data to existing dataset which will keep appending data for the next 10 years as long as the storage limit is ok. Did I not understand this feature correctly?