Incremental Data Loads
Is it possible to add incremental load functionality to Power Query? Pull and load just new or changed rows from source system
We’re pleased to announce that in the May release of Power BI Desktop you can now define incremental refresh policies for your models.
At present these only apply once the model is published to a Premium workspace in the Power BI service; we’ll be bringing it to all Pro users in the future.
You can read more about it in our blog here: https://powerbi.microsoft.com/en-us/blog/power-bi-desktop-may-2018-feature-summary/#incrementalRefresh
We’re going to leave this idea open for now, so please give us your feedback while we’re in this preview phase!
Are there any plans to add incremental refresh to the Power BI Report Server implementation?
What about on the Power BI Desktop?
This is a must! Top priority.
Michael Bristol commented
i am looking forward to this coming to Pro. Our organization is not big enough to justify Premium as of yet.
Por favor que esté disponible en power bi desktop.
Earnestly looking forward to have this feature available in Power BI Desktop.
It would be fantastic to have this also in Power BI Desktop. Currently we have to use some not ideal DAX workarounds how to load new data only :(
Have to agree with what Withold mentioned earlier. Also was my understanding that this idea was to solve the problem of taking snapshots of a database, that involves having to continually download excel files and dump them into a folder.
great work! Need for Power BI PRO, please...
Please introduce this for Power BI Desktop to connect to SharePoint Lists for ODataFeed and SharePoint List Connectors.
Joe Champagne commented
This is a great feature. Why can't you download a pbix file from the service once incremental refresh is on?
This is a great feature, however, for certain reference data sources that don't need a daily refresh or a refreshed schedule at all, it would be great to be able to designate a lesser frequency for the entire data source. An example might be - a refresh frequency setting - daily, monthly, quarterly?
Is there a scheduled date for PRO users?
When will we have other users?
Michael Hughes commented
Hi, coulf you make it possible to keep data for a rolling 24 hours and load new data every two hours (as the relevent old data is removed from the service)?
One of our API's makes a call to one of our internal systems on a ~2 hour basis, extracting the most relevant information for say 10% of our requests (eg the next 200 combinations of city pairs out of a total ranked 2,000) and places the results in a Oracle database (200 city pairs equals output of around 12m records).
Our Oracle database is emptied everyday at midnight meaning that we dont get the vast majority of the 'city pairs' until later in the day when everyone has gone home (because the API starts at city pair '1' and goes down the list).
I want to keep the data for say 22 hours, and load the new data as it comes in using Inremental loading, whilst dropping off the previous days 1-200 city pairs when todays pull of 1-200 city pairs is added.
Any News on this? It has been opened since 2015 . . .
Excited about this feature! One issue from initial testing: using incremental refresh seems to make a datasource ineligible for the UpdateDatasources api call--I get a BadRequest response if the report has an incremental refresh set. Will that be changed? Incremental refresh will drastically improve our ability to work with larger datasets, but only if we can still programmatically change the database connection being used.
When can we expect this for Pro users? TBH I think this is poor form to only allow the feature for Premium users only
hello, this seems to be a powerful update, but id doesn´t meet the need I originally understood from this post.
What we need is in Power BI desktop the possibility to do shots of full dataset everytime you refresh data that appends into the existing table, including the refresh date in an additional column.
we have a SharePoint database with changing content and want to monitor the evolution of the data periodically. Currently, to obtain this we need to download from SharePoint to Excel manually to a library twice a month, add refresh date from creation date of the excel file and import to PowerBI.
This is a highly manual process that could be very easily solved with an incremental refresh of the SharePoint list directly to PowerBi desktop, by appending current dataset and giving it a timestamp.
matthew Jin commented
this feature will allow appending new data to existing as far as I can tell. For example, you can set "store rows in the last" to 10 years, and "refresh rows in the last" to 1 day will continue to add last one day of data to existing dataset which will keep appending data for the next 10 years as long as the storage limit is ok. Did I not understand this feature correctly?
Good news and this is a good start, but incremental refresh on time cycle is not exactly what we had in mind.... we want to append new data to existing data from sources like CSV.
The use case is adding a period's data to an existing model. In other words instead of truncate (delete) and replace data we simply want to tack it onto the table that's already there. Otherwise accumulation of time series data has to be done outside the tool.
Of course you don't want to accumulate forever, so being able to selectively delete date ranges of data within specific tables would be a natural companion functional requirement.
The only other choice at present is to use another database to accumulate time series data. This is fine but adds complexity for users. Access can only scale to 2GB so it's no good for this, which puts you into the realm of "real" databases, which are too complex for a lot of smaller enterprises.
Anyone doing periodic time series updates would benefit from this easy to implement feature... and that means almost any small to medium size business, which i believe is important to success.