How can we improve Power BI?

Incremental Data Loads

Is it possible to add incremental load functionality to Power Query? Pull and load just new or changed rows from source system

3,146 votes
Sign in
Check!
(thinking…)
Reset
or sign in with
  • facebook
  • google
    Password icon
    Signed in as (Sign out)

    We’ll send you updates on this idea

    Javier Recasens shared this idea  ·   ·  Flag idea as inappropriate…  ·  Admin →

    We’re pleased to announce that in the May release of Power BI Desktop you can now define incremental refresh policies for your models.

    At present these only apply once the model is published to a Premium workspace in the Power BI service; we’ll be bringing it to all Pro users in the future.

    You can read more about it in our blog here: https://powerbi.microsoft.com/en-us/blog/power-bi-desktop-may-2018-feature-summary/#incrementalRefresh

    We’re going to leave this idea open for now, so please give us your feedback while we’re in this preview phase!

    365 comments

    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      Signed in as (Sign out)
      Submitting...
      • Josh commented  ·   ·  Flag as inappropriate

        Are there any plans to add incremental refresh to the Power BI Report Server implementation?

      • Michael Bristol commented  ·   ·  Flag as inappropriate

        i am looking forward to this coming to Pro. Our organization is not big enough to justify Premium as of yet.

      • Hanka commented  ·   ·  Flag as inappropriate

        It would be fantastic to have this also in Power BI Desktop. Currently we have to use some not ideal DAX workarounds how to load new data only :(

      • Anonymous commented  ·   ·  Flag as inappropriate

        Have to agree with what Withold mentioned earlier. Also was my understanding that this idea was to solve the problem of taking snapshots of a database, that involves having to continually download excel files and dump them into a folder.

      • Preeti commented  ·   ·  Flag as inappropriate

        Please introduce this for Power BI Desktop to connect to SharePoint Lists for ODataFeed and SharePoint List Connectors.

      • Joe Champagne commented  ·   ·  Flag as inappropriate

        This is a great feature. Why can't you download a pbix file from the service once incremental refresh is on?

      • Andrew commented  ·   ·  Flag as inappropriate

        This is a great feature, however, for certain reference data sources that don't need a daily refresh or a refreshed schedule at all, it would be great to be able to designate a lesser frequency for the entire data source. An example might be - a refresh frequency setting - daily, monthly, quarterly?

      • David commented  ·   ·  Flag as inappropriate

        Is there a scheduled date for PRO users?
        When will we have other users?

      • Michael Hughes commented  ·   ·  Flag as inappropriate

        Hi, coulf you make it possible to keep data for a rolling 24 hours and load new data every two hours (as the relevent old data is removed from the service)?

        One of our API's makes a call to one of our internal systems on a ~2 hour basis, extracting the most relevant information for say 10% of our requests (eg the next 200 combinations of city pairs out of a total ranked 2,000) and places the results in a Oracle database (200 city pairs equals output of around 12m records).

        Our Oracle database is emptied everyday at midnight meaning that we dont get the vast majority of the 'city pairs' until later in the day when everyone has gone home (because the API starts at city pair '1' and goes down the list).

        I want to keep the data for say 22 hours, and load the new data as it comes in using Inremental loading, whilst dropping off the previous days 1-200 city pairs when todays pull of 1-200 city pairs is added.

      • Anonymous commented  ·   ·  Flag as inappropriate

        Excited about this feature! One issue from initial testing: using incremental refresh seems to make a datasource ineligible for the UpdateDatasources api call--I get a BadRequest response if the report has an incremental refresh set. Will that be changed? Incremental refresh will drastically improve our ability to work with larger datasets, but only if we can still programmatically change the database connection being used.

      • Anonymous commented  ·   ·  Flag as inappropriate

        When can we expect this for Pro users? TBH I think this is poor form to only allow the feature for Premium users only

      • Withold commented  ·   ·  Flag as inappropriate

        hello, this seems to be a powerful update, but id doesn´t meet the need I originally understood from this post.

        What we need is in Power BI desktop the possibility to do shots of full dataset everytime you refresh data that appends into the existing table, including the refresh date in an additional column.

        Scenario:
        we have a SharePoint database with changing content and want to monitor the evolution of the data periodically. Currently, to obtain this we need to download from SharePoint to Excel manually to a library twice a month, add refresh date from creation date of the excel file and import to PowerBI.
        This is a highly manual process that could be very easily solved with an incremental refresh of the SharePoint list directly to PowerBi desktop, by appending current dataset and giving it a timestamp.

      • matthew Jin commented  ·   ·  Flag as inappropriate

        this feature will allow appending new data to existing as far as I can tell. For example, you can set "store rows in the last" to 10 years, and "refresh rows in the last" to 1 day will continue to add last one day of data to existing dataset which will keep appending data for the next 10 years as long as the storage limit is ok. Did I not understand this feature correctly?

      • Anonymous commented  ·   ·  Flag as inappropriate

        Good news and this is a good start, but incremental refresh on time cycle is not exactly what we had in mind.... we want to append new data to existing data from sources like CSV.

        The use case is adding a period's data to an existing model. In other words instead of truncate (delete) and replace data we simply want to tack it onto the table that's already there. Otherwise accumulation of time series data has to be done outside the tool.

        Of course you don't want to accumulate forever, so being able to selectively delete date ranges of data within specific tables would be a natural companion functional requirement.

        The only other choice at present is to use another database to accumulate time series data. This is fine but adds complexity for users. Access can only scale to 2GB so it's no good for this, which puts you into the realm of "real" databases, which are too complex for a lot of smaller enterprises.

        Anyone doing periodic time series updates would benefit from this easy to implement feature... and that means almost any small to medium size business, which i believe is important to success.

      ← Previous 1 3 4 5 18 19

      Feedback and Knowledge Base

      Ready to get started?

      Try new features of Power BI today by signing up and learn more about our powerful suite of apps.