Matt on 05 Feb 2020 05:15:52
Currently the Data Lineage View will only show links between dataflows if the linked entity is included in the second dataflow. When the linked dataflow hides the linked entity, the link disappears in lineage view and the dataflows appear unrelated. This can also hide sources that were only used in hidden entities. This ends up with a very incomplete data lineage view.
Hiding entities is a key method of control to restrict access to sensitive data since linked entities cannot be changed. It's also important to keep a clean data pipeline and ensure consumers of the final dataflow only use the desired entities.
Adding a toggle to show/hide links in Data Lineage View for hidden entities back to their original dataflow or source would solve this problem.
Showing these at all times and greying out the names of hidden entities would also solve the problem.
- Comments (2)
RE: Show Links using Hidden Entities in Data Lineage View for Dataflows
I have data flows which pull data from linked Data Flows, but this is not displayed within the lineage view. The Upstream dataflows just show that they do not go anywhereAnd the downstream dataflow they are linked to does not show them coming in either. The only way to get that insight is to go into the Data Flow and look at the diagram view. I want to be able to see all of the linkages, accurately, within the Lineage view.As far as I can tell, the lineage doesn't show linked dataflows, regardless whether they are loaded or not. This is a bug.
RE: Show Links using Hidden Entities in Data Lineage View for Dataflows
I'm just adding something to make it more clear:
When linking entities between dataflows, we have the option to "enable load" or "disable load".
When using "enable load" we need to have a premium subscription, otherwise the dataflow can't be refreshed, but disabling the load is enough for most of the cases using Pro subscription.
However, when we build this scenario with many relations between dataflows, the lineage view doesn't follow relations which are not using the "enable load", reducing a lot the possible usages of data lineage by anyone with Pro subscription.