Microsoft Fabric is an end-to-end data analytics solution in the cloud, and one of its workloads is called Data Factory. In this article, you will learn what Data Factory is, how it works with the rest of Microsoft Fabric, and what are elements and functions of Data Factory.

Video

Microsoft Fabric

To understand Data Factory, it is best to understand Microsoft Fabric first. Microsoft Fabric is an end-to-end Data Analytics software-as-a-service offering from Microsoft. Microsoft Fabric combined some products and services to cover an end-to-end and easy-to-use platform for data analytics. Here are the components (also called workloads) of Microsoft Fabric.

To learn more about Microsoft Fabric and enable it in your organization, I recommend reading the articles below;

• What is Microsoft Fabric
• How to enable Fabric in your organization

Data Factory Origin

Microsoft Fabric has a workload for Data Integration. Any end-to-end data analytics system should have a data integration component. Microsoft has been a strong data integration tool and service leader for decades. This started with SQL Server tools such as DTS (Data Transformation Service) and SSIS (SQL Server Integration Services) and then stepped into cloud-based technologies such as ADF (Azure Data Factory). Microsoft also used a data transformation engine that first targeted citizen data analysts called Power Query.

Data Factory is the data integration component of Microsoft Fabric which brings the power of Azure Data Factory and Power Query Dataflows into one place. For many years, we had these two technologies doing data transformations separately. But now, these two are combined under Fabric, called Data Factory.

Power Query

Power Query Dataflows was first announced a few years ago as an additional component to Power BI for data transformation as a cloud technology that is simple to use for data analysts. But soon, it became more than just for Power BI; it became Power Platform Dataflows. These days, Power Query Dataflows are used for data transformations in Power BI projects and data migration in Power Apps projects.

Although Power Query Dataflows is also on the dataflow side, it needed some enhancements on scalability and the control of execution with some control flow elements (such as loop structures, conditional execution, etc.).

Azure Data Factory

Azure Data Factory came into the market many years ago as the next generation of SSIS for in-the-cloud ETL. However, the data transformation engine of Azure Data Factory was not built on a strong basis, so most of the time, ADF was used for data ingestion, and then with the help of SQL stored procedures, etc., for doing the transformation afterward. ADF was not a tool for citizen data analysts. It was instead for data engineers and developers. ADF used data pipelines to execute a group of activities as a flow, and among those activities, there were tasks such as copy data, running a stored procedure, etc.

For the past few years, we have always had this split; If you wanted a simple-to-use data transformation engine but not much data, use Power Query Dataflows. If you want scalable data ingestion, then use Azure Data Factory.

Best of Both Worlds

Now in Microsoft Fabric, We combine the best things from Power Query Dataflows and Azure Data Factory Data Pipelines into one stream: Data Factory. Data Factory ensures that you still have a simple-to-use and powerful transformation engine of Power Query for data transformation, but on the other hand, you also have the scalability of Data Pipelines and can build a control flow for execution of the ETL using the Data Pipelines. In other words, Data Factory is a state-of-the-art ETL software-as-a-service offering for Microsoft Fabric.

Elements of Data Factory

Combining these two services brings great features that make the Data Factory an ultimate ETL service. Here are some of those below;

Data Connectors

For an ETL (Extract, Transform, Load) system, one of the most important aspects is what sources the data can be fetched from. Data Factory offers hundreds of data connectors, enabling you to get data from sources such as databases, files, folders, software-as-a-service systems, etc.

It is also possible to create your connector if you are keen.

Dataflows

Dataflows are the heart of Data Factory. This is where you get the data from the sources, define the data transformation and prepare it in any shape needed, and finally load it into destinations. Dataflows use the Power Query data transformation engine and the user interface for creating it using the simple-to-use Power Query Editor online.

Power Query Editor online is not only powerful in the graphical interface, it also enables the developer to write code in M language, which is the data transformation language for Power Query.

To learn more about Dataflows, I suggest reading my article below.

Dataflows support a few destinations at the time of writing this article which are;

• Azure Data Explorer (Kusto)
• Azure SQL Database
• Data Warehouse
• Lakehouse

Data Pipelines

Although Dataflows are the main ETL component of the Data Factory, they can be enhanced when wrapped by a control flow execution component. This control flow execution component is called Data Pipeline. A Data Pipeline is a group of activities (or tasks) defined by a particular flow of execution. The activities in a Pipeline can involve copying data, running a Dataflow, executing a stored procedure, looping until a certain condition is met, or executing a particular set of activities if a condition is met, etc.

Data Pipelines can then be scheduled, and there is a monitoring tool to check the execution stage of the pipeline in addition to the activity-state-outputs where you can define what happens if a certain task fails or succeeds.

As mentioned, one of the most important activities that can be done in a Pipeline is the execution of a Dataflow. This is where Dataflows and Data Pipelines work together in their best way.

To learn more about Data Pipelines, read my article below;

Summary

Data Factory is an ETL-in-cloud solution that is the data integration workload of Microsoft Fabric. Data Factory is not a new product or service; it comes from many years of Microsoft data transformation tools and services. It is built on top of Power Query and Azure Data Factory. Data Factory uses two main components to deliver the best ETL scenarios possible; Dataflows and Data Pipelines. Dataflows are for the main get data, transform, and load process, and the Data Pipeline can control the rest of the execution with control flow activities.

I highly recommend reading the articles below to study more about Data Factory;

• What is Microsoft Fabric
• How to enable Fabric in your organization
• What is Lakehouse, and a sample Dataflow Gen2 created to load data into it.
• Getting started with Data Pipelines in Microsoft Fabric Data Factory
• Getting started with Dataflows in Microsoft Fabric Data Factory

The post What is Data Factory in Microsoft Fabric appeared first on RADACAD.

Follow Source

The post What is Data Factory in Microsoft Fabric appeared first on Microsoft Dynamics 365 Blog.

Do you have a Power BI dataset that gets data from a dataflow? have you ever thought; “can I get the dataset refreshed only after the refresh of dataflow completed and was successful?” The answer to this question is yes, you can. One of the recent updates from the data integration team of Power BI made this available for you. Let’s see in this blog and video, how this is possible.

The scenario

If you are using both dataflows and datasets in your Power BI architecture, then your datasets are very likely getting part of their data from Power BI dataflows. It would be great if you can get the Power BI dataset refreshed right after a successful refresh of the dataflow. In fact, you can do a scenario like below.

Power Automate connector for dataflow

Power Automate recently announced availability of a connector that allows you to trigger a flow when a dataflow refresh completes.

Choosing the dataflow

You can then choose the workspace (or environment if you are using Power Platform dataflows), and the dataflow.

Condition on success or fail

The dataflow refresh can succeed or fail. You can choose the proper action in each case. For doing this, you can choose the result of refresh to be Success.

Refresh Power BI dataset

In the event of successful refresh of the dataflow, you can then run the refresh of the Power BI dataset.

Refreshing Power BI dataset through Power Automate is an ability that we had for sometime in the service.

Capture the failure

You can also capture the failure details and send a notification (or you can add a record in a database table for further log reporting) in the case of failure.

Overall flow

The overall flow seems a really simple but effective control of the refresh as you can see below.

My thoughts

Making sure that the refresh of the dataset happens after the refresh of the dataflow, was one of the challenges of Power BI developers if they use dataflow. Now, using this simple functionality, you can get the refresh process streamlined all the way from the dataflow.

Dataflow refresh can be done as a task in the Power Automate as well. Which might be useful for some scenarios, such as running the refresh of the dataflow after a certain event.

This is not only good for refreshing the Power BI dataset after the dataflow, it is also good for refreshing a dataflow after the other one. Especially in best practice scenarios of dataflow, I always recommend having layers of the dataflow for staging, data transformation, etc, as I explained in the below article.

Although, dataflow is not a replacement for the data warehouses. However, having features like this helps the usability and the adoption of this great transformation service.

Do you think of any scenarios that you use this for? let me know in the comments below, I’d love to hear about your scenarios.

Video

The post Streamline Power BI Refresh: Refresh dataset after a successful refresh of dataflow appeared first on RADACAD.

Follow Source

The post Streamline Power BI Refresh: Refresh dataset after a successful refresh of dataflow appeared first on Microsoft Dynamics 365 Blog.

In the last Post, I will explain how to analyze a JSON file that has been generated in the Sentiment Analysis process .

some explanation, this is a JSON file that contains the sentiment analysis for the comments one traveler put on the hotel website as below

The suite was awesome. We did not have much interaction with the staff. We did sleep on the queen size sofa bed in the living room instead of the queen size bed in the actual bedroom due to the temperature. It was very hot and humid. The air conditioner in the living room does not cool off the bedroom. This suite is very close to shopping and dining. After each day of adventures, we would walk to dinner at different joints. I will stay here again when I return.

the JSON file is like below

{“predictionOutput”:{“result”:{“sentiment”:”mixed”,”documentScores”:{“positive”:0.64,”neutral”:0.05,”negative”:0.31},”sentences”:[{“sentiment”:”positive”,”sentenceScores”:{“positive”:1.0,”neutral”:0.0,”negative”:0.0},”offset”:0,”length”:22},{“sentiment”:”neutral”,”sentenceScores”:{“positive”:0.01,”neutral”:0.83,”negative”:0.16},”offset”:23,”length”:48},{“sentiment”:”neutral”,”sentenceScores”:{“positive”:0.0,”neutral”:1.0,”negative”:0.0},”offset”:72,”length”:134},{“sentiment”:”neutral”,”sentenceScores”:{“positive”:0.03,”neutral”:0.93,”negative”:0.04},”offset”:207,”length”:26},{“sentiment”:”negative”,”sentenceScores”:{“positive”:0.01,”neutral”:0.06,”negative”:0.93},”offset”:234,”length”:69},{“sentiment”:”positive”,”sentenceScores”:{“positive”:0.92,”neutral”:0.08,”negative”:0.0},”offset”:304,”length”:48},{“sentiment”:”neutral”,”sentenceScores”:{“positive”:0.02,”neutral”:0.96,”negative”:0.02},”offset”:353,”length”:73},{“sentiment”:”neutral”,”sentenceScores”:{“positive”:0.03,”neutral”:0.92,”negative”:0.05},”offset”:427,”length”:37}]}},”operationStatus”:”Success”,”error”:null}

The related JSON file allocate an overall score to the whole comment, then for each sentence, you can get a separate sentiment score, the result in the JSON format is not able to analyze, so I decided to use Power Query to solve this problem

First Open Power BI desktop and navigate to Power Query, import the JSON file, then load the data, click on the record to expand it and to see the record and list. Right-click on both of them and add them as a separate query.

Then click on the To Table, and expand the records to see the inside, finally, you should see the 7 rows of data with label positive, negative and neutral, and the two last columns for the offset and length of each sentence.

Now we need to add the original text to merge them together, import the text file, then use the dot separator and Transpose to make them as a table. Then use the Add Column and add index column, to both the Sentence query and the hotel comment. Then in the Home Tab click on the Merge columns and merge them based on the Index

The post Analyse the JSON File with Power Query appeared first on RADACAD.

Follow Source

The post Analyse the JSON File with Power Query appeared first on Microsoft Dynamics 365 Blog.

If you don’t have a Power BI Premium capacity license, then you are limited to refresh your dataflows up to eight times a day, with the frequency of 30 minutes. The good news for you is that you have a way to do unlimited refreshes and with whatever frequency you like using Power Platform Dataflows. Read the rest of this blog article to learn how.

What is Dataflow?

In the world of Power BI, You can separate the data transformation layer using Dataflows. Dataflows can run one or multiple Power Query scripts on the cloud and store the result into Azure Data Lake. The result of the dataflow (which is stored in Azure Data Lake) can then be used in a Power BI dataset using Get Data from Power BI dataflows. If you want to learn more about Dataflows, I highly recommend reading my dataflows series that starts with this article.

Creating Power Platform Dataflows

Dataflow is not only a concept for Power BI, but It is also available as Power Platform dataflow. You can check out one of the announcements of features added in Power Platform Dataflows by Miguel Llopis (one of the product managers of the Data Integration team at Microsoft), here. You can create Power Platform Dataflows, very similar to the way that you create Power BI dataflows. The only difference at the beginning is where you create it. Power BI dataflows are created in Power BI service, Power Platform Dataflows can be created in Power Apps (or Power Platform) portal.

Login to Power Platform Portal here.

Then go to Dataflows under the Data section,

Start a New dataflow

just provide a name for the dataflow, and then create

At this step, you can either go ahead and get data from anywhere you want or use Blank Query and copy a Power Query script from Power BI Desktop into here.

after preparing your queries, you can go to the step of loading it.

Load to Entity

The load to entity step in Power Platform Dataflows is a bit different from Power BI Dataflows. In Power BI Dataflows, the result will be loaded into CSV files into Azure Data Lake, so no more configuration is needed. However, Power Platform Dataflows stores the data into CDS (Common Data Services), and you have to select the entity you want to load this data into it.

You can either choose one of the existing entities or create a new entity, then map fields. Because these are CDS entities, there are some rules for having key fields, name fields, etc. If you don’t know what the CDS is, and where it stores the data, read my article about CDS here.

Refresh Settings

After setting up the load to entity and mapping fields correctly, you will get into the Refresh settings. This is where you can refresh your dataflow even with the frequency of a minute. You can refresh it as many times as you want. You are not limited to eight times or even 48 times a day.

Refresh as much as you want

Here you can see the frequency of the test dataflow that I set up, which ran every minute.

What about Licensing?

I know what you are going to ask now! What about licensing? What type of license do you need for this? Well, for Power Platform dataflows, you don’t need a Power BI license at all, not Premium, not even Pro. You do need, however, to have a Power Apps license. at the time of writing this blog post, there are two options, $10 a month, and $40 a month. Both of these can be cheaper than Power BI Premium if you are a single user (or even a few users). The main difference between the two plans of Power Apps, as I see is the database and file size for CDS;

How to use the results in Power BI?

You can use the result of dataflows in Power BI (similar to the way that you can use the result of Power BI dataflows in Power BI). You need to Get Data from Power Platform Dataflows.

However, at the time of writing this blog post, this feature is still beta and under development, and might not show the Power Platform dataflows.

Another option is also to get data from the Common Data Service. (Because Power Platform Dataflows stores the data into CDS)

Note that Power BI Dataset Refresh Limites Still Applies

Having a Power Query script that can refresh even every minute is great. However, you have to note that the dataflow refresh is different from the dataset refresh. If you use the result of the dataflow in a Power BI dataset, Still you need to get your dataset refreshed, and that means you are limited to eight times a day or 48 times a day or API refreshes depends on your Power BI license limits.

However, having the ability to refresh even the dataflow more frequent, still can be useful for some scenarios. I bet if you came to this post by searching for such functionality, you have some ideas about that already .

Video

The post Refresh Power BI Queries Through Power Platform Dataflows: Unlimited Times with Any Frequency appeared first on RADACAD.

Follow Source

The post Refresh Power BI Queries Through Power Platform Dataflows: Unlimited Times with Any Frequency appeared first on Microsoft Dynamics 365 Blog.