Exploring how to use and deploy Azure Data Factory

In this blog, I will introduce how to create an automation script that can deploy the Data Factory’s resources to Azure with a key press 😎. So, I will reuse the resources on Data Factory – 3 basic things post for demonstration.

All steps that the script will run are:

  1. Read config.json file to gather configuration information.
  2. Login user with Azure account.
  3. Setup datafactory.
  4. Install Linked services.
  5. Install Datasets.
  6. Install Pipelines.

Setup Config File

Since a powershell script can load a json file quickly, we create a config.json file like this

{
    "location": "EastUS",
    "resourceGroup": "InDark",
    "subscriptionName": "Developer Program Benefit", 
    "dataFactoryName": "indarkdatafactory",
    "dataFactoryLinkedServices": [
        "./ProdSQLLinkedService.json",
        "./DevSQLLinkedService.json",
    ],
    "dataFactoryDatasets": [
        "./Input_ProdUsersTable.json",
        "./Output_DevUsersTable.json",
    ],
    "dataPipelines": [
        "./CopyPipeline.json"
    ],
}

The resourceGroup and subscriptionName both are information required to fill in while datafactory is installing.

Scripts to read config file on powershell

$config = get-content config.json | ConvertFrom-Json;
$location = $config.location;
$subscriptionName = $config.subscriptionName;
$resourceGroupName = $config.resourceGroup;
$dataFactoryName = $config.dataFactoryName;

Authentication

In the first step, the script will ask the user to log in

### Authenticate ###
$user = Login-AzureRmAccount;
if ($user -eq $null) { Return; }

Create Data Factory

Scripts to setup Data Factory

### Create data factory ###
$dataFactory = Get-AzureRmDataFactory -ResourceGroupName $resourceGroupName -Name $dataFactoryName -ErrorAction SilentlyContinue;
if ($dataFactory -eq $null) {
    "creating data factory $dataFactoryName...";
    $dataFactory = New-AzureRmDataFactory -ResourceGroupName $resourceGroupName -Name $dataFactoryName -Location $location -Force;
    "data factory $dataFactoryName has been created successfully.";
    if ($dataFactory -eq $null)
    {
        Write-Error "Cannot create data factory $dataFactoryName";
        Stop-Transcript;
        return;
    }
}
else {
    "data factory $dataFactoryName found";
}

Install Linked Services

Scripts to install Linked services

$linkedServiceFiles = $config.dataFactoryLinkedServices;
foreach($fileName in $linkedServiceFiles) {
    "installing linked services $fileName...";
    $linkedService = New-AzureRmDataFactoryLinkedService -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -File $fileName -Force -ErrorAction SilentlyContinue;
    if ($linkedService -eq $null)
    {
        Write-Error "Cannot create linked service $fileName";
        Stop-Transcript;
        return;
    }
}

Install Datasets

Scripts to install Datasets

$datasetFiles = $config.dataFactoryDatasets;
foreach($fileName in $datasetFiles) {
    "installing dataset $fileName...";
    $dataset = New-AzureRmDataFactoryDataset -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -File $fileName -Force -ErrorAction SilentlyContinue;
    if ($dataset -eq $null)
    {
        Write-Error "Cannot create dataset $fileName";
        Stop-Transcript;
        return;
    }
}

Install Pipelines

Scripts to install Pipelines

$pipelineFiles = $config.dataPipelines;
foreach($fileName in $pipelineFiles) {
    "installing pipeline $fileName...";
    $pipeline = New-AzureRmDataFactoryPipeline -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -File $fileName -Force -ErrorAction SilentlyContinue;
    if ($pipeline -eq $null)
    {
        Write-Error "Cannot create pipeline $fileName";
        Stop-Transcript;
        return;
    }
}