Exploring how to use and deploy Azure Data Factory
In this blog, I will introduce how to create an automation script that can deploy the Data Factory’s resources to Azure with a key press “Cool”
. So, I will reuse the resources on Data Factory – 3 basic things post for demonstration.
All steps that script will run are
- Read config.json file to gather configuration information.
- Login user with Azure account.
- Setup datafactory.
- Install Linked services.
- Install Datasets.
- Install Pipelines.
Setup Config File
Since powershell script can load a json file quickly so we create a config.json file like this
{
"location": "EastUS",
"resourceGroup": "InDark",
"subscriptionName": "Developer Program Benefit",
"dataFactoryName": "indarkdatafactory",
"dataFactoryLinkedServices": [
"./ProdSQLLinkedService.json",
"./DevSQLLinkedService.json",
],
"dataFactoryDatasets": [
"./Input_ProdUsersTable.json",
"./Output_DevUsersTable.json",
],
"dataPipelines": [
"./CopyPipeline.json"
],
}
The resourceGroup
and subscriptionName
both are information requires to fill while datafactory is installing.
Scripts to read config file on powershell
$config = get-content config.json | ConvertFrom-Json;
$location = $config.location;
$subscriptionName = $config.subscriptionName;
$resourceGroupName = $config.resourceGroup;
$dataFactoryName = $config.dataFactoryName;
Authentication
In the first step, script will ask user to login
### Authenticate ###
$user = Login-AzureRmAccount;
if ($user -eq $null) { Return; }
Create Data Factory
Scripts to setup Data Factory
### Create data factory ###
$dataFactory = Get-AzureRmDataFactory -ResourceGroupName $resourceGroupName -Name $dataFactoryName -ErrorAction SilentlyContinue;
if ($dataFactory -eq $null) {
"creating data factory $dataFactoryName...";
$dataFactory = New-AzureRmDataFactory -ResourceGroupName $resourceGroupName -Name $dataFactoryName -Location $location -Force;
"data factory $dataFactoryName has been created successfully.";
if ($dataFactory -eq $null)
{
Write-Error "Cannot create data factory $dataFactoryName";
Stop-Transcript;
return;
}
}
else {
"data factory $dataFactoryName found";
}
Install Linked Services
Scripts to install Linked services
$linkedServiceFiles = $config.dataFactoryLinkedServices;
foreach($fileName in $linkedServiceFiles) {
"installing linked services $fileName...";
$linkedService = New-AzureRmDataFactoryLinkedService -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -File $fileName -Force -ErrorAction SilentlyContinue;
if ($linkedService -eq $null)
{
Write-Error "Cannot create linked service $fileName";
Stop-Transcript;
return;
}
}
Install Datasets
Scripts to install Datasets
$datasetFiles = $config.dataFactoryDatasets;
foreach($fileName in $datasetFiles) {
"installing dataset $fileName...";
$dataset = New-AzureRmDataFactoryDataset -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -File $fileName -Force -ErrorAction SilentlyContinue;
if ($dataset -eq $null)
{
Write-Error "Cannot create dataset $fileName";
Stop-Transcript;
return;
}
}
Install Pipelines
Scripts to install Pipelines
$pipelineFiles = $config.dataPipelines;
foreach($fileName in $pipelineFiles) {
"installing pipeline $fileName...";
$pipeline = New-AzureRmDataFactoryPipeline -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -File $fileName -Force -ErrorAction SilentlyContinue;
if ($pipeline -eq $null)
{
Write-Error "Cannot create pipeline $fileName";
Stop-Transcript;
return;
}
}