AIRFLOW

1. Type virtual machines in the search.
2. Under Services, select Virtual machines.
3. In the Virtual machines page, select Add. The Create a virtual machine page opens.
4. In the Basics tab, under Project details, make sure the correct subscription is selected and then choose to Create new resource group. Type myResourceGroup for the name.*.
5. Under Instance details, type myVM for the Virtual machine name, choose East US for your Region, and choose Ubuntu 18.04 LTS for your Image. Leave the other defaults.
6. Under Administrator account, select SSH public key, type your user name, then paste in your public key. Remove any leading or trailing white space in your public key.
7. Under Inbound port rules > Public inbound ports, choose Allow selected ports and then select SSH (22) and HTTP (80) from the drop-down.
8. Leave the remaining defaults and then select the Review + create button at the bottom of the page.
9. On the Create a virtual machine page, you can see the details about the VM you are about to create. When you are ready, select Create.

It will take a few minutes for your VM to be deployed. When the deployment is finished, move on to the next section.

Connect to virtual machine

Create an SSH connection with the VM.

1. Select the Connect button on the overview page for your VM.
2. In the Connect to virtual machine page, keep the default options to connect by IP address over port 22. In Login using VM local account a connection command is shown. Select the button to copy the command. The following example shows what the SSH connection command looks like:

bashCopy

ssh azureuser@10.111.12.123

1. Using the same bash shell you used to create your SSH key pair (you can reopen the Cloud Shell by selecting >_ again or going to https://shell.azure.com/bash), paste the SSH connection command into the shell to create an SSH session.

Usage/Deployment Instructions

Step 1:  Access the Airflow in Azure Marketplace and click on Get it now button.

Click on Continue and then click on Create.

Step 2: Now to create a virtual machine, enter or select appropriate values for zone, machine type, resource group and so on as per your choice.

Click on Review + create.

Step 3:  The below window confirms that VM was deployed.

Step 4:  Open port 8080 in security group by going to resource group –

1. Select your network security group.
2. Select Inbound security rules from the left menu, then select Add.
3. You can limit the Source as needed or leave the default of Any.
4. Limit the Source port range as 8080. 
5. You can limit the Destination as needed or leave the default of Any.
6. Choose a common Service from the drop-down menu, TCP. You can also select Custom if you want to provide a specific port to use like Port_4000.
7. Optionally, change the Priority or Name. The priority affects the order in which rules are applied: the lower the numerical value, the earlier the rule is applied.
8. Select Add to create the rule.

Step 5: Open putty and connect with your machine. Add IP address of the running virtual machine.

Step 6: Login with user name and password that you provided during machine creation.

Creating a New Airflow User

• Developers must create a new user on the first startup of airflow.
• It can be done with the help of the “users create” command.
• To create a new user with a username as admin with Admin role, we can run the following code:

$ airflow users create –username admin –password your_password –firstname your_first_name –lastname your_last_name –role Admin –email your_email@domain.com

• Run the following command to check if the user was created successfully:

$ airflow users list

Running of the Airflow Scheduler and Webserver

• Now we will start the airflow scheduler using the airflow scheduler command after activating the virtual environment:

$ airflow scheduler

• Open a new terminal, activate the virtual environment, go to the airflow directory, and start the web server.

$ source airflow_env/bin/activate

$ cd airflow

$ airflow webserver

• Once the scheduler and webserver get initialized, open any browser and go to http://IP-Address:8080/.
• Port 8080 should be the default port for Airflow, and you see the following page:

• If it doesn’t work or shows occupied by some other program, go to airflow.cfg file and change the port number.
• After logging in using our airflow username and password, we should see the following web server UI.

• These are some prebuilt dags you will observe when you log in for the first time.
• If you see this page, congratulations! you have successfully installed Apache Airflow in your system.
• You can explore the UI, experiment with the dags, and see their workings.