Airflow Testing & Local/Dev Setup using Kubernetes

Airflow Testing

Airflow unit tests, DAG integrity tests and Pipeline definition tests

Install Python 2.7.13 using pyenv:

brew install pyenv
pyenv install 2.7.13
pyenv global 2.7.13
pyenv global (Verify the python version)

Install Pip and PyBuilder

easy_install pip
pip install pybuilder

Set AIRFLOW_HOME

export AIRFLOW_HOME={project dir}/src/main/python

Go to project root directory and run these commands:

pyb install_dependencies
airflow initdb

Please update $AIRFLOW_HOME/airflow.cfg:

load_examples = False

Run tests:

pyb run_unit_tests

Airflow Local/Dev Setup using Kubernetes

Follow this steps to install and run airflow on dev machine. This will setup following components:

• Postgres (To store the metadata of airflow)
• Redis (Broker for celery executors)
• Airflow Scheduler
• Celery Workers
• Airflow Web Server
• Flower

Prerequisite:

git clone https://github.com/chandulal/airflow-testing.git
brew cask install virtualbox (run if you don't have virtual box installed)

1. Install minikube

    brew cask install minikube
    brew install kubernetes-cli
    minikube start --cpus 4 --memory 8192
    

2. Mount DAGs,Plugins, etc. in minikube

    
    minikube mount {project dir}/src/main/python/:/data
    

3. Open new terminal. Go to project root dir and run:

    
    kubectl apply -f airflow.kube.yaml
    

   wait for 3-4 min to start all airflow components.

4. Get Minikube ip:

    minikube ip
    

5. Now you can access:

   Airflow UI: {minikube-ip}:31317

   Flower: {minikube-ip}:32081

How it works?