Coding HAMA

This post will go through steps to run airflow webserver and scheduler on kubernetes using minikube. If you don't have minikube installed yet, please have a look at this post first.  As introduced in this post , Airflow consists of different components, and therefore we need to spin up multiple pods. For this practice, we will create service for Webserver Scheduler Postgres DB (meta database) For workers, we will use LocalExecutor for now. Deploy Airflow on Kubernetes 1. Deploy Postgres database We first need to have postgres up and running. We can create a pod and a service be defining in postgres.yaml. kind: Deployment apiVersion: apps/v1 metadata: name: postgres-airflow spec: replicas: 1 selector: matchLabels: deploy: postgres-airflow template: metadata: labels: name: postgres-airflow deploy: postgres-airflow spec: restartPolicy: Always containers: - name: postgres image: postgres:13.4 port...

What is Minikube?  Minikube is a tool to run Kubernetes(k8s) on the local machine easily.  It creates a single-node k8s cluster  without requiring much time and resources. All we need to start are: 2 CPUs or more 2GB of free memory 20GB of free disk space Internet connection Container or virtual machine manager, such as:  Docker ,  Hyperkit ,  Hyper-V ,  KVM ,  Parallels ,  Podman ,  VirtualBox , or  VMware Fusion/Workstation 1. Installation 1. To install the latest minikube  stable  release on  x86-64   macOS  using  Homebrew : brew install minikube If the  installation fails via brew, you may have to remove the old minikube links and link the newly installed binary: brew unlink minikube brew link minikube 2. Install kubectl minikube kubectl --get po -A 2. Run cluster Start a cluster by running following command on the terminal. minikube start 3. Interact with your cluster You...