Kubernetes

Overview

Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.

We recommend kubernetes as a production environment. Because of different clients’ requirements we are not able to provide the best configuration and each client should adjust configuration to their needs.

In this document we share general Kubernetes manifests skeleton which is runnable in Minikube. If you want to deploy Open Loyalty on AWS you have to adjust these scripts. For example you have to replace services as standalone containers to external services provided by AWS.

Manifests

Credentials for docker registry used in other manifests

apiVersion: v1
data:
    .dockerconfigjson: __REPLACE_IT__
kind: Secret
metadata:
    name: registry
    namespace: openloyalty
type: kubernetes.io/dockerconfigjson

Configuration files for services (e.g. parameters.yml for Symfony)

apiVersion: v1
kind: ConfigMap
metadata:
  name: app
  namespace: openloyalty
data:
  config.js: |-
    const config = {
        "apiUrl": "http://api.example.com/api",
        "dateFormat": "YYYY-MM-DD",
        "dateTimeFormat": "YYYY-MM-DD HH:mm",
        "perPage": 20,
        "debug": false,
        "modules": []
    };
    window.OpenLoyaltyConfig = config;
  .env.prod: |
    APP_ENV=prod
    APP_SECRET=__REPLACE_IT__
    RABBITMQ_URL=amqp://guest:guest@rabbitmq:5672
    DATABASE_URL=pgsql://openloyalty:openloyalty@db:5432/openloyalty
    DATABASE_VERSION='9.6'
    MAILER_URL=smtp://mail:1025
    GELF_SERVER=logstash
    GELF_PORT=12201
    ELK_HOST=elk:9200
    VARNISH_HOST=varnish
    VARNISH_CACHE_BASE_URL=localhost
    VARNISH_TTL=30
    VARNISH_INVALIDATION=true
    VARNISH_DEBUG=false
    REDIS_DSN=redis://redisserver:6379

    api_url=http://openloyalty.localhost/api
    admin_url=http://openloyalty.localhost:8182/
    customer_url=http://openloyalty.localhost:8183/
    merchant_url=http://openloyalty.localhost:8184/
    reset_password_url="#!/password/reset"
    frontend_activate_account_url="#!/customer/panel/customer/registration/activate"
    frontend_invitation_url="#!/register/"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: php
  namespace: openloyalty
data:
  www.conf: |
    [www]
    user = www-data
    group = www-data
    listen = 127.0.0.1:9000
    pm = dynamic
    pm.max_children = 80
    pm.start_servers = 15
    pm.min_spare_servers = 15
    pm.max_spare_servers = 30
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: varnish
  namespace: openloyalty
data:
  settings.vcl: |
    import directors;

    backend default_server {
      .host = "api";
      .port = "80";
      .max_connections = 300;

      .first_byte_timeout     = 300s;
      .connect_timeout        = 5s;
      .between_bytes_timeout  = 2s;
    }

    acl purge {
      # ACL we'll use later to allow purges
      "localhost";
      "127.0.0.1";
      "::1";
      "php";
      "api";
    }

    sub vcl_init {
      new vdir = directors.round_robin();
      vdir.add_backend(default_server);
    }
---

Persistent volumes

kind: PersistentVolume
apiVersion: v1
metadata:
    name: database-pv
    namespace: openloyalty
spec:
    storageClassName: manual
    capacity:
        storage: 20Gi
    accessModes:
        - ReadWriteOnce
    hostPath:
        path: "/data/openloyalty/database"
        type: DirectoryOrCreate
---
kind: PersistentVolume
apiVersion: v1
metadata:
    name: elasticsearch-pv
    namespace: openloyalty
spec:
    storageClassName: manual
    capacity:
        storage: 25Gi
    accessModes:
        - ReadWriteOnce
    hostPath:
        path: "/data/openloyalty/elasticsearch"
        type: DirectoryOrCreate
---
kind: PersistentVolume
apiVersion: v1
metadata:
    name: logs-pv
    namespace: openloyalty
spec:
    storageClassName: manual
    capacity:
        storage: 5Gi
    accessModes:
        - ReadWriteMany
    hostPath:
        path: "/data/openloyalty/logs"
        type: DirectoryOrCreate
---
kind: PersistentVolume
apiVersion: v1
metadata:
    name: jwt-pv
    namespace: openloyalty
spec:
    storageClassName: manual
    capacity:
        storage: 5Gi
    accessModes:
        - ReadWriteMany
    hostPath:
        path: "/data/openloyalty/jwt"
        type: DirectoryOrCreate
---
kind: PersistentVolume
apiVersion: v1
metadata:
    name: uploads-pv
    namespace: openloyalty
spec:
    storageClassName: manual
    capacity:
        storage: 200Mi
    accessModes:
        - ReadWriteMany
    hostPath:
        path: "/data/openloyalty/uploads"
        type: DirectoryOrCreate
---
kind: PersistentVolume
apiVersion: v1
metadata:
    name: rabbitmq-pv
    namespace: openloyalty
spec:
    storageClassName: manual
    capacity:
        storage: 1Gi
    accessModes:
        - ReadWriteMany
    hostPath:
        path: "/data/openloyalty/rabbitmq"
        type: DirectoryOrCreate

Persistent volumes claims

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: database-volume
  namespace: openloyalty
  labels:
    storage-tier: database
spec:
  storageClassName: manual
  resources:
    requests:
      storage: 20Gi
  accessModes:
    - ReadWriteOnce
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: elasticsearch-volume
  namespace: openloyalty
  labels:
    storage-tier: elasticsearch
spec:
  storageClassName: manual
  resources:
    requests:
      storage: 25Gi
  accessModes:
    - ReadWriteOnce
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: logs-volume
  namespace: openloyalty
  labels:
    storage-tier: logs
spec:
  storageClassName: manual
  resources:
    requests:
      storage: 5Gi
  accessModes:
    - ReadWriteMany
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jwt-volume
  namespace: openloyalty
  labels:
    storage-tier: jwt
spec:
  storageClassName: manual
  resources:
    requests:
      storage: 5Gi
  accessModes:
    - ReadWriteMany
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: uploads-volume
  namespace: openloyalty
  labels:
    storage-tier: uploads
spec:
  storageClassName: manual
  resources:
    requests:
      storage: 200Mi
  accessModes:
    - ReadWriteMany
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: rabbitmq-volume
  namespace: openloyalty
  labels:
    storage-tier: rabbitmq
spec:
  storageClassName: manual
  resources:
    requests:
      storage: 1Gi
  accessModes:
    - ReadWriteMany

Deployment and services. This file contains manifests for all services which are required to run OpenLoyalty and additional ones like Kibana, Logstash etc. You can comment it.

##########
# Database
##########
apiVersion: extensions/v1
kind: Deployment
metadata:
  name: db
  namespace: openloyalty
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
        - env:
            - name: POSTGRES_DB
              value: openloyalty
            - name: POSTGRES_PASSWORD
              value: openloyalty
            - name: POSTGRES_USER
              value: openloyalty
          image: postgres:12.6-alpine
          name: db
          ports:
            - containerPort: 5432
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: database
      restartPolicy: Always
      volumes:
        - name: database
          persistentVolumeClaim:
            claimName: database-volume
---
apiVersion: v1
kind: Service
metadata:
  name: db
  namespace: openloyalty
  labels:
    app: db
spec:
  ports:
    - port: 5432
      targetPort: 5432
  selector:
    app: db
---
##########
# Frontend
##########
apiVersion: extensions/v1
kind: Deployment
metadata:
  name: frontend
  namespace: openloyalty
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: frontend
    spec:
      imagePullSecrets:
        - name: registry
      containers:
        - image: registry-1.divante.pl:5000/openloyalty/frontend:4.2.0
          name: openloyalty-frontend
          ports:
            - containerPort: 80
          volumeMounts:
            - mountPath: /var/www/openloyalty/front/config.js
              name: config
              subPath: config.js
      restartPolicy: Always
      volumes:
        - name: config
          configMap:
            name: app
            items:
              - key: config.js
                path: config.js
---
apiVersion: v1
kind: Service
metadata:
  name: frontend-admin
  namespace: openloyalty
  labels:
    app: frontend-admin
spec:
  ports:
    - name: http
      port: 3001
      targetPort: 3001
  selector:
    app: frontend
---
apiVersion: v1
kind: Service
metadata:
  name: frontend-pos
  namespace: openloyalty
  labels:
    app: frontend-pos
spec:
  ports:
    - name: http
      port: 3003
      targetPort: 3003
  selector:
    app: frontend
---
#####
# API
#####
apiVersion: extensions/v1
kind: Deployment
metadata:
  name: api
  namespace: openloyalty
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: api
    spec:
      imagePullSecrets:
        - name: registry
      containers:
        - image: registry-1.divante.pl:5000/openloyalty/api-framework:4.2.0
          name: openloyalty-api
          ports:
            - containerPort: 80
          volumeMounts:
            - mountPath: /var/www/openloyalty/front/config.js
              name: config
              subPath: config.js
      restartPolicy: Always
      volumes:
        - name: config
          configMap:
            name: app
            items:
              - key: config.js
                path: config.js
---
apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: openloyalty
  labels:
    app: api
spec:
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: api
---
#######
# PWACC
#######
apiVersion: extensions/v1
kind: Deployment
metadata:
  name: pwacc
  namespace: openloyalty
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: pwacc
    spec:
      imagePullSecrets:
        - name: registry
      containers:
        - image: registry-1.divante.pl:5000/openloyalty/pwacc:4.2.0
          name: openloyalty-pwacc
          env:
            - name: OL_HOST
              value: api.example.com
            - name: OL_API_PROTOCOL
              value: http
            - name: NODE_ENV
              value: production
          ports:
            - containerPort: 80
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: pwacc
  namespace: openloyalty
  labels:
    app: pwacc
spec:
  ports:
    - port: 3004
      targetPort: 80
  selector:
    app: pwacc
---
########
# PHPFPM
########
apiVersion: extensions/v1
kind: Deployment
metadata:
  labels:
    app: php
  name: php
  namespace: openloyalty
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: php
    spec:
      imagePullSecrets:
        - name: registry
      containers:
        - image: registry-1.divante.pl:5000/openloyalty/fpm-framework:4.2.0
          name: php
          ports:
            - containerPort: 9000
          volumeMounts:
            - mountPath: /var/www/openloyalty/var/log
              name: logs
              subPath: logs
            - mountPath: /var/www/openloyalty/config/jwt
              name: jwt
              subPath: jwt
            - mountPath: /var/www/openloyalty/var/uploads
              name: uploads
              subPath: uploads
            - mountPath: /var/www/openloyalty/.env.prod
              name: parameters
              subPath: .env.prod
            - mountPath: /usr/local/etc/php-fpm.d/www.conf
              name: php-pool-config
              subPath: www.conf
      restartPolicy: Always
      volumes:
        - name: logs
          persistentVolumeClaim:
            claimName: logs-volume
        - name: jwt
          persistentVolumeClaim:
            claimName: jwt-volume
        - name: uploads
          persistentVolumeClaim:
            claimName: uploads-volume
        - name: parameters
          configMap:
            name: app
            items:
              - key: .env.prod
                path: .env.prod
        - name: php-pool-config
          configMap:
            name: php
            items:
            - key: www.conf
              path: www.conf
---
apiVersion: v1
kind: Service
metadata:
  name: php
  namespace: openloyalty
  labels:
    app: php
spec:
  ports:
    - port: 9000
      targetPort: 9000
  selector:
    app: php
---
#########
# Elastic
#########
apiVersion: extensions/v1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: openloyalty
  labels:
    app: elasticsearch
    component: elastcsearch
spec:
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
      component: elasticsearch
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: elasticsearch
        component: elasticsearch
    spec:
      imagePullSecrets:
      - name: registry
      securityContext:
        fsGroup: 1000
      initContainers:
      - name: init-sysctl
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      - name: volume-mount-hack
        image: busybox
        command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
        volumeMounts:
        - name: elasticsearch
          mountPath: /usr/share/elasticsearch/data
      containers:
      - image: registry-1.divante.pl:5000/openloyalty/elasticsearch:7.17.4
        name: elasticsearch
        securityContext:
          privileged: false
          capabilities:
            add:
              - IPC_LOCK
              - SYS_RESOURCE
        ports:
        - containerPort: 9200
        - containerPort: 9300
        env:
        - name: discovery.type
          value: single-node
        - name: ES_JAVA_OPTS
          value: "-Xms1024m -Xmx1024m"
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/data
          name: elasticsearch
      restartPolicy: Always
      volumes:
        - name: elasticsearch
          persistentVolumeClaim:
            claimName: elasticsearch-volume
---
apiVersion: v1
kind: Service
metadata:
  name: elk
  namespace: openloyalty
  labels:
    app: elasticsearch
spec:
  ports:
  - port: 9200
    targetPort: 9200
    name: http
  selector:
    app: elasticsearch
---
##############
# MAIL CATCHER
##############
apiVersion: extensions/v1
kind: Deployment
metadata:
  labels:
    app: mail
  name: mail
  namespace: openloyalty
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mail
    spec:
      containers:
        - image: mailhog/mailhog
          name: mail
          ports:
            - containerPort: 1025
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: mail
  namespace: openloyalty
  labels:
    app: mail
spec:
  ports:
    - port: 1025
      targetPort: 1025
  selector:
    app: mail

---
########
# WORKER
########
apiVersion: extensions/v1
kind: Deployment
metadata:
  labels:
    app: worker
  name: worker
  namespace: openloyalty
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: worker
    spec:
      imagePullSecrets:
      - name: registry
      containers:
      - image: registry-1.divante.pl:5000/openloyalty/worker-framework:4.2.0
        name: worker
        imagePullPolicy: Always
        volumeMounts:
          - mountPath: /var/www/openloyalty/var/log
            name: logs
            subPath: logs
          - mountPath: /var/www/openloyalty/config/jwt
            name: jwt
            subPath: jwt
          - mountPath: /var/www/openloyalty/var/uploads
            name: uploads
            subPath: uploads
          - mountPath: /var/www/openloyalty/.env.prod
            name: parameters
            subPath: .env.prod
      restartPolicy: Always
      volumes:
        - name: logs
          persistentVolumeClaim:
            claimName: logs-volume
        - name: jwt
          persistentVolumeClaim:
            claimName: jwt-volume
        - name: uploads
          persistentVolumeClaim:
            claimName: uploads-volume
        - name: parameters
          configMap:
            name: app
            items:
              - key: .env.prod
                path: .env.prod
---
##########
# RABBITMQ
##########
apiVersion: extensions/v1
kind: Deployment
metadata:
  name: rabbitmq
  namespace: openloyalty
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      imagePullSecrets:
      - name: registry
      containers:
      - image: rabbitmq:3.9-management
        name: rabbitmq
        ports:
          - containerPort: 5672
        volumeMounts:
          - mountPath: /var/lib/rabbitmq
            name: rabbitmq
      restartPolicy: Always
      volumes:
        - name: rabbitmq
          persistentVolumeClaim:
            claimName: rabbitmq-volume
---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
  namespace: openloyalty
  labels:
    app: rabbitmq
spec:
  ports:
  - port: 5672
    targetPort: 5672
  selector:
    app: rabbitmq
---
##########
# LOGSTASH
##########
apiVersion: extensions/v1
kind: Deployment
metadata:
    name: logstash
    namespace: openloyalty
spec:
    replicas: 1
    strategy:
        type: Recreate
    template:
        metadata:
            labels:
                app: logstash
        spec:
            imagePullSecrets:
                - name: registry
            containers:
                - image: registry-1.divante.pl:5000/openloyalty/logstash:7.5.2
                  name: logstash
                  ports:
                      - containerPort: 5000
                  env:
                    - name: LS_JAVA_OPTS
                      value: "-Xmx256m -Xms256m"
                    - name: xpack.monitoring.elasticsearch.hosts
                      value: http://elk:9200
                    - name: xpack.monitoring.elasticsearch.url
                      value: http://elk:9200
                    - name: ELASTICSEARCH_HOST
                      value: http://elk:9200
            restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
    name: logstash
    namespace: openloyalty
    labels:
        app: logstash
spec:
    ports:
        - name: tcp
          port: 5000
          targetPort: 5000
        - name: udp
          port: 9600
          targetPort: 9600
    selector:
        app: logstash
---
#######
# REDIS
#######
apiVersion: extensions/v1
kind: Deployment
metadata:
  name: redisserver
  namespace: openloyalty
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: redisserver
    spec:
      imagePullSecrets:
        - name: registry
      containers:
        - image: registry-1.divante.pl:5000/openloyalty/redis:5.0.7-alpine
          name: redis
          ports:
            - containerPort: 6379
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: redisserver
  namespace: openloyalty
  labels:
    app: redisserver
spec:
  ports:
    - port: 6379
      targetPort: 6379
  selector:
    app: redisserver
---
#######
# KIBANA
#######
apiVersion: extensions/v1
kind: Deployment
metadata:
  name: kibana
  namespace: openloyalty
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
        - image: docker.elastic.co/kibana/kibana:7.5.0
          name: kibana
          ports:
            - containerPort: 5601
          env:
            - name: ELASTICSEARCH_HOSTS
              value: http://elk:9200
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: openloyalty
  labels:
    app: kibana
spec:
  ports:
    - port: 5601
      targetPort: 5601
  selector:
    app: kibana
---
#########
# Varnish
#########
apiVersion: extensions/v1
kind: Deployment
metadata:
  name: varnish
  namespace: openloyalty
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: varnish
    spec:
      imagePullSecrets:
        - name: registry
      containers:
        - image: registry-1.divante.pl:5000/openloyalty/varnish:6
          name: openloyalty-varnish
          ports:
            - containerPort: 80
          volumeMounts:
            - mountPath: /etc/varnish/settings.vcl
              name: varnish
              subPath: settings.vcl
      restartPolicy: Always
      volumes:
        - name: varnish
          configMap:
            name: varnish
            items:
              - key: settings.vcl
                path: settings.vcl
---
apiVersion: v1
kind: Service
metadata:
  name: varnish
  namespace: openloyalty
  labels:
    app: varnish
spec:
  ports:
    - name: http
      port: 8081
      targetPort: 80
  selector:
    app: varnish
---

Ingress configuration

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: openloyalty-ingress
  namespace: openloyalty
  annotations:
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
    - host: admin.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: frontend-admin
              servicePort: 3001
    - host: client.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: pwacc
              servicePort: 3004
    - host: pos.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: frontend-pos
              servicePort: 3003
    - host: api.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: varnish
              servicePort: 8081

How to run OL on Minikube

We assume that you have experience with kubernetes; some details were skipped.

  1. Install Minikube (go to installation guideline)
  2. Start minikube
minikube start
  1. Obtain minikube’s IP by
minikube ip
  1. Add bellow addresses to /etc/hosts (replace IP addresses to obtained above)
192.168.99.100 admin.example.com
192.168.99.100 client.example.com
192.168.99.100 pos.example.com
192.168.99.100 api.example.com
  1. Create a namespace
kubectl create namespace openloyalty
  1. Create new or reuse __secret.yml file
apiVersion: v1
data:
    .dockerconfigjson: __REPLACE_IT__
kind: Secret
metadata:
    name: registry
    namespace: openloyalty
type: kubernetes.io/dockerconfigjson
``kubectl create secret docker-registry registry --docker-server=<server_name:port> --docker-username=<username> --docker-password=<password> --docker-email=example@example.com --dry-run -o yaml``
  1. Apply secret file
kubectl -n openloyalty apply -f __secret.yml

8. Create persistant volumes. In this example we provide storages with manual class name. Make sure that persistent volumes paths defined in manifests are empty. If not, try to log in to minikube (minikube ssh) and remove it.

kubectl -n openloyalty apply -f pv.yml

9. Apply the rest of manifests. Default registry is registry-1.divante.pl:5000/openloyalty/. You can change it in deployment.yml. Make sure that you have correct credentials.

kubectl -n openloyalty apply -f storage.yml
kubectl -n openloyalty apply -f config.yml
kubectl -n openloyalty apply -f deployment.yml
kubectl -n openloyalty apply -f ingress.yml
  1. Make sure that all containers are running
  1. Log in to php-* container

and execute bellow commands, which set proper ownership of directories:

root@php-6c4b5b8cb9-vq252:/var/www/openloyalty# chown -R www-data:www-data .

Before initializing application sample data make sure that current user is www-data:

root@php-6c4b5b8cb9-vq252:/var/www/openloyalty# su www-data

If you wish to set up full sample data, good for demo purposes:

www-data@php-6c4b5b8cb9-vq252:~/openloyalty$ phing setup

And if you wish to initialize the application with just basic data, which is good for production purposes:

www-data@php-6c4b5b8cb9-vq252:~/openloyalty$ phing basic-setup
  1. Open the panels in your browser

http://admin.example.com

http://client.example.com

http://pos.example.com

http://api.example.com