Kubernetes¶
Overview¶
Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.
We recommend kubernetes as a production environment. Because of different clients’ requirements we are not able to provide the best configuration and each client should adjust configuration to their needs.
In this document we share general Kubernetes manifests skeleton which is runnable in Minikube. If you want to deploy Open Loyalty on AWS you have to adjust these scripts. For example you have to replace services as standalone containers to external services provided by AWS.
Manifests¶
Credentials for docker registry used in other manifests
apiVersion: v1
data:
.dockerconfigjson: __REPLACE_IT__
kind: Secret
metadata:
name: registry
namespace: openloyalty
type: kubernetes.io/dockerconfigjson
Configuration files for services (e.g. parameters.yml for Symfony)
apiVersion: v1
kind: ConfigMap
metadata:
name: app
namespace: openloyalty
data:
config.js: |-
const config = {
"apiUrl": "http://api.example.com/api",
"dateFormat": "YYYY-MM-DD",
"dateTimeFormat": "YYYY-MM-DD HH:mm",
"perPage": 20,
"debug": false,
"modules": []
};
window.OpenLoyaltyConfig = config;
.env.prod: |
APP_ENV=prod
APP_SECRET=__REPLACE_IT__
RABBITMQ_URL=amqp://guest:guest@rabbitmq:5672
DATABASE_URL=pgsql://openloyalty:openloyalty@db:5432/openloyalty
DATABASE_VERSION='9.6'
MAILER_URL=smtp://mail:1025
GELF_SERVER=logstash
GELF_PORT=12201
ELK_HOST=elk:9200
VARNISH_HOST=varnish
VARNISH_CACHE_BASE_URL=localhost
VARNISH_TTL=30
VARNISH_INVALIDATION=true
VARNISH_DEBUG=false
REDIS_DSN=redis://redisserver:6379
api_url=http://openloyalty.localhost/api
admin_url=http://openloyalty.localhost:8182/
customer_url=http://openloyalty.localhost:8183/
merchant_url=http://openloyalty.localhost:8184/
reset_password_url="#!/password/reset"
frontend_activate_account_url="#!/customer/panel/customer/registration/activate"
frontend_invitation_url="#!/register/"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: php
namespace: openloyalty
data:
www.conf: |
[www]
user = www-data
group = www-data
listen = 127.0.0.1:9000
pm = dynamic
pm.max_children = 80
pm.start_servers = 15
pm.min_spare_servers = 15
pm.max_spare_servers = 30
---
apiVersion: v1
kind: ConfigMap
metadata:
name: varnish
namespace: openloyalty
data:
settings.vcl: |
import directors;
backend default_server {
.host = "api";
.port = "80";
.max_connections = 300;
.first_byte_timeout = 300s;
.connect_timeout = 5s;
.between_bytes_timeout = 2s;
}
acl purge {
# ACL we'll use later to allow purges
"localhost";
"127.0.0.1";
"::1";
"php";
"api";
}
sub vcl_init {
new vdir = directors.round_robin();
vdir.add_backend(default_server);
}
---
Persistent volumes
kind: PersistentVolume
apiVersion: v1
metadata:
name: database-pv
namespace: openloyalty
spec:
storageClassName: manual
capacity:
storage: 20Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/data/openloyalty/database"
type: DirectoryOrCreate
---
kind: PersistentVolume
apiVersion: v1
metadata:
name: elasticsearch-pv
namespace: openloyalty
spec:
storageClassName: manual
capacity:
storage: 25Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/data/openloyalty/elasticsearch"
type: DirectoryOrCreate
---
kind: PersistentVolume
apiVersion: v1
metadata:
name: logs-pv
namespace: openloyalty
spec:
storageClassName: manual
capacity:
storage: 5Gi
accessModes:
- ReadWriteMany
hostPath:
path: "/data/openloyalty/logs"
type: DirectoryOrCreate
---
kind: PersistentVolume
apiVersion: v1
metadata:
name: jwt-pv
namespace: openloyalty
spec:
storageClassName: manual
capacity:
storage: 5Gi
accessModes:
- ReadWriteMany
hostPath:
path: "/data/openloyalty/jwt"
type: DirectoryOrCreate
---
kind: PersistentVolume
apiVersion: v1
metadata:
name: uploads-pv
namespace: openloyalty
spec:
storageClassName: manual
capacity:
storage: 200Mi
accessModes:
- ReadWriteMany
hostPath:
path: "/data/openloyalty/uploads"
type: DirectoryOrCreate
---
kind: PersistentVolume
apiVersion: v1
metadata:
name: rabbitmq-pv
namespace: openloyalty
spec:
storageClassName: manual
capacity:
storage: 1Gi
accessModes:
- ReadWriteMany
hostPath:
path: "/data/openloyalty/rabbitmq"
type: DirectoryOrCreate
Persistent volumes claims
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: database-volume
namespace: openloyalty
labels:
storage-tier: database
spec:
storageClassName: manual
resources:
requests:
storage: 20Gi
accessModes:
- ReadWriteOnce
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: elasticsearch-volume
namespace: openloyalty
labels:
storage-tier: elasticsearch
spec:
storageClassName: manual
resources:
requests:
storage: 25Gi
accessModes:
- ReadWriteOnce
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: logs-volume
namespace: openloyalty
labels:
storage-tier: logs
spec:
storageClassName: manual
resources:
requests:
storage: 5Gi
accessModes:
- ReadWriteMany
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: jwt-volume
namespace: openloyalty
labels:
storage-tier: jwt
spec:
storageClassName: manual
resources:
requests:
storage: 5Gi
accessModes:
- ReadWriteMany
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: uploads-volume
namespace: openloyalty
labels:
storage-tier: uploads
spec:
storageClassName: manual
resources:
requests:
storage: 200Mi
accessModes:
- ReadWriteMany
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: rabbitmq-volume
namespace: openloyalty
labels:
storage-tier: rabbitmq
spec:
storageClassName: manual
resources:
requests:
storage: 1Gi
accessModes:
- ReadWriteMany
Deployment and services. This file contains manifests for all services which are required to run OpenLoyalty and additional ones like Kibana, Logstash etc. You can comment it.
##########
# Database
##########
apiVersion: extensions/v1
kind: Deployment
metadata:
name: db
namespace: openloyalty
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: db
spec:
containers:
- env:
- name: POSTGRES_DB
value: openloyalty
- name: POSTGRES_PASSWORD
value: openloyalty
- name: POSTGRES_USER
value: openloyalty
image: postgres:12.6-alpine
name: db
ports:
- containerPort: 5432
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: database
restartPolicy: Always
volumes:
- name: database
persistentVolumeClaim:
claimName: database-volume
---
apiVersion: v1
kind: Service
metadata:
name: db
namespace: openloyalty
labels:
app: db
spec:
ports:
- port: 5432
targetPort: 5432
selector:
app: db
---
##########
# Frontend
##########
apiVersion: extensions/v1
kind: Deployment
metadata:
name: frontend
namespace: openloyalty
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: frontend
spec:
imagePullSecrets:
- name: registry
containers:
- image: registry-1.divante.pl:5000/openloyalty/frontend:4.2.0
name: openloyalty-frontend
ports:
- containerPort: 80
volumeMounts:
- mountPath: /var/www/openloyalty/front/config.js
name: config
subPath: config.js
restartPolicy: Always
volumes:
- name: config
configMap:
name: app
items:
- key: config.js
path: config.js
---
apiVersion: v1
kind: Service
metadata:
name: frontend-admin
namespace: openloyalty
labels:
app: frontend-admin
spec:
ports:
- name: http
port: 3001
targetPort: 3001
selector:
app: frontend
---
apiVersion: v1
kind: Service
metadata:
name: frontend-pos
namespace: openloyalty
labels:
app: frontend-pos
spec:
ports:
- name: http
port: 3003
targetPort: 3003
selector:
app: frontend
---
#####
# API
#####
apiVersion: extensions/v1
kind: Deployment
metadata:
name: api
namespace: openloyalty
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: api
spec:
imagePullSecrets:
- name: registry
containers:
- image: registry-1.divante.pl:5000/openloyalty/api-framework:4.2.0
name: openloyalty-api
ports:
- containerPort: 80
volumeMounts:
- mountPath: /var/www/openloyalty/front/config.js
name: config
subPath: config.js
restartPolicy: Always
volumes:
- name: config
configMap:
name: app
items:
- key: config.js
path: config.js
---
apiVersion: v1
kind: Service
metadata:
name: api
namespace: openloyalty
labels:
app: api
spec:
ports:
- port: 80
targetPort: 80
selector:
app: api
---
#######
# PWACC
#######
apiVersion: extensions/v1
kind: Deployment
metadata:
name: pwacc
namespace: openloyalty
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: pwacc
spec:
imagePullSecrets:
- name: registry
containers:
- image: registry-1.divante.pl:5000/openloyalty/pwacc:4.2.0
name: openloyalty-pwacc
env:
- name: OL_HOST
value: api.example.com
- name: OL_API_PROTOCOL
value: http
- name: NODE_ENV
value: production
ports:
- containerPort: 80
restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
name: pwacc
namespace: openloyalty
labels:
app: pwacc
spec:
ports:
- port: 3004
targetPort: 80
selector:
app: pwacc
---
########
# PHPFPM
########
apiVersion: extensions/v1
kind: Deployment
metadata:
labels:
app: php
name: php
namespace: openloyalty
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: php
spec:
imagePullSecrets:
- name: registry
containers:
- image: registry-1.divante.pl:5000/openloyalty/fpm-framework:4.2.0
name: php
ports:
- containerPort: 9000
volumeMounts:
- mountPath: /var/www/openloyalty/var/log
name: logs
subPath: logs
- mountPath: /var/www/openloyalty/config/jwt
name: jwt
subPath: jwt
- mountPath: /var/www/openloyalty/var/uploads
name: uploads
subPath: uploads
- mountPath: /var/www/openloyalty/.env.prod
name: parameters
subPath: .env.prod
- mountPath: /usr/local/etc/php-fpm.d/www.conf
name: php-pool-config
subPath: www.conf
restartPolicy: Always
volumes:
- name: logs
persistentVolumeClaim:
claimName: logs-volume
- name: jwt
persistentVolumeClaim:
claimName: jwt-volume
- name: uploads
persistentVolumeClaim:
claimName: uploads-volume
- name: parameters
configMap:
name: app
items:
- key: .env.prod
path: .env.prod
- name: php-pool-config
configMap:
name: php
items:
- key: www.conf
path: www.conf
---
apiVersion: v1
kind: Service
metadata:
name: php
namespace: openloyalty
labels:
app: php
spec:
ports:
- port: 9000
targetPort: 9000
selector:
app: php
---
#########
# Elastic
#########
apiVersion: extensions/v1
kind: Deployment
metadata:
name: elasticsearch
namespace: openloyalty
labels:
app: elasticsearch
component: elastcsearch
spec:
replicas: 1
selector:
matchLabels:
app: elasticsearch
component: elasticsearch
strategy:
type: Recreate
template:
metadata:
labels:
app: elasticsearch
component: elasticsearch
spec:
imagePullSecrets:
- name: registry
securityContext:
fsGroup: 1000
initContainers:
- name: init-sysctl
image: busybox
imagePullPolicy: IfNotPresent
command: ["sysctl", "-w", "vm.max_map_count=262144"]
securityContext:
privileged: true
- name: volume-mount-hack
image: busybox
command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
volumeMounts:
- name: elasticsearch
mountPath: /usr/share/elasticsearch/data
containers:
- image: registry-1.divante.pl:5000/openloyalty/elasticsearch:7.17.4
name: elasticsearch
securityContext:
privileged: false
capabilities:
add:
- IPC_LOCK
- SYS_RESOURCE
ports:
- containerPort: 9200
- containerPort: 9300
env:
- name: discovery.type
value: single-node
- name: ES_JAVA_OPTS
value: "-Xms1024m -Xmx1024m"
volumeMounts:
- mountPath: /usr/share/elasticsearch/data
name: elasticsearch
restartPolicy: Always
volumes:
- name: elasticsearch
persistentVolumeClaim:
claimName: elasticsearch-volume
---
apiVersion: v1
kind: Service
metadata:
name: elk
namespace: openloyalty
labels:
app: elasticsearch
spec:
ports:
- port: 9200
targetPort: 9200
name: http
selector:
app: elasticsearch
---
##############
# MAIL CATCHER
##############
apiVersion: extensions/v1
kind: Deployment
metadata:
labels:
app: mail
name: mail
namespace: openloyalty
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: mail
spec:
containers:
- image: mailhog/mailhog
name: mail
ports:
- containerPort: 1025
restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
name: mail
namespace: openloyalty
labels:
app: mail
spec:
ports:
- port: 1025
targetPort: 1025
selector:
app: mail
---
########
# WORKER
########
apiVersion: extensions/v1
kind: Deployment
metadata:
labels:
app: worker
name: worker
namespace: openloyalty
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: worker
spec:
imagePullSecrets:
- name: registry
containers:
- image: registry-1.divante.pl:5000/openloyalty/worker-framework:4.2.0
name: worker
imagePullPolicy: Always
volumeMounts:
- mountPath: /var/www/openloyalty/var/log
name: logs
subPath: logs
- mountPath: /var/www/openloyalty/config/jwt
name: jwt
subPath: jwt
- mountPath: /var/www/openloyalty/var/uploads
name: uploads
subPath: uploads
- mountPath: /var/www/openloyalty/.env.prod
name: parameters
subPath: .env.prod
restartPolicy: Always
volumes:
- name: logs
persistentVolumeClaim:
claimName: logs-volume
- name: jwt
persistentVolumeClaim:
claimName: jwt-volume
- name: uploads
persistentVolumeClaim:
claimName: uploads-volume
- name: parameters
configMap:
name: app
items:
- key: .env.prod
path: .env.prod
---
##########
# RABBITMQ
##########
apiVersion: extensions/v1
kind: Deployment
metadata:
name: rabbitmq
namespace: openloyalty
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: rabbitmq
spec:
imagePullSecrets:
- name: registry
containers:
- image: rabbitmq:3.9-management
name: rabbitmq
ports:
- containerPort: 5672
volumeMounts:
- mountPath: /var/lib/rabbitmq
name: rabbitmq
restartPolicy: Always
volumes:
- name: rabbitmq
persistentVolumeClaim:
claimName: rabbitmq-volume
---
apiVersion: v1
kind: Service
metadata:
name: rabbitmq
namespace: openloyalty
labels:
app: rabbitmq
spec:
ports:
- port: 5672
targetPort: 5672
selector:
app: rabbitmq
---
##########
# LOGSTASH
##########
apiVersion: extensions/v1
kind: Deployment
metadata:
name: logstash
namespace: openloyalty
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: logstash
spec:
imagePullSecrets:
- name: registry
containers:
- image: registry-1.divante.pl:5000/openloyalty/logstash:7.5.2
name: logstash
ports:
- containerPort: 5000
env:
- name: LS_JAVA_OPTS
value: "-Xmx256m -Xms256m"
- name: xpack.monitoring.elasticsearch.hosts
value: http://elk:9200
- name: xpack.monitoring.elasticsearch.url
value: http://elk:9200
- name: ELASTICSEARCH_HOST
value: http://elk:9200
restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
name: logstash
namespace: openloyalty
labels:
app: logstash
spec:
ports:
- name: tcp
port: 5000
targetPort: 5000
- name: udp
port: 9600
targetPort: 9600
selector:
app: logstash
---
#######
# REDIS
#######
apiVersion: extensions/v1
kind: Deployment
metadata:
name: redisserver
namespace: openloyalty
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: redisserver
spec:
imagePullSecrets:
- name: registry
containers:
- image: registry-1.divante.pl:5000/openloyalty/redis:5.0.7-alpine
name: redis
ports:
- containerPort: 6379
restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
name: redisserver
namespace: openloyalty
labels:
app: redisserver
spec:
ports:
- port: 6379
targetPort: 6379
selector:
app: redisserver
---
#######
# KIBANA
#######
apiVersion: extensions/v1
kind: Deployment
metadata:
name: kibana
namespace: openloyalty
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: kibana
spec:
containers:
- image: docker.elastic.co/kibana/kibana:7.5.0
name: kibana
ports:
- containerPort: 5601
env:
- name: ELASTICSEARCH_HOSTS
value: http://elk:9200
restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
name: kibana
namespace: openloyalty
labels:
app: kibana
spec:
ports:
- port: 5601
targetPort: 5601
selector:
app: kibana
---
#########
# Varnish
#########
apiVersion: extensions/v1
kind: Deployment
metadata:
name: varnish
namespace: openloyalty
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: varnish
spec:
imagePullSecrets:
- name: registry
containers:
- image: registry-1.divante.pl:5000/openloyalty/varnish:6
name: openloyalty-varnish
ports:
- containerPort: 80
volumeMounts:
- mountPath: /etc/varnish/settings.vcl
name: varnish
subPath: settings.vcl
restartPolicy: Always
volumes:
- name: varnish
configMap:
name: varnish
items:
- key: settings.vcl
path: settings.vcl
---
apiVersion: v1
kind: Service
metadata:
name: varnish
namespace: openloyalty
labels:
app: varnish
spec:
ports:
- name: http
port: 8081
targetPort: 80
selector:
app: varnish
---
Ingress configuration
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: openloyalty-ingress
namespace: openloyalty
annotations:
ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: admin.example.com
http:
paths:
- path: /
backend:
serviceName: frontend-admin
servicePort: 3001
- host: client.example.com
http:
paths:
- path: /
backend:
serviceName: pwacc
servicePort: 3004
- host: pos.example.com
http:
paths:
- path: /
backend:
serviceName: frontend-pos
servicePort: 3003
- host: api.example.com
http:
paths:
- path: /
backend:
serviceName: varnish
servicePort: 8081
How to run OL on Minikube¶
We assume that you have experience with kubernetes; some details were skipped.
- Install Minikube (go to installation guideline)
- Start minikube
minikube start
- Obtain minikube’s IP by
minikube ip
- Add bellow addresses to /etc/hosts (replace IP addresses to obtained above)
192.168.99.100 admin.example.com
192.168.99.100 client.example.com
192.168.99.100 pos.example.com
192.168.99.100 api.example.com
- Create a namespace
kubectl create namespace openloyalty
- Create new or reuse __secret.yml file
apiVersion: v1
data:
.dockerconfigjson: __REPLACE_IT__
kind: Secret
metadata:
name: registry
namespace: openloyalty
type: kubernetes.io/dockerconfigjson
``kubectl create secret docker-registry registry --docker-server=<server_name:port> --docker-username=<username> --docker-password=<password> --docker-email=example@example.com --dry-run -o yaml``
- Apply secret file
kubectl -n openloyalty apply -f __secret.yml
8. Create persistant volumes. In this example we provide storages with manual class name. Make sure that persistent volumes paths defined in manifests are empty. If not, try to log in to minikube (minikube ssh) and remove it.
kubectl -n openloyalty apply -f pv.yml
9. Apply the rest of manifests. Default registry is registry-1.divante.pl:5000/openloyalty/. You can change it in deployment.yml. Make sure that you have correct credentials.
kubectl -n openloyalty apply -f storage.yml
kubectl -n openloyalty apply -f config.yml
kubectl -n openloyalty apply -f deployment.yml
kubectl -n openloyalty apply -f ingress.yml
- Make sure that all containers are running
- Log in to php-* container
and execute bellow commands, which set proper ownership of directories:
root@php-6c4b5b8cb9-vq252:/var/www/openloyalty# chown -R www-data:www-data .
Before initializing application sample data make sure that current user is www-data:
root@php-6c4b5b8cb9-vq252:/var/www/openloyalty# su www-data
If you wish to set up full sample data, good for demo purposes:
www-data@php-6c4b5b8cb9-vq252:~/openloyalty$ phing setup
And if you wish to initialize the application with just basic data, which is good for production purposes:
www-data@php-6c4b5b8cb9-vq252:~/openloyalty$ phing basic-setup
- Open the panels in your browser