OpenFaaS Function Troubleshooting
If a function is not behaving as expected, you can look at the logs to troubleshoot. First, SSH or console to the appliance as shown in the Requirements section.
List out the pods.
kubectl get pods -A
This is the function output:
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-584795fc57-4bp2s 1/1 Running 1 6d4h
kube-system coredns-584795fc57-76pwr 1/1 Running 1 6d4h
kube-system etcd-veba01 1/1 Running 2 6d4h
kube-system kube-apiserver-veba01 1/1 Running 2 6d4h
kube-system kube-controller-manager-veba01 1/1 Running 3 6d4h
kube-system kube-proxy-fvf2n 1/1 Running 2 6d4h
kube-system kube-scheduler-veba01 1/1 Running 2 6d4h
kube-system weave-net-v9jss 2/2 Running 6 6d4h
openfaas-fn powercli-entermaint-d84fd8d85-sjdgl 1/1 Running 1 6d4h
openfaas alertmanager-58f8d787d9-nqwm8 1/1 Running 1 6d4h
openfaas basic-auth-plugin-dd49cd66b-rv6n7 1/1 Running 1 6d4h
openfaas faas-idler-59ff9778fd-84szz 1/1 Running 4 6d4h
openfaas gateway-74f6f9489b-btgz8 2/2 Running 5 6d4h
openfaas nats-6dfbf45d77-9swph 1/1 Running 1 6d4h
openfaas prometheus-5f5494b54f-srs2d 1/1 Running 1 6d4h
openfaas queue-worker-59b67bf4-wqhm5 1/1 Running 4 6d4h
projectcontour contour-5cddfc8f6-hpzn8 1/1 Running 1 6d4h
projectcontour contour-5cddfc8f6-tdv2r 1/1 Running 2 6d4h
projectcontour contour-certgen-wrgnb 0/1 Completed 0 6d4h
projectcontour envoy-8mdhb 1/1 Running 2 6d4h
vmware tinywww-7fcfc6fb94-v7ncj 1/1 Running 1 6d4h
vmware vmware-event-router-5dd9c8f858-9g44h 1/1 Running 4 6d4h
First, we want to see if the event router is capturing events and forwarding them on to a function.
Use this command to follow the live Event Router log.
kubectl logs -n vmware vmware-event-router-5dd9c8f858-9g44h --follow
For this sample troubleshooting, we have the sample hostmaintenance alarms function running. To see if the appliance is properly handling the event, we put a host into maintenance mode.
When we look at the log output, we see various entries regarding EnteredMaintenanceModeEvent, ending with the following:
[OpenFaaS] 2020/03/11 22:15:09 invoking function(s) on topic: EnteredMaintenanceModeEvent
[OpenFaaS] 2020/03/11 22:15:09 successfully invoked function powercli-entermaint for topic EnteredMaintenanceModeEvent
This lets us know that the function was invoked. If we still don’t see the expected result, we need to look at the function logs.
Each OpenFaaS function will have its own pod running in the openfaas-fn namespace. We can examine the logs with the following command.
kubectl logs -n openfaas-fn powercli-entermaint-d84fd8d85-sjdgl
We don’t need the –follow switch because we are just trying to look at recent logs, but –follow would work too.
Some other useful switches are --since and --tail.
This command will show you the last 5 minutes worth of logs.
kubectl logs -n openfaas-fn powercli-entermaint-d84fd8d85-sjdgl --since=5m
This command will show you the last 20 lines of logs.
kubectl logs -n openfaas-fn powercli-entermaint-d84fd8d85-sjdgl --tail=20
Log output showing a succesful function invocation:
Connecting to vCenter Server ...
Disabling alarm actions on host: esx01.labad.int
Disconnecting from vCenter Server ...
2020/03/11 22:15:15 Duration: 6.085448 seconds
An alternative way to troubleshoot OpenFaaS logs is to use faas-cli.
This faas-cli command will show all available functions in the appliance. --tls-no-verify bypasses SSL certificate validation
faas-cli list --tls-no-verify
The command output is:
Function Invocations Replicas
powercli-entermaint 3 1
We can look at the logs with this command.
faas-cli logs powercli-entermaint --tls-no-verify
The logs are the same:
2020-03-11T22:15:15Z Connecting to vCenter Server ...
2020-03-11T22:15:15Z
2020-03-11T22:15:15Z Disabling alarm actions on host: esx01.labad.int
2020-03-11T22:15:15Z Disconnecting from vCenter Server ...
2020-03-11T22:15:15Z
2020-03-11T22:15:15Z 2020/03/11 22:15:15 Duration: 6.085448 seconds
All of the same switches shown in the kubectl commands such as --tail and --since work with faas-cli.
Note:
faas-cliwill stop tailing the log after a fixed period.
OpenFaaS Gateway not available
Self-signed certificate errors
The most common issue with OpenFaaS is related to certificates. The appliance certificate is self-signed. Attempting to connect to it without ignoring certificate errors results in an error, shown below
faas-cli secret list
The command output is:
Cannot connect to OpenFaaS on URL: https://veba02.lab.int
Adding the switch --tls-no-verify allows you to bypass SSL errors
faas-cli secret list --tls-no-verify
The command output is:
NAME
vc-hostmaint-config
Alternatively, you can replace the self-signed certificate with a signed certificate
Requirements
- Root access to the appliance
- A public/private key pair copied to a folder on the appliance filesystem
Run the following commands
cd /path/to/your/cert/files
CERT_NAME=eventrouter-tls
KEY_FILE=yourkeyfile.pem
CERT_FILE=yourcertfile.cer
#recreate the tls secret
kubectl --kubeconfig /root/.kube/config -n vmware delete secret ${CERT_NAME}
kubectl --kubeconfig /root/.kube/config -n vmware create secret tls ${CERT_NAME} --key ${KEY_FILE} --cert ${CERT_FILE}
#reapply the config to take the new certificate
kubectl --kubeconfig /root/.kube/config apply -f /root/ingressroute-gateway.yaml
Still having trouble?
Please submit bug reports and feature requests by using our GitHub Issues page or Join us on slack #vcenter-event-broker-appliance on vmwarecode.slack.com.