tmp is an emptyDir volume mount with 0777 permissions. Some versions
of Ruby require the sticky bit to be set.
Change-Id: Ib6f4daa2068e7b29c62b5858848774b7117f6808
Based on spec in openstack-helm repo,
support-OCI-image-registry-with-authentication-turned-on.rst
Each Helm chart can configure an OCI image registry and
credentials to use. A Kubernetes secret is then created with these
info. Service Accounts then specify an imagePullSecret specifying
the Secret with creds for the registry. Then any pod using one
of these ServiceAccounts may pull images from an authenticated
container registry.
Change-Id: Iebda4c7a861aa13db921328776b20c14ba346269
The broker attribute should use a comma separated list with the port
definition included
Example: kafka1:9092,kafka2:9092,kafka:9092
The kafka client will connect to the first available host this
will provide resiliency if a host is not available
Change-Id: I5f82e96f2aa274379b6d808291d4b5109709bf72
This change enables TLS between Elasticsearch and Kibana
data path. Note that TLS terminates at apache-proxy container
of the Elasticsearch-client pod, not directly to port 9200 of
elasticsearch-client container.
Since all data traffic goes through apache-proxy container,
fluentd output to Elasticsearch are configured to have TLS
enabled as well.
In additon, other Elasticsearch pods that communicate with
Elasticsearch-client endpoint are modified to provide
the cacert option with curl.
Change-Id: I3373c0c350b30c175be4a34d25a403b9caf74294
This change updates the fluentd chart to use HTK probe templates
to allow configuration by value overrides
Change-Id: I97a3cc0832554a31146cd2b6d86deb77fd73db41
This chart could deploy fluentd either as a Deployment
or a Daemonset. Both options would use the deployment-fluentd
template with various sections toggled off based on values.yaml
I'd like to know - Does anyone run this chart as a Deployment?
We can simplify the chart, and zuul gates, by changing the chart
to deploy a Daemonset specifically.
Change-Id: Ie88ceadbf5113fc60e5bb0ddef09e18fe07a192c
Unrestrict octal values rule since benefits of file modes readability
exceed possible issues with yaml 1.2 adoption in future k8s versions.
These issues will be addressed when/if they occur.
Also ensure osh-infra is a required project for lint job, that matters
when running job against another project.
Change-Id: Ic5e327cf40c4b09c90738baff56419a6cef132da
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
By tying the fluent condfiguration to the release, it will be re-rendered
if the release is upgraded. This is useful in combination with [0], allowing
powerful configuration updates using helm upgrade. For example
Values:
.Values.pod.env.fluentd.vars.OUTPUT_ENABLED: true
fluent.conf:
...
{{- if .Values.pod.env.fluentd.vars.OUTPUT_ENABLED }}
<match **>
# Output Configuration here
</match>
{{- end }}
To disable this output section, issue a helm upgrade command and set the
apprpriate value to false.
helm upgrade fluentd ./fluentd --set pod.env.fluentd.vars.OUTPUT_ENABLED=false
[0] https://review.opendev.org/#/c/726880/
Change-Id: I3dce9e5c4eaf588569e8cc3e1ea3cf3bebd0c3c5
This change updates the fluentd configmap-etc to render
.Values.conf.fluentd.template as a template, allowing for greater
flexibility in configuration.
Change-Id: I8809767c679c377e319ecc53960c55ae18e1b558
The current copyright refers to a non-existent group
"openstack helm authors" with often out-of-date references that
are confusing when adding a new file to the repo.
This change removes all references to this copyright by the
non-existent group and any blank lines underneath.
Change-Id: I1882738cf9757c5350a8533876fd37b5920b5235
This change updates the fluentd chart to use the native
fluent-plugin-prometheus for metric production. This plugin
provides more detailed metrics about fluentd's operations,
specifically regarding input and output statistics.
https://github.com/fluent/fluent-plugin-prometheus
Using the plugin, each fluentd pod produces metrics, so
scape annotations have been added to the pod spec. The
zuul check on metric producers has been updated to account
for this.
Depends-On: https://review.opendev.org/714167
Change-Id: I809356d92b0cff1e31cb2062102bbedefd4843fd
This patch set updates and tests the apiVersion for rbac.authorization.k8s.io
from v1beta1 to v1 in preparation for its removal in k8s 1.20.
Change-Id: I4e68db1f75ff72eee55ecec93bd59c68c179c627
Signed-off-by: Tin Lam <tin@irrational.io>
This change adds in missing network policy overrides for
fluent-daemonset and prometheus-exporter, as well as removes
existing mariadb network policies overrides that were causing
the network policy check job to fail.
Change-Id: Ib7a33f3d14617f9a9fda264f32cde7729a923193
This adds a helm-toolkit util for consuming arbitrary secret env
variables via pod env variables. It also updates the Fluentd chart
to add a release secret that is used to house the secret env
variables defined in the chart's values.yaml. This can be used as
an example to expand to other charts where this functionality is
desired
Change-Id: I9ef606840af92e54b2204e637c58442085e2c748
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This proposes adding a kafka chart to osh-infra that aligns
with the design patterns laid out by the other charts in osh-infra
and osh.
danielqsj's kafka-exporter image is leveraged to deploy a prometheus
exporter for kafka alongside the main application if enabled in
values.yaml
Change-Id: I5997b0994fc3aef9bd1b222c373cc3a013112566
Co-Authored-By: Meghan Heisler <mh783g@att.com>
This updates the kubeadm and minikube Kubernetes deployments to
deploy version 1.16.2
Change-Id: I324f9665a24c9383c59376fb77cdb853facd0f18
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This adds support for configuring fluentd's update strategy when
deployed as a daemonset, as this was previously missed when
the changes to support both daemonsets and deployments were made
Change-Id: I5ac4fbfc0e64caaf207de42cd71c893f8d0f6ff1
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This updates the Fluentd deployment template to use the helm
toolkit util for generating environment variables through the
chart's values.yaml. This adds flexibility in defining fluentd
outputs, as arbitrary environment variables can be injected and
consumed in fluentd's filters and outputs
Change-Id: I72a2c476378cc555bde1387781b4a06f13b51bc6
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This updates the fluentd chart to add support for leveraging a
Kafka output. This required adding a kafka endpoint entry to the
chart's values.yaml, as well as the required template updates to
the fluentd deployment template and the addition of a secret for
kafka credentials
Depends-On: https://review.opendev.org/#/c/679297/
Change-Id: I80a487a0538f0b3704fb598da38c07feedaccb0e
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This updates the Fluentd clusterrole to allow for getting
namespaces, as this is required for the fluentd kubernetes
plugin to function correctly
Change-Id: Id9d735310c53a922a62c6a82121edd332e7df724
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This fixes the whitespace chomps for adding extra volumes and
volume mounts via values.yaml for the Fluentd chart, as currently
too much whitespace is removed and the extra volumes and mounts
are not added correctly
Change-Id: I9cf67c3321339078ac795a7290f441b16cc41d41
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This adds a conditional check on the deployment type of the
Fluentd chart to determine whether to enable the current liveness
and readiness probes or not. The current probes are designed
around using fluentd as an aggregator and do not function properly
when fluentd is deployed as a daemonset. When run as a daemonset
and configured to tail files via the tail input plugin, fluentd
will prioritize reading the entirety of those files before
processing other input types, including opening the forward source
socket required for the current probes to function correctly. This
results in scenarios where the current probes will fail when in
fact fluentd is functioning correctly.
Daemonset focused probes to come as a follow on once a proper path
forward has been determined
Change-Id: I8a164bd47ce1950e0bd6c5043713f4cde9f85d79
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This removes the readOnly flag from the /var/log mount for the
fluentd pod to allow for using the file buffer mechanism when
desired
Change-Id: I23f0f03824eec5b142d3f2e8e42e8d07cddfe618
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This updates the default fluentd configuration to include
recommended settings for preventing the elasticsearch plugin from
reloading the connection after 10000 requests (default for the
ruby gem). This also updates the configuration overrides for the
fluentd-daemonset deployment to provide input parity with the
default fluentbit configuration by adding inputs for the docker
and kubelet systemd units, inputs for ceph, libvirt, kernel logs,
and auth logs on the host. Finally, this updates the fluentd
template to include environment variables for the host name and
the fluentd pod name so they can be added to logged events through
fluentd filter plugins
Change-Id: I21f7a89a325c44f8b058ff01a20191bea1a210b4
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This moves Fluentd to use the helm-toolkit endpoint lookup for
using the fqdn for the Elasticsearch hostname instead of the
standard short host name
Change-Id: Ibe640979002331693f0a9b6155c9014572294664
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This adds required changes to the Fluentd chart to allow for
deploying Fluentd as either a deployment or a daemonset. This
follows the pattern laid out by the ingress chart. This also
updates the single and multinode jobs to deploy fluentd as both
a daemonset and a deployment for validation
Change-Id: I84353a2daa2ce56ff59882a8d33203286ed27e06
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This begins to split the fluent-logging chart into two separate
charts, one for fluentbit and one for fluentd. This is to help
isolate each chart and its dependencies better, and to treat each
service as its own entity.
This also moves the job for creating Elasticsearch templates to
the Elasticsearch chart, as the elasticsearch chart should have
ownership of creating the templates for its indices.
This also performs some general cleanup of values keys that are
not currently used
Change-Id: I827277d5faa62b8b59c5960330703d23c297ca47
Signed-off-by: Steve Wilkerson <sw5822@att.com>
