Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Readme updates for 4.0 and project migration #1199

Merged
merged 3 commits into from
Feb 17, 2023
Merged

Readme updates for 4.0 and project migration #1199

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a047fa4
Readme updates for 4.0 and project migration
fekete-robert Feb 14, 2023
ba830e6
Update README.md
fekete-robert Feb 16, 2023
f211865
Merge branch 'master' into patch-2
ahma Feb 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 37 additions & 24 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,21 @@

# Logging operator

Logging operator for Kubernetes based on Fluentd and Fluent-bit.
The Logging operator solves your logging-related problems in Kubernetes environments by automating the deployment and configuration of a Kubernetes logging pipeline.

The Logging operator automates the deployment and configuration of a Kubernetes logging pipeline. The operator deploys and configures a Fluent Bit daemonset on every node to collect container and application logs from the node file system. Fluent Bit queries the Kubernetes API and enriches the logs with metadata about the pods, and transfers both the logs and the metadata to Fluentd. Fluentd receives, filters, and transfer logs to multiple outputs. Your logs will always be transferred on authenticated and encrypted channels.
1. The operator deploys and configures a log collector (currently a Fluent Bit DaemonSet) on every node to collect container and application logs from the node file system.
1. Fluent Bit queries the Kubernetes API and enriches the logs with metadata about the pods, and transfers both the logs and the metadata to a log forwarder instance.
1. The log forwarder instance receives, filters, and transforms the incoming the logs, and transfers them to one or more destination outputs. The Logging operator supports Fluentd and syslog-ng as log forwarders.

Your logs are always transferred on authenticated and encrypted channels.

This operator helps you bundle logging information with your applications: you can describe the behavior of your application in its charts, the Logging operator does the rest.

## What is this operator for?

This operator helps you bundle logging information with your applications: you can describe the behavior of your application in its charts, the Logging operator does the rest.

<p align="center"><img src="https://banzaicloud.com/docs/one-eye/logging-operator/img/logging_operator_flow.png" ></p>
<p align="center"><img src="https://kube-logging.github.io/docs/img/logging_operator_flow.png" ></p>

## Feature highlights

Expand All @@ -38,50 +44,56 @@ This operator helps you bundle logging information with your applications: you c
- [x] Secure communication (TLS)
- [x] Configuration validation
- [x] Multiple flow support (multiply logs for different transformations)
- [x] Multiple [output](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/plugins/outputs/) support (store the same logs in multiple storage: S3, GCS, ES, Loki and more...)
- [x] Multiple output support (store the same logs in multiple storage: S3, GCS, ES, Loki and more...)
- [x] Multiple logging system support (multiple fluentd, fluent-bit deployment on the same cluster)

## Architecture

You can define `outputs` (destinations where you want to send your log messages, for example, Elasticsearch, or and Amazon S3 bucket), and `flows` that use filters and selectors to route log messages to the appropriate outputs. You can also define cluster-wide outputs and flows, for example, to use a centralized output that namespaced users cannot modify.
The Logging operator manages the log collectors and log forwarders of your logging infrastructure, and the routing rules that specify where you want to send your different log messages.

The **log collectors** are endpoint agents that collect the logs of your Kubernetes nodes and send them to the log forwarders. Logging operator currently uses Fluent Bit as log collector agents.

The **log forwarder** instance receives, filters, and transforms the incoming the logs, and transfers them to one or more destination outputs. The Logging operator supports Fluentd and syslog-ng as log forwarders. Which log forwarder is best for you depends on your logging requirements.

You can filter and process the incoming log messages using the **flow** custom resource of the log forwarder to route them to the appropriate **output**. The outputs are the destinations where you want to send your log messages, for example, Elasticsearch, or an Amazon S3 bucket. You can also define cluster-wide outputs and flows, for example, to use a centralized output that namespaced users can reference but cannot modify. Note that flows and outputs are specific to the type of log forwarder you use (Fluentd or syslog-ng).

You can configure the Logging operator using the following Custom Resource Definitions.

- [logging](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/crds/v1beta1/logging_types/) - Represents a logging system. Includes `Fluentd` and `Fluent-bit` configuration. Specifies the `controlNamespace`. Fluentd and Fluent-bit will be deployed in the `controlNamespace`
- [output](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/crds/v1beta1/output_types/) - Defines an Output for a logging flow. This is a namespaced resource. See also `clusteroutput`.
- [flow](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/crds/v1beta1/flow_types/) - Defines a logging flow with `filters` and `outputs`. You can specify `selectors` to filter logs by labels. Outputs can be `output` or `clusteroutput`. This is a namespaced resource. See also `clusterflow`.
- [clusteroutput](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/crds/v1beta1/clusteroutput_types/) - Defines an output without namespace restriction. Only effective in `controlNamespace`.
- [clusterflow](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/crds/v1beta1/output_types/) - Defines a logging flow without namespace restriction.
- [logging](https://kube-logging.github.io/docs/logging-infrastructure/logging/) - The `logging` resource defines the logging infrastructure (the log collectors and forwarders) for your cluster that collects and transports your log messages. It also contains configurations for Fluent Bit, Fluentd, and syslog-ng.
- CRDs for Fluentd:
- [output](https://kube-logging.github.io/docs/configuration/output/) - Defines a Fluentd Output for a logging flow, where the log messages are sent using Fluentd. This is a namespaced resource. See also `clusteroutput`. To configure syslog-ng outputs, see `SyslogNGOutput`.
- [flow](https://kube-logging.github.io/docs/configuration/flow/) - Defines a Fluentd logging flow using `filters` and `outputs`. Basically, the flow routes the selected log messages to the specified outputs. This is a namespaced resource. See also `clusterflow`. To configure syslog-ng flows, see `SyslogNGFlow`.
- [clusteroutput](https://kube-logging.github.io/docs/configuration/output/) - Defines a Fluentd output that is available from all flows and clusterflows. The operator evaluates clusteroutputs in the `controlNamespace` only unless `allowClusterResourcesFromAllNamespaces` is set to true.
- [clusterflow](https://kube-logging.github.io/docs/configuration/flow/) - Defines a Fluentd logging flow that collects logs from all namespaces by default. The operator evaluates clusterflows in the `controlNamespace` only unless `allowClusterResourcesFromAllNamespaces` is set to true. To configure syslog-ng clusterflows, see `SyslogNGClusterFlow`.
- CRDs for syslog-ng (these resources like their Fluentd counterparts, but are tailored to features available via syslog-ng):
- [SyslogNGOutput](https://kube-logging.github.io/docs/configuration/output/#syslogngoutput) - Defines a syslog-ng Output for a logging flow, where the log messages are sent using Fluentd. This is a namespaced resource. See also `SyslogNGClusterOutput`. To configure Fluentd outputs, see `output`.
- [SyslogNGFlow](https://kube-logging.github.io/docs/configuration/flow/#syslogngflow) - Defines a syslog-ng logging flow using `filters` and `outputs`. Basically, the flow routes the selected log messages to the specified outputs. This is a namespaced resource. See also `SyslogNGClusterFlow`. To configure Fluentd flows, see `flow`.
- [SyslogNGClusterOutput](https://kube-logging.github.io/docs/configuration/output/#syslogngoutput) - Defines a syslog-ng output that is available from all flows and clusterflows. The operator evaluates clusteroutputs in the `controlNamespace` only unless `allowClusterResourcesFromAllNamespaces` is set to true.
- [SyslogNGClusterFlow](https://kube-logging.github.io/docs/configuration/flow/#syslogngflow) - Defines a syslog-ng logging flow that collects logs from all namespaces by default. The operator evaluates clusterflows in the `controlNamespace` only unless `allowClusterResourcesFromAllNamespaces` is set to true. To configure Fluentd clusterflows, see `clusterflow`.

See the [detailed CRDs documentation](https://banzaicloud.com/docs/one-eye/logging-operator/configuration/crds/).
See the [detailed CRDs documentation](https://kube-logging.github.io/docs/configuration/crds/).

<p align="center"><img src="https://banzaicloud.com/docs/one-eye/logging-operator/img/logging-operator-v2-architecture.png" ></p>
<p align="center"><img src="hhttps://kube-logging.github.io/docs/img/logging-operator-v2-architecture.png" ></p>

## Quickstart

[![asciicast](https://asciinema.org/a/315998.svg)](https://asciinema.org/a/315998)

Follow these [quickstart guides](https://banzaicloud.com/docs/one-eye/logging-operator/quickstarts/) to try out the Logging operator!
Follow these [quickstart guides](https://kube-logging.github.io/docs/quickstarts/) to try out the Logging operator!

### Install

Deploy Logging Operator with [Kubernetes manifests](https://banzaicloud.com/docs/one-eye/logging-operator/install/) or [Helm chart](https://banzaicloud.com/docs/one-eye/logging-operator/install/#deploy-logging-operator-with-helm).
Deploy Logging Operator with our [Helm chart](https://kube-logging.github.io/docs/install/#deploy-logging-operator-with-helm).

> Caution: The **master branch** is under heavy development. Use [releases](https://github.com/banzaicloud/logging-operator/releases) instead of the master branch to get stable software.
> Caution: The **master branch** is under heavy development. Use [releases](https://github.com/kube-logging/logging-operator/releases) instead of the master branch to get stable software.

## Support

### Community support

If you encounter problems that the documentation does not address, [open an issue](https://github.com/banzaicloud/logging-operator/issues) or talk to us on the Banzai Cloud Slack channel [#logging-operator](https://banzaicloud.com/invite-slack/).

### Commercial support

If you are using the Logging operator in a production environment and [require commercial support, contact Banzai Cloud](https://banzaicloud.com/contact/), the company backing the development of the Logging operator. If you are looking for the ultimate observability tool for multi-cluster Kubernetes infrastructures to automate the collection, correlation, and storage of logs and metrics, check out [One Eye](https://banzaicloud.com/products/one-eye/).
If you encounter problems while using the Logging operator the documentation does not address, [open an issue](https://github.com/kube-logging/logging-operator/issues) or talk to us on the Banzai Cloud Slack channel [#logging-operator](https://banzaicloud.com/invite-slack/).

## Documentation

You can find the complete documentation of the Logging operator on the [Banzai Cloud Documentation Page](https://banzaicloud.com/docs/one-eye/logging-operator) :blue_book: <br>
You can find the complete documentation on the [Logging operator documentation page](https://kube-logging.github.io/docs/) :blue_book: <br>

## Contributing

Expand All @@ -92,10 +104,11 @@ If you find this project useful, help us:
- Help new users with issues they may encounter :muscle:
- Send a pull request with your new features and bug fixes :rocket:

*For more information, read the [developer documentation](https://banzaicloud.com/docs/one-eye/logging-operator/developers)*.
*For more information, read the [developer documentation](https://kube-logging.github.io/docs/developers)*.

## License

Copyright (c) 2021-2023 [Cisco Systems, Inc. and its affiliates](https://cisco.com)
Copyright (c) 2017-2020 [Banzai Cloud, Inc.](https://banzaicloud.com)

Licensed under the Apache License, Version 2.0 (the "License");
Expand Down