[](https://loggie-io.github.io/docs/)
[](https://bestpractices.coreinfrastructure.org/projects/569)
[](https://hub.docker.com/r/loggieio/loggie/)
> English | [中文](./README_cn.md)
Loggie is a lightweight, high-performance, cloud-native agent and aggregator based on Golang.
- Supports multiple pipeline and pluggable components, including data transfer, filtering, parsing, and alerting.
- Uses native Kubernetes CRD for operation and management.
- Offers a range of observability, reliability, and automation features suitable for production environments.
Based on Loggie, we can build a cloud-native scalable log data platform.
## Features
### Next-generation cloud-native log collection and transmission
#### Building pipelines based on CRD
Loggie includes LogConfig/ClusterLogConfig/Interceptor/Sink CRDs, allowing for the creation of data collection, transfer, processing, and sending pipelines through simple YAML file creation.
eg:
```yaml
apiVersion: loggie.io/v1beta1
kind: LogConfig
metadata:
name: tomcat
namespace: default
spec:
selector:
type: pod
labelSelector:
app: tomcat
pipeline:
sources: |
- type: file
name: common
paths:
- stdout
- /usr/local/tomcat/logs/*.log
sinkRef: default
interceptorRef: default
```
#### Multiple architectures
- **Agent**: Deployed via DaemonSet, Loggie can collect log files without the need for containers to mount volumes.
- **Sidecar**: Supports non-intrusive auto-injection of Loggie sidecars, without the need to manually add them to the Deployment/StatefulSet templates.
- **Aggregator**: Supports deployment as an independent intermediate machine, which can receive aggregated data sent by Loggie Agent and can also be used to consume and process various data sources.
But regardless of the deployment architecture, Loggie still maintains a simple and intuitive internal design.
### High Performance
#### Benchmark
Configure Filebeat and Loggie to collect logs, and send them to a Kafka topic without using client compression, with the Kafka topic partition configured as 3.
With sufficient resources for the Agent specification, modify the number of files collected, the concurrency of the sending client (configure Filebeat worker and Loggie parallelism), and observe their respective CPU, memory, and pod network card transmission rates.
| Agent | File Size | File Count | Sink Concurrency | CPU | MEM (rss) | Transmission Rates |
|----------|-----------|------------|------------------|----------|-----------|--------------------|
| Filebeat | 3.2G | 1 | 3 | 7.5~8.5c | 63.8MiB | 75.9MiB/s |
| Filebeat | 3.2G | 1 | 8 | 10c | 65MiB | 70MiB/s |
| Filebeat | 3.2G | 10 | 8 | 11c | 65MiB | 80MiB/s |
| | | | | | | |
| Loggie | 3.2G | 1 | 3 | 2.1c | 60MiB | 120MiB/s |
| Loggie | 3.2G | 1 | 8 | 2.4c | 68.7MiB | 120MiB/s |
| Loggie | 3.2G | 10 | 8 | 3.5c | 70MiB | 210MiB/s |
#### Adaptive Sink Concurrency
With sink concurrency configuration enabled, Loggie can:
- Automatically adjust the downstream data sending parallelism based on the actual downstream data response, making full use of the downstream server's performance without affecting its performance.
- Adjust the downstream data sending speed appropriately when upstream data collection is blocked to relieve upstream blocking.
### Lightweight Streaming Data Analysis and Monitoring
Logs are a universal data type and are not related to platforms or systems. How to better utilize this data is the core capability that Loggie focuses on and develops.
#### Real-time parsing and transformation
With the configuration of transformer interceptors and the configuration of functional actions, Loggie can achieve:
- Parsing of various data formats (json, grok, regex, split, etc.)
- Conversion of various fields (add, copy, move, set, del, fmt, etc.)
- Support for conditional judgment and processing logic (if, else, return, dropEvent, ignoreError, etc.)
eg:
```yaml
interceptors:
- type: transformer
actions:
- action: regex(body)
pattern: (?\S+) (?\S+) (?\S+) (?
[](https://loggie-io.github.io/docs/)
[](https://bestpractices.coreinfrastructure.org/projects/569)
[](https://hub.docker.com/r/loggieio/loggie/)
> English | [中文](./README_cn.md)
Loggie is a lightweight, high-performance, cloud-native agent and aggregator based on Golang.
- Supports multiple pipeline and pluggable components, including data transfer, filtering, parsing, and alerting.
- Uses native Kubernetes CRD for operation and management.
- Offers a range of observability, reliability, and automation features suitable for production environments.
Based on Loggie, we can build a cloud-native scalable log data platform.
## Features
### Next-generation cloud-native log collection and transmission
#### Building pipelines based on CRD
Loggie includes LogConfig/ClusterLogConfig/Interceptor/Sink CRDs, allowing for the creation of data collection, transfer, processing, and sending pipelines through simple YAML file creation.
eg:
```yaml
apiVersion: loggie.io/v1beta1
kind: LogConfig
metadata:
name: tomcat
namespace: default
spec:
selector:
type: pod
labelSelector:
app: tomcat
pipeline:
sources: |
- type: file
name: common
paths:
- stdout
- /usr/local/tomcat/logs/*.log
sinkRef: default
interceptorRef: default
```
#### Multiple architectures
- **Agent**: Deployed via DaemonSet, Loggie can collect log files without the need for containers to mount volumes.
- **Sidecar**: Supports non-intrusive auto-injection of Loggie sidecars, without the need to manually add them to the Deployment/StatefulSet templates.
- **Aggregator**: Supports deployment as an independent intermediate machine, which can receive aggregated data sent by Loggie Agent and can also be used to consume and process various data sources.
But regardless of the deployment architecture, Loggie still maintains a simple and intuitive internal design.
### High Performance
#### Benchmark
Configure Filebeat and Loggie to collect logs, and send them to a Kafka topic without using client compression, with the Kafka topic partition configured as 3.
With sufficient resources for the Agent specification, modify the number of files collected, the concurrency of the sending client (configure Filebeat worker and Loggie parallelism), and observe their respective CPU, memory, and pod network card transmission rates.
| Agent | File Size | File Count | Sink Concurrency | CPU | MEM (rss) | Transmission Rates |
|----------|-----------|------------|------------------|----------|-----------|--------------------|
| Filebeat | 3.2G | 1 | 3 | 7.5~8.5c | 63.8MiB | 75.9MiB/s |
| Filebeat | 3.2G | 1 | 8 | 10c | 65MiB | 70MiB/s |
| Filebeat | 3.2G | 10 | 8 | 11c | 65MiB | 80MiB/s |
| | | | | | | |
| Loggie | 3.2G | 1 | 3 | 2.1c | 60MiB | 120MiB/s |
| Loggie | 3.2G | 1 | 8 | 2.4c | 68.7MiB | 120MiB/s |
| Loggie | 3.2G | 10 | 8 | 3.5c | 70MiB | 210MiB/s |
#### Adaptive Sink Concurrency
With sink concurrency configuration enabled, Loggie can:
- Automatically adjust the downstream data sending parallelism based on the actual downstream data response, making full use of the downstream server's performance without affecting its performance.
- Adjust the downstream data sending speed appropriately when upstream data collection is blocked to relieve upstream blocking.
### Lightweight Streaming Data Analysis and Monitoring
Logs are a universal data type and are not related to platforms or systems. How to better utilize this data is the core capability that Loggie focuses on and develops.
#### Real-time parsing and transformation
With the configuration of transformer interceptors and the configuration of functional actions, Loggie can achieve:
- Parsing of various data formats (json, grok, regex, split, etc.)
- Conversion of various fields (add, copy, move, set, del, fmt, etc.)
- Support for conditional judgment and processing logic (if, else, return, dropEvent, ignoreError, etc.)
eg:
```yaml
interceptors:
- type: transformer
actions:
- action: regex(body)
pattern: (?