ELK (logs collecting)

Here's a brief overview of each component and how they work together:

Logstash:
Logstash is a data collection and processing pipeline tool. It's used to ingest, transform, and enrich data from various sources. These sources can include log files, databases, APIs, and more. Logstash allows you to apply filters and transformations to the incoming data before sending it to Elasticsearch for indexing. For example, you can use Logstash to parse and structure unstructured log data, convert data formats, and even enrich the data by adding additional fields.

Elasticsearch:
Elasticsearch is a powerful and distributed search and analytics engine. It's designed for real-time search and analysis of large volumes of data. Elasticsearch is used to index and store the data ingested by Logstash, making it highly searchable and performant. It employs a distributed architecture that allows data to be split across multiple nodes, providing scalability, fault tolerance, and fast search capabilities. Elasticsearch also provides a RESTful API for querying and interacting with the indexed data.

Kibana:
Kibana is a data visualization and exploration tool that works in conjunction with Elasticsearch. It provides a user-friendly interface for creating dashboards, visualizations, and reports based on the data stored in Elasticsearch. With Kibana, you can easily create charts, graphs, maps, and tables to help you analyze and understand your data. Kibana dashboards can display real-time information and help users gain insights from the data stored in Elasticsearch.

What is ElasticSearch? (ELK stack)

How they work together:

1.  Data Collection and Ingestion:
Logstash collects data from various sources, processes it, and sends it to Elasticsearch. It can handle various data formats and apply filters and transformations as needed. This ensures that the data is structured and indexed properly in Elasticsearch.

2.  Data Storage and Indexing:
Elasticsearch stores structured data in a distributed index, making it highly searchable and performant. It handles the indexing, querying, and retrieval of data based on the user's requests.

3.  Data Visualization and Exploration:
Kibana connects to Elasticsearch to visualize and analyze the indexed data. Users can create custom dashboards and visualizations, allowing them to monitor trends, anomalies, and patterns in the data.

In summary, Logstash ingests and processes data, Elasticsearch indexes and stores the data, and Kibana provides the interface for visualizing and exploring the data.

JBT already has the required dependencies for Logstash, you just need to configure logback-spring.xml by providing Logstash details such as host and port.

logback-spring.xml path: configuration/src/main/resources/logback-spring.xml

fill in <remoteHost/> and <port/> by values based on your running Logstash service.
You could also use <destination> instead of <remoteHost/> and <port/>.

Don't forget to change the spring profile name.

JBT uses the docker profile just for testing, you should substitute it with a name based on your environment such as local/dev/prod