Platform

Core Components

SkaLogs is a self-hosted entreprise grade Open Source – Big Data – Real-Time Platform design to help you deploy, manage, and scale a centralized log management solution. It is based on 3 core components (Apache V2):

  1. SkaETL : an Open Source (SkaETL GitHub Repo) real-time ETL developed by SkaLogs,
  2. ELK Stack : Elasticsearch, Logstash, Kibana,
  3. Kafka.

Core Features

  • Self-hosted (on-premise or cloud) complete end-to-end centralized Log Management Platform
  • Scripted and Automated deployment (Ansible, Shell and yaml scripts)
  • Container management (Rancher container management platform) and Orchestration (Kubernetes)
  • Guided workflows for
    • data and Log ingestion and transformation (structured and unstructured)
    • real-time metrics computations and insights
    • interfacing with your own ML algorithms

Functional Architecture Diagram

SkaLogs - Architecture-Functional
SkaLogs - Architecture-Functional

Bundle

The SkaLogs Platform consists of a bundle which deploys many services, and scales them according to the ressources (cloud, on-premise) allocated to the instance deployed :

SkaLogs Bundle (GitHub repo):

The entire platform is deployed via a single Ansible script which:

  • installs a bundle consisting of the above-mentioned 3 core components (SkaETL, ELK, Kafka),
  • adds multiple side components,
  • assembles the pieces into a scalable, automated, resilient, self-monitored, and complete end-to-end Log Management Platform,
  • provides an entirely managed infrastructure (Rancher) with containerized (Docker) and orchestrated (Kubernetes) components.

The SkaLogs bundle includes:

  • Rancher as a container management platform,
  • SkaETL as an advanced log-dedicated ETL (SkaETL, developed by SkaLogs) with multiple guided workflows to help you with all the difficult tasks:
    • Logs: collect, transform, normalize, parse, aggregate,
    • Metrics: compute (before ES ingestion), store, search, investigate,
    • Alerts: create thresholds with alerts and notifications.
    • Visualize:
      • before ES ingestion: monitor data before ingestion and indexing in Elasticsearch
      • after ES ingestion:
        • Kibana as a front-end to Elasticsearch
        • Grafana as a front end for technical monitoring

SkaETL Features

SkaETL is a specialized 100% log-dedicated ETL developed by SkaLogs, allolwing you to process structured or unstructured data. The difficult task of data transformation is completely simplified thanks to multiple guided workflows:

  • Ingest, parse, transform, enrich, normalize, aggregate, index, archive
  • Compute (simple statistics, complex functions, and ML algorithms)
  • Search and investigate
  • Visualize and monitor
  • Alerts and notifications

Technical Features

  • Microservices based architecture
  • Packages multiple Open Source Libraries and framework
  • Entirely managed infrastructure with containerized and orchestrated components
  • Base deployment assembles 50+ services into 150+ docker containers
  • Scalable, automated, resilient, self-monitored
  • Error retry mechanism
  • Volume: Scales without limits
  • Speed: ingest at +100 K EPS (events / second or json documents / second)