Architecture

The platform is splitted into micro-services, each one providing a specific function.

graph LR ui[fa:fa-user User interface] --> backend(Backend) backend[fa:fa-cogs Backend] --> rmq(fa:fa-bars Message broker) backend --> DB(fa:fa-database Database) rmq --> W1[fa:fa-cog Worker 1] rmq --> W2[fa:fa-cog Worker 2] rmq --> W3[fa:fa-cog Worker 3] classDef blue fill:#155799,stroke: none,color: #fff; classDef green fill:#159957,stroke: none,color: #fff; class ui green class backend blue class rmq blue class DB blue class W1 green class W2 green class W3 green

Backend and User interface

The core component is the backend, which provides user management, workflow engine and monitoring.

Workflows

The backend component manages workflows structured like that:

• Workflow: a description including a list of steps
  • Step: static declaration of a process
    • Job: instance of a step, many jobs can be generated at the runtime for 1 step (1 per file for example)

graph LR wf[fa:fa-list-ul Workflow] --> |0..*| step[Step] step --> |0..*| job[fa:fa-cogs Job] classDef blue fill:#155799,stroke: none,color: #fff; classDef green fill:#159957,stroke: none,color: #fff; class wf green class step green class job blue

Complete workflow documentation of the schema is described here.

Messaging protocol

A communication standard is required to match messages between backend and workers. To ensure the quality of integration, it is recommended to use everywhere rs_amqp_worker which provides a worker SDK.

All details about communication between the backend and workers is detailed here.

Deployment

The entire project is currently deployed using Kubernetes.
Every part of the platform is built into Docker images and can be deployed easily with the startup project.

Developement stack

User interface

The frontend is designed using Angular.io

Backend

The backend uses the power of Elixir, based on Erlang to provide a resilient and efficient backend.

Workers

Most of workers are using Rust, high-typed language with performances similar to C/C++ applications.

Some other workers can use C/C++ using the c_amqp_worker project