In our previous article, we presented the GamayunAI solution, which uses machine learning to help predict outages and improve overall network performance. We also presented the potential applications in other areas such as energy infrastructure.

The main challenge we faced in developing the GamayunAI solution was addressing the complexity of systems in which the number of components is increasing, some elements remain obsolete and the interdependence of individual elements only adds to the unmanageability of the situation.

The final solution is a working GamayunAI platform that enables the extraction, transformation, storage, and visualization of data or events and provides insight into the functioning of the information system. Let's take a look at how it works.

How GamayunAI platform and its building blocks function?

The figure below shows the GamayunAI platform, with the layers from left to right showing the flow of data (events) across the platform. On the left, we see the data flow platform, which is fed by different devices and applications, and then events are sent to the GamayunAI platform through the service layer. Through various pipelines, the data travels towards the storage layer where it becomes available for use in machine learning. As you can see from the figure, the data for machine learning comes from both the data storage and the data transfer layers, allowing us to evaluate the data in real-time, while at the same time learning from real-time and archive data. The last layer consists of dedicated visualizations and notifications of anomalies detected in the platform.

For the development of the GamayunAI platform, we have chosen open-source components that have wide community support, are resilient to bugs, and offer scalability. As input to the GamayunAI platform, we have chosen the open-source streaming platform Kafka, which has a number of built-in mechanisms to ensure resilience. It also allows us to easily scale up the data stream when needed. For the data lake, we have chosen ElasticSearch or OpenSearch, which is a distributed and highly scalable data repository for fast data search and analysis. The final layer of the platform is the GamayunEvents application, which aims to visualize, search and analyze the captured data in a simple and transparent way.

Machine learning, KYS, and system architecture

The whole GamayunAI platform was created for the purpose of machine learning, as we prepare the collected data from different sources through Kafka to learn models. Enriched and real-time refined data is key to enabling search through the data and its use for machine learning. Having the data easily accessible and in a usable format with the GamayunAI platform decreases model preparation and evaluation time and therefore saves money and time that can be spent solving other problems.

We have mentioned the complexity of the system and the dependencies between the components that affect the malfunctioning of the system. When we come to a situation where a part of our system is not working, but we do not know why, we need to ask the question: how well does the company know its system and the dependencies between the components? The term we use for this is KYS or "Know Your System". GamayunKYS takes care of showing the architecture of the ecosystem and the communication between applications. The display gives us an overview of multi-level potential dependencies, even within a chosen time period. The key advantage is, of course, the representation of detected faults in the system.

Procedure definitions, critical path, and prediction

The definition of a procedure could be described as a sequence of steps or activities that are executed when a user or system action is taken, i.e. it is a procedure that is executed when one action is triggered. It can be simple communication between two applications, or it can be multi-level and involve several applications. An example would be an online purchase, where a click by the user triggers a series of sequential steps and activities between applications to complete the purchase. The definition of procedures is essential to know the critical path, which becomes critical the moment a complication occurs in the path. Without a clear overview, real-time detection of critical paths would be impossible. GamayunAI goes one step further by trying to predict in real-time that a critical path will happen, allowing us to prevent it. When it comes to preventing outages on critical infrastructure, such a system is indispensable.

We have come to the conclusion that GamayunAI is a complex but robust and transparent platform that uses machine learning algorithms to predict outages and improve the overall performance of our client's system. Despite the many challenges, we have demonstrated once again through the development of the GamayunAI platform that with a lot of research, the right approaches, and a good team, there is virtually no problem that cannot be solved.