Modeling & Simulation of Computational Systems for Process Scheduling in Resource Orchestration Scenarios

Abstract

The rapid evolution and adoption of Large Language Models in real-time applications have highlighted inference request scheduling as a critical factor in the performance of serving systems. Due to the autoregressive nature of text generation, the final length of a response is not known in advance, resulting in requests with radically different computational demands competing for the same resources. Despite this heterogeneity, several modern serving systems continue to route requests on a first-come basis, exposing the infrastructure to the phenomenon of head-of-line blocking. This diploma thesis studies, models, and evaluates dynamic scheduling policies to effectively address the aforementioned problem. Instead of a direct implementation on physical hardware, this work leverages the rigorous mathematical formalism of Petri Nets. Specifically, Stochastic, Coloured, and Timed-Arc Petri Nets are utilized within the TAPAAL tool environment to develop four models representing different scheduling approaches: traditional non-preemptive policies and dynamic, multi-level preemptive policies . Through Statistical Model Checking , the models were evaluated across a sequence of four experiments that covered performance under eight workload scenarios, scalability with respect to available capacity, average response latency, and the verification of structural correctness properties. The results demonstrate that the determining performance factor is the capability of preemption rather than the prior knowledge of request sizes. Multi-level policies more than doubled the completion rate compared to non-preemptive policies, utilized every additional computational resource in a near-linear fashion, and drastically reduced average latency. Overall, this thesis provides a theoretical framework for the study of scheduling models and proves that dynamic preemption constitutes a necessary condition for the sustainable and scalable operation of modern LLM infrastructures.