Dynamic Scheduling of Real-Time Mixture-of-Experts Systems on Limited Resources.

A Mixture-of-Experts (MoE) system generates an output in each operating cycle by combining results of multiple models (the “experts”). The contribution of any given expert to a final solution depends on a parameter called responsibility, which can vary from cycle to cycle. When resources are insufficient to run all experts, two problems arise: 1) how much utilization is to be allocated to experts and 2) how can a schedule be created based on these allocations. Problem (1) can be formulated as a succession of optimization problems, each of which calculates experts’ allocations in a cycle. Explicit mappings from responsibilities to allocation weights are needed to solve each of these problems in every cycle using a technique called “task compression (TC).” We refer to this baseline approach as TT-TC. Two other proposed heuristics \ssr TT\-\ssr TC^\ast and TT-Top reduce TC’s execution time to \ssr O(\mbiN) for \mbiN experts. To address (2), the proposed EPOC scheduler converts the heuristics’ allocations into schedules that satisfy capacity, execution, and learning constraints across cycles. Simulations demonstrate that our approaches enable real-time computation and significantly decrease the average percentage error of limited-resource outputs (i.e., 0.2%–40% and 0.3%–0.5% when scheduled with \ssr TT\-\ssr TC^\ast and TT-Top, respectively, versus 0.2%–97% when using TT-TC). [ABSTRACT FROM PUBLISHER]