Edge inference, model serving and optimisation.

Technologies

• Embedded and on-device inference Small models on microcontrollers and sensor boards that classify or detect without sending raw data.
• Edge server inference Vision, language and sensor models served from far and near edge servers at bounded latency.
• Quantisation INT8, INT4, FP8 and block-scaled formats that cut memory and power with measured accuracy loss.
• Distillation, pruning and sparsity Training smaller student models and removing redundant weights so models fit edge hardware.
• Compilers and runtimes Graph compilers and portable model formats that target many accelerator types from one model.
• Serving: batching, caching and scheduling Request batching, key-value caching and admission control that raise throughput on shared accelerators.
• Parameter-efficient fine-tuning Adapting large models to a site, language or sensor with small trainable adapters.
• Energy-aware inference Measuring and reducing energy per inference as a design target alongside accuracy and latency.
• Split and collaborative inference Dividing a model between device, edge and cloud to balance latency, bandwidth and privacy.
• Safety-related inference Inference inside functions where failure can cause harm, needing functional safety methods.