← Profile · Projects & contributions · Medium
My research follows the whole inference path: how a model becomes an executable artifact, how a runtime reaches the hardware, how memory moves, and how to measure the result fairly.
My edge-inference research began with object detection and grew into a C++20 framework for detection and RIFE frame interpolation. A stable execution-provider interface separates backend execution from model-family preprocessing and postprocessing. TensorRT, ONNX Runtime, ncnn, MNN, QNN, LiteRT, and IREE are evaluated through that architecture.
The coverage matrix spans ten providers and Windows, Jetson, and Android. Coverage is tracked per model, backend, platform, and precision; unsupported or unmeasured combinations remain explicit. A model that passes on one backend is not assumed to pass on another.
Public entry: Edge Inference Engine: One Native API Across CPU, GPU, NPU, and RISC-V. This is a research framework; no public standalone framework repository is linked here.
The inference kernel is only part of a video's cost. I investigate GPU-resident decode → interpolation → encode pipelines, Vulkan image interoperability, Android AHardwareBuffer import, and the work surrounding a model: format conversion, upload, readback, and CPU processing.
The important comparison holds weights, backend, hardware, and output conditions fixed while changing the pipeline architecture. I distinguish a resident inference core from a complete codec pipeline, and measure latency, CPU work, memory, and energy separately. Results vary by device and workload; a desktop gain is not automatically a mobile gain.
My LM lab is an experimental workspace, not a separate inference product. It studies and improves the existing community engine mudler/vllm.cpp, with llama.cpp and vLLM as reference engines. Selected implementation work is available in my port/vulkan branch and contribution record.
The work includes:
- A common HTTP benchmark client, fixed completion workloads, independent prompts, caching controls, fresh-server isolation, concurrency sweeps, and repeated measurements.
- Separate prefill and decode analysis: compute-bound GEMM, bandwidth-bound GEMV, and dispatch overhead require different experiments.
- Memory hierarchy: model weights, paged KV cache, step arenas, transfer staging, and backend memory classes.
- Vulkan cooperative-matrix paths and subgroup portability, optional Triton CPU dispatch, AVX-512 attention, and Windows/CUDA build support.
- External operation-level controls where useful. An ncnn GEMV microbenchmark or an empty-weight architecture benchmark is not evidence of complete ncnn LLM inference.
I study how IREE AOT modules, Triton-generated kernels, and hand-written RVV implementations reach new targets. The experiments include model conversion gaps, operation-level conformance, ISA/ABI/numerical checks, and the boundary between compiler code generation and vendor kernel libraries.
RISC-V work under QEMU establishes functional and numerical evidence only; it does not establish hardware throughput. A working operation or compiled artifact is reported separately from end-to-end model execution.
I start with a numerical reference, verify that the intended backend and optimization actually run, and control the comparison's model, precision, clocks, threads, caching, and timing boundary. I report startup separately from steady state, GPU timestamps separately from wall time, and total throughput separately from per-stream decode rate.
Surprising results lead to new controls. When a claim fails those controls, I record the correction and retain the failed hypothesis as part of the engineering story.