Open Source
Explore the latest AI open-source projects from GitHub and HuggingFace.
Explore the latest AI open-source projects from GitHub and HuggingFace.
FlashRT is a high-performance real-time inference engine built for small-batch, latency-sensitive AI workloads, a niche general-purpose serving stacks are not shaped for: TensorRT compiles to frozen engines for throughput, while vLLM and SGLang target high-batch LLM serving. FlashRT instead composes hand-written CUDA kernels (normalization, activation, fused residual+norm+quant, RoPE, FP8/NVFP4 GEMM, vendored Flash-Attention 2) into a statically captured CUDA graph, avoiding ONNX export or per-driver compile steps, with one code path running across Jetson AGX Thor, RTX 4090, and RTX 5090. Its flagship integration is real-time VLA (vision-language-action) control for robotics policies including Pi0, Pi0.5, GROOT N1.6, GROOT N1.7, and Pi0-FAST, where the project's benchmarks show per-decision latency on a Jetson AGX Thor dropping from roughly 266 ms on the unmodified host to about 29 ms with FlashRT for a Pi0.5 LIBERO task, raising the policy loop from under 4 Hz to over 34 Hz. The same kernels back NVFP4-quantized LLM serving for Qwen3.6-27B/Qwen3-8B, Wan2.2 video generation, and Higgs Audio v3 TTS. FlashRT Structures can attach the kernel catalog onto an unmodified PyTorch host such as lerobot, Isaac-GR00T, openpi, or transformers without forking it. An August 2026 update added a ROCm/HIP backend for AMD Instinct MI350X GPUs. The project accompanies a published paper and ships under Apache 2.0.