openbox-llm

Punch above the VRAM weight class — drop-in kernels and memory layers so a single 24–32 GB GPU runs and trains larger effective models.

openbox-llm is a reproducible research program with one thesis: decouple a model's effective capability from what must sit in GPU VRAM. The first component is a clean-room implementation of native sparse attention (NSA) with custom Triton kernels — gated against a reference at every step and validated across GPU generations. Everything is Apache-2.0 and built to be read, not just imported.

Validated

• NSA kernels — forward, backward, and fused paths gated against a reference; validated Ampere→Hopper (sm_86→sm_90)
• +53% end-to-end training throughput from a query-tiling optimization
• −45% model VRAM; resident where dense attention runs out of memory
• Honest performance analysis, including the dominant training cost — see the paper

Get it

git clone https://github.com/msb-msb/openbox-llm

Roadmap

• NSA kernels — done
• Selection-backward optimization — next
• Trainable memory layers — planned
• Hot/cold offload — planned
• Knowledge injection vs. retrieval — planned