a-r-r-o-w/input-data-share
Updated
Aryan V S
a-r-r-o-w
AI & ML interests
computer vision, reinforcement learning
Recent Activity
updated
a model
10 days ago
a-r-r-o-w/input-data-share
published
a model
10 days ago
a-r-r-o-w/input-data-share
upvoted
an
article
17 days ago
🕳️ Attention Sinks in LLMs for endless fluency
Organizations
updated
a
model
10 days ago
published
a
model
10 days ago
upvoted
an
article
17 days ago
Article
🕳️ Attention Sinks in LLMs for endless fluency
•
32
reacted to
gokaygokay's
post with 🔥
about 1 month ago
Post
5064
FlashPack: Lightning-Fast Model Loading for PyTorch
https://github.com/fal-ai/flashpack
FlashPack — a new, high-throughput file format and loading mechanism for PyTorch that makes model checkpoint I/O blazingly fast, even on systems without access to GPU Direct Storage (GDS).
With FlashPack, loading any model can be 3–6× faster than with the current state-of-the-art methods like accelerate or the standard load_state_dict() and to() flow — all wrapped in a lightweight, pure-Python package that works anywhere.
https://github.com/fal-ai/flashpack
FlashPack — a new, high-throughput file format and loading mechanism for PyTorch that makes model checkpoint I/O blazingly fast, even on systems without access to GPU Direct Storage (GDS).
With FlashPack, loading any model can be 3–6× faster than with the current state-of-the-art methods like accelerate or the standard load_state_dict() and to() flow — all wrapped in a lightweight, pure-Python package that works anywhere.
- 2 replies
Add images/videos for productionizing diffusion articles
#555 opened 2 months ago
by
a-r-r-o-w
Add images/videos for productionizing diffusion articles
#554 opened 2 months ago
by
a-r-r-o-w
upvoted
an
article
3 months ago
Article
Make your ZeroGPU Spaces go brrr with ahead-of-time compilation
- +2
•
70
liked
a
model
3 months ago
upvoted
a
collection
4 months ago
upvoted
an
article
4 months ago
Article
From Zero to GPU: A Guide to Building and Scaling Production-Ready CUDA Kernels
•
88
liked
2
models
4 months ago
updated
a
model
4 months ago
published
a
model
4 months ago
upvoted
an
article
4 months ago
Article
Say hello to `hf`: a faster, friendlier Hugging Face CLI ✨
- +1
•
83
liked
2
models
4 months ago
replied to
their
post
4 months ago
Thanks for the feedback Sayak! I will be doing a live implementation where I walk through all the optimizations, hopefully in a few weeks. I mean to use these posts to scatter out the overall ideas, but I realize it's probably too vague. Will improve on this
posted
an
update
4 months ago
Post
2324
You would've implemented the 3-loop matrix multiplication many times as a ML practitioner, but the naive implementation is terrible for GPU performance. Modern GPUs achieve peak performance through careful memory access patterns and minimizing scheduling overhead.
In naive matmul (MxK . KxN), the computation happens in tiles - both for the output matrix and for how you read chunks from the input matrices. Each thread-block processes one output tile by loading corresponding tiles from input (for sum-reduction across K dimension), performing the computation, then terminating. The GPU launches many thread-blocks and schedules them across available streaming multiprocessors (SMs). When an SM finishes one tile, it gets assigned a new thread-block for the next uncomputed tile. This way, multiple output tiles are computed in parallel across the SMs, but we pay the cost for launching thread-blocks each time a new tile is computed.
Persistent matmul changes this approach. Instead of launching thread-blocks to compute some output tiles, computing the results on SMs in parallel, and repeating until all output tiles are computed, you launch only as many thread-blocks as you have SMs available (typically 80-132 on modern GPUs). These thread-blocks stay alive until all output tiles are computed, looping through multiple tiles sequentially. Each persistent thread-block may handle multiple output tiles.
The key benefit is the reduced thread-block launch latency. This persistence strategy, combined with other optimizations like coalesced memory loads/stores, block-tiling, warp-tiling, warp-specialization, double-buffering, ping-pong scheduling and other tricks, helps achieve peak performance. More on this in the future!
Code snippet for testing: https://gist.github.com/a-r-r-o-w/28339b442d164084506c0967029968a8
(Bonus: Since I've wanted to learn Manim for a while, this was a great opportunity to make a visualization for Naive VS Persistent matmul. Enjoy ✨)
In naive matmul (MxK . KxN), the computation happens in tiles - both for the output matrix and for how you read chunks from the input matrices. Each thread-block processes one output tile by loading corresponding tiles from input (for sum-reduction across K dimension), performing the computation, then terminating. The GPU launches many thread-blocks and schedules them across available streaming multiprocessors (SMs). When an SM finishes one tile, it gets assigned a new thread-block for the next uncomputed tile. This way, multiple output tiles are computed in parallel across the SMs, but we pay the cost for launching thread-blocks each time a new tile is computed.
Persistent matmul changes this approach. Instead of launching thread-blocks to compute some output tiles, computing the results on SMs in parallel, and repeating until all output tiles are computed, you launch only as many thread-blocks as you have SMs available (typically 80-132 on modern GPUs). These thread-blocks stay alive until all output tiles are computed, looping through multiple tiles sequentially. Each persistent thread-block may handle multiple output tiles.
The key benefit is the reduced thread-block launch latency. This persistence strategy, combined with other optimizations like coalesced memory loads/stores, block-tiling, warp-tiling, warp-specialization, double-buffering, ping-pong scheduling and other tricks, helps achieve peak performance. More on this in the future!
Code snippet for testing: https://gist.github.com/a-r-r-o-w/28339b442d164084506c0967029968a8
(Bonus: Since I've wanted to learn Manim for a while, this was a great opportunity to make a visualization for Naive VS Persistent matmul. Enjoy ✨)
-
-
- 3 replies
upvoted
an
article
4 months ago
Article
Your Own GPU-Powered Image Generator with HF Jobs
•
25