About

About

Kushal Dabbe

LLM inference · ML systems · Computer vision

🗼 Tokyo, Japan

I build ML systems end-to-end. LLM agents, retrieval pipelines, and computer vision. My focus now is the LLM inference and training stack: serving optimization, distributed training, fine-tuning, and benchmarking. This site tracks the projects and notes from that work.

Milestones

kushaldabbe avatar
Merged
2 files changed +26 −3

Embedding LoRA was scaled down by embed_scale. On Transformers-backend models with a scaled input embedding (VaultGemma; Gemma with --model-impl transformers), the base embedding multiplied the lookup by embed_scale, but the LoRA delta was added without it, so an embed_tokens adapter had only 1 / embed_scale of its effect. PEFT scales the delta, and merged adapters are scaled by construction. The fix multiplies lora_b by embed_scale once in set_lora; the delta ratio goes from 0.0625 to 1.0000 and the max diff vs PEFT embeddings from 1.11 to 2.4e-07.

kushaldabbe avatar
Merged
2 files changed +47 −5

NaN poisoning in Triton attention. The softcap computed tanh via raw exponentials; scores above ~4400 overflowed to inf, so the ratio became inf/inf = NaN and the whole attention row died; reachable on Gemma-2 style models that use softcapping precisely because their logits are large. The fix rewrites tanh in a saturating form; the new test fails on main and passes with the fix, with 336 softcap tests green.

kushaldabbe avatar
Merged
2 files changed +56 −3

First merged contribution to vLLM. MoE expert packing recomputed LoRA scaling as alpha/rank and dropped the stored rsLoRA factor (alpha/√r), so adapters trained with use_rslora applied a delta √r times weaker - silently. The fix stores and reuses the scaling factor; the test suite went from 3 failed / 2 passed to 5 passed.

kushaldabbe avatar
Merged
2 files changed +34 −2

First merged contribution to SGLang. With attention DP on, the MegaMoE startup check divided chunked_prefill_size by attn_dp_size a second time, after handle_data_parallelism had already stored the per-rank value. With --chunked-prefill-size 16384 --attn-dp-size 2 the validator saw 4096 while each rank could receive 8192 tokens, so an undersized token cap passed startup and later failed the runtime assertion or fell back off the MegaMoE path. The fix uses the resolved value and adds a regression test.

Experience

  • Jan 2024 - Present · Tokyo, Japan
    Software Engineer · Rakuten Mobile
  • Aug 2023 - Dec 2023 · Remote (Delhi, India)
    NLP Engineer · Counsello AI
  • 2019 - 2023
    IIT (BHU) Varanasi
    B.Tech, Mechanical Engineering

Skills

LanguagesPython · C++ · C · SQL · JavaScript · Go
AI / LLMPyTorch · HuggingFace · RAG pipelines · LangChain · LangGraph · Pinecone/ChromaDB)
VisionYOLOv8 fine-tuning · Grounding DINO · SAM 2 · Depth Anything V2 · BoT-SORT tracking · pose estimation · OpenCV
ML optimizationONNX export · graph optimization · CoreML / CUDA deployment · mAP / IoU evaluation
InfraFastAPI · Next.js · Spark · Airflow · AWS · Docker

Currently focused on

TrainingDDP · FSDP · DeepSpeed · mixed precision
Fine-tuningLoRA · QLoRA · SFT · DPO · dataset curation
InferencevLLM · SGLang · TensorRT-LLM · quantization
ServingTriton · Kubernetes
Evalslm-eval-harness · benchmark design