IST-DASLab

Code for the ICLR 2023 paper "GPTQ: Accurate Post-training Quantization of Generative Pretrained Transformers".

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.

Code for the ICML 2023 paper "SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot".

Code for the paper "QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models".

Quantized LLM training in pure CUDA/C++.

QuTLASS: CUTLASS-Powered Quantized BLAS for Deep Learning

Repository for the QUIK project, enabling the use of 4bit kernels for generative inference - EMNLP 2024

Code for the NeurIPS 2022 paper "Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning".

Gumbel-Softmax post-training quantization for LLMs (1–3 bit scalar, INT/GGUF-compatible).

Boosting 4-bit inference kernels with 2:4 Sparsity

Quartet II Official Code

Work in progress.

Code for data-aware compression of DeepSeek models

RCO: Optimization of Non-Decomposable Objectives under Exact Budget Constraints for LLM Quantization and Pruning

Efficient non-uniform quantization with GPTQ for GGUF

Statistically-Lossless Quantization of Large Language Models

Code accompanying the NeurIPS 2020 paper: WoodFisher (Singh & Alistarh, 2020)