thomasip/Qwen3-Omni-30B-A3B-Instruct-GPTQ-4bit

Qwen3-Omni

This is a 4-bit GPTQ quant for Qwen/Qwen3-Omni-30B-A3B-Instruct.

Quantized using the following script:

from datasets import load_dataset
from gptqmodel import GPTQModel, QuantizeConfig

model_id = "Qwen/Qwen3-Omni-30B-A3B-Instruct"
quant_path = "Qwen3-Omni-30B-A3B-Instruct-GPTQ-4bit"

calibration_dataset = load_dataset(
    "allenai/c4",
    data_files="en/c4-train.00001-of-01024.json.gz",
    split="train"
  )

calibration_dataset = calibration_dataset.filter(lambda x: 256 <= len(x["text"]) <= 2048)
calibration_dataset = calibration_dataset.select(range(1024))["text"]

quant_config = QuantizeConfig(bits=4, group_size=128)

model = GPTQModel.load(model_id, quant_config)

model.quantize(calibration_dataset, batch_size=8)

model.save(quant_path)