zhaav-gemma3-4B

The alifzl/zhaav-gemma3-4B_q8_0.gguf model is a Persian specific model, fine tuned based on the Gemma 3 architecture. By utilizing QLoRA’s 4-bit quantization, it reduces computational demands while delivering strong performance in generating and understanding Persian text. Thus it is suitable for running on commodity hardware with no GPUs.

Usage

This model is compatible with both the Hugging Face Transformers library and Ollama.

Running with Ollama

ollama run hf.co/alifzl/zhaav-gemma3-4B:Q8_0

Running with Hugging Face Transformers

1. Install Dependencies:

   pip install git+https://github.com/huggingface/[email protected] accelerate
   

2. Load Model and Tokenizer:

   from transformers import AutoModelForCausalLM, AutoTokenizer
   import torch
   
   model_id = "alifzl/zhaav-gemma3-4B_q8_0.gguf"
   
   model = AutoModelForCausalLM.from_pretrained(
       model_id,
       device_map="auto",  # Use "cuda" for GPU usage if available
       torch_dtype=torch.bfloat16,  # Alternatively, use torch.float16
   )
   tokenizer = AutoTokenizer.from_pretrained(model_id)
   
   messages = [
       {
           "role": "user",
           "content": "تفاوت قهوه موکا با آمریکانو چیه؟"
       }
   ]
   inputs = tokenizer.apply_chat_template(
       messages,
       add_generation_prompt=True, tokenize=True, return_tensors="pt"
   ).to(model.device)
   
   outputs = model.generate(**inputs, max_new_tokens=200)
   print(tokenizer.decode(outputs[0], skip_special_tokens=True))
   

Training Data and Fine-Tuning

Training Dataset

Fine-Tuning was made via mshojaei77/Persian_sft dataset, which contains approximately 680k rows of Persian text focused on instruction-following and conversational interactions.

Fine-Tuning

• Method: Supervised Fine-Tuning (SFT) using QLoRA (4-bit quantization)
• Hardware: one T4 GPU
• Software: Utilizes Hugging Face Transformers, with supporting libraries like peft for QLoRA and bitsandbytes for quantization

Evaluation Results

Metric	Value
Avg.	22.04
IFEval (0-Shot)	43.58
BBH (3-Shot)	31.87
MATH Lvl 5 (4-Shot)	11.10
GPQA (0-shot)	6.49
MuSR (0-shot)	9.49
MMLU-PRO (5-shot)	29.70

Future Work

• Additional evaluation metrics and benchmarks
• Expanded documentation and usage examples