Fine‑Tune BERT for Low‑Resource Indian Languages HuggingFace

A quick story that starts with a broken sentence

I was staring at a tiny text file of 3 kB from a community‑sourced Marathi corpus, trying to make sense of why my BERT model was throwing a RuntimeError: Expected input batch size of 32 but got 1 every time I ran it. The culprit? I’d fed the raw text straight into a tokenizer that was built for English. That night, after a cup of chai, I decided to write a quick guide so that no one else has to go through the same 2 am debugging loop.

Understanding the Target Language and Dataset

When you’re dealing with a low‑resource Indian language, the first thing you need to do is understand the linguistic quirks and the data you have. Marathi, for example, has a rich morphology and a fairly large set of post‑positions that can change the meaning of a sentence drastically. If you ignore those, the model will just learn noise.

# Wrong: Treat Marathi like English
from datasets import load_dataset
dataset = load_dataset('csv', data_files='marathi_sentences.csv')

# What you see:
# TypeError: The dataset must have a column named 'text' or 'sentence'

The error happens because the default load_dataset expects a text column. The fix is simple: rename the column or tell the loader what to use.

# Right: Explicitly map the column name
dataset = load_dataset('csv', data_files='marathi_sentences.csv', split='train')
dataset = dataset.rename_column('marathi_text', 'text')

Tip: Keep an eye on the dataset schema. If you’re unsure, run dataset.column_names and double‑check that the field you’ll feed into the tokenizer exists.

Setting Up the Environment

I always start with a clean virtual environment. On my laptop I use conda because it handles CUDA versions nicely. For a BERT fine‑tune you’ll need at least PyTorch 2.0 and Transformers 4.36.

conda create -n bert-marathi python=3.11
conda activate bert-marathi
pip install torch==2.0.1 torchvision torchaudio
pip install transformers==4.36.2 datasets==2.14.4

# Wrong: Using an old transformer version that doesn’t support `tokenizer.pad_token_id`
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-multilingual-cased')

You’ll get an error like:

AttributeError: 'BertTokenizer' object has no attribute 'pad_token_id'

Fix it by pinning a newer version:

# Right: Use a recent version
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-multilingual-cased', use_fast=True)

Side note: If you’re on a GPU, make sure CUDA is correctly installed. Run torch.cuda.is_available() to confirm.

Preparing the Data

Tokenization is where most low‑resource pitfalls happen. The default BertTokenizerFast can handle Marathi, but you might want to add a few language‑specific tokens or adjust the max length.

# Wrong: Tokenizing without truncation or padding
encoded = tokenizer(dataset['text'], return_tensors='pt')
# This will raise:
# RuntimeError: Expected 32 samples in the batch, but got 1

Instead, pad to a fixed length and truncate longer sequences:

# Right: Pad and truncate
encoded = tokenizer(
    dataset['text'],
    return_tensors='pt',
    padding='max_length',
    truncation=True,
    max_length=128
)

After tokenization, convert the dataset to a torch.utils.data.DataLoader for efficient batching.

from torch.utils.data import DataLoader

train_loader = DataLoader(
    encoded,
    batch_size=32,
    shuffle=True,
    num_workers=4,
    pin_memory=True
)

Real‑world scenario: Imagine building a Marathi sentiment‑analysis API for a local e‑commerce platform. If your data loader mis‑handles padding, the API will return wrong predictions for short reviews, hurting user trust.

Tokenizer Customization

Sometimes the pretrained tokenizer doesn’t know how to split a new word that appears in your corpus. The trick is to add it to the tokenizer’s vocabulary and keep the embedding matrix small.

# Wrong: Adding a new token without resizing the embeddings
tokenizer.add_tokens(['नवीनशब्द'])
model.resize_token_embeddings(len(tokenizer))
# But you forgot to load the model first

You’ll see:

RuntimeError: size mismatch, got 30522, expected 30523

The right way is to first load the model, then add tokens, then resize:

# Right: Proper order
from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained('bert-base-multilingual-cased')
tokenizer.add_tokens(['नवीनशब्द'])
model.resize_token_embeddings(len(tokenizer))

Now the new token gets a random embedding that the model can fine‑tune.

Model Selection and Configuration

For low‑resource languages, a smaller model can sometimes generalize better because it has fewer parameters to overfit. I usually start with distilbert-base-multilingual-cased.

# Wrong: Using a large model with limited GPU memory
model = AutoModelForSequenceClassification.from_pretrained('bert-base-multilingual-cased')

You’ll hit out‑of‑memory errors or have to drop batch sizes to 4, which slows training drastically.

# Right: Use DistilBERT
from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(
    'distilbert-base-multilingual-cased',
    num_labels=2  # For binary sentiment
)

Set the training arguments:

from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir='./results',
    evaluation_strategy='epoch',
    learning_rate=2e-5,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=64,
    num_train_epochs=5,
    weight_decay=0.01,
    push_to_hub=False
)

Training the Model

I love using the Trainer API because it handles a lot of the boilerplate. The key is to pass a compute_metrics function for proper evaluation.

# Wrong: Forgetting to set `label_names`
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=encoded,
    eval_dataset=encoded
)

You’ll get:

ValueError: The dataset does not contain a column named 'labels'.

Add labels to the dataset and define metrics:

# Right: Add labels and metrics
from datasets import load_metric
metric = load_metric('accuracy')

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = logits.argmax(axis=-1)
    return metric.compute(predictions=preds, references=labels)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=encoded,
    eval_dataset=encoded,
    compute_metrics=compute_metrics
)

Run training:

trainer.train()

You’ll see logs like:

[2026-09-01 12:00:00] INFO - Training loss: 0.62
[2026-09-01 12:00:10] INFO - Evaluation accuracy: 0.78

Tip: If you hit RuntimeError: CUDA out of memory, reduce per_device_train_batch_size or use gradient accumulation.

Evaluation and Error Analysis

After training, the Trainer will give you an overall accuracy, but that’s the “nice” number. The real value is in looking at misclassified samples.

# Wrong: Only print the overall score
print(trainer.evaluate()['eval_accuracy'])

You’ll get a single float that hides the nuances.

# Right: Inspect individual errors
eval_results = trainer.evaluate()
print(f"Accuracy: {eval_results['eval_accuracy']:.2%}")

predictions = trainer.predict(encoded)
for i, (pred, true) in enumerate(zip(predictions.predictions.argmax(-1), predictions.label_ids)):
    if pred != true:
        print(f"Sample {i} misclassified: {encoded['text'][i]}")

You’ll see something like:

Sample 42 misclassified: मला वाटतं की हे उत्पादन चांगलं नाही

Now you can decide whether to add more data, tweak the tokenizer, or adjust the class weights.

Deploying the Fine‑Tuned Model

When the model looks good, it’s time to serve it. I usually go with FastAPI and Docker for portability. The trick is to keep the tokenizer and the model in the same container.

# Wrong: Trying to load the tokenizer from a relative path that doesn’t exist
from fastapi import FastAPI
app = FastAPI()

@app.post("/predict")
async def predict(text: str):
    tokens = tokenizer(text, return_tensors='pt')
    logits = model(**tokens).logits
    return {"prediction": logits.argmax().item()}

You’ll get a 500 error at runtime:

FileNotFoundError: [Errno 2] No such file or directory: 'tokenizer.json'

Make sure you copy the tokenizer files into the image.

# Right: Dockerfile snippet
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY ./model /app/model
COPY ./tokenizer /app/tokenizer
COPY main.py /app
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

After building the image:

docker build -t marathi-bert .
docker run -p 8000:8000 marathi-bert

You can test the endpoint with curl or the API Tester. The response should look like:

{
  "prediction": 1
}

Real‑world deployment scenario: A startup in Karnataka wants a quick Marathi chatbot for customer support. The API must handle thousands of requests per minute, so packaging the model with FastAPI in Docker gives you a consistent environment across dev, staging, and production.

Wrapping Up

Fine‑tuning BERT for low‑resource Indian languages is a lot like learning a new dialect: you have to pay attention to the small differences that matter most. By carefully preparing the data, customizing the tokenizer, choosing the right model size, and rigorously evaluating, you can turn a handful of sentences into a robust sentiment classifier or any other downstream task.

If you’re curious about how to handle even smaller vocabularies or want to experiment with other transformer families, feel free to explore the Hugging Face model hub. And remember, the first time you hit a weird error, it’s usually a small oversight just like that time I forgot to add a padding token and the whole pipeline crashed.

Happy training!

FAQs

Q1: Why does my training crash with “CUDA out of memory” even though I have 8 GB of GPU?
A1: The default batch size of 32 for DistilBERT may still be too high. Try per_device_train_batch_size=16 or enable gradient accumulation with gradient_accumulation_steps=2.

Q2: How can I add my own Marathi stop‑words to the tokenizer?
A2: Use tokenizer.add_tokens(['तुम्ही', 'आम्ही']) before resizing embeddings. Don’t forget to update the tokenizer.pad_token if you’re adding special tokens.

Q3: My evaluation accuracy is low; should I increase the number of epochs?
A3: Often the issue is overfitting or class imbalance. Check the eval_loss trend; if it’s increasing while training loss decreases, you’re overfitting. Try num_train_epochs=3 and add class_weight or weight_decay.

Q4: How do I deploy the model on a serverless platform?
A4: Pack the model into a Lambda layer or an Azure Function, ensuring you include the tokenizer files. For AWS, you can reference the model from S3 and load it at cold‑start time. See also the article on Zero‑Trust Architecture in Serverless Environments for security considerations.

Q5: Can I use a multilingual BERT trained on Hindi to help with Marathi?
A5: Yes, cross‑lingual transfer works surprisingly well. Just fine‑tune the Hindi‑trained BERT on your Marathi data; you’ll likely see faster convergence and better generalization.