Low accuracy (~50%) with SSL (BYOL/MAE/VICReg) on hyperspectral crop stress data — what am I missing? [R]

I’m working on a hyperspectral dataset of cabbage crops for nitrogen deficiency detection. The dataset has 3 classes:

Healthy

Mild nitrogen stress

Severe nitrogen stress

I’m trying to use self-supervised learning (SSL) for representation learning and then fine-tune for classification.

What I’ve done:

Tried multiple SSL methods: BYOL, MAE, VICReg

Used data augmentation (spectral noise, masking, scaling, etc.)

Fine-tuned with a classifier head

Evaluated using accuracy and F1-score

Problem:

No matter what I try, the performance is stuck around:

Accuracy: ~45–50%

F1-score: also low (~0.5)

This is barely better than random (since 3 classes ≈ 33%).

My setup:

Hyperspectral data (hundreds of bands)

1D/patch-based model (ViT-style)

SSL pretraining → fine-tuning pipeline

Tried k-NN and linear probe as well (still weak)

What I suspect:

Classes might not be well separable spectrally

SSL methods designed for RGB may not adapt well

Augmentations might be hurting instead of helping

Model not capturing spectral-specific patterns

What I’m looking for:

Would really appreciate suggestions on:

Better SSL methods for hyperspectral data

Is VICReg actually the best choice here?

Should I try masked spectral modeling instead?

Feature engineering

Should I include vegetation indices (NDVI, etc.)?

PCA before training?

Model architecture

1D CNN vs ViT vs hybrid?

Any proven architectures for hyperspectral?

Evaluation

Best way to validate SSL representations?

Any tricks to improve linear probe results?

General advice

Anyone worked on plant stress / hyperspectral classification?

Common