What is self-supervised learning, and how is it different from unsupervised learning?
Self-supervised learning creates its own labels from the data, such as hiding a word and predicting it. Unsupervised learning has no target at all and looks for structure like clusters. Large language models are pre-trained in a self-supervised way.
Explain it simply
Think of a fill-in-the-blanks exercise made from any newspaper. Cover one word in each sentence and ask the student to guess it. The answer is already printed on the page, so nobody had to label anything by hand, yet the student learns a lot of language.
That is self-supervised learning: the data supplies its own answers. The task is still "predict a target from inputs", just with a target built automatically.
Example
Text : "The train to Mumbai leaves at 6 pm"
Input : "The train to ____ leaves at 6 pm"
Target : "Mumbai" (taken from the original sentence)
Image : cut a photo into patches, hide some
Target : the hidden patches
Audio : mask a short segment, predict itCompare with unsupervised learning, such as k-means, where there is no prediction target and the output is groupings or a lower-dimensional view.
The usual pipeline is: pre-train on huge unlabelled data with a self-supervised task, then fine-tune on a small labelled dataset for the real job, for example sentiment classification.
What to say in the interview
"Self-supervised learning builds labels from the data itself, so it can use huge unlabelled datasets. It behaves like supervised learning during training but needs no human annotation. That is how models such as BERT and GPT get general language knowledge before being fine-tuned."
Common mistakes
- Saying it is the same as clustering.
- Thinking no loss function is involved; there is one, computed against the automatic target.