One-sentence definition
Fase inicial de entrenamiento a gran escala que aprende patrones generales antes de ajustes o usos especializados.
Definition: What Does It Mean to Pretrain?
Pretraining is a training stage in which a model is trained before being used for certain tasks or adapted for a later use. The word “pre” describes the stage’s position relative to a later stage or purpose; it does not mean that this is always the first training the system has received, or that every model must follow the same sequence.
The U.S. National Institute of Standards and Technology (NIST) glossary defines pretraining in machine learning as a training step in which a general-purpose model is trained on publicly available data; it adds that this model is sometimes called a foundation model. This is a concise and useful definition, but it does not describe every possible variation or establish that all pretraining uses public data. Nor does it allow us to conclude that “foundation model” and “pretraining” are synonyms: one may name the model, while the other names a stage of its training.
Put simply, a system receives data and is trained according to a particular objective. The result may be a model that is reused, adapted, or evaluated on a task. The data, objective, architecture, and method of reuse vary across projects. So saying that a model is “pretrained” tells us something about its training history, but it is not enough to know what it learned, what data it used, how well it performed, or which situations it is suited to.
The General Outline: Data, Objective, and Reuse
A practical way to understand this stage is to separate four questions. First: what data are presented to the model? Second: what learning objective guides training? Third: what change is the training intended to produce in the model? And fourth: what will be done with the result afterward? Answering these questions helps avoid treating “pretraining” as if it named a specific algorithm.
In some systems, the objective may involve predicting or reconstructing parts of the data; in others, the model may learn a representation that can later be used in another way. These are general possibilities, not an exhaustive list and not a claim that all pretraining works this way. For example, the work on unsupervised generative language pretraining presents a particular strategy for that domain; its title and description do not justify turning that strategy into a definition of the whole field.
The result may serve as a starting point for later adaptation, evaluation, or other training stages. Pretraining can even be continuous or multimodal: a NeurIPS guide explicitly addresses continual multimodal pretraining. This qualifies the introductory account of “an initial stage followed by fine-tuning”: that is one possible outline, not a mandatory sequence.
The simplified process below is a conceptual map. It does not specify hyperparameters, dataset sizes, or a technical sequence that applies to every model.
A conceptual map, not a recipe
This sequence represents decisions commonly discussed when describing pretraining. In a real system, some may be repeated, reordered, or omitted.
- 01Define the data family and the capability to be learned or improved.
- 02Choose a training objective suited to the architecture and intended use.
- 03Train the model and evaluate the result; training duration or data volume alone does not demonstrate quality.
- 04Decide whether to reuse the model, adapt it through another stage, or use it in an evaluation or a specific application.
Three Examples That Help Explain the Term
The following examples are explanatory scenarios. They show how a description changes with the modality, but they do not claim to document a specific dataset or attribute a particular recipe to a project. The evidence supplied for this article supports a language example and the existence of research on clinical and multimodal pretraining; it does not provide enough detail to reconstruct the procedures used in each case.
blocks
Pretraining, Foundation Models, and Fine-Tuning
Pretraining names a training stage or process. “Foundation model” generally refers to a type of general-purpose model that can serve as the basis for later uses. NIST notes that a general-purpose model trained in a pretraining stage is sometimes called a foundation model. The phrase “sometimes called” matters: the wording does not establish an absolute equivalence between the terms.
Fine-tuning is a form of later model adaptation through additional training. A common sequence is to start with a pretrained model and continue training it to orient it toward a more specific task or context. However, not every pretrained model has to undergo fine-tuning, and not every later process must necessarily be described by that name. Lenovo’s secondary source presents pretraining as training on broad data and mentions later adaptation, but this explanation is not enough to establish a universal technical definition.
Self-supervised learning is not a synonym for pretraining either. “Pretraining” indicates the position or function of a stage; “self-supervised” describes a family of ways to construct learning signals from the data itself, rather than necessarily relying on labels prepared for each example. A system may be described using both terms when appropriate, but one does not automatically follow from the other. The example of unsupervised generative language learning documents one particular approach, not a rule for every modality.
Inference is the use of an already-trained model to produce an output from an input—for example, generating a response or producing a prediction. It is not the same as training: during inference, the immediate purpose is to use the model, not update it through a new learning stage. In practice, a product may combine inference with tools, information retrieval, or other components; therefore, observing a response alone does not reveal how the model was pretrained.
Distinguishing Related Concepts
The table summarizes the functional differences. A specific system may combine several stages; consult its technical documentation to classify it correctly.
| Term | What it describes | A useful question |
|---|---|---|
| Pretraining | A training stage that comes before a later purpose or adaptation. | What objective and data were used at this stage? |
| Foundation model | A general-purpose model that may serve as a basis for later uses; the term does not itself name a recipe. | What capabilities and uses are documented by its evaluation? |
| Fine-tuning | Additional training to adapt a model to a task or context. | What changed during adaptation, and what evidence was used to evaluate it? |
| Self-supervised learning | A way to obtain learning signals from data; it is not inherently a chronological stage. | Where do the signals that guide learning come from? |
| Inference | Using a model to produce an output from an input; this does not mean that the model is being trained. | What input did it receive, and which components contributed to the output? |
What Pretraining Does Not Allow Us to Conclude
The fact that a model is pretrained does not guarantee that it will respond correctly, be safe, work in every language or modality, or be suitable for a particular context. “Pretrained” describes a development history or stage; it is not a quality seal. Usefulness must be assessed through tests relevant to the intended use and with information about the model’s behavior.
Nor can quality be inferred from dataset size, training duration, or the fact that data are public alone. NIST’s definition mentions publicly available data, but that does not show that every stage uses this type of data or that availability guarantees representativeness, permission to use the data, freedom from errors, or good performance. These questions require specific evidence about the data and the system.
Dataset construction can affect results. A paper from the Association for Computational Linguistics on pretraining dataset construction addresses, among other matters, deduplication and filtering. This supports treating data preparation as an important issue, but it does not establish that there is a universal deduplication or filtering recipe that guarantees better results in every project.
It is also worth avoiding the claim that pretraining automatically produces general competence. The JMLR paper on the role of pretraining specifically frames an investigation into what role pretraining plays in continual learning. Its presence in the literature is a reason not to turn an intuition into a guarantee: capabilities and limitations must be established with evidence for the model and task in question.
Common Mistakes and Practical Criteria
A common mistake is to call any earlier learning “pretraining” without clarifying what later stage it precedes. Another is to assume that all models follow the same path: initial training, fine-tuning, and use. Research on continual multimodal pretraining shows that researchers also study processes that do not fit neatly into this linear outline.
Scale and quality are also easily confused: a large dataset does not, by itself, establish that the data are suitable or that the model will generalize. Likewise, a training objective should not be mistaken for a guarantee of capability. Claims about results should be supported by evaluations, not inferred solely from the name of a technique.
When reading a model card or comparing base models, check what “pretrained” means in that particular case. Look for the modality and type of data described; the learning objective; whether the stage is one-off or continuous; which later adaptations were made; and which evaluations support the stated capabilities. If this information is unavailable, the responsible conclusion is that the procedure cannot be determined from the label alone.
In short, pretraining can create a reusable foundation, but its specific function depends on the design and the available evidence. Distinguishing the stage, objective, model, and later use prevents us from attributing more information to the term than it actually provides.
Sources and Scope of This Article
The introductory definition is based on the NIST glossary. The academic work of the Association for Computational Linguistics is considered for the context of data construction; the article by Radford and colleagues supports the language example; and the JMLR and NeurIPS papers among the verified sources provide context for limits and variations. The article on clinically relevant pretraining supports the claim that biomedical research in this area exists, but the information provided for this article is not sufficient to detail its procedure.
For that reason, the vision example is explicitly presented as a conceptual illustration rather than a documented case. We also do not attribute results, datasets, or mechanisms to the papers unless those details are supported by the verified information available. This caution is especially important when comparing models: a source about one particular technique does not show that the technique is universal.