Photo by editor
# Introduction
Machine learning Systems, in essence, consist of models—such as decision trees, linear registers, or neural networks, among many others—that are trained on a set of Data Examples of learning a series of patterns or relationships, for example, to predict the price of an apartment in sunny Seville (Spain) based on its attributes. But the quality or performance of a machine learning model on this task largely depends on its own “appearance” or “shape”. Even two models of the same type, for example, two linear regression models, depend on one important aspect: their parameters depend.
This article eliminates the concept of a Parameter in Machine Learning Models and explains what they are, how many parameters a model has (spoiler alert: it depends!), and what can go wrong when setting model parameters during training. Let’s explore these basic ingredients.
# Eliminating parameters in machine learning models
Parameters are like internal Dials and knobs of a machine learning model: They define the behavior of your model. Just as a barista’s coffee machine depends on the quality of the coffee beans it grinds, the parameters of a machine learning model are set differently by the nature—and, to a large extent, by the quality—of the training data examples used to learn to perform a task.
For example, in the case of predicting apartment prices, if the training dataset of apartment examples with known prices contains noisy, irrelevant or biased information, the training process can yield a model whose parameters (remember, internal settings) capture misleading patterns or input-output relationships, resulting in poor price forecasts. Meanwhile, if the dataset contains clean, representative and high-quality examples, chances are that the training process will produce a model whose parameters are fine-tuned with real factors that influence high or low housing prices, leading to great predictions.
Notice that now I used italics to emphasize the word “internal“Many times? That was purely intentional and necessary to distinguish between machine learning model parameters and hyperparameters. Compared to parameters, a hyperparameter in a machine learning model is like a dial, a knob, or even a button or switch. Externally and manually adjusted (not learned from data), usually by a human but also as a result of a search process to find the best setting of relevant hyperparameters in your model. You can learn more about hyperparameters here This machine learning master article.
Parameters are like the internal dials and knobs of a machine learning model – they define the “personality” or “behavior” of the model, that is, the data it participates in, and to what extent.
Now that we have a better understanding of machine learning model parameters, some questions that arise are:
- Parameters look like?
- How many parameters are there in a machine learning model?
Parameters are usually numeric values, looking like weights that are between 0 and 1 in some model types, and can take any other real values in others. This is why the terms parameter and weight are often used to refer to the same concept in machine learning, especially in neural network-based models. The higher the weight, the more strongly this “knob” within the model affects the outcome or prediction. In simple machine learning models, such as linear regression models, parameters are associated with characteristics of the input data.
For example, suppose we want to predict the price of an apartment based on four attributes: size in square meters, proximity to the city center, number of bedrooms, and building age in years. A linear regression model trained for this predictive task will have four parameters – one linked to the input predictor – plus an additional parameter called a bias term (or intercept), which is not linked to any input feature of your data but is typically required to have more “freedom” to effectively learn from diverse data in multiple machine learning models. Thus, each parameter or weight value indicates the strength of influence of the input feature associated with it in the prediction process with that model. If the highest weight for “Proximity to city center” is one, it means that the price of an apartment in Seville is mostly affected by how far they are from the city center.
More generally, and in mathematical terms, the parameters in a simple model such as a multiple linear regression model are denoted by \(\Theta_i \) in an equation like this:
\(
\cap{y} = \Theta_0 + \Theta_1x_1 + \dot + \theta_x_n
\)
Of course, this is just the simplest type of machine learning model with a small number of parameters. As the complexity of the data increases, more sophisticated models such as support vector machines, random forest ensembles, or neural networks are usually required, which introduce additional layers of structural complexity to be able to learn challenging relationships and patterns. As a result, large models have a large number of parameters, which are now linked not only to the inputs, but to the complex and abstract interrelationships between the inputs that are stacked and built into the interior of the model. For example, a deep neural network can have hundreds to millions of parameters, and the largest machine learning models to date. Transformer architecture Behind large language models (LLMs) – they typically have billions of learning parameters!
# Learning the parameters and troubleshooting potential issues
When the training process of a machine learning model begins, the parameters are usually initialized as random values. The model makes predictions using examples of training data with predictable results, such as apartments with known prices, determining the error and adjusting some parameters accordingly to gradually reduce the errors. For example, machine learning models learn instance after instance: the parameters are slowly and iteratively updated during training, making them more consistent with the set of training examples that the model is exposed to.
Unfortunately, some difficulties and problems can arise in practice when training a machine learning model – in other words, while gradually adjusting the values of its parameters. Some common problems include: More than necessary and reducing its counterpart, and they are characterized by some finally learned parameters that are not at their optimal state, resulting in a model that may perform poor predictions. These problems can also partly arise from human choices, such as choosing a model that is too complex or too simple for the training data, that is, the number of parameters in the model is too small or too large. A model with too many parameters can be slow, expensive to train and use, and difficult to control if it degrades over time. Meanwhile, a model with too few parameters does not have enough flexibility to learn useful patterns from the data.
# wrap up
This article provided an explanation in simple and friendly terms about an essential element in Machine Learning Models: Parameters. They are like the DNA of your models, and understanding what they are, how they are learned, and how they relate to model behavior and performance is a critical skill toward becoming machine learning savvy.
Ivan Palomares Carrascosa Is a leader, author, speaker, and consultant in AI, Machine Learning, Deep Learning, and LLMS. He trains and guides others in real-world applications of AI.