.jpg)
Testing data and training data are two of the main pillars of the machine learning process. Without one there can not be the other. In machine learning, an unknown universal dataset is assumed to exist, which contains all the possible data pairs as well as their probability distribution of appearance in the real world. When we are dealing with real applications, what we observe is only a subset of the universal dataset. This acquired dataset is called the training set and used to learn the properties and knowledge of the universal dataset.
In machine learning, what we desire is that these learned “properties” can not only explain the training set, but also be used to predict unseen samples or future events. In order to examine the performance of learning, another dataset may be reserved for testing, called the test set.
For example, before final exams, the teacher may give students several questions for practice (training set), and the way he judges the performances of students is to examine them with another problem set (test set). That is why you must split your data set into a training and a testing data set.
What are training data and testing data
Training data is a set of samples (such as a collection of photos or videos) used to train machine learning models. Training datasets are fed to machine learning algorithms, in order to learn. They are necessary to teach the algorithm how to make accurate predictions in accordance with the goals of an AI project.
Just like people learn better from examples, machines also need to start isolating patterns in data. Unlike human beings, however, computers need a lot more examples because they do not think in the same way as humans do. They do not see objects in the pictures or can not recognize people in the photos as we can. They speak their own programming languages.
Testing data, as the name suggests, helps you validate the progress of the algorithm’s training and adjust or optimize it for improved results.The testing dataset is a subset of the training initial one, and it is “shown” to the model just after it has completed its training. It is very important to keep the test dataset separate from the training one. It is used to provide an unbiased evaluation of the performance of a model and ensure that it can generalise well to new, unseen data.
In simple words, when collecting a data set that you’ll be using to train your algorithm, you should keep in mind that part of the data will be used to check how well the training goes. This means that your data will be split into two parts: one for training and the other for testing.
Why do we need train and test sample
Creating different data samples for training and testing the model is the most common approach that can be used to identify these sorts of issues. The simplest way to split the modelling dataset into training and testing sets is to assign two thirds of the data to the former and the remaining one-third to the latter. Therefore, we train the model using the training set and then apply the model to the test set.
In this way, we can evaluate the performance of our model. For instance, if the training accuracy is extremely high while the testing accuracy is poor then this is a good indicator that the model is probably overfitted.
Overfitting and underfitting: what they are
A very common issue when training a model is overfitting. This phenomenon occurs when a model performs really well on the data that we used to train it but it fails to generalise well to new, unseen data. There are numerous reasons why this can happen — it could be due to the noise in data or it could be that the model learned to predict specific inputs rather than the predictive parameters that could help it make correct predictions. Typically, the higher the complexity of a model the higher the chance that it will be overfitted.
On the other hand, underfitting occurs when the model has poor performance even on the data that was used to train it. In most cases, underfitting occurs because the model is not suitable for the problem you are trying to solve. In general, an underfit model will be less flexible and cannot account for the data.
Creating different data samples for training and testing the model is the most common approach that can be used to identify these sorts of issues. The simplest way to split the modelling dataset into training and testing sets is to assign two thirds of the data to the former and the remaining one-third to the latter. Therefore, we train the model using the training set and then apply the model to the test set.
In this way, we can evaluate the performance of our model. For instance, if the training accuracy is extremely high while the testing accuracy is poor then this is a good indicator that the model is probably overfitted.
Conclusions
To summarize, in a dataset a training set is implemented to build up a model, while a test set is to validate the model built. In the processes of training and testing there could be problems that you want to avoid. Make sure not to undertrain (underfitting) or overtrain (overfitting) your algorithm, because this will impact the testing, and hand you back inaccurate or irrelevant predictions.
Also keep in mind your work doesn’t stop after you’ve trained your machine learning algorithm. Fine-tuning and maintenance of the model is a non-stop task, since the innovative power of machine learning models is in the fact that they learn and improve over time.
