This happens because auto-generated numerical features that are based on categorical features are calculated differently for the training and validation datasets: Training dataset: the feature is calculated differently for every object in the dataset. For each i-th object the feature is calculated based on data from the first i-1 objects (the first i-1 objects in some random permutation). Validation dataset: the feature is calculated equally for every object in the dataset. For each object the feature is calculated using data from all objects of the training dataset.

Go deeper into our methodology for solving a business problem with machine learning