How to Use K-Fold Cross Validation in Python: A Comprehensive Guide

Cross-validation is an essential technique in machine learning that helps in evaluating the performance of predictive models. It enables the estimation of the accuracy of a model and determines how well the model would generalize to new data. One of the most popular cross-validation techniques is the k-fold cross-validation. In this article, we will delve into what k-fold cross-validation is, how it works, and how to use it in Python.

Table of Contents

What is k-fold cross-validation?

K-fold cross-validation is a resampling technique used to assess the performance of a machine learning model. It involves splitting the data into k subsets or folds of equal sizes. The model is then trained on k-1 subsets and validated on the remaining subset. This process is repeated k times, with each subset serving as the validation set once. The k results are then averaged to produce a single estimation.

The value of k is usually chosen such that k is a positive integer that is less than or equal to the total number of observations in the dataset. A common value for k is 10.

Why use k-fold cross-validation?

One of the main advantages of k-fold cross-validation is that it provides a more accurate estimate of the model’s performance than a simple train-test split. In a train-test split, the data is divided into a training set and a test set. The model is trained on the training set and evaluated on the test set. However, this approach can lead to overfitting, where the model fits the training data too well and performs poorly on new data.

K-fold cross-validation reduces the risk of overfitting by training and testing the model on different subsets of the data. It also provides a more robust estimate of the model’s performance as it uses multiple iterations to evaluate the model.

How does k-fold cross-validation work?

The k-fold cross-validation process can be broken down into the following steps:

  1. Shuffle the dataset randomly.
  2. Split the dataset into k equal subsets or folds.
  3. For each fold, train the model on k-1 folds and validate it on the remaining fold.
  4. Repeat this process k times, with each fold serving as the validation set once.
  5. Calculate the average performance of the model across all k folds.

The figure below illustrates how the k-fold cross-validation process works:

How to implement k-fold cross-validation in Python

To implement k-fold cross-validation in Python, we will use the scikit-learn library, which provides a convenient function for this purpose. Here’s how to do it:

Step 1: Import the necessary libraries

import pandas as pd
from sklearn.model_selection import KFold
from sklearn.linear_model import LinearRegression

Step 2: Load the dataset

data = pd.read_csv('dataset.csv')
X = data.iloc[:, :-1] # features
y = data.iloc[:, -1] # target variable

Step 3: Initialize the k-fold object

kf = KFold(n_splits=10, shuffle=True, random_state=42)

Here, we have initialized the k-fold object with 10 splits, shuffled the data randomly, and set a random seed of 42 for reproducibility.

Step 4: Train and test the model using k-fold cross-validation

model = LinearRegression()
scores = []
for train_index, test_index in kf.split(X):
    X_train, X_test = X.iloc[train_index], X.iloc[test_index]
    y_train, y_test = y.iloc[train_index], y.iloc[test_index], y_train)
    score = model.score(X_test, y_test)

print(f'Average R-squared score: {sum(scores)/len(scores):.2f}')

In this step, we have trained and tested the linear regression model using k-fold cross-validation. We have used the split() method of the k-fold object to get the indices of the training and testing sets for each fold. We have then trained the model on the training set and evaluated it on the testing set. We have calculated the R-squared score for each fold and stored it in a list. Finally, we have calculated the average R-squared score across all folds.


In this article, we have covered the basics of k-fold cross-validation, its advantages over a simple train-test split, and how to implement it in Python using the scikit-learn library. K-fold cross-validation is a powerful tool that can help in improving the accuracy and robustness of machine learning models. By using k-fold cross-validation, we can evaluate the performance of a model more accurately and reduce the risk of overfitting.

Leave a Comment

Your email address will not be published. Required fields are marked *