League: Can we use individual stats to predict if they won or lost?

20 Oct 2022 in Projects / Machine learning

Summary of Findings¶

Introduction¶

The prediction problem that we aim to answer in this project, is to predict whether an individual wins or loses based on their stats during the game.

This is a classification problem as each game can only be classified as a win or a lost. The metric that we decided to use to score our model, is accuracy. We chose accuracy as the knowledge of the false positives and the false negatives are not as important to the effectiveness of the model as compared to the accuracy.

Baseline Model¶

For our baseline model, we first looked at the overall dataset to determine which columns would have the most weightage to predicting a win or lost.

We selected 12 features columns, of which 2 are nominals and 10 are quantitative.

Nominals¶

• patch
  • For patch we OneHotEncoded
• position
  • For position we OneHotEncoded
• champion
  • For Champion we OneHotEncoded

Quatitative¶

• gamelength
• kills
• deaths
• assists
• teamkills
• teamdeaths
• total cs
• dpm(Damage per Minute)
• damagetakenperminute
• earned gpm

We then fit the features into a Decision Tree Classifier Pipeline, which yielded a accuracy of 0.96209. Just the baseline model it seems to be fairly accurate predicting results correctly 96.2% of the time.

Final Model¶

Additional Features¶

To improve the model, first we binarized dpm, damagetakenperminute and earned gpm as we didn't want overly highly/low values to weight heavily on the result. An example of how this can occur is when carries will tend to have higher dpm as they are the damage dealers in a team as compared to a non-carry position.

In the previous project we did an analysis on how kda (Kill, death and assists ratio) varies between roles so we thought it might have an impact on the likelihood of winning a game as well. Therefore we transformed the columns kills, deaths and assists into one column called kda.

Model and Hyperparameter Selection¶

To determine the best classifier model and the best hyperparameters for the prediction. We did a grid multiple gridsearches and compared the cross validation scores on all of them to select the best model.

Models and Cross validation scores

• Decision Tree
  1. Best Params
     • Criterion: Entropy
     • max_depth: None
     • min_sample_split: 4
  2. Cross Validation Score: 0.95716...
  3. Accuracy Score: 0.961708..
• Random Forest Classifier
  1. Best Params
     • Criterion: Entropy
     • max_depth: 15
     • min_samples_split: 4
     • n_estimators: 200
  2. Cross Validation Score: 0.94109...
  3. Accuracy Score: 0.94997...
• Logistical Regression
  1. Best Params
     • C: 1
     • max_iter: 100
     • penalty: l2
  2. Cross Validation Score: 0.95844...
  3. Accuracy Score: 0.957925...
• KNeighbors Classifier
  1. Best Params
     • n_neighbors: 5
     • weights: distance
  2. Cross Validation Score: 0.93053...
  3. Accuracy Score: 0.929379...

We chose Decision Decision with the best parameters as our final model, as looking at all the grid searches, it had the highest accuracy on test data and second best cross vaildation score losing out to Logistical Regression by a negligible margin.

Therefore we chose Decision Tree as our Final Model

Fairness Evaluation¶

For our fairness evaluation we chose the subset as 'High' teamkills and 'Low' teamkills to test whether the model is fair in terms of a players teams total kills or not. To determin the threshold for 'High' / 'Low' kills, we took the 75th percentile of all teamkills as the threshold.

The parity measure that we will be using is the accuracy score as we only care about correct predictions within each subset.

Permuation Test¶

Null: The model is fair, as the accuracy between the 2 subsets are roughly the same Alternative: The model is unfair, as the accuracy between theh 2 subsets are not equal.

As seen in the code, we fail to reject the Null. Which means that there is no biasness between 'High' teamkills and 'Low' teamkill meaning our model is fair.

Code¶

In [ ]:

import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
import seaborn as sns
from sklearn.preprocessing import FunctionTransformer, Binarizer, OneHotEncoder
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
%matplotlib inline
%config InlineBackend.figure_format = 'retina'  # Higher resolution figures

Baseline Model¶

In [ ]:

df = pd.read_csv('data/2022_LoL.csv')
df_clean = df[df['position'] != 'team']
df_clean = df_clean[df_clean['datacompleteness'] != 'partial']

In [ ]:

# Features that were determined were relevant
feats = df_clean[[
    'patch', 'position', 'champion', 'gamelength', 
    'kills', 'deaths', 'assists', 
    'teamkills', 'teamdeaths', 'total cs',
    'dpm', 'damagetakenperminute', 'earned gpm'
]]
results = df_clean['result']

# Train Test split
train_x, test_x, train_y, test_y = train_test_split(feats, results)
train_x.head()

Out[ ]:

	patch	position	champion	gamelength	kills	deaths	assists	teamkills	teamdeaths	total cs	dpm	damagetakenperminute	earned gpm
91704	12.11	top	Camille	1959	5	1	8	18	8	239.0	414.0582	739.2649	313.3538
115256	12.13	bot	Draven	1355	1	4	1	3	15	178.0	364.7380	390.5535	200.5461
144674	12.18	mid	Akali	1626	3	4	5	9	21	191.0	468.8192	771.2546	215.1661
141552	12.18	top	Ornn	2331	3	9	11	18	30	207.0	699.0991	1260.7979	203.5779
22016	12.03	bot	Aphelios	1994	2	0	11	18	9	314.0	583.7212	282.3671	327.1414

In [ ]:

# Preprocessing steps for model
preproc = ColumnTransformer([
    ('ohe_champs_pos', OneHotEncoder(sparse=False, handle_unknown='ignore'), ['champion', 'position', 'patch']),
], remainder='passthrough')

preproc.fit_transform(train_x)[:10]

Out[ ]:

array([[  0.    ,   0.    ,   0.    , ..., 414.0582, 739.2649, 313.3538],
       [  0.    ,   0.    ,   0.    , ..., 364.738 , 390.5535, 200.5461],
       [  0.    ,   0.    ,   1.    , ..., 468.8192, 771.2546, 215.1661],
       ...,
       [  0.    ,   0.    ,   0.    , ..., 287.4164, 692.8632, 253.8602],
       [  0.    ,   0.    ,   0.    , ..., 132.6464, 525.9766, 120.6464],
       [  0.    ,   0.    ,   0.    , ..., 531.1989, 546.1397, 257.6548]])

In [ ]:

# Baseline model using Decision Tree
from sklearn.tree import DecisionTreeClassifier

pl = Pipeline([
    ('preprocessing',  preproc),
    ('dt', DecisionTreeClassifier())
])

pl.fit(train_x, train_y)
pl.predict(test_x)
pl.score(test_x, test_y) # Accuracy

Out[ ]:

0.9679379394680526

In [ ]:

from sklearn.metrics import plot_confusion_matrix
import warnings
pl = Pipeline([
    ('preprocessing',  preproc),
    ('dt', DecisionTreeClassifier())
])

warnings.simplefilter('ignore')

pl.fit(train_x, train_y)
y_pred = pl.predict(train_x)
pl.score(test_x, test_y)
plot_confusion_matrix(pl, test_x, test_y)

Out[ ]:

<sklearn.metrics._plot.confusion_matrix.ConfusionMatrixDisplay at 0x2169cca2df0>

Final Model¶

In [ ]:

from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier

In [ ]:

feats[['dpm', 'damagetakenperminute', 'earned gpm']].describe()

Out[ ]:

	dpm	damagetakenperminute	earned gpm
count	104670.000000	104670.000000	104670.000000
mean	418.668644	587.590700	228.937327
std	232.170156	244.839004	87.520314
min	0.000000	0.000000	9.411800
25%	234.578350	396.144325	166.017300
50%	388.976300	545.355050	231.035450
75%	560.765850	764.966800	290.740200
max	1947.272700	2126.743800	690.265100

In [ ]:

def kda(df):
    """Function to calculate kda

    Args:
        df (pandas dataframe): Dataframe with the columns, kills, deaths, assists

    Returns:
        Pandas Dataframe: Dataframe of the kda
    """
    df = pd.DataFrame((df['kills'] + df['assists'])/df['deaths'])
    max_not_inf = df[df[0] != np.inf].max()
    return df.replace(np.inf, max_not_inf).fillna(0).astype(float)

kda_trans = FunctionTransformer(kda)

pl_kda_standardize = Pipeline([
    ('kda', kda_trans)
])

In [ ]:

preproc = ColumnTransformer([
    ('kda_standardize', pl_kda_standardize, ['kills', 'deaths', 'assists']),
    ('ohe_champs_pos', OneHotEncoder(sparse=False, handle_unknown='ignore'), ['champion', 'position', 'patch']),
    # Thresholds for dpm, dtpm, gpm are the means
    ('binarize_dpm', Binarizer(threshold=419), ['dpm']),
    ('binarize_dtpm', Binarizer(threshold=588), ['damagetakenperminute']),
    ('binarize_gpm', Binarizer(threshold=229), ['earned gpm'])
], remainder='passthrough')

preproc.fit_transform(train_x)[:10]

Out[ ]:

array([[ 13.        ,   0.        ,   0.        , ...,  18.        ,
          8.        , 239.        ],
       [  0.5       ,   0.        ,   0.        , ...,   3.        ,
         15.        , 178.        ],
       [  2.        ,   0.        ,   0.        , ...,   9.        ,
         21.        , 191.        ],
       ...,
       [  1.66666667,   0.        ,   0.        , ...,   7.        ,
         20.        , 175.        ],
       [  4.75      ,   0.        ,   0.        , ...,  23.        ,
         15.        ,  38.        ],
       [  3.33333333,   0.        ,   0.        , ...,  15.        ,
         16.        , 284.        ]])

In [ ]:

# Decision Tree pipeline
pl_dt = Pipeline([
    ('preprocessing',  preproc),
    ('DT', DecisionTreeClassifier())
])
#Hyperparameters for the grid search on decision tree
hyperparameters_dt = {
    'DT__max_depth': [10, 15, 18, None],
    'DT__criterion': ['gini', 'entropy'],
    'DT__min_samples_split': [2, 4, 8, 16]
}
# Gridsearch
searcher_dt = GridSearchCV(pl_dt, hyperparameters_dt)
searcher_dt.fit(train_x, train_y)
searcher_dt.best_params_ #Final Model

Out[ ]:

{'DT__criterion': 'entropy', 'DT__max_depth': 18, 'DT__min_samples_split': 4}

In [ ]:

# {'DT__criterion': 'entropy', 'DT__max_depth': 18, 'DT__min_samples_split': 4}
searcher_dt.score(test_x, test_y) #Accuracy

Out[ ]:

0.96170895750535

In [ ]:

searcher_dt.best_score_ #Cross Validation Score

Out[ ]:

0.9571603569410362

In [ ]:

#Pipeline for Random Forest
pl_rf = Pipeline([
    ('preprocessing',  preproc),
    ('RF', RandomForestClassifier())
])
#Hyperparameters for Random Forest
hyperparameters_rf = {
    'RF__n_estimators': [100,200],
    'RF__max_depth': [15],
    'RF__criterion': ['entropy'],
    'RF__min_samples_split': [4]
}
searcher_rf = GridSearchCV(pl_rf, hyperparameters_rf)
searcher_rf.fit(train_x, train_y)
searcher_rf.best_params_

Out[ ]:

{'RF__criterion': 'entropy',
 'RF__max_depth': 15,
 'RF__min_samples_split': 4,
 'RF__n_estimators': 200}

In [ ]:

# {'RF__criterion': 'entropy','RF__max_depth': 15,'RF__min_samples_split': 4,'RF__n_estimators': 200}
searcher_rf.score(test_x, test_y) # Accuracy

Out[ ]:

0.9499757967949861

In [ ]:

searcher_rf.best_score_ # Cross Validation Score

Out[ ]:

0.9410970724003542

In [ ]:

# Logistical Regression Pipeline
pl_LR = Pipeline([
    ('preprocessing',  preproc),
    ('LR', LogisticRegression())
])
# HyperParameters for Logistical Regressio
hyperparameters_LR = {
    'LR__penalty': ['l2'],
    'LR__C': [1],
    'LR__max_iter': [50, 100, 200]
}
searcher_LR = GridSearchCV(pl_LR, hyperparameters_LR)
searcher_LR.fit(train_x, train_y)
searcher_LR.best_params_

Out[ ]:

{'LR__C': 1, 'LR__max_iter': 100, 'LR__penalty': 'l2'}

In [ ]:

# {'LR__C': 1, 'LR__max_iter': 100, 'LR__penalty': 'l2'}
searcher_LR.score(test_x, test_y) # Accuracy

Out[ ]:

0.9579257107918068

In [ ]:

searcher_LR.best_score_ # Cross Validation Score

Out[ ]:

0.9584469430118656

In [ ]:

# KNeighbors Classifiers Pipeline
pl_KNC = Pipeline([
    ('preprocessing',  preproc),
    ('KNC', KNeighborsClassifier())
])
# Hyperparameters for KNeighbors
hyperparameters_KNC = {
    'KNC__n_neighbors': [2, 3, 5],
    'KNC__weights': ['uniform', 'distance'],
}
searcher_KNC = GridSearchCV(pl_KNC, hyperparameters_KNC)
searcher_KNC.fit(train_x, train_y)
searcher_KNC.best_params_

Out[ ]:

{'KNC__n_neighbors': 5, 'KNC__weights': 'distance'}

In [ ]:

searcher_KNC.score(test_x, test_y) # Accuracy

Out[ ]:

0.9293793946805259

In [ ]:

searcher_KNC.best_score_ # Cross Validation score

Out[ ]:

0.9305368524946888

Fairness Evaluation¶

In [ ]:

feats.describe()[['kills', 'teamkills']]

Out[ ]:

	kills	teamkills
count	104670.000000	104670.000000
mean	2.912554	14.562769
std	2.752270	7.566761
min	0.000000	0.000000
25%	1.000000	8.000000
50%	2.000000	14.000000
75%	4.000000	20.000000
max	28.000000	60.000000

In [ ]:

# Using Gridsearch output model to get predicted values for fairness evaluation
preds = searcher_dt.predict(test_x)

In [ ]:

from sklearn.metrics import accuracy_score
# Adding actual, predicted and split
results = test_x.copy()
results['over_20_kills'] = results['teamkills'].apply(lambda x: 1 if x > 20 else 0)
results['predictions'] = preds
results['actual'] = test_y
accuracy_score(test_y, preds)

Out[ ]:

0.96170895750535

In [ ]:

# Dataframe to see relationship between precision of the split
results.groupby('over_20_kills')\
       .apply(lambda x: accuracy_score(x['actual'], x['predictions']))\
       .rename('accuracy').to_frame()
# 0 is less than equal 20, 1 is more than

Out[ ]:

	accuracy
over_20_kills
0	0.963096
1	0.956914

In [ ]:

# Performing our Permuatation Test
obsv = results.groupby('over_20_kills')\
              .apply(lambda x: accuracy_score(x['actual'], x['predictions']))\
              .diff().iloc[-1]
obsv # Our observed value

Out[ ]:

-0.006182007649189747

In [ ]:

# Permuatation Test
diff_in_acc = []
for _ in range(1000):
    # Shuffling and calculation difference in precision score between players whose team had over 20 kills
    # And those that didn't
    s = (
        results[['over_20_kills', 'predictions', 'actual']]
        .assign(over_20_kills=results.over_20_kills.sample(frac=1.0, replace=False).reset_index(drop=True))
        .groupby('over_20_kills')
        .apply(lambda x: accuracy_score(x['actual'], x['predictions']))
        .diff()
        .iloc[-1]
    )
    
    diff_in_acc.append(s)
    
# Calculating the Pvalue
sims = pd.Series(diff_in_acc)
p_val = len(sims[sims >= obsv]) / len(sims)
p_val

Out[ ]:

0.818

In [ ]:

# Plot of observations
plt.figure(figsize=(10, 5))
pd.Series(diff_in_acc).plot(kind='hist',
                            ec='w',
                            density=True,
                            bins=15,
                            title='Difference in Accuracy (Low Kills - High Kills)',
                            label = 'Difference in Accuracy')
plt.axvline(x=obsv, color='red', label='observed difference in Accuracy')
plt.legend(loc='upper left');

In [ ]: