Back to jeromewilliams.net
Predicting BGG ratings from boardgame characteristics¶
Jerome Williams
2024 March 9
This notebook investigates how well average boardgame ratings from BoardGameGeek (BGG) can be predicted from boardgame metadata (number of players, playing time, mechanics, etc.).
We will consider the prediction question from the point of view of a hypothetical boardgame publisher, trying to predict whether the game will be a hit or not in advance of release. Thus, we will rely only on intrinsic characteristics of the game, such as the number of players, playing time, themes, mechanics, and designers, and will not consider as possible predictors any "post-release" data, such as the number of ratings on BGG or the game's publisher.
Preamble¶
Let's first load some modules we will need.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import xgboost as xgb
input_path = 'input/bggdata/'
plt.rcParams["font.family"] = "Arial"
plt.rcParams['grid.linestyle'] = 'solid'
plt.rcParams['axes.labelcolor'] = '0.2'
plt.rcParams['grid.color'] = '0.9'
plt.rcParams['xtick.color'] = '0.3'
plt.rcParams['ytick.color'] = '0.3'
plt.rcParams['axes.edgecolor'] = '0.3'
plt.rcParams['axes.titleweight'] = 'bold'
plt.rcParams['axes.facecolor'] = 'white'
Data¶
We are using the BGG data from boardgamegeek.com, which I downloaded from Jen Watkins' page on Kaggle.
The dataset has several tables, described below.
| Filename | Description |
|---|---|
| artists_reduced.csv | Artists associated with each game, stored in one-hot format. |
| designers_reduced.csv | Designers associated with each game, stored in one-hot format. |
| games.csv | Tabular data for each boardgame, including year published, minimum and maximum number of players, and average rating. |
| mechanics.csv | Game mechanics associated with each boardgame, stored in one-hot format. |
| publishers_reduced.csv | Publishers associated with each game, stored in one-hot format. |
| ratings_distribution.csv | The distribution of ratings for each game, stored as, for each numerical rating (e.g., 7.5), the number of times the boardgame has received that rating. |
| subcategories.csv | Subcategories associated with each game. Example subcategories are "Exploration," "Miniatures," "Territory Building," and "Card Game." Categories, as opposed to subcategories, are stored in games.csv. |
| themes.csv | Themes associated with each game. Example themes are "World War I", "Humor", and "Traffic". |
| user_ratings.csv | This seems to be a complete set of ratings data. Each row is a boardgame, username, and numerical rating. |
For this analysis, we will use games.csv, mechanics.csv, designers_reduced.csv, subcategories.csv and themes.csv.
games = pd.read_csv(input_path + 'games.csv')
mechanics = pd.read_csv(input_path + 'mechanics.csv')
subcategories = pd.read_csv(input_path + 'subcategories.csv')
designers = pd.read_csv(input_path + 'designers_reduced.csv')
themes = pd.read_csv(input_path + 'themes.csv')
Initial exploration¶
Let's first split our dataset into training and test datasets. The plots that follow will be based on the training data only.
train = games.sample(frac=0.8, random_state=12345)
test = games.drop(train.index)
print(f"Total: {games.shape}")
print(f"Train: {train.shape}")
print(f"Test: {test.shape}")
Total: (21925, 48) Train: (17540, 48) Test: (4385, 48)
Our dataset has 21,925 games. We create an 80%/20% train/test split, which results in a training set of 17,540 observations and a test set of 4,385 observations.
Rating data¶
First, let's look at the histogram of average ratings.
Our dataset includes both the "Average Rating" (raw average of user ratings, AvgRating in games.csv) and the “BGG Rating", a modified version of "Average Rating". The BGG Rating (BayesAvgRating in games.csv) modifies "Average Rating" by adding some number of dummy ratings with scores of 5.5 (out of 10) to the set of ratings, which has the effect of pulling ratings--especially those for games with few votes--closer to the middle (BoardGameGeek).
According to BoardGameGeek, the BGG Rating is used on the site to “prevent games with relatively few votes climbing to the top of the BGG Ranks” (BoardGameGeek).
Let's look at the distribution of AvgRating and BayesAvgRating.
plot_data = train.melt(id_vars = ['BGGId'],value_vars=['AvgRating', 'BayesAvgRating'], var_name='Rating Type', value_name='Rating')
ax = sns.kdeplot(data=plot_data, x="Rating", hue="Rating Type", fill=True)
# plt.show()
fig = ax.get_figure()
ax.grid(True)
ax.set_axisbelow(True)
# title
fig.text(0.5, 1, "BGG Boardgame Ratings", ha='center', weight='bold', size = 16)
# subtitle
fig.text(0.5, 0.94, "Distribution of Ratings", ha='center', weight='normal', size = 16)
# caption
caption_str = f"Source: BoardGameGeek.com; Jerome Williams\nNote: Plot based on training dataset of {len(train)} games."
fig.text(0.05, -0.1, caption_str, ha='left', size = 10)
ax.set_xlabel('Rating')
ax.set_ylabel('Density')
plt.savefig("output/03_avg_vs_bgg_rating_distn.pdf", format="pdf", bbox_inches = 'tight')
Boardgames with at least 1,000 reviews¶
If we filter to games with at least 1,000 reviews, the two distributions should be more similar.
plot_data = (train[train['NumUserRatings'] > 1000]
.melt(id_vars = ['BGGId'],
value_vars=['AvgRating', 'BayesAvgRating'],
var_name='RatingType',
value_name='Rating')
)
ax = sns.kdeplot(plot_data, x="Rating", hue="RatingType", fill=True)
fig = ax.get_figure()
ax.grid(True)
ax.set_axisbelow(True)
# title
fig.text(0.5, 1, "BGG Boardgame Ratings", ha='center', weight='bold', size = 16)
# subtitle
fig.text(0.5, 0.94, "Distribution of Ratings, Games with at least 1000 ratings", ha='center', weight='normal', size = 16)
# caption
caption_str = f"Source: BoardGameGeek.com; Jerome Williams\nNote: Plot based on training dataset of {len(train)} games, \
filtered to games with at least 1,000 ratings."
fig.text(0.05, -0.1, caption_str, ha='left', size = 10)
ax.set_xlabel('Rating')
ax.set_ylabel('Density')
plt.savefig("output/04_avg_vs_bgg_rating_distn_1000.pdf", format="pdf", bbox_inches = 'tight')
Although it is noiser, in this exercise, we will try to predict the "Average Rating" rather than the "BGG Rating".
Feature engineering¶
Now that we've examined the target variable, let's look at some of the potential feature variables.
Mechanics¶
First, let's examine the game mechanics variables in mechanics.csv.
Before anything else, let's join the mechanics data to our training and test set, and focus our review on the training set only, so that we don't contaminate the analysis.
mechanics = pd.read_csv('input/bggdata/mechanics.csv')
train = train.merge(mechanics, on = 'BGGId')
test = test.merge(mechanics, on = 'BGGId')
print(f"After joining mechanics, train: {train.shape}")
print(f"After joining mechanics, test: {test.shape}")
After joining mechanics, train: (17540, 205) After joining mechanics, test: (4385, 205)
The mechanics data is one-hot encoded, with a column for each mechanic and a row for each game, and a 1 or 0 in each cell indicating whether a certain game uses that mechanic. Thus, we can find the total number of games with each mechanic by summing over all rows.
mechanics_list = mechanics.columns[1:].to_list()
mechanics_sum = train[mechanics_list].sum(axis=0).sort_values(ascending=False)
print('Top 10 mechanics:')
print(mechanics_sum[:10])
print(f'Number of unique mechanics: {len(mechanics_list)}')
Top 10 mechanics: Dice Rolling 5207 Hand Management 3596 Set Collection 2352 Variable Player Powers 2172 Hexagon Grid 1975 Simulation 1725 Drafting 1612 Tile Placement 1458 Modular Board 1355 Grid Movement 1310 dtype: int64 Number of unique mechanics: 157
We have 157 different mechanics. The most common mechanic is 'Dice Rolling', which is used by 5,207 games (in the training set).
Subcategories¶
Next, let's join the subcategories data. Let's add a prefix of "subcat_" to the column names so that there are no naming conflicts.
subcategories = pd.read_csv('input/bggdata/subcategories.csv')
subcategories_new_columns = ['subcat_' + col for col in subcategories.columns[1:].to_list()]
subcategories.columns = ['BGGId'] + subcategories_new_columns
print(len(subcategories_new_columns))
print(subcategories.columns)
train = train.merge(subcategories, on = 'BGGId')
test = test.merge(subcategories, on = 'BGGId')
print(train.shape, test.shape)
10
Index(['BGGId', 'subcat_Exploration', 'subcat_Miniatures',
'subcat_Territory Building', 'subcat_Card Game', 'subcat_Educational',
'subcat_Puzzle', 'subcat_Collectible Components', 'subcat_Word Game',
'subcat_Print & Play', 'subcat_Electronic'],
dtype='object')
(17540, 215) (4385, 215)
We have 10 different subcategories: Exploration, Miniatures, Terrritory Building, Card Game, Educational, Puzzle, Collectible components, Word Game, Print & Play, and Electronic.
Designers¶
Next, let's join the designers data.
designers = pd.read_csv('input/bggdata/designers_reduced.csv')
designers_columns = designers.columns.to_list()
# print(designers_columns.index('BGGId'))
# print(designers_columns.index('Low-Exp Designer'))
designers_new_columns = ['designer_' + col for col in designers.columns[:1592].to_list()]
designers.columns = designers_new_columns + ['BGGId', 'Low-Exp Designer']
train = train.merge(designers, on = 'BGGId')
test = test.merge(designers, on = 'BGGId')
print(train.shape, test.shape)
(17540, 1808) (4385, 1808)
designers_sum = train[designers_new_columns + ['Low-Exp Designer']].sum(axis=0).sort_values(ascending=False)
print('Top 10 designers:')
print(designers_sum[:10])
print(f'Number of unique designers: {len(designers_new_columns)}')
Top 10 designers: Low-Exp Designer 6652 designer_(Uncredited) 1147 designer_Reiner Knizia 260 designer_Joseph Miranda 113 designer_Wolfgang Kramer 103 designer_Richard H. Berg 89 designer_Jim Dunnigan 81 designer_James Ernest 80 designer_Martin Wallace 75 designer_Frank Chadwick 74 dtype: int64 Number of unique designers: 1592
Our designers data does not include data on every designer on BGG. Instead, designers with 3 or fewer entries are lumped together as "Low-Exp Designer". 6,652 have a Low-Exp Designer among their designers. An additional 1,147 have "(Uncredited)" listed as their designer. Aside from these entry types, the most common designers in the training data are Reiner Knizia, Joseph Miranda, and Wolfgang Kramer.
Themes¶
Lastly, let's join the themes data.
themes = pd.read_csv('input/bggdata/themes.csv')
themes_new_columns = ['theme_' + col for col in themes.columns[1:].to_list()]
themes.columns = ['BGGId'] + themes_new_columns
train = train.merge(themes, on = 'BGGId')
test = test.merge(themes, on = 'BGGId')
print(train.shape, test.shape)
(17540, 2025) (4385, 2025)
designers_sum = train[themes_new_columns].sum(axis=0).sort_values(ascending=False)
print('Top 10 themes:')
print(designers_sum[:10])
print(f'Number of unique themes: {len(themes_new_columns)}')
Top 10 themes: theme_Fantasy 2110 theme_Science Fiction 1332 theme_Fighting 1330 theme_Economic 1221 theme_Animals 1061 theme_World War II 996 theme_Humor 963 theme_Adventure 924 theme_Movies / TV / Radio theme 859 theme_Medieval 824 dtype: int64 Number of unique themes: 217
Next let's collect our features. We have 5 player count and playtime variables, 157 mechanics, 8 categories, 10 subcategories, 217 themes, and 1593 designers.
feature_vars = (['MinPlayers', 'MaxPlayers', 'MfgPlaytime', 'ComMinPlaytime', 'ComMaxPlaytime',
'Cat:Thematic',
'Cat:Strategy',
'Cat:War',
'Cat:Family',
'Cat:CGS',
'Cat:Abstract',
'Cat:Party',
'Cat:Childrens'] +
mechanics_list +
subcategories_new_columns +
designers_new_columns + ['Low-Exp Designer'] +
themes_new_columns)
X_train = train[feature_vars].copy()
X_test = test[feature_vars].copy()
target_var = 'AvgRating'
y_train = train[target_var]
y_test = test[target_var]
print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)
# print(X_train.head(5))
(17540, 1990) (4385, 1990) (17540,) (4385,)
Playing time and player count¶
Let's also create some different representations of the player count variables. Our data records the minimum and maximum player counts, but the model may do better with explicit variables indicating whether a game can be played 2/3/4/etc. players.
X_train['plays1'] = ((X_train['MinPlayers'] == 1) & (X_train['MaxPlayers'] >= 1)).astype(int)
X_train['plays2'] = ((X_train['MinPlayers'] <= 2) & (X_train['MaxPlayers'] >= 2)).astype(int)
X_train['plays3'] = ((X_train['MinPlayers'] <= 3) & (X_train['MaxPlayers'] >= 3)).astype(int)
X_train['plays4'] = ((X_train['MinPlayers'] <= 4) & (X_train['MaxPlayers'] >= 4)).astype(int)
X_train['plays5'] = ((X_train['MinPlayers'] <= 5) & (X_train['MaxPlayers'] >= 5)).astype(int)
X_train['plays6'] = ((X_train['MinPlayers'] <= 6) & (X_train['MaxPlayers'] >= 6)).astype(int)
X_train['plays7plus'] = (X_train['MaxPlayers'] >= 7).astype(int)
X_train['num_player_counts'] = X_train['MaxPlayers'] - X_train['MinPlayers'] + 1
X_test['plays1'] = ((X_test['MinPlayers'] == 1) & (X_test['MaxPlayers'] >= 1)).astype(int)
X_test['plays2'] = ((X_test['MinPlayers'] <= 2) & (X_test['MaxPlayers'] >= 2)).astype(int)
X_test['plays3'] = ((X_test['MinPlayers'] <= 3) & (X_test['MaxPlayers'] >= 3)).astype(int)
X_test['plays4'] = ((X_test['MinPlayers'] <= 4) & (X_test['MaxPlayers'] >= 4)).astype(int)
X_test['plays5'] = ((X_test['MinPlayers'] <= 5) & (X_test['MaxPlayers'] >= 5)).astype(int)
X_test['plays6'] = ((X_test['MinPlayers'] <= 6) & (X_test['MaxPlayers'] >= 6)).astype(int)
X_test['plays7plus'] = (X_test['MaxPlayers'] >= 7).astype(int)
X_test['num_player_counts'] = X_test['MaxPlayers'] - X_test['MinPlayers'] + 1
print(X_train.shape, X_test.shape)
(17540, 1998) (4385, 1998)
Variable scaling¶
Before fitting our model, let's scale the features and target variables.
# scaling
from sklearn.preprocessing import RobustScaler
X_scaler = RobustScaler()
features = X_train.columns
Xtr2 = X_scaler.fit_transform(X_train[features])
Xte2 = X_scaler.transform(X_test[features])
y_scaler = RobustScaler()
ytr2 = y_scaler.fit_transform(y_train.values.reshape(-1, 1))
yte2 = y_scaler.transform(y_test.values.reshape(-1, 1))
Next, we setup the training loop. We create a cross-validation split with 5 folds. We also define an objective function for our parameter tuning. The objective function takes the current parameters as arguments, fits the model, and fits the mean score (in this case, RMSE) across the 5 folds.
We use optuna to tune the parameters (learning rate and number of trees) and use a TPE (Tree-structured Parzen Estimator) sampler (arxiv.org).
import optuna
from optuna.samplers import TPESampler
from sklearn.model_selection import KFold
from sklearn.metrics import mean_squared_error
n = 5
cv = KFold(n, shuffle = True, random_state = 12345)
def objective(trial):
learning_rate = trial.suggest_float("learning_rate", 0, 0.1)
n_estimators = trial.suggest_int('n_estimators', 500, 2000)
scores = []
for i, (train_index, val_index) in enumerate(cv.split(Xtr2, ytr2)):
Xtr2_fold, X_val = Xtr2[train_index], Xtr2[val_index]
ytr2_fold, y_val = ytr2[train_index], ytr2[val_index]
xgb_model = xgb.XGBRegressor(objective = "reg:squarederror",
verbosity = 0,
learning_rate = learning_rate,
n_estimators = n_estimators
)
xgb_model.fit(Xtr2_fold, ytr2_fold)
xgb_preds = xgb_model.predict(X_val)
score = mean_squared_error(y_val, xgb_preds)
scores.append(score)
return np.mean(scores)
study = optuna.create_study(direction = "minimize", sampler = TPESampler())
study.optimize(func = objective, n_trials = 30)
print(study.best_params)
/Users/jeromew/.pyenv/versions/3.9.1/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
[I 2024-03-19 22:44:06,244] A new study created in memory with name: no-name-aaabedc9-46b1-4d96-8380-55cbd091e661
[I 2024-03-19 22:45:04,341] Trial 0 finished with value: 0.30738036954574566 and parameters: {'learning_rate': 0.060833069383555406, 'n_estimators': 1880}. Best is trial 0 with value: 0.30738036954574566.
[I 2024-03-19 22:45:30,533] Trial 1 finished with value: 0.31015399413744166 and parameters: {'learning_rate': 0.09460936319049677, 'n_estimators': 774}. Best is trial 0 with value: 0.30738036954574566.
[I 2024-03-19 22:46:15,825] Trial 2 finished with value: 0.307577481719661 and parameters: {'learning_rate': 0.07204047491631509, 'n_estimators': 1449}. Best is trial 0 with value: 0.30738036954574566.
[I 2024-03-19 22:46:56,794] Trial 3 finished with value: 0.3092449938464673 and parameters: {'learning_rate': 0.06923893229177526, 'n_estimators': 1261}. Best is trial 0 with value: 0.30738036954574566.
[I 2024-03-19 22:47:25,644] Trial 4 finished with value: 0.3092600644066167 and parameters: {'learning_rate': 0.09754983694887447, 'n_estimators': 857}. Best is trial 0 with value: 0.30738036954574566.
[I 2024-03-19 22:47:50,051] Trial 5 finished with value: 0.3191691204250052 and parameters: {'learning_rate': 0.0530247042495883, 'n_estimators': 665}. Best is trial 0 with value: 0.30738036954574566.
[I 2024-03-19 22:48:33,966] Trial 6 finished with value: 0.30931469706954123 and parameters: {'learning_rate': 0.06444077495302468, 'n_estimators': 1328}. Best is trial 0 with value: 0.30738036954574566.
[I 2024-03-19 22:49:26,250] Trial 7 finished with value: 0.3097184429918773 and parameters: {'learning_rate': 0.051063539675557296, 'n_estimators': 1553}. Best is trial 0 with value: 0.30738036954574566.
[I 2024-03-19 22:50:01,194] Trial 8 finished with value: 0.3249103086468331 and parameters: {'learning_rate': 0.026099593905584306, 'n_estimators': 900}. Best is trial 0 with value: 0.30738036954574566.
[I 2024-03-19 22:50:57,406] Trial 9 finished with value: 0.3151928740535336 and parameters: {'learning_rate': 0.0294326492052586, 'n_estimators': 1603}. Best is trial 0 with value: 0.30738036954574566.
[I 2024-03-19 22:52:09,694] Trial 10 finished with value: 0.32227640400023777 and parameters: {'learning_rate': 0.014564746218513225, 'n_estimators': 1973}. Best is trial 0 with value: 0.30738036954574566.
[I 2024-03-19 22:53:12,429] Trial 11 finished with value: 0.30630432455435286 and parameters: {'learning_rate': 0.0805902980585565, 'n_estimators': 1938}. Best is trial 11 with value: 0.30630432455435286.
[I 2024-03-19 22:54:17,010] Trial 12 finished with value: 0.3058130642860485 and parameters: {'learning_rate': 0.08048588227575734, 'n_estimators': 1998}. Best is trial 12 with value: 0.3058130642860485.
[I 2024-03-19 22:55:14,999] Trial 13 finished with value: 0.3062823091602316 and parameters: {'learning_rate': 0.08374472433204481, 'n_estimators': 1750}. Best is trial 12 with value: 0.3058130642860485.
[I 2024-03-19 22:56:14,012] Trial 14 finished with value: 0.306195857233181 and parameters: {'learning_rate': 0.08597443652607173, 'n_estimators': 1730}. Best is trial 12 with value: 0.3058130642860485.
[I 2024-03-19 22:57:12,271] Trial 15 finished with value: 0.30630024779280285 and parameters: {'learning_rate': 0.08288170579717015, 'n_estimators': 1729}. Best is trial 12 with value: 0.3058130642860485.
[I 2024-03-19 22:57:51,274] Trial 16 finished with value: 0.31709825247952195 and parameters: {'learning_rate': 0.03924650352231968, 'n_estimators': 1025}. Best is trial 12 with value: 0.3058130642860485.
[I 2024-03-19 22:59:42,742] Trial 17 finished with value: 0.3696725984291914 and parameters: {'learning_rate': 0.001513941838738729, 'n_estimators': 1755}. Best is trial 12 with value: 0.3058130642860485.
[I 2024-03-19 23:00:30,425] Trial 18 finished with value: 0.30697863645312873 and parameters: {'learning_rate': 0.08877811645345303, 'n_estimators': 1437}. Best is trial 12 with value: 0.3058130642860485.
[I 2024-03-19 23:01:09,064] Trial 19 finished with value: 0.309753023607684 and parameters: {'learning_rate': 0.07389433722377375, 'n_estimators': 1082}. Best is trial 12 with value: 0.3058130642860485.
[I 2024-03-19 23:02:11,566] Trial 20 finished with value: 0.3051665343626448 and parameters: {'learning_rate': 0.09816410797966438, 'n_estimators': 1831}. Best is trial 20 with value: 0.3051665343626448.
[I 2024-03-19 23:03:13,171] Trial 21 finished with value: 0.30573173424293654 and parameters: {'learning_rate': 0.09605043241004642, 'n_estimators': 1797}. Best is trial 20 with value: 0.3051665343626448.
[I 2024-03-19 23:04:16,130] Trial 22 finished with value: 0.30560561322170504 and parameters: {'learning_rate': 0.09908057813759082, 'n_estimators': 1859}. Best is trial 20 with value: 0.3051665343626448.
[I 2024-03-19 23:05:18,786] Trial 23 finished with value: 0.3063965449381953 and parameters: {'learning_rate': 0.0996817965426739, 'n_estimators': 1830}. Best is trial 20 with value: 0.3051665343626448.
[I 2024-03-19 23:06:12,928] Trial 24 finished with value: 0.30577930717232704 and parameters: {'learning_rate': 0.0925811010598444, 'n_estimators': 1595}. Best is trial 20 with value: 0.3051665343626448.
[I 2024-03-19 23:07:08,321] Trial 25 finished with value: 0.3060915718375415 and parameters: {'learning_rate': 0.09977579176829841, 'n_estimators': 1632}. Best is trial 20 with value: 0.3051665343626448.
[I 2024-03-19 23:08:12,869] Trial 26 finished with value: 0.3051846627192057 and parameters: {'learning_rate': 0.09038473485505345, 'n_estimators': 1871}. Best is trial 20 with value: 0.3051665343626448.
[I 2024-03-19 23:09:02,370] Trial 27 finished with value: 0.30722468026563676 and parameters: {'learning_rate': 0.0766276264869035, 'n_estimators': 1421}. Best is trial 20 with value: 0.3051665343626448.
[I 2024-03-19 23:10:09,002] Trial 28 finished with value: 0.3064995308658916 and parameters: {'learning_rate': 0.09012860503999219, 'n_estimators': 1893}. Best is trial 20 with value: 0.3051665343626448.
[I 2024-03-19 23:11:12,457] Trial 29 finished with value: 0.3062500412000902 and parameters: {'learning_rate': 0.06389813461055553, 'n_estimators': 1859}. Best is trial 20 with value: 0.3051665343626448.
{'learning_rate': 0.09816410797966438, 'n_estimators': 1831}
Results¶
The optimal set is a learning rate of 0.098 and 1,831 for the number of trees.
Let's now fit the model again (with the best parameters) to the whole training set.
study.best_params
xgb_model_best = xgb.XGBRegressor(objective = "reg:squarederror",
verbosity = 0,
importance_type = 'gain',
**study.best_params
)
xgb_model_best.fit(Xtr2, ytr2)
test_preds = xgb_model_best.predict(Xte2)
test_score = mean_squared_error(yte2, test_preds)
print(f"Test score: {test_score:.4f}")
print(f"RMSE: {np.sqrt(test_score):.4f}")
Test score: 0.3003 RMSE: 0.5480
We have an RMSE value of 0.548. However, this RMSE is given in terms of the scaled target variable, so is not straightforward to interpret. Let's convert back to our original scale and recalculate the RMSE.
y_test_inverted = y_scaler.inverse_transform(yte2).flatten()
test_preds_inverted = y_scaler.inverse_transform(test_preds.reshape(-1, 1)).flatten()
test_df = pd.DataFrame({'y_test': y_test_inverted, 'test_preds': test_preds_inverted})
RMSE_inverted = np.sqrt(mean_squared_error(test_df['y_test'], test_df['test_preds']))
print(f"RMSE: {RMSE_inverted:.4f}")
RMSE: 0.6633
On our original scale, the RMSE is 0.663. This means that the average absolute difference between our predicted ratings and actual ratings is (roughly) 0.66, where ratings are on the 1-10 BGG scale.
Predictions¶
Next, let's plot predicted ratings against actual ratings.
ax = sns.scatterplot(data = test_df, x = 'y_test', y = 'test_preds', alpha = 0.6, marker = '.', edgecolor=None)
fig = ax.get_figure()
ax.set_xlabel('Actual rating')
ax.set_ylabel('Predicted rating')
ax.set_facecolor('white')
ax.grid(True)
ax.set_axisbelow(True)
# title
fig.text(0.5, 1, "Predicting BGG Boardgame Ratings", ha='center', weight='bold', size = 16)
# subtitle
fig.text(0.5, 0.94, "Predicted Ratings vs. Actual Ratings", ha='center', weight='normal', size = 16)
# caption
caption_text = """
Source: BoardGameGeek.com; Jerome Williams
Note: Predicted ratings from XGBoost model."""
fig.text(0.1, -0.10, caption_text, ha='left', size=9)
plt.savefig("output/01_predicted_vs_actual_XGB01.pdf", format="pdf", bbox_inches = 'tight')
Reassuringly, we see an upward-sloping between predicted and actual ratings. One interesting fact is that predicted ratings never go below 3.5, even though there are a handful of actual ratings in that range.
Feature importance¶
Now that we've fit our model, let's examine feature importance, the degree to which different features contribute to the predicted rating.
The feature importance for our XGBoost model corresponds to the average information gain) across all splits the feature is used in.
importance_df = pd.DataFrame({'feature' : features, 'importance' : xgb_model_best.feature_importances_})
importance_df.sort_values('importance', ascending = False, inplace = True)
ax = sns.barplot(data = importance_df.head(20), x = 'importance', y = 'feature')
fig = ax.get_figure()
ax.set_axisbelow(True)
ax.grid(True)
# title
fig.text(0.5, 1, "Predicting BGG Boardgame Ratings", ha='center', weight='bold', size = 16)
# subtitle
fig.text(0.5, 0.94, "Top 20 features by importance, XGBoost model", ha='center', weight='normal', size = 16)
# caption
caption_text = """Source: BoardGameGeek.com; Jerome Williams
Note: Feature importance is based on the gain metric, which measures the average improvement in accuracy
across all splits the feature is used in."""
fig.text(-0.2, -0.15, caption_text, ha='left', size = 9)
ax.set_xlabel('Feature Importance')
ax.set_ylabel('Feature')
plt.show()
# plt.savefig("output/02_variable_importances_XGB01.pdf", format="pdf", bbox_inches = 'tight')
