Transformer-based ORL Applied to Chess - Part 4: Reward Targets, Manipulation and Conclusion

This last part covers the forward-looking Return-To-Go reward formulations, the Decision Transformer, the two experiments that manipulate moves during training to verify that the models really adhere to the reward signal, and a supervised model that receives the player’s Elo instead of a reward. It closes with the discussion, the conclusion, the limitations, directions for future work and the appendix.

About this paper

Master Thesis for the Master of Science - Artificial Intelligence at IU International University.

The original LaTeX source, the bibliography, the full code, the data preparation and the compiled PDF are available on GitLab: gitlab.com/iu-msc-ai/transformer-based-offline-reinforcement-learning-applied-to-chess

This is part 4 of 4:

1. Foundations
2. Experimental setup
3. Results - Chess-Transformer baseline and base UDRL
4. Results - reward targets, manipulation and conclusion (this part)

RTG models (5 & 6)

While until now the reward was the immediate reward of the action, the following models used a forward-looking return-to-go (RTG) signal. Two reward types were used, one was the evaluation difference-based reward signal and the other the material difference-based reward signal. Both were trained with a trajectory length of three moves and five moves. In all cases the RTG was not discounted.

5_UDRL-rtg3_eval

In this experiment the reward signal is a cumulation of three rewards without discount. As this RTG was still clamped between -1 and +1 in the data preprocessing (compare reward distribution), the reward targets during inference were still set in this range. As always, the first experiment is again against Stockfish on level 3, with target rewards between -1 and +1, in steps of size 0,1, for 100 games.

It is already interesting to see that the model plays better for reward targets below 0. An explanation for this could be that because of the cumulation of three rewards, the reward target is not as clean of a signal anymore, by summing up multiple rewards which must not be directly related to the current state or action. Further the playing strength seems to be lower than the earlier models. The statistical test results are shown below for selected reward targets.

L3, 5_UDRL-rtg3_eval compared to 2_UDRL-eval, with different reward targets - source: own depiction

This confirms the assumption from the visualization, that the RTG based model plays significantly stronger for lower reward targets and not as strong for the reward target of 0,0. Next the inferences are analyzed. In this case just for the reward target of 0,0.

The most notable difference in the image are the values of the logits for legal moves. Here quite a few legal moves have low or negative logits, which was uncommon in the previous models. This could indicate that the model excludes legal moves better, by having more information of their performance in the future. The attention parts of the image look similar to the earlier models, especially 2_UDRL-eval. The reward attention does not show a clear pattern here. Probably because the RTG signal is more complex than the direct reward signal before.

Key findings:

This result was surprising, as the RTG signal was expected to improve the models planning strength, which is somewhat visible in the inference plots, but does not result in better playing strength. An explanation for this could be that the evaluation signal, which was provided by Stockfish already includes a look-ahead, as Stockfish is a search-based engine. Therefore, the model does not benefit from the RTG signal, as it already gets forward-looking information from the evaluation signal.

5b_UDRL-rtg5_eval

The just described experimental setup was conducted again, but this time with a trajectory length of five moves for the RTG signal. The results are visualized below.

The visualization shows even less capability of steering via the reward target, but slightly higher playing strength compared to the previous experiment with a RTG length of 3. The results of the statistical tests shown below, comparing this model against the 5_UDRL-rtg3_eval model and for the reward target of 0 against the 2_UDRL-eval model.

L3, 5b_UDRL-rtg5_eval compared to 2_UDRL-eval, with different reward targets - source: own depiction

This shows an improvement in playing strength over the trajectory length of three moves, for lower and higher reward targets, but the signal does not appear to help the model much for steering anymore. This makes sense, as it intensifies the effect already seen with the trajectory length of three moves, by averaging over an even longer time horizon.

The inference plots for the reward target of 0,0 are shown below.

Here the same pattern in the logits is visible as in the RTG model with three steps, especially for the legal moves. For illegal moves a higher amount moves have a small positive logit value though. The reward attention shows a more convoluted pattern in the last layer, compared to the earlier model. Probably again because the RTG signal is more complex than before.

Key findings:

Here the reduced steering capability seen in the 5_UDRL-rtg3_eval model with the trajectory length of three moves is intensified. This confirms that an evaluation difference-based RTG is not optimal for steering. Further the playing strength does not improve for the reward target of 0 either, which may be due to the base signal already incorporating a look ahead.

6_UDRL-rtg3_material

In this experiment the base reward was the material difference and the reward signal was calculated again as a RTG with three moves. The results are shown below.

Here the steering capabilities are visible again, with a stronger play around reward target 0. The statistical test results shown below compare this model against 2_UDRL-eval and 3_UDRL-material.

L3, 6_UDRL-rtg3_material compared to 2_UDRL-eval, with different reward targets - source: own depiction

L3, 6_UDRL-rtg3_material compared to 3_UDRL-material, with different reward targets - source: own depiction

Here the RTG has not such a negative effect than in the evaluation difference-based model, as the steering capability is still intact. This confirms the earlier assumption, that the combination of an already forward-looking evaluation difference-based signal with a RTG signal does not help the model for steering. The statistical tests show that the 6_UDRL−rtg3_material model shows no statistical difference to the immediate-reward material model (3_UDRL−material) across all targets. Therefore the RTG based on the material difference-based reward signal has no effect on playing strength.

The inference plots for the reward target of 0,0 are shown below.

Here the logits show a similar pattern as in the model with a direct material-based reward, but have less negative values than in the evaluation-based RTG model. The reward attention does not show a clear pattern, but is comparable to the visualizations shown for the 3_UDRL−material model. A noticeable difference is the low reward key attention in the last layer. This could indicate that the model relies less on the reward signal.

Key findings:

The results show that the RTG signal successfully retains steering capability when paired with a material difference-based reward signal. Further the RTG signal does not appear to have any effect on playing strength when compared to the 3_UDRL-material model, which uses the material difference-based reward signal.

6b_UDRL-rtg5_material

In this experiment the base reward signal was the material difference and the reward was calculated again as an RTG with five moves. The results are shown below.

Here the playing strength seems to collapse. This suggests that for the material difference-based reward signal a trajectory length of 5 is too long for the model to learn meaningful patterns between the state, RTG and action. The statistical test results shown below compare this model against the 2_UDRL-eval model.

L3, 6b_UDRL-rtg5_material compared to 2_UDRL-eval, with different reward targets - source: own depiction

This confirms the assumption from the visualization, that the RTG based model plays significantly weaker for the reward target of 0,0. Next the inferences are analyzed. In this case again just for the reward target of 0,0.

The logits are showing a few more legal moves with a negative logit. Which may indicate their long term performance being worse, by looking further into the future. The reward queries attention to the board squares show a rather sparse pattern in the last layer for queries and keys. Such a sparse pattern was in other models an indication of following the reward signal. Here the attention was not on the source or target square of the played move though. This could indicate that the model is not able to utilize the reward signal well.

Key findings:

These results show that increasing the trajectory length to five moves for the sparse material reward signal severely degrades performance, leading to a significant collapse in playing strength. Therefore a RTG signal of five moves is too long for the model to learn meaningful patterns from the data. The steering capability stayed somewhat intact though.

Decision Transformer (7_DT-eval)

With having successfully implemented the Upside Down Reinforcement Learning (UDRL) architecture, which already utilizes the RTG signal, the step to implementing a full Decision Transformer (DT) architecture should be small. However, the training results of the DT architecture were disappointing. The DT architecture learns substantially less effectively than UDRL-based models under the same data, compute, and model size constraints.

The loss value converges at a higher level then in all other models, while the top-1 accuracy plateaus on a very low level. This indicates that, although the model captures some regularities in the data, it fails to exploit them as effectively as architectures that predict actions from a single state-reward pair. During inference this model was not able to win a single game on the lowest available Stockfish level. To ensure that the different performance in learning was not due to an implementation error, the model was trained a second time using the DT implementation from “Hugging Face”.¹ The code was adjusted to take categorical variables and showed similar, poor results.²

A significant amount of work was put into trying to get the DT architecture to learn better, including changing hyperparameters and model size. None of these attempts showed a significant improvement in training performance. Further, due to the results from earlier experiments, it was tested if a direct reward instead of an RTG improved the learning capabilities, which was not the case. Therefore the degradation in performance is hypothesized to be due to the added action, state and reward trajectory.

It can just be assumed why adding the trajectory context did not help the model. The additional trajectories are more complex in chess compared to Atari games, where the DT architecture was successfully implemented. The game Pong may provide easier to learn trajectories, e.g. a once correctly identified upwards movement would continue in the next states for a while. Chess is a rather complex game with a high-branching factor. Therefore the trajectory context could add too much noise to the input and the model would not be able to identify relevant parts of the trajectory.

Key findings:

The DT architecture had a substantially weaker learning outcome than the UDRL-based models. Although the model does capture some structure from the data, trajectory conditioning did not translate into improved playing strength or decision quality in chess. This may indicate that the added trajectory context adds too much complexity and noise to the input for the model and therefore appears to exceed the learning capacity of the model.

Manipulating moves during training (8-9)

The following experiments are designed to provide a better understanding on whether the UDRL architecture really adheres to the reward signal. This is done by manipulating specific moves during training, either via removing them from the training data or via adjusting their reward.

8_UDRL-eval-manipulated_reward

In this experiment the reward of specific moves was set to -1, regardless of the evaluation found in the data. The selected moves are classical opening moves in chess. The first one being e2e4 and the other one being e2e3. The goal is to confirm that the model adheres to the reward signal and adjusts the move selection accordingly. Therefore, the model is expected to execute these moves preferentially when the reward target matches the artificially assigned reward, and to avoid them otherwise. This would obviously worsen the playing strength of the model to a certain degree. As increasing the playing strength is not the target of this experiment it will not be tested here.

The resulting amount of the specific moves played in 100 games against Stockfish on level 3 with different reward targets are shown below.

e2e3 / e2e4 moves - 8_UDRL-eval-manipulated_reward compared to 2_UDRL-eval, with different reward targets - source: own depiction

These results confirm that the model adheres to the reward signal during inference and adjusts its move selection accordingly. Further the model with the manipulated reward, and the specific reward target set, played the moves even more often than the baseline model in a single game. With a reward target of -1,0, the 8_UDRL-eval-manipulated_reward model played the specific moves 764 times across 500 games, compared to a maximum of 538 occurrences in the 2_UDRL-eval model using a reward target of +1,0.

Below the inference visualization for this model and the 2_UDRL-eval model without the artificial reward signal modification is shown for the first move in the game, which is often one of the modified e2e3 or e2e4 moves.

It is visible that the modification in the reward signal has an impact on the model’s attention. The baseline model focuses on the manipulated moves in the first and last layer in multiple heads and even plays e2e3 in this situation. The modified model focuses less on this move, with just having the highest attention value for the e2e3 move in the first layer, head 4, while still being numerically low with 0,007.

The reward attention patterns on the 2_UDRL-eval model looks very focused on single squares. It even shows the source square with the highest attention in the reward query of the last layer for the second head. For the manipulated model the patterns are more diffuse and less focused on single squares.

Key findings:

With a reward target of -1 the model plays the specific moves more often than the comparable 2_UDRL-eval model. For the other tested reward targets the model ignored these moves. This verifies that the model adheres to the reward signal and target and therefore just plays the move when the reward target is set accordingly. This again confirms the second research question.

9_UDRL-eval-hidden_action

In this experiment the same moves as in the previous experiment were manipulated, but instead of adjusting the reward signal the specific moves have been removed from the training data altogether. This was done by having the data loader choose another move from the selected game, whenever one of these moves were selected. The goal is to see how the model behaves, when it never saw these moves during training, what makes them out-of-distribution (OOD), compared to the previous experiment where the reward was adjusted.

The results below show how many times these moves have been played for different reward targets.

e2e3 / e2e4 moves - 9_UDRL-eval-hidden_action compared to 2_UDRL-eval, with different reward targets - source: own depiction

It is shown that the model almost never selected these moves, regardless of the reward target. This is the expected behavior, as the model never experienced these moves during training, which makes them OOD for the model. Since the model has no learned representation or associated reward signal for these moves, it does not select them during inference.

The reward attention in the visualization looks again very similar to the patterns identified in the 2_UDRL-eval and 8_UDRL-eval-manipulated_reward models, indicating that the removal of these actions did not fundamentally alter the overall attention.

Key findings:

Here the hidden moves are almost never played, which is plausible, as the model never saw these moves in the training data, which makes them OOD. This confirms the earlier mentioned problem of offline reinforcement learning (ORL) with OOD actions.

Supervised model with Elo input (10_TEO-Elo_signal)

In this experiment a supervised model was trained with the playing strength of the individual player, represented by the Elo value, as an additional input feature. This setup is somewhat related to the original Chess-Transformer project, where the Elo was used as a filter on the input data. The model was augmented with the Elo as input, instead of using a reward, to test a different method of adjusting the playing strength during inference. In this case an Elo signal was provided during training and set as targeted during inference. This would show that a steering capability is not limited to just using a reward signal, and therefore a reinforcement type architecture, but could also be achieved by other inputs, e.g. higher level abstractions of move quality, like the Elo rating.

Below the results of playing against Stockfish on level 3 with different Elo targets are shown.

The image shows that the model is not able to adjust its playing strength based on the provided Elo signal. The next table tests these differences for being statistically significant.

L3, Elo rewards - 10_TEO-Elo_signal compared to each other - source: own depiction

As expected from the visualization, the model is not able to adjust its playing strength based on the provided Elo signal. Now the strength is compared to the 2_UDRL-eval mode, which uses a move-based reward signal and the best performing reward target of 0 and the 1_TEO model, which does not use any additional input.

L3, 10_TEO-Elo_signal compared to 2_UDRL-eval, with reward target 0 - source: own depiction

This shows no significant difference in playing strength between the models. Which means the Elo signal does not help the model to improve its playing strength to a higher value than with the move-based reward signal, but plays on a similar level.

L3, 10_TEO-Elo_signal compared to 1_TEO - source: own depiction

This shows that just one of the Elo targets increased the playing strength of the model significantly, compared to the base model 1_TEO without any additional input signal.

Next the inferences for the different Elo targets are shown. In this case the plot shows the Elo signal instead of the reward target used in the previous experiments.

The reward plots, which use the Elo target as input, show very similar patterns between the different Elo targets. Which makes sense, as the different Elo targets do not make the model differ much in playing strength.

Key findings:

The Elo signal has a minor positive effect on playing strength, compared to the 1_TEO model without any additional input, but does not allow adjusting the playing strength based on the provided Elo target. This shows that the UDRL architecture benefits specifically from a direct reward signal, instead of a higher level abstraction of move quality, like the Elo rating.

Discussion

To contextualize and interpret the experimental results, the table below provides a consolidated overview of the main findings across all evaluated model variants. The following discussion focuses on explaining the observed behaviors, identifying patterns, and relating them to architectural and methodological choices.

Summary of main findings for each trained model - source: own depiction

A central finding of this thesis is that a reward-conditioned architecture is able to adhere to an externally specified reward target during inference. This confirms that the UDRL architecture does not merely learn a static policy, but instead conditions its action selection on the provided reward signal. This capability is highly dependent on the chosen reward definition though. The dataset was rather unbalanced for the evaluation difference-based reward signal, with a low percentage of values above 0. Therefore models based on this reward type saw more negative than positive values. This could have led to the models degrading playing strength when seeing higher reward target values, which should have instead led to better play according to the reward design. The material difference-based reward signal had a more balanced distribution, which did not lead to stronger playing strength either though. This could be due to the fact that a high reward for material differences is captured material, which is not possible in all positions. Therefore the reward design could be further improved to better reflect achievable goals in a wider range of positions.

Further, the results the UDRL architectures demonstrated were not repeatable when using the players Elo rating as an input signal instead of a direct action-based reward. This indicates that high-level abstractions of playing strength are insufficient to guide fine-grained action selection, and that direct reward signals tied to state transitions are essential for the observed steering capability in the UDRL-based architectures.

Direct reward signals derived from evaluation differences or material differences consistently enabled controllable behavior, whereas forward-looking reward formulations based on a RTG sometimes degraded both steering capability and playing strength. This suggests that forward-looking reward aggregation is not universally advantageous in deterministic, high-branching domains, especially when the reward already encodes long-term performance information through the chess engines evaluation, as it is the case with the evaluation difference-based reward. Introducing an additional aggregation via a RTG may therefore lead to redundant or conflicting signals. This effect was particularly visible for longer RTG horizons, where the model with evaluation-based reward signal exhibited a collapse in steering capability.

Improvements in playing strength remained within the same discrete Stockfish difficulty level. This indicates that reward conditioning enhances move selection quality, but does not yet enable breakthroughs to substantially higher levels of play. Further research is needed to explore whether e.g. more sophisticated reward designs or larger model capacities can unlock greater performance gains.

The experiments further confirm the known limitation of ORL approaches regarding OOD data. When specific moves were systematically removed from the training data, the model failed to select these moves during inference. This behavior highlights the reliance of offline-trained policies on high state-action coverage and demonstrates that generalization across unseen actions remains limited.

Interpreting the attention visualizations yielded mixed results. While attention patterns did highlight relevant board squares mostly in the reward queries, the findings were not always sufficient to draw a definitive understanding about the internal reasoning processes of the models. This suggests that the final feed-forward network plays a substantial role in mapping the aggregated attention representations to concrete move decisions.

Conclusion

Each experiment conducted in this thesis provided insights into the capabilities and limitations of transformer-based offline reinforcement learning architectures applied to chess. This section summarizes the main findings, discusses the limitations of the current work, and outlines directions for future research.

Summary

This thesis investigated two primary research questions. First, whether a UDRL-based architecture can improve playing strength in chess compared to a supervised learning based architecture and second, whether playing strength can be adjusted dynamically via a reward target during inference. Both questions were addressed and confirmed by the experimental results.

Most of the evaluated UDRL models outperformed the baseline 1_TEO architecture for certain reward targets, which matched the expectations based on the reward modeling. Furthermore, they exhibited clear steering behavior when varying the reward target during inference. These findings demonstrate that incorporating a reward signal as model input enables both, improved performance and controllable behavior via a reward target in an offline learning setting.

The experiments showed that reward-conditioned architectures are sensitive to the formulation of the reward signal. Evaluation difference-based rewards, used in the 2_UDRL-eval model, and material difference-based rewards, used in the 3_UDRL-material model, both enabled steering capabilities. Adding legal moves as additional input features in model 4_UDRL-eval-legal_input did not improve playing strength or the attention patterns. This suggests that the architecture was already able to learn to focus on legal moves from the training data alone, and that explicit legal move inputs did not provide additional benefit in the current setup.

Forward-looking reward aggregation via RTG did not increase the playing strength significantly for either reward type, shown by the models 5_UDRL-rtg3_eval and 6_UDRL-rtg3_material. While the 5_UDRL-rtg_eval model already showed degrading performance for the steering capability, an increased RTG horizon in the 5b_UDRL-rtg5 model even worsened it. The longer RTG horizon in the 6b_UDRL-rtg5_material model led to a collapse in playing strength, while still showing some steering capability.

The DT architecture evaluated in model 7_DT_eval showed substantially worse training results than all other architectures. It is likely that the added complexity of the trajectory context was too high for the model to extract meaningful patterns from, especially in a high-branching domain like chess. Further research would be needed to confirm this assumption.

Model adherence to the reward signal was further confirmed through reward manipulation during training and inference in model 8_UDRL-eval-manipulated_reward. When rewards were artificially altered for specific moves, the model adjusted its behavior accordingly. This is strong evidence that the learned policy conditions directly on the reward input rather than merely memorizing state-action relationships.

OOD effects were clearly observed when specific actions were removed from the training data in model 9_UDRL-eval-hidden_action, leading to the model almost never executing these actions during inference. This confirms a fundamental limitation of the ORL approach and highlights the importance of comprehensive state and action coverage in the training dataset.

Finally, providing Elo values as an input signal in model 10_TEO-Elo_signal did not replicate the steering behavior observed with reward-conditioned models. This indicates that a high-level indicator of playing strength is insufficient in settings where fine-grained action control is needed. This confirms the importance of direct reward-based conditioning in the UDRL framework.

Limitations

The DT architecture was not able to perform similarly to the UDRL-architectures. This could be due to the added complexity of the action trajectory context, which the model was not able to extract meaningful patterns from. Further research would be needed to confirm this assumption.

The unbalanced dataset for the evaluation difference-based reward signal, with a low number of values above 0, may have led to the model not showing improved or even further improved playing strength when seeing higher reward target values. On the other hand, not all moves are possible to increase the evaluation by substantial level, especially in already good positions. More research would be needed to answer whether just more data is needed or the reward design itself needs to be improved to better reflect achievable goals in a wider range of positions.

The legal moves input did not improve the playing strength or attention patterns. This may be due to the fact that the model already learned to focus on legal moves via the training data. A different way of providing this information, e.g. via masking illegal moves in the attention mechanism, could be explored.

Furthermore, the inference visualizations did not show very conclusive patterns. This may be due to the complexity of chess and the limited model size used in the experiments. Larger models may be able to learn more distinct and therefore separable attention patterns, as they provide more attention heads to capture such patterns.

Directions for future work

Based on these findings, several areas for further research can be identified. First, more research on why the DT architecture was underperforming is needed. As experiments with different model sizes and different hyperparameters have already been executed, a training run with a significantly larger amount of training data would be interesting.

Furthermore, using one of the trained models as base model for fine-tuning in an online setting could be explored. This would formally remove the OOD problem and may improve the playing strength of the model further.

As already mentioned, improving the reward design to better reflect achievable goals in a wider range of positions could be explored. This may help the model to improve playing strength when seeing higher reward target values.

Another angle already mentioned would be experimenting with more and higher dimensional reward signals to guide the model during inference. An example would be to add a reward signal for controlling the playing style, e.g. more aggressive or more defensive or a reward signal for activating a specific piece like the queen more. This capability was already shown on a basic level by manipulating a small amount of actions during training.

The created codebase and experiments provide a solid foundation for further research in this area. The code is modular and extensible, making it easy to add new features and experiment with different architectures and training methods. The code executes the experiments based on configuration files, making it easy to reproduce the results, compare different approaches and build upon them.

Appendix

Code, visualizations, data and models

Links to the code repository, stored visualizations, the dataset, and all trained models used in the experiments are provided below.

Code repository

Link to the Git repository: https://gitlab.com/iu-msc-ai/transformer-based-offline-reinforcement-learning-applied-to-chess.

All code after the initial commit, which was the fork from the source project, was written by the author of this thesis (compare overview of code changes).

Visualizations

Direct link to the visualizations shown in this document: https://gitlab.com/iu-msc-ai/transformer-based-offline-reinforcement-learning-applied-to-chess/-/blob/main/written_document/images/experiments/inference_plot.

Link to visualizations of the shown moves (masked, not masked, limited layers and all layers) and visualizations of a full game for each model: https://drive.google.com/drive/u/0/folders/1hdufR-XJG6hZEuTcZ9coDUh5oRrGmwcY.

Datasets and trained models

Link to source dataset: https://database.lichess.org/standard/lichess_db_standard_rated_2025-07.pgn.zst.

Link to the pre-processed dataset used for training and evaluation: https://huggingface.co/datasets/micha-net/chess_dataset_500k_games.arrow/blob/main/chess_dataset_500k_games.arrow.

Link to the trained models: https://huggingface.co/micha-net/transformer-based-offline-reinforcement-learning-applied-to-chess

Overview of game results

The following table shows all game results for all models, including the confidence intervals. The confidence intervals are calculated according to the Wilson score interval method.

Overview of game results for all models - source: own depiction

Overview of code changes

The following table shows the added and deleted lines of code in comparison to the original Chess-Transformer project. Git treats edited and rearranging lines as deletion of the original line and insertion of the modified line. Not relevant changes, like adding a new single token to the list of UCI moves, which generates over 1.000 insertions and deletions have been excluded from the table. Further duplicate changed, e.g. the model configs which are technically very similar, or other irrelevant files like the Latex document have been removed from the table.³

Overview of code changes compared to original project - source: own depiction

Individual training plots of all experiments

Below the individual training plots for all models are shown. Each plot shows the training loss and top-1 accuracy over the training steps.

1_TEO

2_UDRL-eval

3_UDRL-material

4_UDRL-eval-legal_input

5_UDRL-rtg3_eval

5b_UDRL-rtg5_eval

6_UDRL-rtg3_material

6b_UDRL-rtg5_material

8_UDRL-eval-manipulated_reward

9_UDRL-eval-hidden_action

10_TEO-Elo_signal

Back to Part 1: Foundations

References