LIPADE

The game of the Amazons is a fairly young member of the class of territory-games. Since there is very few human play, it is difficult to estimate the level of current programs. However, it is believed that humans could play much stronger than todays programs, given enough training and incentives. With the more general goal of improving the playing level of Amazons programs in mind, we focused here on the playing of endgames situations. Our comparative study of two solvers, DFPN and WPNS, and three game-playing algorithms, Minimax with Alpha/Beta, Monte-Carlo Tree-Search and Temperature Discovery Search, shows that even though their computing process is quite expensive, traditional PNS-based solvers are best suited for the task of finding moves in a subgame, while no specific improvement is needed to classical game-playing engines to play well combinations of subgames. Even the new Amazons standard of Monte-Carlo Tree-Search, despite showing often weaknesses in precise tasks like solving, handles Amazons endgames pretty well.

The game of Go is one of the games that still withstand classical Artificial Intelligence approaches. Hence, it is a good testbed for new AI methods. Amongst them, Monte-Carlo led to promising results. This method consists of building an evaluation function by averaging the outcome of several randomized games. The paper introduces a new strategy, which we call Objective Monte-Carlo, to improve this evaluation. Objective Monte-Carlo is composed of two parts. The first one is a move-selection strategy that adjusts the amount of exploration and exploitation automatically. We show experimentally that it outperforms the two classical strategies previously proposed for Monte-Carlo Go: Simulated Annealing and Progressive Pruning. The second part of our algorithm is a new backpropagation strategy. We show that it gives better results than Minimax in this context. Finally we discuss the extension of this method to other problems.

This paper describes the generation and utilisation of a pattern database for 19x19 go with the Knearest-neighbor representation. Patterns are generated by browsing recorded games of professional players. Meanwhile, their matching and playing probabilities are estimated. The database created is then integrated into an existing go program, INDIGO, either as an opening book or as an enrichment of other pre-existing hand-crafted databases used by INDIGO move generator. The improvement brought about by the use of this pattern database is estimated at 15 points on average, which is significant on go standards.

This paper experimentally evaluates multiagent learning algorithms playing repeated matrix games to maximize their cumulative return. Previous works assessed that Qlearning surpassed Nash-based multi-agent learning algorithms. Based on all-againstall repeated matrix game tournaments, this paper updates the state of the art of multiagent learning experiments. In a first stage, it shows that M-Qubed, S and bandit-based algorithms such as UCB are the best algorithms on general-sum games, Exp3 being the best on cooperative games and zero-sum games. In a second stage, our experiments show that two features -forgetting the far past, and using recent history with statesimprove the learning algorithms. Finally, the best algorithms are two new algorithms, Qlearning and UCB enhanced with the two features, and M-Qubed.

This paper describes multi-agent learning experiments performed on tactical sequences of the pursuit evasion game on very small grids. It underlines the performance difference between a centralized approach and a distributed approach when using Rmax, a model-based reinforcement learning algorithm. The prey's goal is to go out of the grid and the predators' goal is to kill the prey. The prey may learn or not. The predators learn in two ways: in the centralized approach they are part of one single learning agent, and in the distributed approach, each predator is a learning agent in itself. Every agents learn to accomplish its goal by using Rmax. Our results compare the centralized approach with the distributed approach. Future works mainly include scaling up to larger boards using model-free algorithms, and exploring partial observability of agents.

Playing repeated matrix games (RMG) while maximizing the cumulative returns is a basic method to evaluate multi-agent learning (MAL) algorithms. Previous work has shown that U CB, M 3, S or Exp3 algorithms have good behaviours on average in RMG. Besides, hedging algorithms have been shown to be effective on prediction problems. An hedging algorithm is made up with a top-level algorithm and a set of basic algorithms. To make its decision, an hedging algorithm uses its top-level algorithm to choose a basic algorithm, and the chosen algorithm makes the decision. This paper experimentally shows that well-selected hedging algorithms are better on average than all previous MAL algorithms on the task of playing RMG against various players. S is a very good top-level algorithm, and U CB and M 3 are very good basic algorithms. Furthermore, two-level hedging algorithms are more effective than one-level hedging algorithms, and three levels are not better than two levels.

In this paper, we introduce a new heuristic search algorithm based on mean values for real-time planning, called MHSP. It consists in associating the principles of UCT, a banditbased algorithm which gave very good results in computer games, and especially in Computer Go, with heuristic search in order to obtain a real-time planner in the context of classical planning. MHSP is evaluated on different planning problems and compared to existing algorithms performing on-line search and learning. Besides, our results highlight the capacity of MHSP to return plans in a real-time manner which tend to an optimal plan over the time which is faster and of better quality compared to existing algorithms in the literature.

Cet article décrit la génération automatique et l'utilisation d'une base de patterns pour le go 19x19. La représentation utilisée est celle des K plus proches voisins. Les patterns sont engendrés en parcourant des parties de professionnels. Les probabilités d'appariement et de jeu des patterns sontégalement estiméesà ce moment là. La base créée est intégrée dans un programme existant, Indigo. Soit elle est utilisée comme un livre d'ouvertures en début de partie, soit comme une extension des bases pré-existantes du générateur de coups du programme. En terme de niveau de jeu, le gain résultant est estiméà 15 points en moyenne.

Dans cet article, nous présentons un nouvel algorithme de recherche heuristique basé sur le calcul de moyennes pour la planification temps réel, appelé MHSP (Mean-Based Heuristic Search Planning). Il associe les principes d'UCT (Upper Confidence for Tree), un algorithme de type bandit ayant donné de très bons résultats dans le domaine des jeux, et plus particulièrement dans le cadre du jeu de Go, et une recherche heuristique en vue d'obtenir un planificateur temps réel dans le contexte de la planification d'actions. MHSP est évalué sur différents problèmes de planification et comparé aux algorithmes de recherche en ligne et d'apprentissage existants. Nos résultats mettent en évidence la capacité de MHSP à retourner en temps réel des plans qui tendent vers un plan optimal au cours du temps. Ils montrent de plus que MHSP est plus rapide et retourne des plans de meilleure qualité que les algorithmes existants dans la littérature.

Cet article propose deux contributions pour améliorer la phase de sélection d'actions dans le cadre de la planification temps réel. Tout d'abord, la première amélioration s'appuie sur un agenda de buts pour classer les buts par ordre de difficulté croissante et tente ensuite de les résoudre de manière incrémentale. La seconde amélioration consiste à effectuer des sauts, c'est-à-dire à choisir à chaque pas de décision une séquence d'actions qui sera exécutée et non plus une seule action. Pour évaluer ces deux améliorations, nous avons développé un algorithme de planification temps réel dans lequel la phase de sélection d'actions peut être guidée par un agenda de buts et également capable d'exécuter un saut à chaque pas de décision. Les résultats expérimentaux, réalisés sur les problèmes de planification classiques issus des différentes compétitions de planification, montrent que l'agenda de buts ainsi que les sauts améliorent de manière importante la sélection d'actions dans le contexte de la planification temps réel. Utilisées simultanément, ces deux améliorations permettent d'augmenter de manière significative la qualité des plans trouvés ainsi que la vitesse pour les obtenir.

This paper describes experiments using reinforcement learning techniques to compute pattern urgencies used during simulations performed in a Monte-Carlo Go architecture. Currently, Monte-Carlo is a popular technique for computer Go. In a previous study, Monte-Carlo was associated with domain-dependent knowledge in the Go-playing program Indigo. In 2003, a 3x3 pattern database was built manually. This paper explores the possibility of using reinforcement learning to automatically tune the 3x3 pattern urgencies. On 9x9 boards, within the Monte-Carlo architecture of Indigo, the result obtained by our automatic learning experiments is better than the manual method by a 3-point margin on average, which is satisfactory. Although the current results are promising on 19x19 boards, obtaining strictly positive results with such a large size remains to be done.

In the context of real-time planning, this paper investigates the contributions of two enhancements for selecting actions. First, the agenda-driven planning enhancement ranks relevant atomic goals and solves them incrementally in a best-first manner. Second, the committed actions enhancement commits a sequence of actions to be executed at the following time steps. To assess these two enhancements, we developed a real-time planning algorithm in which action selection can be driven by a goal-agenda, and committed actions can be done. Experimental results, performed on classical planning problems, show that agenda-planning and committed actions are clear advantages in the real-time context. Used simultaneously, they enable the planner to be several orders of magnitude faster and solution plans to be shorter.

The game of Amazons is a fairly young member of the class of territory-games. The best Amazons programs play now at a high level, but can still be defeated by humans expert of the game. Our focus here is on the solving of endgames, with the goal of improving the general playing level of Amazons programs. Our comparative study of four solvers, DFPN, WPNS, Alpha/Beta and Monte-Carlo Tree-Search, shows that usual game-playing methods used for Amazons, namely Alpha-Beta and Monte-Carlo Tree-Search, are best suited for this task. This result also emphasizes the need of an evaluation function for the game of Amazons.

Monte-Carlo Tree Search (MCTS) is a powerful tool in games with a finite branching factor. This paper describes an artificial player playing the Voronoi game, a game with an infinite branching factor. First, this paper shows how to use MCTS on a discretization of the Voronoi game, and the effects of enhancements such as RAVE and Gaussian processes (GP). A first set of experimental results shows that MCTS with UCB+RAVE or with UCB+GP are first good solutions for playing the Voronoi game without domain-dependent knowledge. Second, this paper shows how to greatly improve the playing level by using geometrical knowledge about Voronoi diagrams, the balance of diagrams being the key concept. The second set of experimental results shows that a player using MCTS and geometrical knowledge outperforms the player without knowledge.

In this paper, we introduce a new heuristic search algorithm based on mean values for anytime planning, called MHSP. It consists in associating the principles of UCT, a bandit-based algorithm which gave very good results in computer games, and especially in Computer Go, with heuristic search in order to obtain an anytime planner that provides partial plans before finding a solution plan, and furthermore finding an optimal plan. The algorithm is evaluated in different classical planning problems and compared to some major planning algorithms. Finally, our results highlight the capacity of MHSP to return partial plans which tend to an optimal plan over the time.