The 2004 Competition

Consulting Committee

  • Stefan Edelkamp and Jörg Hoffmann (co-chairs, ``classical'' part)
  • Michael Littman and Hakan Younes (co-chairs, ``probabilistic'' part)
  • Fahiem Bacchus
  • Drew McDermott
  • Maria Fox
  • Derek Long
  • Jussi Rintanen
  • David Smith
  • Sylvie Thiebaux
  • Daniel Weld

The International Planning Competition has been a bi-annual event, hosted at the Artificial Intelligence Planning and Scheduling Conference series. The objectives of the competition are to provide a forum for empirical comparison of planning systems, to highlight challenges to the community in the form of problems at the edge of current capabilities, to propose new directions for research and to provide a core of common benchmark problems and a representation formalism that can aid in the comparison and evaluation of planning systems. Although the series has a competitive style (individual systems are identified for exceptional performance at the event itself), the focus is on data-collection and presentation, with interpretation of results being understated. The real goal of the competition is to make as much data as possible available to the community.

The competition event began in 1998 when Drew McDermott and a committee created a common specification language (PDDL) and a collection of problems forming a first benchmark. Five systems participated in the competition. In 2000, Fahiem Bacchus continued this work, and the event attracted 16 competitors. The event was extended to include both fully automatic and hand-tailored planning systems. Both STRIPS and ADL domains were used but no further extensions were made to the language. In 2002, the competition was run by Derek Long and Maria Fox. The event attracted 14 competitors, and focussed on planning in temporal and metric domains. McDermott's PDDL was not expressive enough to enable the modelling of durative actions and continuous resource consumption, so it was extended to enable the modelling of these features. The resulting competition language PDDL2.1 consisted of three levels of expressive power. Level 1 was ADL planning as before, level 2 added numerical variables, level 3 added durational constructs.

The 4th International Planning Competition, IPC-4, was hosted at ICAPS-2004. IPC-4 built on the previous efforts, in particular the language PDDL2.1. The competition event was extended and revised in several respects. In particular, IPC-4 featured, for the first time, a competition for probabilistic planners, so that the overall competition was split into a classical part -- a continuation of the previous events -- as well as a probabilistic part. In the latter part, co-organized by Michael Littman and Hakan Younes, the main objective of the event was to introduce a common representation language for probabilistic planners, and to establish some first benchmarks and results. For more information on the probabilistic part of IPC-4, see Michael Littman's page.

The classical part of IPC-4 was co-organized by Stefan Edelkamp and Joerg Hoffmann. The event attracted 19 competing systems (22 when counting different system versions). Several important revisions to the previous classical competitions were made:

  1. Our main effort was to devise a range of benchmark domains that are close to applications, and diverse in structure (see details on the ``Domains'' page).
  2. We made two relatively minor language extensions to PDDL2.1 levels 1 to 3, namely derived predicates and timed initial literals. The resulting language is called PDDL2.2. Both new language features are practically motivated and were put to use in some of the IPC-4 domains.
  3. We provided domain formulations where problem constraints were compiled from the more complex into the less complex PDDL2.2 subsets. In particular, in many of our domains the most natural domain formulation comes with complex precondition formulas and conditional effects, i.e., in ADL. In difference to the previous IPCs, were the more interesting (ADL) problem constraints were dropped in order to obtain the STRIPS domains, we compiled the ADL constructs into (fully grounded, most of the time) STRIPS. The resulting STRIPS benchmarks are structurally much more interesting than most of the traditional ones.
  4. We separated the optimal planners -- those that prove a guarantee on the quality of the found solution -- from the sub-optimal planners -- those that don't. Clearly, the huge gap between the runtime performances currently achieved by these two kinds of planners (in most of the commonly used benchmark domains) demands such a separation. Seven out of the 19 (respectively 22) competing systems were optimal planners.
  5. We made the competition more understandable at conference time. At ICAPS'04, a booklet describing all competing systems was distributed to all conference participants. Also, the detailed results of the competition were available in the form of posters showing runtime and plan quality graphs (the posters can be downloaded from the ``Results'' page).