IPC 2014 - rules

Rules

This page summarizes the rules for the deterministic part of IPC-2014. It is meant to be the central place where participants can find everything they need to know to build a good entry for the competition and it will be frequently updated. In particular, you will find information or links about:

the different tracks/subtracks of the competition
the modeling language used to represent competition domains and problems

Other useful information such as deadlines, updates, and supporting documents/information can be found by following links from the competition homepage. If you still do not find the answer to your question, please use the mailing list to discuss the issue. Issues discussed on the mailing list which are of general importance will be reflected in the contents of this page.

Tracks

The deterministic part currently considers three tracks: sequential, temporal and preferences. In all cases, two different subtracks shall be arranged: optimal and satisficing; an additional multi-core and agile subtracks shall be arranged for the sequential part as well. Awards will be given to planners in each subtrack and not aggregated across several subtracks, so each subtrack essentially defines a competition on its own.

The eight subtracks differ in which PDDL features are required of the participating planners, which objective function planners are expected to optimize, and whether suboptimal solutions are considered acceptable.

In each track, there are certain "core features" (e.g. durative actions in the temporal track) that are prerequisites for entering the track, and there are also optional features (e.g. ADL, derived predicates) that may appear in some of the domains. For planners that do not support the optional features, we will offer equivalent domain versions where these features have been compiled away. So it is enough to support the core features to fully participate in a given track. Note, however, that compiled problem formulations are often significantly larger than the original problem formulations, so it may be advantageous to support the optional features directly.

Sequential: satisficing | optimal | multi-core. The objective of this track is to minimize total action cost of a plan.
Sequential: agile. The objective of this track is to minimize the CPU time needed for finding a plan.
Temporal: satisficing | optimal. The objective of this track is to minimize plan makespan.
Preferences: satisficing | optimal. The objective of this track is to find a plan with minimum penalization (a combination of total action cost and penalization for not achieving some goals).

Competition Rules

Potential participants are requested to send an email to both Lukas Chrpa (l.chrpa at hud.ac.uk) and Mauro Vallati (m.vallati at hud.ac.uk) by the 31st October 2013 with the registration form completed.

The evaluation process will comprise two phases: Initially, the competitors will be given a set of representative domain/problem instances to test their planners on their own machines. Then, they will submit a final version of the source code of their planners that will be run by the organizers on the actual competition domains/problems (unknown to the competitors till this time). This way no fine-tuning of the planners will be possible.

PDDL in this year's competition

IPC-2014 will be based on PDDL 3.1, i.e. the same language used in IPC-2011 with no extensions at all. A competing team may participate for any track. However, the competitors are strongly encouraged to participate in every track supported by their planners.

Evaluation schema

As in the IPC 2008 and 2011, this competition is going to be arranged as a blind evaluation. Thus, instead of having the competitors run their planners on problems provided (incrementally) by the organizers, we as the organizers will run all planners according to our schedule and only report the final results when the competition ends. The actual problem sets used in the competition will not be known to the competitors and will only be made public at the end of the IPC.

We will provide example problems for all tracks so that the participants can test their planners before submitting the final executable. We hope that the examples will cover all possible constraints that appear in the actual testing problems. However, if there is a problem running a participating planner on a given testing problem, we will try our best to work with the participants to figure out the issue. Please check the IPC-2014 homepage for information on when the example problems for all tracks will be available and when we will start running the submitted planners on testing problems.

Evaluation criteria: For most of the tracks we do preserve the same evaluation criteria used in the last IPC, that was focused on good plan quality. The agile track is the only one that emphasize the solving time. The details on the evaluation criteria are summarized below.

The following rules apply for optimal tracks:

Each planner is given 30 minutes to solve each problem. The time needed to solve each problem is not taken into account; what matters is only whether the problem was solved or not within 30 mins.
The quality of the resulting plans is an on/off criterion, since plans have to be optimal. The organizers will try to obtain optimal solutions for all problems, mainly through the development of specialized domain-specific solvers. If a planner produces at least one non-optimal plan in a given domain, all results of that planner in that domain are ignored, i.e., it is assigned a score of 0 for this domain. If a planner produces at least one non-optimal plan on at least two different domains, the planner is disqualified. However, the organizers will try to give the participants a chance to repair the bug that caused the problem and rerun the planner on the domain(s), in so far as this is possible within the time constraints and the blind evaluation mode of the competition.
So, the only thing that matters is the number of solved problems. All problems (either hard or easier) are considered equivalent. The winner of each category (sequential, temporal, preferences) is simply the planner that solves most of the problems.

The following rules apply for satisficing tracks (including multi-core sequential satisficing):

Each planner is given 30 minutes to solve each problem. The time needed to solve each problem is not taken into account; what matters is only whether the problem was solved or not within 30 mins.
The quality of the resulting plans is important! Suppose a problem with an optimal solution having quality Q* and a planner finding a plan with quality Q<Q* (assume higher qualities are considered best). This planner would aggregate the quality ratio Q/Q* to its total score (use Q*/Q in case lower qualities are considered best). The organizers will try to find optimal solutions for all problems, through the development of specialized domain-specific solvers. In cases where this is not feasible, Q* will be considered the best plan quality found by any competing planner or by the organizers.
The planner with the highest sum of quality ratios is declared the winner in each category.

The following rules apply for agile track:

Each planner is given 5 minutes to solve each problem. The quality of the resulting plans is not taken into account; what matters is only whether the problem was solved or not within 5 mins, and the CPU time required.
For a given problem let T* be the minimum time required by any planner to solve the problem. A planner that solves the problem in time T will get a score of 1/(1 + log10(T/T*)) for the problem. Those that do not solve the problem get a score of 0.
Runtimes below 1 sec get the same score.
The planner with the highest sum of scores over all evaluation problems is declared the winner in each category.

Miscellaneous Issues

Submission of source code: We require all competitors to make public, through the IPC-2014 website, the source code of the version competing in the competition. This will encourage information sharing and allow independent evaluation and double checking of the competition results by the planning community.

Hardware Platform: The competition will be run in a cluster of 256 cores. Each core is an AMD Processor 2.39 Ghz. Up to 4GB of RAM and 200 GB of hard disk memory will be available for each planner. Each planner will be run in a single node and no planner will be allowed to run in more than one simultaneously (although in the multicore track it will be allowed to use all the 4 cores of a node). No GPU is available. Memory and time will be externally limited.

Using external memory: External memory search is another trend in the search community that can have a big impact on planner performance, when the amount of main memory is limited. We do not disallow planners that use external memory search in the competition. However, the I/O time needs to be counted against the total allocated time for each run (expected to be 30 minutes at the moment).