Call for AM Comparisons

Call in PDF format

Compare Your Segmentation Algorithm to the COST211quat Analysis Model

COST 211 would like to provide you with the opportunity to show that your segmentation algorithm yields better segmentation results than the COST211quat Analysis Model. It is planned to invite a number authors of promising approaches to present their comparison results in a workshop or a publication as well as to join COST 211quat group and enrich the Analysis Model with your ideas. In the following, it is described how to compare your segmentation results to the results from the COST 211quat Analysis Model.

The exhaustive comprehensive results will be reported at WIAMIS 2001 (to be held in Tampere, Finland) but we will permanently show all proposed results on a web page. On the basis of both subjective and objective evaluation, the author of the best proposal will be invited as "keynote speaker" to the workshop. A special journal issue will also publish the best contributions.

The final deadline for submitting a proposal is March 15, 2001.

  1. Context

    Nowadays, with the emergence of the MPEG-4 and MPEG-7 standards, (semi-)automatic image and video segmentation is more and more relevant. Unfortunately, video segmentation is still an unsolved problem despite the effort of many researchers. Among these, COST 211 has set up a generic algorithm (referred to as COST AM) for segmentation of moving images. In parallel COST 211 has defined some metric that helps comparing different segmentation algorithms on an objective basis.

    In order to help research to advance and have a better idea of the performances of the various existing software, COST 211 has decided to issue the present call for comparisons. The COST211quat group (http://www.iva.cs.tut.fi/COST211/index.html) has also set up a web site intended as an exchange platform for algorithms and sequences related to video segmentation and indexing: http://www.tele.ucl.ac.be/EXCHANGE/.

    The following of the paper presents the call objectives, the experimental conditions for the comparison as well as the used metric. All material necessary to perform the comparison is available from the exchange web site.

    For any additional information, please contact the chairman of the group:
    Prof. Moncef Gabbouj
    Tampere University of Technology
    Signal Processing Laboratory
    P.O. Box 553 Tampere
    FIN-33101
    Finland
    Moncef.Gabbouj@tut.fi

  2. What is called for

    The primary aim of the call for comparison is to compare various segmentation algorithms one with each other in order to see the pros and cons of every technique. To this aim, COST has defined a procedure that is presented hereunder. Nevertheless, in order to better fit with the needs of different application domains, it is also possible to submit new sequences and also to contribute to the definition of the objective evaluation criteria.

    All three types of proposals have to be submitted via the exchange web site (http://www.tele.ucl.ac.be/EXCHANGE/). Please note that you have to register to this web site first.

    1. Segmentation algorithms

      The main target of the call is a segmentation algorithm. For your proposal to be complete, you have to:

      • Provide an executable version of your algorithm (under Windows, Linux or Solaris) along with a short but comprehensive description of the way to use it. If some parameters are to be filled, this should be clearly indicated. However, only a unique set of parameters, valid for all sequences, will be accepted.

      • Provide the resulting segmentation masks for all test sequences. The input material is listed in section 3.2 while the formats are described in section 3.1.

      • Submit all these files via ftp (ftp ftp.tele.ucl.ac.be, login = exchange, password = exchange).

      • Fill the on-line questionnaire.

      As a result, evaluation will be performed on the results (segmented sequences only) while the algorithm performances in terms of computational time… are only used for the purpose of information. This will also guarantee that you do not need to disclose the technical details of your proposal. However, in order to ensure the correctness of the submitted result, an executable version of the algorithm is mandatory.

    2. Sequences to segment

      If you think that your segmentation algorithm is better suited for one precise type of applications, please feel free to submit your own sequences along with perfect (hand-segmented or user-assisted) segmentation masks. We will be happy to include them into our test set.

      A proposal for a new sequence should contain:

      • The sequence along with the reference masks, in the format described in section 3.1. This is to be submitted via ftp (ftp ftp.tele.ucl.ac.be, login = exchange, password = exchange).

      • A short description of the sequence via the on-line questionnaire.

      • A copyright release stating that you agree to release the copyright on your material for research purposes.

    3. Objective criteria for segmentation evaluation

      Like the PSNR measure is used to roughly evaluate and compare the quality of coded sequences, the objective criteria defined by COST 211 do not provide a final judgment about the quality of segmentation results. It will never replace subjective evaluation (tape viewing). Nevertheless, it aims at providing a first approximation.

      Since it is for sure a very hot topic, COST 211 considers that one has the right not to fully agree with the objective criteria of section 3.4, and everybody is encouraged to contribute to it.

      Proposals with respect to the objective criteria can be submitted under any format that appears appropriate. Please submit your material via ftp (ftp ftp.tele.ucl.ac.be, login = exchange, password = exchange) and do not forget to fill the appropriate part of the questionnaire.

  3. Experimental conditions for algorithm comparison

    1. Format of sequences and segmentation masks

      Original sequences are all proposed in CIF format (352x288 pels, 4:2:0 (chrominance is only 176x144 pels)) within a RAW file: all Y pels (8 bits, 1 byte par pel) are first written in raster scan order, then all Cb and finally all Cr pels. The second image directly follows the first one, and so on. The file has no header.

      Sequences of masks undergo the same format but have no chrominance components. The luminance value have the following meaning: 0 means that the pixel belongs to the background. Any positive value refers to a (foreground) moving objects. Pixels sharing the same positive value are part of the same object.

      IMPORTANT NOTICE: Mask sequences must have the same number of frames than the original sequence they are related to. It means that if your algorithm is unable to offer any segmentation for the first image you have to provide a black mask (all zero pixels) for this image.

    2. Test Material

      So far, COST 211 has defined the following sequences to be part of the test set for the call for comparison. All sequences are in CIF format (see section 3.1).

      Sequence name

      Target objet(s)

      Sequence owner

      Related information

      Mother & Daughter

      the mother, the daughter

      AT&T

      300 images, 30Hz

      Hall-Monitor

      the two men

      Texas Instruments

      300 images, 30Hz

      Container-Ship

      the two ships, the bird

      Hughes

      300 images, 30Hz

      Coastguard

      two ships

      Hughes

      300 images, 30Hz - pan

      Table-Tennis

      The two players, the ball, the spectators

      CCETT

      300 images, 25Hz - pan and zoom

      Erik

      the person

      Universität Hannover

      50 images, 12.5Hz

      Roadsurveillance1

      the road vehicles

      MODEST (AC 304)

      300 images, 25Hz

      Roadsurveillance2

      the road vehicles

      MODEST (AC 304)

      300 images, 25Hz

      Urbicande

      the pedestrians

      MODEST (AC 304)

      300 images, 12.5Hz

      Penguin

      the penguins

      MoMuSys (AC 098)

      240 images, 25Hz - (small) pan

      Giraffe

      the giraffes

      MoMuSys (AC 098)

      300 images, 25Hz - hand-held cam.

      Pelwalking

      the pelwalkings

      MoMuSys (AC 098)

      220 images, 25Hz - (small) pan

      Lion

      the lions

      MoMuSys (AC 098)

      250 images, 25Hz - pan and zoom

      Stair_wide

      the persons

      MoMuSys (AC 098)

      1475 images, 25Hz

      All sequences, along with the (already existing) reference masks, are available from the exchange web site (http://www.tele.ucl.ac.be/EXCHANGE/.

      CAUTION: By downloading the various sequences, the following conditions are accepted: the utilisation of the sequences is allowed only in the framework of COST 211 or in the framework of the COST 211 Call for Comparison. A redistribution to third parties is not allowed. Every sequence is provided with authorization of its owning organization (cfr. above list); and is (c) to this organization. The same conditions apply to all segmentation masks.

    3. Subjective Evaluation of Segmentation Results

      Tape viewing allows performing subjective evaluation of the segmentation results. The human evaluator has to watch the segmentation results, i.e. a sequence of masks with all target objects blended and a sequence of masks with blended background. He/she should consider the spatial accuracy and the temporal coherency of the estimated object masks.

      During COST 211quat AG meeting, the group will systematically compare the results of the various proposals. If you want to do it also, please feel free to download the sequences of object masks generated by the COST 211quat Analysis Model (only background vs foreground segmentation) in order to compare with your own results. For convenience, a Java program to compare side by side two different segmentation results is provided on the exchange site.

    4. Objective evaluation of segmentation results

      For objective evaluation of the segmentation results the three criteria proposed in [1] have to be used. These evaluation criteria compare the spatial accuracy and the temporal coherency of the estimated object masks to the reference object masks (provided in the COST211 test set). The three criteria are:

      (1) Spatial quality as a measure of the number and location of deviating pels between the estimated and the original mask, related to the number of pels belonging to the reference mask.

      (2) Temporal stability of the estimated masks with respect to the reference ones in terms of fluctuating spatial quality.

      (3) Temporal coherency as a measure of the relative variation of the gravity centers of both the reference and the estimated masks.

      Using the provided reference masks, one could compute the objective criteria for his/her own algorithm, and compare it to the ones of the COST AM. COST 211 will take care of summarizing all objective criteria for all submitted algorithms on the exchange web site. A tool to perform this objective evaluation along with more details about the algorithm is available on the exchange site.

  4. What COST is doing with your proposal

    Proposals for new test sequences or better objective criteria will always be evaluated during the next COST 211quat AG. After the meeting, some member will come back to you with comments, proposals...

    Concerning the proposals of segmentation algorithm, your resulting segmentation masks and the algorithm executable (if you agree to do so) will be placed on the exchange web site, along with your contact information. Every meeting, COST 211 will establish a ranking of all proposals. No other information concerning your algorithm will be divulged without your prior and explicit agreement.

    If any new sequence is inserted in the test set, COST 211 will take care of generating segmentation masks for all previously submitted algorithms thanks to the executable that has been provided.

  5. References

    [1]P. Villegas, X. Marichal, A. Salcedo, "Objective Evaluation of Segmentation Masks in Video Sequences", WIAMIS 99 workshop, Berlin, May 1999, pp. 85-88.