Call for Challenges
The IEEE International Symposium on Biomedical Imaging (ISBI) 2022 is soliciting proposals for scientific challenges. Challenges have become an intrinsic part of ISBI in the past few years. The aim of the challenges is to accelerate the pace of biomedical science on demanding research and clinical problems. To achieve this goal, a challenge should aid quantitative comparisons of competing approaches to cutting-edge research problems in biomedical image analysis, using standardized datasets, evaluation metrics, and multi-institutional collaborations.
Each challenge proposal needs an organizer who will be responsible for providing training and testing datasets, defining the tasks, specifying the performance metrics, managing and evaluating entries (through appropriate software environments), and organizing on-site presentations in the challenge workshop. ISBI will assist in advertising the challenges, but the organizer is ultimately responsible for the success of the challenge.
The organization of a challenge and the agenda for each workshop will be determined by the challenge organizers. The challenge organizers should not rely solely on the on-site evaluation. We encourage scientific presentations at the workshop by the challenge participants. Multi-institutional collaborations are encouraged to ensure the wider applicability of the solutions. New challenge topics as well as topics that were addressed during previous challenges are acceptable. New challenge topics could introduce new imaging devices, new biomedical applications, or existing applications that would benefit from focused attention from the biomedical imaging research community. Challenges that address previous challenge topics could feature, for example, repeating a challenge to track how the field has advanced, addressing bottlenecks in existing processing pipelines, processing larger datasets, analyzing specific sub-populations, or exploring the utility of particular technologies such as deep learning. We also encourage the investigation of alternative formats for hosting grand challenges.
Prospective challenge organizers are encouraged to discuss their plans with the challenge co-chairs at an early stage before the acceptance of the proposal. After acceptance, the selected challenge proposals will be hosted on OpenReview.net to invite comments from the community. Challenge organizers will have the opportunity to present the summaries of their challenges during the main conference.
How to Submit a Challenge Proposal
Challenge proposals must be submitted on CMT submission portal as a single PDF file (not exceeding four A4-sized pages excluding references and biosketches) as per the guidelines below
Please include the following in the PDF document in IEEE Transactions format:
- Header: Include contest title, abstract, and keywords (up to five), as well as the name, affiliation, and contact information of the organizers.
- Abstract
- Introduction: Include the description of and motivation for the problem being addressed by the challenge, including the clinical context.
- Data and Challenge: Describe the dataset that will be used, and how it will be allocated for training and testing (with approximate number of samples), including the format of the training target and desired output. If there are differences in the sources or sub-populations of the training and the testing data, then the motivation for the differences should be described (see #5 below). Also, include a description of the challenge tasks. For example, the challenge task could be segmentation where the segmentation map has to be submitted in a specific format.
- Baseline and Evaluation: Include the details of the reference standard, such as the organizers’ basic or a previously published solution, and evaluation metrics. We encourage the contest organizers to, either themselves or through the challenge participants, test the robustness of the results. This may include, but need not be limited to, factoring in variance in results on various bootstrapped samples or testing on external cohorts from institutions that did not contribute to the training cohort. Additionally, we encourage challenge organizers to mention which portions of the BIAS checklist were relevant and considered.[https://arxiv.org/ftp/arxiv/papers/1910/1910.04071.pdf]
- Plan and schedule: A plan for organizing the contest should include the schedule for the release of the following: training data, algorithm or code for metric computation, single or multiple phases of test data, format and dummy examples of the submission files, format of the participant workshop manuscripts, submission of test results, submission of the manuscripts, declaration of the leaderboard, presentations at the workshop, and post-workshop leaderboard release (if any). Please also clarify if the participants are expected to release their source code, and if so, under what license and in what format or on which platform. An estimate of the number of participants based on the difficulty of the challenge and previous comparable events may also be included.
- References
- One-paragraph biosketch of each of the organizers.