Re: Fwd: ICSME 2025 Research Papers review assignments
by Boll, Marian Alexander (INF)
Hi all,
I'd like to take (122) Configurable Ensembles for Software Similarity: Challenging the Notion of Universal Metrics.
I feel the review guideline is quite a bit too extensive and restrictive but try to adhere to it.
Cheers
Belix
________________________________
From: Timo Kehrer <timo.kehrer(a)unibe.ch>
Sent: Monday, March 17, 2025 10:17 PM
To: seg-staff(a)list.inf.unibe.ch <seg-staff(a)list.inf.unibe.ch>
Subject: [SEG-staff] Fwd: ICSME 2025 Research Papers review assignments
Dear all,
if anyone wants to serve as sub-reviewer for ICSME 2025, please let me know.
These are the papers I got (PDFs attached, please do not share):
(79) ADPP: Automated Data-centric Program Partitioning
(81) Towards Better Understanding of Code Changes: An Algorithm to Remove Unintended Moves in GumTree
(117) Understanding Commercial Low-code Application Bugs
(122) Configurable Ensembles for Software Similarity: Challenging the Notion of Universal Metrics
(146) Predicting Clone-proneness: An Exploratory Study with Deep Learning Models
Best,
Timo
-------- Weitergeleitete Nachricht --------
Betreff: ICSME 2025 Research Papers review assignments
Datum: Mon, 17 Mar 2025 13:48:39 +0100
Von: ICSME Research Papers Track <icsme2025_research(a)easychair.org><mailto:icsme2025_research@easychair.org>
An: Timo Kehrer <timo.kehrer(a)inf.unibe.ch><mailto:timo.kehrer@inf.unibe.ch>
[Sie erhalten nicht häufig E-Mails von icsme2025_research(a)easychair.org<mailto:icsme2025_research@easychair.org>. Weitere Informationen, warum dies wichtig ist, finden Sie unter https://aka.ms/LearnAboutSenderIdentification ]
Dear Timo Kehrer,
Thanks again for serving on the PC of ICSME 2025.
This is a rather long email regarding the paper assignments, review instructions and review criteria, but please read it in full as it will make your job as a reviewer easier.
We have received 188 submissions. 21 papers were withdrawn by the full paper deadline. We removed 18 other submissions ourselves for which the authors did not submit a full paper. Out of the remaining 149 papers, we desk-rejected 3 papers for violating submission guidelines. This leaves 146 papers for review. Given the PC size, we have assigned 4-5 papers per PC member. You will find your paper assignment at the end of this email.
1) FORMAT AND DOUBLE-BLIND REQUIREMENTS
Papers must strictly adhere to the two-column IEEE conference proceedings format. Also, papers must be in PDF, anonymized, and must not exceed 10 pages (including figures and appendices) plus up to 2 pages that contain ONLY references.
We have already screened papers regarding these requirements. If you identify any missed cases, please report them to us as soon as possible.
For minor format violations (e.g., slightly different font or template), we have decided to handle these during the camera-ready stage. Please note such font/template violations in your reviews, but do not reject a paper because of them.
Some papers are shorter than 10 pages. While these are shorter than the typical full paper length, our CfP does not have a minimum page length. Therefore, please judge these papers as full papers (i.e., whether the provided content contains all the necessary information for a full paper) rather than forming your opinion solely based on their length.
Also, report to us immediately any missed cases of Conflicts of Interest. We ask you to NOT actively try to determine the authors of the papers.
2) REVIEW FORM
Following the tradition of ICSME and other major conferences, our goal is not to reject papers, but to accept them. Therefore, please be *positive* and provide *constructive* reviews. To support you in preparing quality reviews and to provide transparent reviews to authors, the review form on EasyChair includes the following fields (for your convenience we also include the descriptions of fields in the form on EasyChair):
* Overall recommendation: Reject, Weak reject, Weak accept, Accept, Accept & nominate for distinguished paper (only use ‘Accept & nominate for distinguished paper’ if you believe a paper is outstanding and should be considered for a Distinguished Paper award)
* Reviewer’s confidence: Expert, Knowledgeable, Some familiarity, No familiarity
* Paper summary: Brief summary of the paper (as you understand it)
* Strengths (bullet list): Summarize the key strengths of the paper, in light of the review criteria.
* Weaknesses (bullet list): Summarize the key weaknesses of the paper, in light of the review criteria.
* Artifacts: Artifact provided and in line with what is declared in the paper. Artifact provided, but not inline with what is declared in the paper، Artifact not provided, but authors provide a convincing argument as to why it cannot be provided. Artifact not provided, without explanation (or a non-convincing explanation)
* Summary comments for authors: Please provide a summary of your assessment to justify your overall recommendation. You may elaborate on the strengths and weaknesses and refer to the evaluation criteria (see below). Please do not mention your overall recommendation in the comments.
* Questions for the authors: Please fill in this section ONLY if answers to these questions will affect your score/decision for this paper. Otherwise, clearly indicate "No Questions."
* Confidential remarks for the program committee
* Review criteria: see below
3) REVIEW CRITERIA
For ICSME 2025 we use structured reviews, i.e., there will be a field for each review criterion on the review form. For your convenience, we also explain each criterion in the review form itself.
3.1) Originality and novelty
The extent to which the contribution is sufficiently original and is clearly explained with respect to the state-of-the-art. Note that originality is not about providing surprising or unexpected results or the complexity of a proposed solution, but how the work advances the body-of-knowledge. If a paper lacks important references, we ask reviewers to provide suggestions, but avoid self-citations. When a reviewer’s own work is extremely relevant, they should always contact the PC co-chairs and provide potential alternatives of other related work. Replications that bring new knowledge are welcome.
Do *not* reject papers just because:
– the work is preliminary (especially if an idea is very novel)
– the work is incremental (we always build on top of previous research)
– it reports negative results
– the idea is simple
– you think some other technique that doesn’t exist yet would be better
3.2) Importance of contribution (significance)
The extent to which the paper’s contributions can impact the field of software maintenance and evolution, and, if needed, under which assumptions.
Do *not* reject papers just because:
– it does not include empirical evidence of impact (potential impact is more important)
– you don’t like the topic or don’t find it interesting (would someone in the community?)
– the impact is not immediate for the software industry, the immediate impact may be in the software engineering research community (e.g., methodological contribution)
3.3) Soundness (proper use of research methods)
The extent to which the paper’s contributions and/or innovations address its research questions and are supported by rigorous application of appropriate research methods. Papers should employ rigor in their research methodology (including choosing appropriate methods and procedures). Soundness is relative to claimed contributions (e.g., if a paper finds a correlation, and that is a notable discovery, do not critique it for not also demonstrating causality).
Do *not* reject papers just because:
– the methods are not the ones you would have selected (are they appropriate?)
– the results may not generalize (papers should clearly explain assumptions and scope of contribution)
– Balance the novelty with the extent of the evaluation (very novel papers may be more preliminary)
Furthermore:
– Avoid applying criteria for quantitative methods to qualitative methods or industrial studies (e.g., critiquing a case study for a “small N”).
– Avoid critiquing a lack of a statistically significant difference for case study research; a lack of statistical difference can be a discovery, too (but must be supported by enough statistical power).
– Avoid asking for the paper to do more than it claims if the demonstrated claims are sufficiently publishable (e.g., “I would publish this if it had also demonstrated knowledge transfer”).
– Avoid relying on inexpert, anecdotal judgements (e.g., “I don’t know much about this but I played with it once and it didn’t work”).
– Do take into account the effort it took to run the study; this contributes to the value of results.
3.4) Evaluation (where applicable)
The paper’s claimed contributions are supported by empirical evidence. Papers claiming improvements (e.g., over state of the art or baseline techniques) should also design proper empirical evaluations to confirm that such improvements are achieved. Judge papers based on the match between claims and evaluation.
Do *not* reject papers just because:
– further experiments would be possible, if the included ones are sufficient to support the claims
some hypothetical baseline was not considered, if such a baseline is not available and is not easily re-implementable
– alternative design choices or implementations could provide even larger improvements
– there is no empirical evaluation in a paper claiming a purely theoretical contribution (e.g., the authors demonstrate theoretically an improved computational complexity)
3.5) Quality of presentation
The quality of writing, including clear descriptions, adequate use of the English language, absence of major ambiguity, clearly readable figures and tables, and adherence to the formatting instructions provided. Papers should be clear and concise, comprehensible to diverse audiences, contain sufficient information to understand how an innovation works and to understand how data was obtained, analyzed, and interpreted.
Do *not* reject papers just because:
– it has easily fixable spelling and grammar issues.
– it does not use all of the page count
– it does not follow a particular paper structure or order of sections
Avoid asking for more detail unless you are certain there is space; if there is not enough space, provide concrete suggestions for what to cut.
3.6) Comparison to related work
The extent to which the submission adequately reviews the prior literature.
Do *not* reject papers just because:
– minor prior related work is missing (only if missed work would have altered the study)
– non-peer reviewed work (including arXiv preprints, theses, blog posts, or tech reports) or work that has been accepted after the submission deadline (you can point these out to the authors if relevant but they are not alone grounds to reject)
3.7) Replicability (if relevant)
The extent to which the paper provides sufficient detail on methods and experiments, and shares information and artifacts that are practical and reasonable to share, to support replication and reproducibility.
4) AUTHOR QUESTIONS AND EARLY ACCEPTS/REJECTS
When filling in the review form, please use the “Questions for the Authors” field wisely and ask questions *only* if you think that the answers will affect your scores and decision about the paper. Otherwise, clearly indicate "No Questions."
Also, author responses will be only required for papers where answers to reviewers’ questions can affect the outcomes. For papers whose decision is already clear, we will send out direct accept/reject notifications at the beginning of the author response period.
5) REVIEW QUALITY
ICSME aims for an inclusive and transparent review process. We encourage reviewers to be open, positive and professional:
* Review authorship: PC members were invited because of their expertise. Therefore, we expect PC members to author their reviews, asking for sub-reviewers only for additional feedback. This means that reviewers may solicit help from others. However, reviewers should rewrite the review in their own words, and adjust the scores accordingly. The opinions should be represented as the PC member’s opinions, not those of a sub-reviewer.
* Be clear about what is missing: Even if, in the view of a reviewer, a paper does not meet the standards required for acceptance, we encourage reviewers to highlight what, in their opinion, would be necessary to make it acceptable for ICSME (while acknowledging that ICSME submissions are subject to the limitations of conference papers regarding lengths, etc.).
* Ethical issues: PC members should inform PC co-chairs if they detect any evidence related to plagiarism, concurrent submission, etc.
* Update reviews: Reviews can be updated at any time, i.e., we encourage reviewers to follow the submitted reviews of submissions assigned to them and make adjustments even before the official discussion period.
In summary, quality reviews are:
* Constructive, explicitly identifying the merits of the work, as well as feasible ways of addressing any of its weaknesses.
* Insightful, going deeply into the topic and the research methods.
* Organized, helping the authors clearly understand the reviewer’s opinions of strengths and weaknesses of the work.
* Impartial, demonstrating a commitment to the reviewing criteria of the conference, and not personal interests, speculation, or bias.
To protect authors' rights and research confidentiality, ICSME 2025 does not currently allow the use of Generative AI or AI-assisted technologies such as ChatGPT or similar services for peer review.
Reviewers, as well as authors and organizers, are expected to uphold the IEEE Code of Conduct.
6) LEAD REVIEWERS / DISCUSSION LEADS
You will be designated as the lead reviewer for at most 2 of the papers you review. We ask lead reviewers to do the following:
* Check reviews for any issues (e.g., quality, length) and work with reviewers to improve if needed; if necessary, bring issues to our attention.
* Kickstart and moderate discussion among the reviewers.
* Build consensus where possible.
* Make a recommendation on the paper.
* Write a meta-review to summarize the discussion among reviewers and explain the rationale behind their final recommendation (do not disclose reviewer identities in the meta-review).
7) REVIEWING TIMELINE
* Assignment of papers: March 17, 2025
* First half of reviews due: April 2, 2025
* Second half of reviews due: April 22, 2025
* First review discussion period: April 23, 2025 - April 29, 2025
* Early decisions due: April 30, 2025
* Early decisions notification: May 7, 2025
* Author response period: May 8, 2025 - May 15, 2025
* Final discussion period: May 16, 2025 - May 27, 2025
Thank you again for helping us make ICSME 2025 a success. We appreciate that reviewing is hard work. If you have any questions, please do not hesitate to contact us.
Matthias Galster and Dan Hao
ICSME 2025 Research Track Program Co-Chairs
==============
Paper Assignment
==============
(Should the assignment in this email not match the paper assignment in EasyChair, please reach out and report the mismatch.)
---------------------------------------------
(79) ADPP: Automated Data-centric Program Partitioning
(81) Towards Better Understanding of Code Changes: An Algorithm to Remove Unintended Moves in GumTree
(117) Understanding Commercial Low-code Application Bugs
(122) Configurable Ensembles for Software Similarity: Challenging the Notion of Universal Metrics
(146) Predicting Clone-proneness: An Exploratory Study with Deep Learning Models
