SubSift supports peer review for PAKDD10

SubSift Tools have been used to support the peer review of research papers submitted for the Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD10), which will be taking place from 21-24 June 2010 in Hyderabad, India.

Programme committee (PC) chair, Professor Mohammed Zaki of Rensselaer Polytechnic Institute, New York, used SubSift to produce a custom web page for each of the 151 PC members, ranking the 352 submitted papers in order of their similarity to the reviewer’s published works. The PC members were then able to use this information to assist them in bidding for papers to review. Once reviewers’ bids were submitted, Zaki was able to use SubSift alongside existing tools to allocate the required number of reviewers for each paper.

During the allocation process, SubSift gained a new method of assigning suggested default bids on papers for each reviewer. SubSift currently allocates bids in the range 0 to 3 with the following meanings: 0 – Not Willing, 1 – In-a-pinch, 2 – Willing, 3 – Eager. Prior to PAKDD, SubSift required the PC chair to manually enter three similarity score thresholds such that papers with a similarity score (for a given reviewer) higher than the first threshold became a level 3 bid, papers scoring higher than the second threshold became a level 2 bid and so on down to level 0. Zaki suggested a much simpler alternative method of converting the similarity scores to bid levels:

One thing that would be useful is to automatically adjust the bid-levels per PC member. Say I want 5 papers with bid at level 3, 5 at level 2, 10 at level 1, then the software should automatically separate the bids into those 3 categories (instead of having to set a separate threshold per PC).

Professor Mohammed Zaki, PC Chair PAKDD10.

So, we rapidly added this method to SubSift just in time to help Zaki in the allocation of papers for review. Future SubSift users will now be able to choose between these two methods of calculating default bids.