Research on Information Science and Public Libraries

Automatizing the Assignment of the Submitted Manuscripts to Reviewers: A Systematic Review of Research Texts

Document Type : Original Article

Authors

Somayeh Paknahad 1
Abdolrasoul Jowkar 2
Mahdieh Mirzabeigi 3
Hajar Sotudeh 4
Mostafa Fakhrahmad 5

1 PhD Candidate of KIS, Shiraz University, Shiraz, Iran

2 Professor, Department of KIS, Shiraz University, Shiraz, Iran

3 Assistant Prof., Department of KIS, Shiraz University, Shiraz, Iran

4 Associate Prof., Department of KIS, Shiraz University, Shiraz, Iran

5 Assistant Prof., Department of Computer Science, Shiraz University, Shiraz, Iran

Abstract
Purpose: To systematicly review the automatazation of the assignment of the submitted manuscripts to reviewers in order to identify the status of research studies in this field in terms of types of evidence of expertise, types of retrieval models used, and the research gaps, and finally some suggestions for has been offered for future research.
Method: The current research followed the systematic review framework presented by Kitchenham & Charters (2007). To collect the research data, scientific databases were searched by using related keywords, and thus 50 records in English from 1992 to 2018 were retrieved. After applying the preliminary refinements to the entrance and exit criteria to the study and applying the experts’ reviews, 43 English research studies were selected for systematic review. Then, a checklist was designed and the required information was extracted from the studies.
Findings: The systematic review showed that the studies on automatization of the assignment of the submitted manuscripts to reviewers typically used four sources as the evidence of the candidate's expertise, including: the candidates’ self-expression, the candidates’ documents, collaborative networks and scientific relations between the candidates and the hybrid resource. In addition, Boolean model, expanded Boolean model, Fuzzy model, Vector Space model, Probabilistic model, the Probabilistic Latent Semantic Indexing, the Latent Dirichlet allocation, and Author-Subject model were among the information retrieval models used in the expert reviewer finding systems. The lack of attention to the design of the expert finding system for reviewers in national journals and conferences, lack of attention to knowledge resources, and lack of attention to the users' views on the design of the existing systems were among the research gaps in this field.
Originality/value: The use of hybrid resources in extracting the candidate's expert evidence, using knowledge resources, and paying attention to the users' perspectives in designing the expert finding system for reviewers can help to improve the performance of the existing systems. This research presents an overall picture of both the measures thus far taken and the measures should be taken in desiging the expert reviewer finding systems.

Keywords

Reviewer assignment
Expert finding system
Systematic review
۱. ابراهیمی، سعیدیه و فرج‌پهلو، عبدالحسین (۱۳۸۹). رویکردی سایبرنتیکی به سیستم داوری مقالات در مجلات علمی. فصلنامه مطالعات ملی کتابداری و سازماندهی اطلاعات، ۲۱(۲)، ۱۸۶-۱۹۸.
۲. پرنور، احسان و رضایی‌نور، جلال (۱۳۹۴). بهبودی در سامانه‌های پیشنهادگر خبره با استفاده از بسط پرسش و مدل فضای برداری. نشریه محاسبات نرم، ۳(۲)، ۴۱-۳۱.
۳. ترک‌زاده ماهانی، نفیسه (۱۳۹۳). استخراج خبرگی برای یک سامانۀ خبره‌یابی در حوزه‌های پژوهشی (پایان‌نامۀ کارشناسی ارشد). دانشگاه تهران، تهران، ایران.
۴. حسن زاده، محمد (۱۳۸۳). تأثیر مدل‌های بازیابی اطلاعات بر میزان ربط. مجله اطلاع‌شناسی، ۲(۱)، ۹۰-۶۳.
۵. عطاران، محمد؛ سنگری، علی‌اکبر و دهباشی، اکرم (۱۳۸۹). نگارش علمی در آموزش عالی: نگاهی به فرایند داوری مقالات علمی پژوهشی. نامۀ آموزش عالی، ۳(۹)، ۲۲-۹.
۶. عظیمی، مرجان (۱۳۹۴). شناسایی خبره براساس موضوعات سلسله‌مراتبی (پایان‌نامۀ کارشناسی ارشد). دانشگاه تهران، تهران، ایران.
۷. علی‌محمدزاده، خلیل؛ محبی، فاطمه و لباف، فاطمه (۱۳۹۲). مرور نظام‌مند سه دهه مقالات پژوهشی «علل سزارین» و راهبردهای مدیریت سلامت جمعیت. مطالعات راهبردی زنان، ۱۶(۳)، ۵۷-۷.
۸. فتاحی، سیدرحمت‌الله (۱۳۹۰). مجله‌های علمی پژوهشی و مسئله داوری مقاله‌ها (یادداشت سردبیر). فصلنامه علوم و فناوری اطلاعات، ۲۷(۱):۱-۳.
۹. کلانتری، عبدالحسین؛ روشن‌فکر، پیام و جواهری، جلوه (۱۳۹۰). مرور سه دهه تحقیقات علل طلاق در ایران. مطالعات راهبردی زنان، ۱۴(۳)، ۱۶۲-۱۲۹.
10. Amendola, G., Dodaro, C., Leone, N., & Ricca, F. (2016, November). On the application of answer set programming to the conference paper assignment problem. In Conference of the Italian Association for Artificial Intelligence (pp. 164-178). Springer, Cham.
11. Balog, K. Bogers, T. Azzopardi, L. De Rijke, M. & Van Den Bosch, A. (2007, July). Broad expertise retrieval in sparse data environments. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval (pp. 551-558). ACM.
12. Benferhat, S., & Lang, J. (2001). Conference paper assignment. International Journal of Intelligent Systems, 16(10), 1183-1192.
13. Biswas, H. K., & Hasan, M. M. (2007, March). Using publications and domain knowledge to build research profiles: An application in automatic reviewer assignment. In 2007 International Conference on Information and Communication Technology (pp. 82-86). IEEE.
14. Callaham, M. L., & Tercier, J. (2007). The relationship of previous training and experience of journal peer reviewers to subsequent review quality. PLoS medicine, 4(1), e40.
15. Champiri, Z. D., Shahamiri, S. R., & Salim, S. S. B. (2015). A systematic review of scholar context-aware recommender systems. Expert Systems with Applications, 42(3), 1743-1758.
16. Conry, D., Koren, Y., & Ramakrishnan, N. (2009, October). Recommender systems for the conference paper assignment problem. In Proceedings of the third ACM conference on Recommender systems (pp. 357-360). ACM.
17. Daş, G. S., & Göçken, T. (2014). A fuzzy approach for the reviewer assignment problem. Computers & industrial engineering, 72, 50-57.
18. Dumais, S. T., & Nielsen, J. (1992, June). Automating the assignment of submitted manuscripts to reviewers. In Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval (pp. 233-244). ACM.
19. Dung, N. D., Cong, N. H., & Anh, N. T. (2017). Algorithm of Dynamic Programming For Paper-reviewer Assignment problem.
20. Dutta, A. (1992). AUTOREF: A deductive database for automatic referee selection. Information & management, 22(6), 371-381.
21. Ebrahimi, A., & Farajpahlou, A., (2010). Cybernetic Approach to the Procedures of Peer- Reviewing inScholarly Journals. National Studies on Librarianship and Information Organization, 21(2), 186-198.
22. Ferilli, S., Di Mauro, N., Basile, T. M. A., Esposito, F., & Biba, M. (2006, June). Automatic topics identification for reviewer assignment. In International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems (pp. 721-730). Springer, Berlin, Heidelberg.
23. Goldsmith, J., & Sloan, R. H. (2007, July). The AI conference paper assignment problem. In Proc. AAAI Workshop on Preference Handling for Artificial Intelligence, Vancouver (pp. 53-57).
24. Hartvigsen, D., Wei, J. C., & Czuchlewski, R. (1999). The Conference Paper‐Reviewer Assignment Problem. Decision Sciences, 30(3), 865-876.
25. Jin, J., Geng, Q., Zhao, Q., & Zhang, L. (2017, April). Integrating the trend of research interest for reviewer assignment. In Proceedings of the 26th International Conference on World Wide Web Companion (pp. 1233-1241). International World Wide Web Conferences Steering Committee.
26. Jin, J., Niu, B., Ji, P., & Geng, Q. (2018). An integer linear programming model of reviewer assignment with research interest considerations. Annals of Operations Research, 1-25.
27. Karimzadehgan, M., & Zhai, C. (2009, November). Constrained multi-aspect expertise matching for committee review assignment. In Proceedings of the 18th ACM conference on Information and knowledge management (pp. 1697-1700). ACM.
28. Kitchenham, B., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering.
29. Kliewer, M. A., Freed, K. S., DeLong, D. M., Pickhardt, P. J., & Provenzale, J. M. (2005). Reviewing the reviewers: comparison of review quality and reviewer characteristics at the American Journal of Roentgenology. American Journal of Roentgenology, 184(6), 1731-1735.
30. Kolasa, T., & Król, D. (2010, June). ACO-GA approach to paper-reviewer assignment problem in CMS. In KES International Symposium on Agent and Multi-Agent Systems: Technologies and Applications (pp. 360-369). Springer, Berlin, Heidelberg.
31. Kou, N. M., Hou, U. L., Mamoulis, N., & Gong, Z. (2015, May a). Weighted coverage based reviewer assignment. In Proceedings of the 2015 ACM SIGMOD international conference on management of data (pp. 2031-2046). ACM.
32. Kou, N. M., Mamoulis, N., Li, Y., Li, Y., & Gong, Z. (2015b). A topic-based reviewer assignment system. Proceedings of the VLDB Endowment, 8(12), 1852-1855.
33. Lantz, B. (2013). Machine learning with R. Packt Publishing Ltd.
34. Li, B., & Hou, Y. T. (2016). The new automated IEEE INFOCOM review assignment system. IEEE Network, 30(5), 18-24.
35. Li, K., Cao, Z., & Qu, D. (2017, July). Fair reviewer assignment considering academic social network. In Asia-Pacific Web (APWeb) and Web-Age Information Management (WAIM) Joint Conference on Web and Big Data (pp. 362-376). Springer, Cham.
36. Li, L., Wang, Y., Liu, G., Wang, M., & Wu, X. (2015). Context-aware reviewer assignment for trust enhanced peer review. PloS one, 10(6), e0130493.
37. Li, X., & Watanabe, T. (2013). Automatic paper-to-reviewer assignment, based on the matching degree of the reviewers. Procedia Computer Science, 22, 633-642.
38. Liu, O., Wang, J., Ma, J., & Sun, Y. (2016). An intelligent decision support approach for reviewer assignment in R&D project selection. Computers in Industry, 76, 1-10.
39. Liu, X., Suel, T., & Memon, N. (2014, October). A robust model for paper reviewer assignment. In Proceedings of the 8th ACM Conference on Recommender systems (pp. 25-32). ACM.
40. Long, C., Wong, R. C. W., Peng, Y., & Ye, L. (2013, December). On good and fair paper-reviewer assignment. In 2013 IEEE 13th International Conference on Data Mining (pp. 1145-1150). IEEE.
41. Merelo-Guervós, J. J., & Castillo-Valdivieso, P. (2004, September). Conference paper assignment using a combined greedy/evolutionary algorithm. In International Conference on Parallel Problem Solving from Nature (pp. 602-611). Springer, Berlin, Heidelberg.
42. Mimno, D., & McCallum, A. (2007, August). Expertise modeling for matching papers with reviewers. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 500-509). ACM.
43. Nguyen, J., Sánchez-Hernández, G., Agell, N., Rovira, X., & Angulo, C. (2018). A decision support tool using Order Weighted Averaging for conference review assignment. Pattern Recognition Letters, 105, 114-120.
44. Nylenna, M., Riis, P., & Karlsson, Y. (1994). Multiple blinded reviews of the same two manuscripts: effects of referee characteristics and publication language. JAMA, 272(2), 149-151.
45. Ondrejka, A., & Zelinka, I. (2014, November). Similarity of authors' profiles and its usage for reviewers' recommendation. In 2014 9th International Workshop on Semantic and Social Media Adaptation and Personalization (pp. 3-8). IEEE.
46. Peng, H., Hu, H., Wang, K., & Wang, X. (2017, March). Time-aware and topic-based reviewer assignment. In International Conference on Database Systems for Advanced Applications. (pp. 145-157). Springer, Cham.
47. Protasiewicz, J., Pedrycz, W., Kozłowski, M., Dadas, S., Stanisławek, T., Kopacz, A., & Gałężewska, M. (2016). A recommender system of reviewers and experts in reviewing problems. Knowledge-Based Systems, 106, 164-178.
48. Rodriguez, M. A., & Bollen, J. (2008, October). An algorithm to determine peer-reviewers. In Proceedings of the 17th ACM conference on Information and knowledge management (pp. 319-328). ACM.
49. Schirrer, A., Doerner, K. F., & Hartl, R. F. (2007). Reviewer assignment for scientific articles using memetic algorithms. In Metaheuristics (pp. 113-134). Springer, Boston, MA.
50. Shon, H. S., Han, S. H., Kim, K. A., Cha, E. J., & Ryu, K. H. (2017). Proposal reviewer recommendation system based on big data for a national research management institute. Journal of Information Science, 43(2), 147-158.
51. Sowa, J. F. (1994). Knowledge representation: logical, philosophical, and computational foundations (Vol. 9, pp. 24-25). Brooks/Cole.
52. Sun, Y. H., Ma, J., Fan, Z. P., & Wang, J. (2007, January). A hybrid knowledge and model approach for reviewer assignment. In 2007 40th Annual Hawaii International Conference on System Sciences (HICSS'07) (pp. 47-47). IEEE.
53. Tang, J., Jin, R., & Zhang, J. (2008, December). A topic modeling approach and its integration into the random walk framework for academic search. In 2008 Eighth IEEE International Conference on Data Mining (pp. 1055-1060). IEEE.
54. Tayal, D. K., Saxena, P. C., Sharma, A., Khanna, G., & Gupta, S. (2014). New method for solving reviewer assignment problem using type-2 fuzzy sets and fuzzy functions. Applied intelligence, 40(1), 54-73.
55. Torkzadeh Mahani, N., Dehghani, M., Mirian, M. S., Shakery, A., & Taheri, K. (2018). Expert finding by the Dempster‐Shafer theory for evidence combination. Expert Systems, 35(1), e12231.
56. Wang, F., Chen, B., & Miao, Z. (2008, June). A survey on reviewer assignment problem. In International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems (pp. 718-727). Springer, Berlin, Heidelberg.
57. Wang, F., Shi, N., & Chen, B. (2010). A comprehensive survey of the reviewer assignment problem. International Journal of Information Technology & Decision Making, 9(04), 645-668.
58. Wang, F., Zhou, S., & Shi, N. (2013). Group-to-group reviewer assignment problem. Computers & Operations Research, 40(5), 1351-1362.
59. Wei, S. C., & Luo, H. Y. (2008). Evaluation of Information Retrieval Based Models for Recommendation of Paper Reviewers. Journal of Educational Media & Library Sciences, 45(4).
60. Xu, Y., Ma, J., Sun, Y., Hao, G., Xu, W., & Zhao, D. (2010). A decision support approach for assigning reviewers to proposals. Expert Systems with Applications, 37(10), 6948-6956.
61. Yang, K. H., Kuo, T. L., Lee, H. M., & Ho, J. M. (2009, September). A reviewer recommendation system based on collaborative intelligence. In Proceedings of the 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology-Volume 01 (pp. 564-567). IEEE Computer Society.
62. Yimam-Seid, D., & Kobsa, A. (2003). Expert-finding systems for organizations: Problem and domain analysis and the DEMOIR approach. Journal of Organizational Computing and Electronic Commerce, 13(1), 1-24.
63. Yin, H., Cui, B., Lu, H., & Zhao, L. (2016, November). Expert team finding for review assignment. In 2016 Conference on Technologies and Applications of Artificial Intelligence (TAAI) (pp. 1-8). IEEE.
64. Young, J. P. (2012). A comparative study of automated reviewer assignment methods.
65. Yunhong, X., & Xianli, Z. (2016, June). A LDA model based text-mining method to recommend reviewer for proposal of research project selection. In 2016 13th International Conference on Service Systems and Service Management (ICSSSM) (pp. 1-5). IEEE.
Research on Information Science and Public Libraries
Volume 25, Issue 3
Autumn 2019
Pages 482-457

Home

Submit Manuscript

Reviewers

Contact Us