Sign up for our newsletter, to get updates regarding the Call for Paper, Papers & Research.
Empowering Educators: Crowdsourcing Development Platforms for Malaysian Teachers
- Khalizul Khalid
- Rosmini Ismail
- 11-21
- Dec 25, 2024
- Educational Management
Empowering Educators: Crowdsourcing Development Platforms for Malaysian Teachers
*Khalizul Khalid, Rosmini Ismail
Faculty of Management and Economics, Universiti Pendidikan Sultan Idris, Malaysia
*Corresponding Author
DOI: https://dx.doi.org/10.47772/IJRISS.2024.ICAME2402
Received: 06 December 2024; Accepted: 10 December 2024; Published: 25 December 2024
ABSTRACT
The rapid advancements in Internet and World Wide Web (WWW) technologies have significantly transformed human life, facilitating collective intelligence activities that leverage the wisdom of crowds. This paper explores the development of a crowdsourcing platform designed to enhance educational assessment tools in Malaysia. The platform, grounded in the principles of collective intelligence, enables Malaysian teachers to create, share, and manage assessment instruments collaboratively. The study employs the System Development Life Cycle (SDLC) model to guide the platform’s development, encompassing phases from requirements analysis to implementation and maintenance. A focus group of schoolteachers provided insights into the current processes of assessment instrument preparation, revealing challenges in sharing and collaboration. The developed platform addresses these challenges by offering a controlled environment where teachers can access, create, edit, and share test questions, thus fostering a culture of knowledge exchange. The platform’s design and functionality were refined based on feedback from focus groups and expert evaluations, ensuring its effectiveness and user-friendliness. Testing phases, including alpha, beta, and user acceptance testing, were conducted to ensure the platform’s reliability and functionality. The launch phase involved promoting the platform and gathering user feedback for continuous improvement. The platform’s implementation aims to empower educators, improve the quality of school assessments, and encourage the widespread adoption of crowdsourcing in educational contexts. This paper provides a comprehensive overview of the platform’s development process, offering valuable insights and guidance for future crowdsourcing initiatives. The platform’s success is anticipated to contribute to the broader adoption of collective intelligence in educational settings, enhancing the quality and accessibility of educational resources in Malaysia.
Keywords: Crowdsourcing Platform, Educational Assessment, Collective Intelligence
INTRODUCTION
The rapid technological changes that have occurred over the past decade have made it easier for most people to accept and adapt to new computer devices and software such as Internet technology, the World Wide Web (WWW), and mobile technology. This development has made Internet and World Wide Web access an essential necessity in our daily life. The high acceptance, especially among netizens (network citizens), is further accelerating the growth of Internet technology in a more dynamic and rapid manner (Hauben, 2019). The impact of this technological advancement has transformed human life in their daily routines.
The rapid progress of Internet and WWW technologies has augmented the capacity for collective intelligence operations that were once conducted conventionally among online users. (D. K. Gupta & Sharma, 2013). Collective intelligence is a method of producing better decisions or activities by using the wisdom obtained collectively from the crowd, also referred to as the wisdom of the crowd by Surowiecki (2004). “Crowd” means “a group of people who are gathered together.” Crowds in the context of this study refer to “a group of people (crowd) that is structured independently and voluntarily based on shared goals, emotions, or experiences” (Prpić, Shukla, Kietzmann, & McCarthy, 2015).
The concept of crowd wisdom believes that the highest intelligence is the intelligence that results from the combination of the intelligence of the crowd (the crowd) to solve a problem. (Surowiecki, 2004). This concept has been elaborated by Malone, Laubacher, and Dellarocas (2009), who emphasize that the use of Internet and WWW technology can be utilized to gather the intelligence of the crowd, forming an audience that can contribute to the form of ideas (intelligence) individually. The ideas that are proposed will be collected, managed, and combined to form a single idea together. The idea that is formed because of the initial spark of an individual idea, combined into a good idea and accepted by the audience, creates what is known as “collective intelligence.” (Malone et al., 2009). Furthermore, advancement of cloud computing technology can be used for education management with dynamic scalability and as a service providing utilization of virtualized resources over the Internet (A. Gupta et al., 2023)
(Thavi, Jhaveri, Narwane, Gardas, & Jafari Navimipour, 2024)
Arshad, Salleh, Jenom, Aris & Mastuki (2014) and Mansor, Halim & Ahmad(2018) noted that crowdsourcing in Malaysia is at its growing stage and is not fully leveraged by Malaysian. The Malaysian government has advocated for this project via programs such as Digital Malaysia; nonetheless, the focus has predominantly been on its commercial uses (Zakariah, Janom, Arshad, Salleh, & Aris, 2014). Over the past five years, the use of crowdsourced initiatives has gained prominence in Malaysia (Mansor, et al., 2018). Malaysia has several advantages for utilizing crowd resources, including a strong Internet network infrastructure and a significant multilingual workforce (Hassan, Aris, Arshad, Janom, & Salleh, 2017).
One of the advantages that can be exploited from the concept of crowdsourcing is by providing a large repository of questions that can be used by teachers in Malaysia. It can be done by creating a controlled platform that is accessible to all registered teachers in Malaysia. The platform provides a platform for sharing assessment instruments (test papers and examinations) by teachers who are the audience of this platform, based on the concept of crowd-sourced resources for free. The developed platform will facilitate teachers in creating, editing, reusing, storing, sharing, and responding to assessment questions collected in the krumun.org database. This platform also aims to raise awareness and foster a culture of collectively sharing information and knowledge to improve the quality of school assessment instruments in Malaysia. This paper will discuss the development process of this platform in detail and is hoped to serve as a guide for other crowd platform developers in the future.
LITERATURE REVIEW
Generally, there are three main components in crowdsourcing, namely (i) the audience (crowd), (ii) the initiator, and (iii) the process. To ensure the success of the developed crowd-sourcing initiative, it is necessary to specify the type of audience composition, who the members of the audience are, the diversity of the audience, and the skills possessed by the audience. Clear objectives are also necessary to ensure that crowd sources receive attention (Howe, 2009). Brabham (2013) states that there are many applications that can be developed based on crowd sourcing to carry out tasks that previously seemed difficult and complicated if entrusted to individuals or a group of workers within an organization.
The main objective of the crowdsourcing initiative is to enable the audience to complete tasks through an open call (Howe, 2009). The crowdsourcing platform uses Internet technology, and the results of the tasks will form a type of intelligence called collective intelligence (Malone et al., 2009). This intelligence can benefit the interested parties. Intelligence or knowledge that has been generated for specific purposes such as creating innovations, solving organizational problems, and triggering product innovations (Nasution et al., 2023). Although it has been practiced in the world of business and innovation, Crowdsourcing initiatives are an ongoing process and are not entirely “completed” in a traditional sense. Instead, their effectiveness and completion depend on the specific goals and objectives they aim to achieve (Karachiwalla & Pinkow, 2021). Many organizations continue to use crowdsourcing as a flexible and scalable solution to address various needs, such as problem-solving, innovation, and data collection (Johnson & Liew, 2020).
The progression of cloud computing and its integration into education has been extensively discussed by the educational community in various workshops and seminars, such as the Cloud-based Smart Technologies for Open Education Workshop (CSTOE 2022) and the Illia O. Teplytskyi Workshop on Computer Simulation in Education (CoSinE 2022) in Kiev, Ukraine (Papadakis et al., 2023), as well as in research reports by Brusilovsky et al. (2023), Rashid, Munawar, & Naveed (2024), and Badshah et al. (2023). Cloud computing technology has been recognized as a facilitator of crowdsourcing initiatives (Shen et al., 2023).
To promote the utilization of crowdsourcing, the development of additional crowdsourcing platforms is necessary. How might optimal platform design facilitate the effective operation of crowd-sourcing initiatives? Brabham (2009) delineates crowdsourcing as a process comprising four principal steps: (i) identifying the problem or task to be addressed; (ii) disseminating an open invitation via a website (either proprietary or utilizing a third-party platform); (iii) collecting responses from the crowd; and (iv) assessing and selecting the optimal solution.
This idea has been extended by Muhdi, at el. (2011) by incorporating two processes before and after the four processes proposed by Bhabham (2009). The process involved is that the initiator needs to determine whether the crowd-sourcing initiative is suitable for solving the problem or carrying out the existing task – known as pre-crowdsourcing. The second added process is the process after the fourth process proposed by Bhabham (2009), where after the evaluation is done, the proposed idea or solution will be implemented/absorbed into the organization – used for the organization’s operational purposes. In summary, the resource reconciliation process is as shown in Table 1.
Table 1: Crowdsourcing Process (Brabham, 2009; Muhdi et al., 2011)
Proses | Activities | |
1 | Decision for crowd sources | Deciding whether to use crowd-sourcing initiatives or not based on alternative considerations such as using internal resources, using external resources, or using crowd-sourcing resources (Afuah & Tucci, 2012; Ranade & Varshney, 2012) |
2 | Identifying Tasks | This stage will determine the type of platform to be used, whether to use a self-developed platform or a third-party platform (Brabham, 2009). |
3 | Making an Open Call | It was decided what type of data is needed to complete the provided reward form assignment, the expected work template, and the required solution format. |
The type of audience is also determined whether it consists of a specific group or can include anyone who wants to help. | ||
Rules and ethics in implementing this initiative are also communicated at this stage (Zhao & Zhu, 2014). | ||
4 | Getting a Response | The audience will respond in the form of ideas and solutions, either individually or through inter-collaboration, by providing critiques, discussions, and votes in the process of generating what they consider the best solution (Nakatsu, Grossman, & Iacovou, 2014). |
5 | Making an Assessment | The initiator will evaluate, analyze, and select the available solutions by considering the criticisms, suggestions, votes, and assessments made on the provided solutions. The selected idea will be considered the winner and will be announced to the audience. (Malone et al., 2009). |
6 | Using solutions generated by the audience in organizational operation | The chosen solution will be used as part of the elements to carry out the organization’s operations in achieving its goals. (Muhdi et al., 2011). |
After it was decided that the crowd-sourcing initiative would be used, a design for a crowd-sourcing platform that can ensure the initiative’s objectives are achieved needs to be developed. The design stage is the stage where all components of the crowd resources are detailed to ensure the success of the developed system, and the relationships for each component must also be clearly stated (Hetmank, 2013). Although there have been research reports related to reference models that can be used in developing the design of crowd-sourced resource systems, they are not comprehensive and can serve as absolute reference sources. According to Fettke & Loos (2003), a reference model in information systems is a blueprint that explains the conceptual framework for the development of the system being developed, there has still been no definitive reference model reported.
Pedersen et al. (2013) proposed a conceptual model for crowd resources that can serve as a reference model for the design of crowd resource applications. The proposed conceptual model is in the form of input-process-output where the problem to be solved is the input, the process consists of “process”, technology, governance, and audience, while the output is the result of the solution produced. Challenges and mechanisms for each element mentioned. Hetmank (2013) proposes that the components of a crowdsourcing system consist of user management, task management, contributor management, and flowchart management. He detailed each component.
However, the study conducted focused solely on technical elements. Zhao and Zhu (2014) have proposed a conceptual framework for crowd sources by incorporating regulation and feedback elements as its components. The study by Zhao and Zhu (2014) is quite comprehensive but still cannot provide a conceptual framework that can serve as an absolute reference for designing crowd resources. In conclusion, to create a crowdsourcing system, numerous reported thoughts and perspectives must be synthesized to generate a thorough design guide.
Crowdsourcing Platform Development
The creation of a crowdsourcing platform from the ground up requires meticulous design and development efforts. A special domain needs to be created to launch this platform. The developed platform is expected to successfully become a platform that can help the crowd in carrying out the task of developing assessment instruments more easily and effectively. Figure 1 shows the conceptual diagram for the crowdsourcing platform developed. The developed crowdsourcing platform can be used by teachers to create questions and sets of exam and test questions. Teachers can access the crowd-sourced platform using the web browser, and then log in to the platform to engage in activities such as creating questions and sets of test and exam questions. All activities carried out will be recorded and stored in the platform’s database. Teachers can access their collection of questions by selecting the desired subject, chapter, and teaching unit. Next, they can save or print the selected set of questions.
The platform design process is carried out by detailing the crowd-sourced platform model obtained from reading previous crowd platform development reports. There are various system development models that have been reported, including the SDLC (System Development Life Cycle), ADDIE Model (Analyse, Design, Develop, Implement, and Evaluate), SAM Model (Successive-Approximation Model), and Agile Model. (Ahituv, Neumann, & Zviran, 2002; Sekgweleo, 2019; Woods, 2019). According to Woods (2019), the ADDIE model is used for instructional development, while the SDLC model is used in information system development. Both models have the same development phases and use the same phase names, namely the analysis phase, design phase, development phase, implementation phase, and maintenance/evaluation phase. For this study, the SDLC model is used for the development of the platform. Generally, the SDLC has six main phases of system development, namely requirement analysis/planning, design, development, testing, implementation, and evaluation (Vishwakarma, Lohar, Desai, & Mate, 2020).
Fig. 1: Conceptual diagram Crowdsourcing platform for teachers to build questions and question sets
Requirements Planning and Analysis Phase
The crowdsourcing platform, developed in Malaysia, aims to empower educators in creating school assessment tools by allowing the access, creation, storage, sharing, editing, and modification of test questions. The platform intends to improve awareness and foster a culture of information and knowledge exchange to boost the quality of school evaluation tools in Malaysia. A focus group including ten secondary school teachers was conducted to gather information on the creation process of assessment instruments. Consequently, the procedure to prepare assessment instruments – test/exam questions as outlined in Figure 2.
The procedure commences when the Principal or Headmaster, along with the Senior Assistant Teacher, directs the Examination Secretary to formulate and disseminate the examination calendar. The question drafter formulates a preliminary question in accordance with the specified format, incorporating the items/questions from the Item/Test Specification Table and instrument schemes. Upon completion of the set, the committee chair evaluates and validates the questions. The focus group discovered that question drafters utilize many informational sources, including textbooks, reference materials, self-subscribed question banks, and online resources. The formulated questions are saved on the teacher’s computer or in private cloud storage, making it challenging to share with other educators.
Figure 2. Flow chart Preparation of assessment instruments – test/exam questions
The focus group also discussed the development of a crowdsourcing platform for teachers in Malaysia to share assessment instruments online. The platform would facilitate the process of building, editing, reusing, saving, sharing, and responding to assessment questions collected in the crowdsourcing database. The platform would be designed with a simple, effective, and cheap way for teachers to formulate test and exam question assessment instruments. The also identified features of crowdsourcing from the point of view of control, interface display, and user needs. The proposed interface display would have two colours, white for background and blue for important information. The scope of the platform was set to avoid “scope creep” and ensure it did not exceed the scope set.
The platform’s domain was registered and owned by the researcher. A work schedule for platform development was implemented to ensure the project could be implemented within the specified period and not exceed the allocated cost. The overall duration of the platform developer is 25 weeks.
Platform Design Phase
The next phase of the SDLC is the design phase. After completing the system requirements analysis, crowdsourcing platform design activities began by detailing the crowdsourcing platform model by examining the information obtained from the results of the reading and the information obtained from the focus group during the analysis phase. Once all the details were finalized, the design and programming phase of the platform began. The process of developing existing assessment instruments (preparation of tests and exam papers) was studied, improved, and adapted to the developed crowdsourcing application. The information obtained during the analysis phase is used as a guide to enable a good design to be produced according to the needs of the potential audience of this platform.
The design phase is divided into two design sub-phases, namely the logical design sub-phase and the physical design sub-phase. The logical design phase is the phase where the activity process, data flow, and storage form that will be implemented in the actual program are determined (Kendall & Kendall, 2019). During this phase, the process carried out to carry out physical activity is not considered. Physical activity is the activity performed by users in performing tasks by using information system applications. The physical design will be explained in detail during the physical design phase. After the design phase is finalized, programming activities will begin to realize the platform developed according to the specifications and requirements that have been set.
As a result of the information obtained during the analysis phase, it was decided that the crowdsourcing platform will focus on the process of developing assessment instruments in the form of test papers and exams only. Therefore, the process of creating, editing, collecting and saving test and exam questions is the focus of this platform. The process of building an assessment instrument has two stages, namely building questions and building test/exam papers. Constructing questions is the teacher’s activity of creating questions that are finally collected to form a set of test/exam question papers. This question-building process will produce a collection of test questions in the form of a question bank. When the teacher wants to create a test/exam paper, this collection of questions will be referred to and the appropriate questions will be used for assessment purposes in the teacher’s classroom.
The logical design sub-phase is a sub-phase where the process of producing test/exam questions and the opposite of producing a set of test/exam questions is detailed so that it can be a guide during the programming process of the crowdsourcing platform. This process begins by clarifying the data flow process when the teacher (audience) uses crowdsourcing.
Process in the Crowdsourcing Platform
The crowdsourcing platform has been developed to create an assessment instrument in the form of a question paper. To ensure data security and quality, all users must register and provide their personal information, teaching subject information, and class information. After confirming registration, the platform audience can browse the user profile interface view, update their personal information, and choose to build questions or build exam papers. The first option is to build individual questions by choosing a subject and filling in the question and answer in the editor box. Questions are stored in the repository according to the hierarchy of subjects, degrees/levels, chapters, units, question types, and question levels. The audience can also edit the questions they have built or build a new set of questions.
When building exam papers, the audience can choose related subjects, chapter menus, and type of question. They can also make comments and edit existing questions. Edited questions are saved as new questions and can be referenced by previous question creators. Once the selected question is completed, the audience can print the constructed question in Microsoft Word format (docx) or portable document format (.pdf). The set of questions that have been printed by the teacher will also be saved as a collection for future use.
The basic entity relationship model consists of entity types and defines the relationships between entities in a particular knowledge domain. This system ensures data security and quality while allowing the audience to access and edit the questions they have built.
Platform Development Phase
The crowdsourcing platform is built to allow the audience to create questions, save, share and otherwise develop assessment instruments for teaching and learning. This platform is developed permanently (permanent call) and refined as suggested by the audience from time to time. In the initial stage, refinement was done because of the suggestions and remarks made by the focus group during the discussions related to crowdsourcing.
The crowdsourcing platform was developed using the PHP programming language which is a popular programming language for general use and suitable for website development purposes. It is a fast, flexible and pragmatic programming language. For the website developed to host crowdsourcing, the JavaScript programming language is used to ensure smooth access to the crowdsourcing platform. Database management and record keeping of all activities that occur in the crowdsourcing platform is managed using a MySQL-based database. MySQL is a database service that manages the database for applications that use technology/cloud computing (cloud) (MySQL, 2020). While the programming is being made, testing will be done continuously, and the completed units will be launched directly to the Internet and will continue to receive responses from the audience.
The platform uses Internet technology and launches through a specially developed website. Therefore, the Uniform Resource Locator (URL) for this platform is registered. The domain krumun.org has been registered by subscribing to the long-standing web service provider HostGator.com. HostGator.com was chosen after considering the subscription cost, security, and advantages of the service provider related to domain registration, long-term web hosting, and data security. Figure 3 shows the domain registration display for krumun.org. The use of the suffix “.org” is appropriate for websites that are developed with non-commercial and non-profit-oriented characteristics.
The krumun.org platform that has been developed and all data and activities on the platform by the audience are stored on servers rented from the web hosting service provider, HostGator (HostGator.com), with website and data security factors managed by SiteLock Essentials provided by HostGator. The domain Krumun.org has also been registered and is owned by the researcher.
Figure 3: Domain Registration Display the platform
Testing Phase
Testing is done to ensure no programming errors occur. Errors such as functional errors, loss of information, confusion of information, inappropriate functions and so on need to be observed in detail (Raghuvanshi, 2020). Apart from that, the process that produces the output also needs to be checked to ensure that there are no errors for each system in the crowdsourcing platform. The efficiency of the system is also seen in ensuring that the developed system does not suffer from “lagging” problems. All programming, arithmetic and algorithmic aspects of the system are examined and ensured to function as they should. A checklist is provided to ensure that testing is done systematically and accurately (Raghuvanshi, 2020). The main phase of crowdsourcing testing is done in two stages, namely (i) alpha testing and (ii) beta testing and (iii) user acceptance testing.
During the alpha testing, all views and improvement suggestions from those involved in the platform’s development were taken into consideration, and the platform was developed with enhancements to ensure audience acceptance. The platform that has been developed was tested at the development stage (alpha-test), and when all functions are working perfectly, the second stage of testing, the beta test, is carried out.
The beta testing was conducted on the same focus group (the focus group for the development of the platform application) to obtain feedback on the interface and flow perception of the platform platform. The results of this testing showed slight improvements based on the feedback, which included sentence corrections and spelling corrections. Generally, the focus group is satisfied with the platform interface and flow.
Upon the application’s completion, system verification is performed through comprehensive testing. Expert assessments were taken into account to guarantee the application’s quality and seamlessness. Upon the completion of testing and the absence of faults, the platform was launched online. The application will proceed to function with public engagement.
Following the incorporation of enhancements derived from focus group and expert feedback, the crowdsourcing platform was reinitiated for user acceptance testing. The platform was made available online to educators, comprising five primary school teachers and five secondary school teachers, to evaluate the enhanced platform. They are requested to evaluate all the functions offered by crowdsourcing. Upon confirming the appropriate functionality of all platform features and the absence of programming faults, the platform is prepared for launch. The study instrument was subsequently developed to examine the acceptance and continuous use of the platform, which was not addressed in this paper.
Platform Implementation/Launch Phase
A platform domain has been created to launch this platform. Subsequently, the campaign gets underway, and participant information is collected via member registration; the activity log is also maintained for the study of crowd activities on the platform. User feedback is systematically collected, and enhancements are perpetually implemented to optimize the platform’s efficacy. The platform has been inaugurated both in-person and online.
Platform Maintenance Phase
At the same time, maintenance and improvement activities, response to audience comments, and “patches” of information and tools for audience use will occur continuously. The platform administrator will concurrently oversee each aspect of security to ensure that the platform always remains safe and secure. A study on user acceptability and continued usage will be conducted to verify the platform is embraced and consistently utilized by users as a tool for developing test papers for students.
CONCLUSION
The crowdsourcing platform was created utilizing the SDLC methodology as a framework for system development. Upon finalization of all details, the design and programming phase is implemented. Testing is conducted continually alongside the platform’s programming. Upon completion of the application, comprehensive testing will be conducted, and feedback from focus groups and experts will be taken into account to ensure the application’s quality and functionality. The campaign is ongoing to promote the platform, and participant data will be gathered through member registration, while activity logs will be maintained for this study. The application will function with public participation. Concurrently, enhancement projects, responses to audience comments, and updates of information and tools for audience use will persist. The platform will grow as additional participation occurs.
ACKNOWLEDGEMENT
This research is funded by Sultan Idris Education University under the Fundamental University Research Grant. (2020-0189-106-01)
REFERENCES
- Afuah, A., & Tucci, C. L. (2012). Crowdsourcing As a Solution to Distant Search. Academy of Management Review, 37(3), 355-375. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=bth&AN=76609307&site=ehost-live
- Ahituv, N., Neumann, S., & Zviran, M. (2002). A system development methodology for ERP systems. Journal of Computer Information Systems, 42(3), 56-67.
- Arshad, N. H., Salleh, S. S., Janom, N., Aris, S. R. S., & Mastuki, N. (2014, April ). Issues and Challenges in Crowdsourcing Platform Implementation in Malaysia. Paper presented at the 13th International Conference on Applied Computer and Applied Computational Science (ACACOS ’14), Kuala Lumpur, Malaysia.
- Badshah, A., Rehman, G. U., Farman, H., Ghani, A., Sultan, S., Zubair, M., & Nasralla, M. M. (2023). Transforming educational institutions: harnessing the power of internet of things, cloud, and fog computing. Future Internet, 15(11), 367.
- Brabham, D. C. (2013). Crowdsourcing. Massachusetts: MIT Press
- Brusilovsky, P., Ericson, B. J., Horstmann, C. S., Servin, C., Vahid, F., & Zilles, C. (2023). The Future of Computing Education Materials. CS2023: ACM/IEEECS/AAAI Computer Science Curricula, Curricula Practices.
- Fettke, P., & Loos, P. (2003). Classification of reference models: a methodology and its application. Information Systems and e-Business Management, 1(1), 35-53.
- Gupta, A., Mazumdar, B. D., Mishra, M., Shinde, P. P., Srivastava, S., & Deepak, A. (2023). Role of cloud computing in management and education. Materials Today: Proceedings, 80, 3726-3729.
- Gupta, D. K., & Sharma, V. (2013). Exploring crowdsourcing: a viable solution towards achieving rapid and qualitative tasks. Library Hi Tech News 2, 14-20.
- Hassan, H., Aris, S., Arshad, N., Janom, N., & Salleh, S. (2017). Conceptual Model for a Sustainable Crowdsourcing Ecosystem. PERTANIKA JOURNAL OF SCIENCE AND TECHNOLOGY, 25, 89-98.
- Hauben, R. (2019) The Netizen as the Emerging: New Form of Citizenship. In: Vol. 32 (2). The Net, Netizens and Netizenship (pp. 1-30): The Amateur Computerist Newsletter.
- Hetmank, L. (2013). Components and Functions of Crowdsourcing Systems-A Systematic Literature Review. Paper presented at the Wirtschaftsinformatik.
- Howe, J. (2009). Crowdsourcing: Why The Power of the Crowd is Driving the Furture Business. New York: Three Rivers Press.
- Johnson, E., & Liew, C. L. (2020). Engagement-oriented design: a study of New Zealand public cultural heritage institutions crowdsourcing platforms. Online Information Review, 44(4), 887-912.
- Karachiwalla, R., & Pinkow, F. (2021). Understanding crowdsourcing projects: A review on the key design elements of a crowdsourcing initiative. Creativity and innovation management, 30(3), 563-584.
- Kendall, K. E., & Kendall, J. E. (2019). Systems Analysis and Design ( 10 ed.). New York: Pearson.
- Malone, T. W., Laubacher, R., & Dellarocas, C. (2009). Harnessing crowds: Mapping the genome of collective intelligence. Retrieved from https://dspace.mit.edu/bitstream/handle/1721.1/66259/SSRN-id1381502.pdf?sequence=1/.
- Mansor, M. F., Halim, H. A., & Ahmad, N. H. (2018). Leveraging crowdsourcing practices in small and medium enterprises (smes). Journal of Entrepreneurship Education, 21(4), 1-11.
- Muhdi, L., Daiber, M., Friesike, S., & Boutellier, R. (2011). The crowdsourcing process: an intermediary mediated idea generation approach in the early phase of innovation. International Journal of Entrepreneurship and Innovation Management, 14(4), 315-332.
- Nakatsu, R. T., Grossman, E. B., & Iacovou, C. L. (2014). A taxonomy of crowdsourcing based on task complexity. Journal of Information Science, 40(6), 823-834. doi:10.1177/0165551514550140
- Nasution, M. D. T. P., Sari, P. B., Aspan, H., Rossanty, Y., Irawan, & Hernawaty. (2023). How do social media-facilitated crowdsourcing and knowledge integration affect new product development? SME agile initiatives. Cogent Business & Management, 10(3), 2265093.
- Papadakis, S., Kravtsov, H. M., Osadchyi, V. V., Marienko, M. V., Pinchuk, O. P., Shyshkina, M. P., Striuk, A. M. (2023). Revolutionizing education: using computer simulation and cloud-based smart technology to facilitate successful open learning. Paper presented at the Cloud-based Smart
- Technologies for Open Education Workshop (CSTOE 2022) and Illia O. Teplytskyi Workshop on Computer Simulation in Education (CoSinE 2022) Kyiv, Ukraine.
- Pedersen, J., Kocsis, D., Tripathi, A., Tarrell, A., Weerakoon, A., Tahmasbi, N., . . . De Vreede, G.-J. (2013). Conceptual foundations of crowdsourcing: a review of IS research. Paper presented at the System Sciences (HICSS), 2013 46th Hawaii International Conference on.
- Prpić, J., Shukla, P. P., Kietzmann, J. H., & McCarthy, I. P. (2015). How to work a crowd: Developing crowd capital through crowdsourcing. Business Horizons, 58(1), 77-85.
- Raghuvanshi, D. (2020). Introduction to Software Testing. International Journal of Trend in Scientific Research and Development (IJTSRD), 4(3), 797-800.
- Ranade, G., & Varshney, L. R. (2012). To crowdsource or not to crowdsource? Paper presented at the Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence.
- Rashid, S., Munawar, S., & Naveed, N. (2024). A Framework of Software as a Service Using a Crowdsourcing Approach: A Case Study of Smart Classroom. International Journal of Innovations in Science & Technology, 6(3), 1151-1167.
- Sekgweleo, T. (2019). Comparing Agile And Traditional System Development Methodologies. IJIRAS, 6(5), 174-179.
Shen, S., Feng, Y., Xu, M., Zhang, C., Wang, X., Wang, W., & Leung, V. C. (2023). A holistic qos view of crowdsourced edge cloud platform. Paper - presented at the 2023 IEEE/ACM 31st International Symposium on Quality of Service (IWQoS).
- Surowiecki, J. (2004). The Wisdom of Crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies and nations little. New York: Doulbleday.
- Thavi, R., Jhaveri, R., Narwane, V., Gardas, B., & Jafari Navimipour, N. (2024). Role of cloud computing technology in the education sector. Journal of Engineering, Design and Technology, 22(1), 182-213.
- Vishwakarma, A. R., Lohar, A. A., Desai, B. S., & Mate, P. (2020). Crowdsourcing platform for website testing. International Research Journal of Modernization in Engineering Technology and Science, 2(7), 362-366.
- Woods, R. (2019). ADDIE vs. SDLC: A Comparison. Paper presented at the E-Learn: World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, New Orleans, Louisiana, United States.
- Zakariah, Z., Janom, N., Arshad, N. H., Salleh, S. S., & Aris, S. R. S. (2014, December). Crowdsourcing: The Trend of Prior Studies. Paper presented at the 4th International Conference on Artificial Ontelligence with Aplication in Engineering and Technology, Kota Kinabalu, Sabah, Malaysia.
- Zhao, Y., & Zhu, Q. (2014). Evaluation on crowdsourcing research: Current status and future direction. Information Systems Frontiers, 16(3), 417-434.