Scale Development and Content Validity Assessment for Manufacturing Performance

Scale Development and Content Validity Assessment for Manufacturing Performance

Masdiah Abdul Hamid¹, Nurul Fadly Habidin²*

¹Department of Accounting and Finance, Universiti Tenaga Nasional, Pahang, Malaysia

²Faculty of Management and Economics, Universiti Pendidikan Sultan Idris, Perak, Malaysia

*Corresponding Author

DOI: https://dx.doi.org/10.47772/IJRISS.2024.ICAME2420

Received: 06 December 2024; Accepted: 17 December 2024; Published: 26 December 2024

ABSTRACT

Assessing the performance of manufacturing companies is crucial for the success of manufacturing businesses. However, the presence of numerous overlapping and inconsistent perspectives among academicians and practitioners regarding the construct of manufacturing performance has led to the development of various measurement instruments that exhibit differences in the variables assessed to measure the performance of manufacturing companies across disciplines and contexts. This study aims to validate the development of an instrument for assessing manufacturing performance through content validity assessment. The instrument of the manufacturing performance model was developed using the quantitative research method through the distribution of questionnaires to managers and executives of manufacturing companies in Malaysia. An assessment of content validity was conducted by seven panels of experts. The face validity, content validity index, and Kappa statistic had been employed to assess the content validity of the instrument. The final instruments for further validation comprised 80 items categorised into three constructs and 14 components; supply chain constructs (strategic supplier partnership, customer relationships, information sharing, information quality), lean accounting constructs (eliminating transactions, value stream costing, performance measurement, financial benefits, and value stream management), and manufacturing performance (quality, time, delivery, flexibility, and financial performance) are identified. The content validity index (I-CVI) varies between 0.86 and 1, while the scale content validity index (S-CVI/Ave) ranges from 0.860 to 1.000. The instrument demonstrates a high level of content validity in its assessment. These findings demonstrate that the instrument has adequate content validity. This study delineates the acceptable quantity indices for content validity of a novel instrument for manufacturing performance. The findings possess significant implications for the utilisation of this instrument in the assessment of manufacturing performance. This study is significant as it aids researchers and practitioners in gaining a deeper understanding of the factors that influence manufacturing performance and in formulating interventions for future research agendas.

Keywords: Scale, validation, content validity, face validity

INTRODUCTION

Validation is a crucial assessment when selecting or using an instrument in research. Three common forms of instrument validation are content validation, construct validation and criterion-related validation. Content validation is necessary for other validity, which the researcher should emphasise during instrument development. Validity proof should be acquired for each study where an instrument is employed through the scores obtained on the developed instrument based on the research objectives of a specific population of respondents (Shrotryia & Dhanda, 2019). Content validity is described as the ability of the chosen objects to reflect the variables of the construct in the measurement. Content validity assesses how well an instrument’s items represent the content domain (Anuar et al., 2024). This also determines the amount to which a sample of an instrument or its items represents the content. Content validity gives preliminary evidence for the construct validity of an instrument development.

Furthermore, it can offer information on the representativeness and clarity of things and assist in the improvement of an instrument by obtaining recommendations from experts (Fuad et al., 2022; Shrotryia & Dhanda, 2019). A low score of content validity represents that the instrument is unreliable enough. Content validation involves validating the items by a designated number of experts to ascertain the content validity of the assessment instrument. The selection of domain experts should be conducted based on criteria including expert knowledge, expertise, or relevant professional experience of the subject matter. The involvement of at least three experts is advised to assess content validity. The maximum number of experts remains undetermined; however, it is improbable that more than 10 experts will be involved in the process, as an increase in the number of experts tends to diminish the likelihood of consensus (Polit & Beck, 2006).

The manufacturing industry is a crucial driver of economic, social, and political growth (Taouab & Issor, 2019). To ensure this growth, manufacturing companies need a balanced development strategy that improves existing practices to remain relevant in the business environment. Manufacturers must understand the key factors for business success and strategically plan their operations to remain competitive and resilient, which, in turn, enhances manufacturing performance. Manufacturing companies need to stay relevant and competitive in the global business environment. Manufacturers can achieve higher competitive performance through effective business strategies accompanied by best accounting, control, and measurement practices within their business activities. Adopting advanced management accounting methods drives business efficiency in manufacturing companies (Fullerton & Kennedy, 2011).

Consequently, this study was undertaken in response to a significant deficiency in the existing understanding of the manufacturing performance model in the context of instrument development for manufacturing performance. The development of a comprehensive instrument for evaluating manufacturing performance is essential for the precise assessment of operational efficiency, quality and productivity of the manufacturing industry, which in turn give a positive effect on organisational performance (Anuar et al., 2024). Despite various performance measurement being developed and used to measure organisational performance (Amrina & Yusof, 2013; Milanovi, 2011; Narkunienė & Ulbinaitė, 2018; Neely et al., 1995; Taouab & Issor, 2019), the existing instruments of performance measurement frequently demonstrate deficiencies in consistency, reliability and adaptability across various industrial contexts (Garengo et al., 2005). Moreover, current performance measurement instruments are concentrated on particular industries, hence resulting in limited generalisability across diverse manufacturing environments (De Toni & Tonchia, 2001; Gomes et al., 2004; Milanovi, 2011). Moreover, there are often overemphasised financial metrics, and there is an imbalance in addressing the multidimensional nature of the manufacturing environment, such as neglecting lean practices domains (Ali et al., 2021; Kennedy et al., 2012; Mcvay et al., 2017).

An analysis of the literature review was employed to identify the constructs of manufacturing performance (Amrina & Yusof, 2013; Bhasin, 2008; Fullerton & Wempe, 2009; Mouzas, 2006; Narkunienė & Ulbinaitė, 2018). Accordingly, the main constructs was formulated as represented by supply chain, lean accounting, and manufacturing performance in the context of manufacturing industries. The integration of these constructs arises from the intricate and multifaceted nature of the operation activities within a manufacturing environment. It is imperative to identify the essential components that will enhance the performance of manufacturing companies. The development and evaluation of this instrument are essential for a better business strategy and competitive performance of manufacturing companies in Malaysia.

This study is organised in the following manner: Section 2 provides instrument development for this study, while Section 3 outlines the quantitative research methodology. Section 4 provides an analysis of the findings, whereas section 5 emphasises the discussions and implications derived from the study. The study culminates in a series of recommendations for prospective research agendas.

METHODOLOGY

Instrument Development

The initial stage of instrument development involves three steps: selecting the content domain, creating sample items, and building the instrument (Zamanzadeh et al., 2014). Thorough literature research was undertaken to identify the appropriate content domain for manufacturing performance model. Explicitly, this study developed a questionnaire by drawing on previous studies and adapted it to align with the research objectives and to answer the research questions. Adapting the items with the help of an extensive literature survey and items from previous studies will increase the validity of the research instrument (Messick, 1987). The developed instruments underwent the validation process including content validity and face validity.

Having extensive and significant items for measuring the variable of interest is crucial to ensure reliability or internal consistency (Nunnally, 1978; Nunnally & Bernstein, 1994). According to Awang (2015) and Awang et al. (2018; 2023), A factor with less than three items in an instrument is typically regarded as unstable and weak. However, if a factor has five or more items that strongly load with a score of 0.60 or higher, it is considered desirable and indicates a substantial factor. However, according to Hair et al. (2010), it is preferable to have a minimum of three or more items in order to cover the theoretical domain of a construct adequately. Hinkin (1998) suggested that sufficient internal consistency can be achieved by using only three items per construct. Following the previous studies (Awang et al., 2016, 2023; Hair et al., 2010, 2018, 2019; Hinkin, 1998), this research has at least three items and not more than eleven items for each component.

The instrument is divided into four sections, represented by the letters A, B, C, and D. Section A (Supply Chain) contains 32 items that measure the supply chain construct. Ten (10) items measure the component of strategic supplier partnership (SSP), eleven (11) items measure the component of the customer relationship (CUR), six (6) items measure the component of information sharing (ISH), and five (5) items measure the component of information quality (INQ). Section B (Lean Accounting) consists of 19 items that measure the lean accounting implementation construct. Out of 19 items, four items (4) measure each component, including eliminating transaction (ELT), value stream costing (VSC), performance measurement (PFM) and financial benefit (FNB), while three items (3) measure the component of value stream management (VSM). The endogenous variable (Manufacturing Performance) is measured in Section C. 29 items used to evaluate the manufacturing performance in Malaysia. Nine (9) items measure component of quality (QLTY), seven (7) items measure component of time (TIME), four (4) items measure both component delivery (DEL) and flexibility (FLX) respectively, and five (5) items measure component financial performance (FNP).

Expert Judgement

A group of experts were appointed to assess the instrument’s content validity. A specialist is chosen based on their expertise, training, and experience to align with the subject matter. It is recommended to involve at least three experts in establishing content validity. Although the maximum number of experts has not been decided, it is doubtful that more than 10 experts will participate in the process due to the decreased likelihood of consensus (Polit & Beck, 2006).

This study uses seven panel of experts to validate its instrument. Six are experienced academicians with over 15 years of experience, actively researching the construct of interest, publishing peer-reviewed papers, and having professional experience in supply chain or lean accounting. One panel is an industrial expert in lean thinking and accounting. The remaining panel includes experts in management accounting, supply chain, performance effect, methodology, and language, as well as an international expert in lean thinking foundation and lean accounting. The validation process involves sending emails with appointment letters and content validation review forms to selected experts. After the agreement, a meeting is held with the panel of experts, who are provided with detailed instructions and asked to improve sentence structure for the overall validity of the research instrument. The content review form includes construct definitions, scales, instructions, and comments. Experts evaluate the relevance of 80 items using a 4-point rating scale. Lynn (1986) suggests using a 4-point scale to avoid neutral points. Content Validity Index (CVI) for item (I-CVI) and CVI for scale (S-CVI), and Kappa statistics based on experts’ evaluation are calculated to assess the degree of relevance between experts. A list of panels of experts selected for content validity assessment is presented in Table 1.

Table 1: Panel of Experts selected for Content Validity Assessment

Notes:
UITM = Universiti Teknologi MARA, Malaysia; UNITEN = Universiti Tenaga Nasional, Malaysia; UPM = Universiti Putra Malaysia, Malaysia; BMA = Brian Maskell Association

The expert panel was requested to provide their professional judgment on each dimension of the construct: supply chain, lean accounting, and manufacturing performance. Panel assessment, feedback on the relevance, necessity, representativeness, and comprehensiveness of the items are collected to confirm their content validity

FINDINGS AND DISCUSSIONS

Face Validity

Face validity refers to the extent to which the items of a measurement instrument linguistically and analytically resemble what is intended to be measured (Bougie & Sekaran, 2019). Hardesty and Bearden (2004) suggested that modified measures should provide evidence of face validity for the used items. Face validity is performed to verify that the items represent the construct to be measured and that the language and terms used in the instrument are accurate (Hecker & Violato, 2009). It involves assessing whether the items in the instrument are appropriate, reasonable, explicit, and clear (Oluwatayo, 2012). Fundamentally, face validity measures whether the instrument appears valid to the respondents. Oluwatayo (2012) developed the criteria for conducting face validity, which includes assessing the appropriateness of the items, ensuring clarity and unambiguity, checking for correct spelling, verifying proper sentence structure, evaluating the appropriateness of font size, and maintaining consistency in the style and formatting of the questionnaire. This study performs face validity to ensure that the instrument appears valid at face value to the respondent. Overall, all experts agree on the items used to assess each construct. Experts recommend splitting double-barrel questions, reducing the number of items, and improving sentence structure for consistency.

Content Validity (CVI)

Content validity (CVI) is a widely accepted measure of content validity, based on expert ratings of relevance. Content validity is the extent to which an instrument accurately measures a construct of interest, aligning with its intended purpose. It is defined as the relevance and representativeness of assessment elements to the targeted construct (Bougie & Sekaran, 2019; Darusalam & Hussin, 2018; Muda et al., 2018; Yusoff, 2019). Content validity is evaluated by assessing the accuracy of test samples in accurately representing the subject matter from which conclusions will be drawn (Messick, 1987). Therefore, content validation is crucial to ensure the items are appropriate to the construct domain (Davis, 1992; Malhotra & Grover, 1998; Polit et al., 2007; Polit & Beck, 2006). CVI is calculated using a 4-point rating scale, with each item rated 3 or 4 for content validity. A CVI value of at least 0.70 is considered higher content validity, while 1.00 is required for five or fewer experts and 0.80 for a new scale. Table 2 provides a summary of recommended experts and acceptable cut-off values for CVI.

Table 2: The number of experts and an acceptable cut-off value of CVI

Kappa Statistics Coefficient

Shrotryia & Dhanda (2019) suggested using cKappa coefficients to address potential invalidation from random chance agreement in the CVI calculation. Kappa is an index of interrater agreement between experts, ensuring consensus is not due to chance (Fuad et al., 2022; Polit & Beck, 2006). The probability of chance agreement (Pc) is derived using the formula.

Where;
N is the number of experts in the panel,
A is the number of experts who agree that the item is relevant
Accordingly, the Kappa statistic is calculated by using the formula:

The acceptance value of Kappa Statistics is considered good (> 0.74), satisfactory (0.60 to 0.74) and moderate (0.4 to 0.59) (Polit & Beck, 2006; Zamanzadeh et al., 2015). Table 3, Table 4, and Table 5 showcase the rating scores for each construct of interest, which consists of items’ content validity index (I-CVI), probability of chance agreement (Pc), Kappa statistics (k*) and scale content validity index (S-CVI) for the constructs of supply chain, lean accounting and manufacturing performance.

Table 3: Rating Score for Supply Chain Construct

Table 4: Rating Score for Lean Accounting Construct

Table 5: Rating Score for Manufacturing Performance Construct

The rating scores for supply chain (Table 3), lean accounting (Table 4), and manufacturing performance (Table 5) constructs shows that the I-CVI results are above the threshold of 0.70, indicating valid items. The S-CVI/Ave for supply chain is 0.9857, while lean accounting and manufacturing performance constructs have 0.9786 and 1.00 respectively. The recommended value for S-CVI is 0.80, indicating content validity. All constructs in the research instrument are maintained, and expert feedback indicates a comprehensive list of constructs without additional ones. The I-CVI results across all tables show consistent values ranging from 0.86 to 1.00, which is above the threshold of 0.70. Therefore, the items in the constructs are valid. The S-CVI/Ave for supply chain construct is 0.9857, while for lean accounting construct and manufacturing performance construct is 0.9786 and 1.00, respectively. The recommended value of S-CVI for construct validity should be 0.80 (Lynn, 1986; Polit et al., 2007; Polit & Beck, 2006). Therefore, the S-CVI for all constructs also meet the threshold value, and content validity is established. Accordingly, all constructs in the research instrument for this study are maintained. Table 3.10 exhibits an overall comment by the expert. Based on the qualitative feedback by the experts, the list of constructs is sufficiently comprehensive, requiring no additional constructs.

Table 6: Overall Comments by Experts

CONCLUSION AND RECOMMENDATION

Content validity confirms that the operationalisation of the construct relies on items derived from the specific domain of content pertinent to the particular measurement context. The assessment instrument for manufacturing performance undergoes systematic stages of processes to establish content validity on the developed instrument. During the initial phase of instrument development, 80 items were identified through the analysis of literature reviews and questionnaire distributions to managers and executives of manufacturing companies in Malaysia. Subsequently, during the evaluation phase, seven domain experts were invited to assess the items according to their relevance. The assessment of content validity, utilising the Content Validity Index (I-CVI and S-CVI), face validity, and Kappa coefficient, demonstrated a high level of content validity for the items evaluated. The computation of content validity for the construct contributed to minimising the disparity between academic and industrial perspectives on manufacturing performance, as the identification of the content domain was thoroughly established to ensure that the development of the instrument is founded on a consensus definition.

The instrument developed for measuring manufacturing performance holds considerable implications for academic research and practical application within the industry. Through the attainment of high content validity, as demonstrated by the results of the Content Validity Index (CVI), the instrument guarantees that the items thoroughly encapsulate the domain of manufacturing performance. This alignment reduces inconsistencies between theoretical frameworks and practical applications, connecting academic constructs with the realities encountered by manufacturing organisations. The systematic validation process, which incorporates the insights of domain experts, guarantees that the instrument is specifically designed for the context of the manufacturing industry, thereby enhancing its relevance and applicability for decision-makers within the manufacturing industry.

This instrument serves as a dependable resource for manufacturing managers and executives to assess their manufacturing performance including operational efficiency, quality, cost management, and innovation investment. Through the utilisation of a validated assessment framework, organisations are able to benchmark their performance, discern strengths and weaknesses, and execute targeted strategies aimed at enhancement. This validated instrument is significant as it aids researchers and practitioners in gaining a deeper understanding of the factors that influence manufacturing performance and in formulating interventions for future research agendas. Moreover, the instrument enhances the existing body of knowledge by providing empirical evidence for assessing the manufacturing performance. Nevertheless, the measurement instruments are required to undergo thorough evaluation of their psychometric properties. Consequently, future research should focus on evaluating the instrument for reliability as well as other types of validity, including construct and criterion validity, to enhance the applicability of the assessment instrument

ACKNOWLEDGEMENT

The researcher appreciates and thanks the Ministry of Higher Education (MOHE) Malaysia for financially supporting this research under the Fundamental Research Grant Scheme (FRGS), project code FRGS/1/2019/SS08/UNITEN/03/1.

REFERENCES

1. Ali, S. B., Khan, Z. S., Shah, Z. A., & Ahmad, M. (2021). Lean Accounting System: Importance and Successful Implementation. Journal of Contemporary Issues in Business and Government, 27(3). https://doi.org/10.47750/cibg.2021.27.03.293
2. Amrina, E., & Yusof, S. M. (2013). An AHP Based-Model for Sustainable Manufacturing Performance Evaluation in Automotive Industry. https://doi.org/10.13033/isahp.y2013.016
3. Anuar, K. F., Mohd Zainudin, N., & Fauzi, M. A. (2024). A Content Validity: Organisational Performance’s Assessment Instrument in the Construction Industry. Built Environment Journal, 21(2). https://doi.org/10.24191/bej.v21i2.947
4. Bhasin, S. (2008). Lean and performance measurement. Journal of Manufacturing Technology Management, 19(5), 670–684. https://doi.org/10.1108/17410380810877311
5. Bougie, R., & Sekaran, U. (2019). Research Methods For Business A Skill Building Approach (8th, Asia ed.). Wiley. https://www.wiley.com/en-sg/Research+Methods+For+Business%3A+A+Skill+Building+Approach%2C+8th+Edition%2C+Asia+Edition-p-9781119683537
6. Darusalam, G., & Hussin, S. (2018). Methodologi Penyelidikan Dalam Pendidikan: Amalan dan Analisis Kajian. In University of Malaya Publication (Second Edi). Penerbit Universiti Malaya. https://doi.org/10.14425/ 9789674880095
7. Davis, L. L. (1992). Instrument review: Getting the most from a panel of experts. Applied Nursing Research, 5(4), 194–197. https://doi.org/10.1016/S0897-1897(05)80008-4
8. De Toni, A., & Tonchia, S. (2001). Performance measurement systems Models, characteristics and measures. International Journal of Operations and Production Management, 2(2), 25–38. https://doi.org/10.1108/01443570110358459
9. Fuad, D. R. S. M., Musa, K., Yusof, H., & Hashim, Z. (2022). The development and validation of the principal innovation leadership scale in Malaysian secondary schools. International Journal of Evaluation and Research in Education, 11(1), 193–200. https://doi.org/10.11591/ijere.v11i1.22038
10. Fullerton, R. R., & Wempe, W. F. (2009). Lean manufacturing, non-financial performance measures, and financial performance. International Journal of Operations and Production Management, 29(3), 214–240. https://doi.org/10.1108/01443570910938970
11. Garengo, P., Biazzo, S., & Bititci, U. S. (2005). Performance measurement systems in SMEs: A review for a research agenda. International Journal of Management Reviews, 7(1), 25–47. https://doi.org/10.1111/j.1468-2370.2005.00105.x
12. Gomes, C. F., Yasin, M. M., & Lisboa, J. V. (2004). A literature review of manufacturing performance measures and measurement in an organizational context: A framework and direction for future research. In Journal of Manufacturing Technology Management. https://doi.org/10.1108/17410380410547906
13. Hardesty, D. M., & Bearden, W. O. (2004). The use of expert judges in scale development. Implications for improving face validity of measures of unobservable constructs. Journal of Business Research, 57(2), 98–107. https://doi.org/10.1016/S0148-2963(01)00295-8
14. Hecker, K., & Violato, C. (2009). Validity, reliability, and defensibility of assessments in veterinary education. Journal of Veterinary Medical Education, 36(3), 271–275. https://doi.org/10.3138/jvme.36.3.271
15. Kennedy, F. A., Widener, S. K., & Fullerton, R. (2012). Accounting for a Lean Environment. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.1659386
16. Lynn, M. R. (1986). Determination and quantification of content validity. In Nursing Research (Vol. 35, Issue 6, pp. 382–386). http://ijoh.tums.ac.ir/index.php/ijoh/article/view/26
17. Malhotra, M. K., & Grover, V. (1998). An assessment of survey research in POM: From constructs to theory. Journal of Operations Management, 16(4), 407–425. https://doi.org/10.1016/s0272-6963(98)00021-7
18. Mcvay, G. J., Kennedy, F. A., & Fullerton, R. R. (2017). Accounting in the Lean Enterprise. In Taylor & Francis Group. Productivity Press. https://doi.org/10.1201/b14867
19. Messick, S. (1987). Validity (Issue October).
20. Milanovi, L. (2011). Performance measurement systems Models, characteristics and measures. Business Systems Research, 2(2), 25–38.
21. Mouzas, S. (2006). Efficiency versus effectiveness in business networks. Journal of Business Research. https://doi.org/10.1016/j.jbusres.2006.09.018
22. Muda, H., Loganathan, N., Awang, Z., Jusoh, H., & Baba, Z. S. (2018). Application of Theory, Methodology and Analysis in Conducting Research: A Practical Guide to Quantitative Research and Thesis Writing. In Penerbit UniSZA.
23. Narkunienė, J., & Ulbinaitė, A. (2018). Comparative analysis of company performance evaluation methods. Entrepreneurship and Sustainability Issues, 6(1), 125–138. https://doi.org/10.9770/jesi.2018.6.1(10)
24. Neely, A., Gregory, M., & Platts, K. (1995). Performance measurement system design A literature review and research agenda. International Journal of Operations & Production Management, 15(4), 80–116.
25. Oluwatayo, J. A. (2012). Validity and Reliability Issues in Educational Research. Journal of Educational and Social Research, 2(May), 391–400.
26. Polit, D. F., & Beck, C. T. (2006). The Content Validity Index: Are You Sure You Know What’s Being Reported? Critique and Recommendations. Research in Nursing & Health, 489–497. https://doi.org/10.1002/nur.20147
27. Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Focus on Research Methods Is the CVI an Acceptable Indicator of Content Validity? Appraisal and Recommendations. Research in Nursing & Health, 30, 459–467. https://doi.org/10.1002/nur
28. Shrotryia, V. K., & Dhanda, U. (2019). Content Validity of Assessment Instrument for Employee Engagement. Sage Open, 9(1), 215824401882175. https://doi.org/10.1177/2158244018821751
29. Taouab, O., & Issor, Z. (2019). Firm Performance: Definition and Measurement Models. European Scientific Journal ESJ, 15(1), 93–106. https://doi.org/10.19044/esj.2019.v15n1p93
30. Yusoff, M. S. B. (2019). ABC of Content Validation and Content Validity Index Calculation. Education in Medicine Journal, 11(2), 49–54. https://doi.org/10.21315/eimj2019.11.2.6
31. Zamanzadeh, V., Ghahramanian, A., Rassouli, M., Abbaszadeh, A., Alavi-Majd, H., & Nikanfar, A.-R. (2015). Design and Implementation Content Validity Study: Development of an instrument for measuring Patient-Centered Communication. Journal of Caring Sciences, 4(2), 165–178. https://doi.org/10.15171/jcs.2015.017