Development , Validity and Reliability Study of the Learning Object Evaluation Scale ( LOES )

The purpose of the current study was to develop a Learning Object Evaluation Scale (LOES) in Turkish language. The sample consisted of 388 sixth grade students in a primary school. In the study, item-total correlation, item-remainder correlation, Cronbach’s alpha, difference between upper and lower group means were analyzed. Results of the exploratory factor analysis (EFA) showed that 30 items were loaded into three factors: perceived learning, usability and engagement. The internal consistency reliability of three factors were .88, .91 and .90 respectively. The total variance explained was 54.42%, and common factor loadings ranged between .39 and .72. Fit index values of the model was χ2/df = 2.74, RMSEA = .07, SRMR = .06, GFI = .87, AGFI = .85, CFI = .97, and NNFI = .97. These findings showed that this scale was valid and reliable instrument to evaluate learning objects.


Introduction
The logic of the learning objects (LOs) concept lies behind the Object-Oriented Programming (OOP) which is a paradigm from computer science (Wagner, 2002).The main idea lying behind the LOs is that instructional designers or educators build small chunks of instructional content that can be reused in or with different contexts (Wiley, 2001).
Because the concept is relatively new, there are almost as many definitions as the number of organizations and research groups (McGreal, 2004).To date, there is not a consensus on the definition of LO concept.Furthermore there is not an agreement even on the terms used to describe LO.The most common and well known definition of LOs was made by IEEE LOs Standards Committee (LTSC) in 2001.LTSC (2001, p.5) defines LOs as: "Learning objects are defined here as any entity, digital or nondigital, which can be used, re-used or referenced during technology supported learning."The emergence of standards, state of the results from the research, and the appearance of theoretical perspectives on LOs in time, more definitions emerged.Kay and Knack (2005) emphasized that the definitions on LOs focus on technological or pedagogical dimensions of LOs.In this study, operationally LOs are defined as "interactive web-based tools that support the learning of specific concepts by enhancing, amplifying, and/or guiding the cognitive processes of learners" (Kay & Knaack, 2007).
The characteristics of learning objects in the literature are reusability, granularity, interoperability, accessibility, adaptability, discoverability, durability, manageability, and generativity (McGreal, 2004).In order to describe the LOs, identify LO components and repurpose them, and provide the interoperability, accessibility and reusability of LOs, learning object models such as Sharable Content Object Reference Model (SCORM) and Cisco System RLO (Reusable Learning Object) were created.
The increase in image, text, and video type digital materials in the Internet made it difficult to search and to attain the intended.As a solution to this problem, metadata described as data about data (Wiley, 2001) were created.Metadata enhances searching of learning objects, facilitates management and maintenance of them, and aids filter and selection of the relevant learning for a specific learning context.Dublin Core Metadata (DCM) defined by Dublin Core Metadata Initiative (DCMI) and Learning Object Metadata (LOM) created by IEEE are some mostly used standardizations in order to facilitate the referencing, searching, accessing and updating of content elements.
There are some methodologies used by researchers to evaluate the students' use of LOs.Bradley and Boyle (2004), Kay and Knaack (2005), and Krauss and Ally (2005) used qualitative techniques in the evaluation process.However, the main concern for these studies was the lack of assessment of validity and reliability.Also, quantitative efforts have been put by Bradley and Boyle (2004), Nesbit and Belfer (2004), Cochrane (2005) and others.One of the most cited evaluation instrument is Learning Object Review Instrument (LORI) which was developed by Vargo, Nesbit, Belfer and Archambault (2003), and improved by Nesbit, Belfer and Leacock (2004) in order to evaluate the LOs in collaboration with experts, teachers, designers, and students.The LORI 1.5 used nine items outlined as content quality, learning goal alignment, feedback and adaptation, motivation, presentation design, interaction usability, accessibility, usability, and standards compliance.However, as Kay and Knaack (2009) emphasized the main problems for those evaluation rubrics were that they lacked an underlying evaluation theory as well as thorough assessment of validity and reliability of the items included in these rubrics.In addition, and they were tested with limited number of participants.
After examining the literature on LOs, Kay and Knaack (2009) developed and improved an evaluation tool called Learning Object Evaluation Scale for Student (LOES-S).It was a student-based learning object evaluation tool and based on three key factors emphasized in 10 years of learning object research; perceived learning, quality or instructional design, and engagement or motivation with LOs.Also internal reliability coefficient of the factors of the instrument was assessed .89,.84 and .78respectively.
In summary, previous efforts to evaluate LOs were limited to small sample size, did not have sufficient validity and reliability support, and were not based on established theoretical frameworks.Therefore this study attempts to develop a comprehensive LO evaluation instrument based on wellknown theories in the field and provide validity and reliability evidence through data collected from a large sample.

Participants
The participants of the study were 388 sixth grade students at four different public primary schools in Bolu, Turkey with the mean age of 11.Before the scale was administered, students used the LOs developed for The Silk Road and Turks unit in Social Science lesson curriculum.The LOs were implemented in Information Technology Classrooms in the schools for 12 hours in 4 weeks period.Before using the LOs, the students were informed about the LOs concept, the instruction process with LOs and about the use of LOs.Additionally, students were told that after using LOs, they were to be requested to evaluate them.The participants were ensured that participation was voluntary, they could withdraw from the study any time they wanted, and their grade would not be affected if they do not participate in the study.In addition, students were ensured that their personal information would not be shared with anyone.After using LOs, students were requested to evaluate the learning objects by using a 5 point Likert type scale with anchors ranging from 1 (strongly disagree) to 5 (strongly agree).

Item Generation
The process of item generation in order to develop the learning object evaluation instrument was based on a thorough review of (a) the literature on LOs, and (b) similar evaluation tools developed by researchers (Bradley & Boyle, 2004;Cochrane, 2005;Kay & Knaack 2009;Nesbit & Belfer;2004).The LOES used in this study was mainly based on Kay and Knaack's (2009) Learning Object Evaluation Scale for Students (LOES-S).As in Kay and Knaack's (2009) study, three key constructs were developed for the scale; (1) perceived learning, (2) usability, and (3) engagement.The perceived learning dimension measures students' perception of how much he/she learned by using the learning object.The usability dimension refers to the quality and ease of use of the learning object.Finally, the engagement dimension investigated how much a student involved in the activity in the learning object.In addition to those items, the scale form was enriched with some items in Webster and Ho's (1997) study about engagement in multimedia presentations, and the researchers added items to the pool.The item pool included a total of 40 items.The required permissions were taken from the authors of the related articles Robin Kay and Jane Webster through e-mail.The items coming from Kay and Knack (2009) and Webster and Ho's (1997) instruments were translated into Turkish by two academicians in the Department of Foreign Language Education at one of the public university.Later, the translators discussed on the items in order to determine the final form of the scale.The language of the survey was also checked by an academician in Turkish Language Teaching Department so that the items can be understood easily and clearly by the students in the 6th grade.
Before implementing LOES, in order to determine the face and content validity of the scale, 4 instructional technology experts were asked their opinions on the items.Based on the feedback from the experts, 6 items were removed from the instrument, and 2 new items were added in the instrument.The scale before the pilot application included 36 items in 3 different constructs.There were 8 items in the perceived learning, 15 items in usability, and 13 items in the engagement construct.

Data Analysis
The factor structure of the Learning Object Evaluation Scale was checked with exploratory factor analysis (EFA) and ensured with confirmatory factor analysis (CFA).With EFA, the underlying relationships between measured variables are identified.The hypothesis of relationship between the variables was tested through CFA (Büyüköztürk, 2002).While SPSS was used for EFA, LISREL was used to implement CFA.

Item Analysis
Established guidelines for scale development by DeVellis (2003) were followed in order to create the LOES form.There are three phases in an exploratory factor analysis: (a) select and measure the variables, (b) determine the number of factors, and (c) interpret the factors (Pohlmann, 2004).
Before the exploratory factor analysis, in order to determine how each item could contribute to the variance of the instrument and to validate the LOES items, item analysis was performed.This was reflected by the item-remainder coefficient, item-total correlation, Cronbach's alpha if item deleted and t-test between upper and lower groups' mean scores calculated for each item (Tezbaşaran, 1997).The findings regarding this analysis were presented in Table 1.Although the t-value for item 7 was significant, as the item-total correlation and item-remainder correlation was less than .30and .25 respectively, and it decreased the internal reliability of the scale form, it was removed from the scale.

Exploratory Factor Analysis
In order to find small number of new and conceptually meaningful factors by gathering certain number of interrelated variables and considering the relationship between them (Field, 2009) exploratory factor analysis (EFA) using principal component analysis was executed.
It was attempted to discover if the data obtained from 388 students were appropriate for exploratory factor analysis.The Kaiser-Meyer-Olkin (KMO) value and Bartlett's Test of Sphericity (BTS) were taken into account respectively for that purpose (Field, 2009).Since the KMO coefficient for LOES was calculated as .94and higher than .90, it could be said that the size of the sample was highly acceptable.When the results of Bartlett's Test of Sphericity test were examined (Chi-square = 8563.80;df =595; p < .05), it is seen that the data are appropriate for the factor analysis (Çokluk, Şekercioğlu & Büyüköztürk, 2010;Field, 2009).
The criteria for determining the number of factors to retain were eigenvalue greater than 1, the amount of common variance explained and the scree-test (Field, 2009).The initial solution yielded that the LOES had a three-factor structure accounting for a total of 54.42% of the variance.Inspection of the scree plot supported the retention of three components as well.
Next the loadings less than 0.4 were suppressed in the output (Field, 2009).When the component matrix was analyzed it was observed that all of the 35 items had factor loadings more than .53,and mostly loaded into the first factor.This loading showed the reason for the first factor accounted for most of the variance (Field, 2009).
In the next phase, in the interpretation of the three factors, Varimax vertical axis rotation was used.Five items were deleted from the scale during the Varimax rotation, since those items could not meet the requirement for the value of the differences of each item between the factor loadings of .10(Çokluk et al., 2010).
After inappropriate items were suppressed, it was found that the common variance for each of the three factors was 20.04%, 19.55% and 14.59% respectively.The value of total variance between 40% and 60% is claimed to be sufficient for social science studies, and for any factor to be meaningful, at least 5% of the total variance explained should be attributable to that factor (Çokluk et al., 2010;Tavşancıl, 2010).Therefore, the total variance explained found as 54.18% in this study might be acceptable.
After Varimax vertical axis rotation, the scale consisted of 30 items in 3 factors named as usability, engagement and perceived learning (Table 3).Usability factor consisted of 12 items, and the item loadings of this factor ranged between .51 and .74.Engagement factor included 11 items having factor loadings ranging from .56 to .72.The last factor, perceived learning, consisted of 7 items and factor loadings of the items ranged between .50 and .76.In addition, the common factor loadings of each item ranged from .39 to .72, which was good (Field, 2009).Lastly, Cronbach's alpha coefficient of each factor and the correlations between the factors in the scale were calculated to determine the internal consistency of the scores obtained from the LOES.The Cronbach's alpha coefficients over .70 are stated as adequate for an instrument to be used (Fraenkel & Wallen, 2006).The internal consistency reliability coefficients of perceived learning, usability and engagement were .88,.91 and .90respectively.The correlations between the learning factor and the usability factors (r = .68,p < .01,N = 388) and engagement (r = .68,p < .001,N = 388) factors were significant, as was the correlation between the usability and engagement factors (r = .61,p < .01,N = 388).All LOES factors showed reasonable reliability.

Confirmatory Factor Analysis for LOES
To test whether the three-factor model obtained in EFA fit to the data (Sümer, 2000), confirmatory factor analysis (CFA) was used.So, the data set for 388 cases used in EFA was loaded in LISREL statistic program and a covariance matrix was prepared.
Path diagram and goodness of fit statistics were produced for the three-factor model with 30 items.The path diagram with standardized solutions illustrated the loadings of each item on respective factor the three-factor model, where learning was manifested by seven, usability was manifested by twelve and engagement was manifested by eleven observed variables.The maximum likelihood estimations appeared between .43 and .80 and all t-values were significant at p < .05.In addition, the error variances ranged from .36 to .82.Kline (2005) suggested that error variances should not exceed the value of .90.Moreover, three factors were allowed to correlate with each other.This showed that the factor loadings of each item on the related factor were at a reasonable size.
As the result of the confirmatory factor analysis conducted using the maximum likelihood method without any limitations, the worth of fit values was found to be χ2/df = 2.74, RMSEA = .07,SRMR = .06,GFI = .87,AGFI = .85,CFI = .97,and NNFI = .97.According to these values, it can be said that GFI observable fit value was slightly lower than acceptable value, RMSEA, SRMR and AGFI fit values indicate an acceptable fit and the other observable fit values indicate a perfect fit (Table 4).In other words, this obtained model indicated that the factors were confirmed by the data (Çokluk et al., 2010;Sümer, 2000;Tabachnick & Fidell, 2001).

Conclusion
In this research, a scale was developed in order that students and teachers evaluate the learning objects.For this purpose, item pool was developed by grounding on existing scales (Kay & Knaack, 2009;Webster & Ho, 1997), examining related literature, and receiving instructional technology experts' opinions.The items were related to students' learning, usability of LOs and motivating characteristics of LOs.
The initial form including 36 items were administered to 388 students at their 6th grade.In the item analysis phase, only one of the items (item 7-Bu öğrenme nesnesi konuyu kendi öğrenme hızımda öğrenmemi sağladı.) was deleted from the survey as it had substantially had low correlation and decreased the internal reliability of the scale.
The construct validity of LOES was analyzed with EFA and CFA.EFA deleted five items and showed that the LOES was a five-point Likert type scale including 30 items in three factors, named as perceived learning, usability and engagement.The factor loadings of items in three factors ranged from .50 to .76.In addition, the common factor loadings of each item ranged from .39 to .72, which was good (Field, 2009).The total variance explained by three factors was found as 54.18%.The internal consistency reliability coefficients of the factors and the scale were .88,.91 and .90respectively and reasonable.
The factors obtained as a result of EFA were tested using CFA.The fit values were found as χ2/df = 2.74, RMSEA = .07,SRMR = .06,GFI = .87,AGFI = .85,CFI = .97,and NNFI = .97,and it can be said that although GFI observable fit value was slightly lower than acceptable value, RMSEA, SRMR and AGFI fit values indicate an acceptable fit and the other observable fit values indicate a perfect fit.
As a result, it can be said that learning object evaluation scale with high level of validity and reliability was developed.This instrument is a valid and reliable tool in order to evaluate LOs.
Researchers and teachers as the potential actual users of LOs can use it to reveal strengths and weaknesses of a LO.

Table 1 .
Item Analysis Results for LOES * Significant at .05 level.

Table 3 .
Factor Loadings and Common Factor Loads of Each LOES Item after Varimax Rotation

Table 4 .
The Fit Values of the Suggested Model and the Standard Fit Criteria