Educational Research (3)
Written by T. Neville Postlethwaite, Institute of Comparative Education University of Hamburg from Educational Research: Some Basic Concepts and Terminology
4. Pilot testing
At the pilot testing stage the instruments (tests, questionnaires,observation schedules, etc.) are administered to a sample of the kinds of individuals that will be required to respond in the final data collection. For example, school principals and/or teachers and/or students in a small number of schools in the target population.If the target population has been specified as, for example, Grade 5 in primary school, knowledge should exist in the Ministry, or in the inspectorate, about which schools are good, average, and poor schools in terms of educational achievement levels or in the general conditions of school buildings and facilities. A ‘judgement sample’of five to eight schools can then be drawn in order to represent a range of achievement levels and school conditions. It is in these schools that the pilot testing should be undertaken. The two main purposes of most pilot studies are:
a. To assess whether a questionnaire has been designed in a manner that will elicit the required information from the respondents. This process allows weaknesses in the questionnaire to be detected so that they can be removed before the final form is prepared. Typical weaknesses that are found in questionnaires include:
• Ambiguities in the phrasing of questions.
• Excessive complexity in the language that has been used.
• Inappropriate response categories for some questions.
• Some questions are redundant.
b. To assess whether test items can be understood by the students,that the items are pitched at the appropriate level of complexity (assessed by the ‘Difficulty Index’), provide a stable measure of student ability (assessed by the ‘Reliability Index’), and lead to the construction of total test scores that are meaningful in terms of the student ability being examined (assessed by the ‘Validity Index’).
Typical weaknesses that are found in tests include:
• Some items have either no correct answer or more than one correct answer.
• Some distractors in multiple choice items are not functioning.
• Some items measure abilities different from the ability measured by other items (assessed by the ‘Discrimination Index’).
• Some items contain internal ‘tricks’ that result in high ability students performing worse than low ability students.
5. Data collection
When a probability sample of schools for the whole of the target population under consideration has been selected, and the instruments have been finalized, the next task is to arrange the logistics of the data collection. If a survey is being undertaken in a large country, this can require the mobilization of substantial resources and many people.
The management of this research stage will depend on the existing infrastructure for data collection. In many countries there are regional planning officers and within each region there are district education officers. These people are often used for collecting data. However, the problem of transport can loom large, especially in situations where there is a shortage of transportation and spare parts, and where office vehicles are booked weeks in advance. In such countries, two to three months of careful prior planning will be required.
The instructions for the completion of tests and questionnaires must be clear. When the testing of students is involved, it is important to have a special “Manual for Test Administration”. This manual explains how to arrange the testing room, and provides standard instructions that are given to the students about how to complete the test, questionnaire, or attitude statements, when to start and when to finish.
Finally, instructions must be clear on how the data collection instruments are to be returned from the field. When data collection instruments are returned to the National Planning Office, checks must be undertaken to ensure that all instruments have the correct identification numbers.
5. Data analysis
If there are unequal probabilities of selection for members of the sample, or if there is a small amount of (random) non-response,then the calculation of sampling weights has to be undertaken. For teacher and school data there are choices which can be made about weighting. For example, if one is conducting a survey and each student in the target population had the same probability of entering the sample, then the school weights can either be designed to reflect the probability of selecting a school, or school weights
can be made proportional to the weighted number of students in the sample in the school. In this latter case, the result for a school variable means the school value given is what the ‘average student’ experiences. This matter has been discussed in more detail in the module on ‘Sample Design’.
a. Descriptive
Typically, the first step in the data analyses is to produce descriptive statistics separately for each variable. These statistics are often called univariates. Some variables are continuous – for example ‘size of school’ which can run from, say, 50 to 2,000. In this case the univariate statistics consist of a mean value for all schools, the standard deviation of the values, and a frequency distribution showing the number of schools of different sizes. Other variables are proportions or percentages. Such a variable could be the percentage of teachers with different types of teacher training.
These descriptive statistics describe the characteristics of the students, teachers, and schools in the sample. If a good probability sample has been drawn, then generalizations (within narrow limits) can be made about the target population.Often comparisons between the Ministry norms and sample averages are made. For example, if the Ministry has stated that each student should have 1.25 square meters of space in the classroom,then this norm can be examined for each school by dividing the total number of square meters of classroom floor space in the school by the total enrolment of students in the school. This statistic may then be used to give direct feedback to the educational planners in charge of buildings about the extent to which their norms are being met.
b. Correlational
In this case product moment correlations or cross tabulations can be calculated. There are statistical tests which can be applied to determine whether the association is more than would occur by chance. When the association between two variables is examined, this is known as bivariate’ analysis.
c. Causal
If the research design used is an experimental one, then tests can be applied to see if the performance of the experimental group (that is,the group subjected to the new treatment) is better than the control group.
There are statistical techniques for determining this. However, the use of this approach depends on the application of randomization in order to ensure that the two groups are ‘equivalent’ in all other respects. If the research design is based on a survey, then it is possible to calculate the influence of one variable on another with other variables being “held statistically constant”. Where calculations are
made of the relationships among more than two variables at the same time, this is known as ‘multivariate analysis’.
6. Research report
There are three major types of research reports. The first is the Technical Report written in great detail and showing all of the research details. This is typically read by other researchers. It is this report that provides evidence that the research was conducted soundly. This is usually the report which is written first.
The second report is for the senior policy makers in the Ministry of Education. It is in the form of an Executive Summary of about 5 or 6 pages. It reports the major findings succinctly and explains, in simple terms, the implications of the findings for future action and/or policy.
The third General Report is usually in the form of a 50 to 100 page booklet and is written for interested members of the public,teachers, and university people. This report presents the results in an easily understood and digestible form.
Finish
