Testing and Assessment
Interview with Christine Coombe
 Christine Coombe has a Ph.D in
Foreign/Second Language Education and Testing from Ohio State University. She is currently
on the English faculty of Dubai Men’s College and works as an Assessment Leader for the
Higher Colleges of Technology. Christine is co-editor of the forthcoming Assessment
Practices volume in the TESOL Case Studies series.
She is the current President of TESOL Arabia and the founder and co-chair of the TESOL
Arabia Testing, Assessment and Evaluation Special Interest Group who organize the Current
Trends in English Language Testing (CTELT) Conference. Christine is the recipient of the
2002 Spaan Fellowship for Research in Second/Foreign Language Assessment and the 2002–03
TOEFL Outstanding Young Scholar Award.
Christine, is this your first visit to Russia?
It’s in fact my third visit to Russia. The first time I visited
Russia was in 1997 when I spoke at the LATEUM conference at Moscow State University. That
was only a brief visit to Moscow. And then I came as a specialist in June this past year
and I visited the Far East (Vladivostok, Khabarovsk, Blagoveschensk and Ekaterinburg). I
gave similar type of training sessions for the teachers out there.
This time you’ll also travel a lot, won’t you?
Yes, this time I’ll also be on the move. My first stop is here in
Moscow. And then I’ll go on to Samara, Rostov-on-Don and St. Petersburg.
What are the main goals of your visit?
Well, the main goals I think are basically just to give teachers
information about different types of assessments. Before I came here I had sent out a list
of the things I could speak about to the places that wanted me to speak, and they chose
what they wanted. So I’m speaking on topics that were selected by different agencies,
different universities, etc. My main interest however is training teachers in how to write
tests.
You’ve met a lot of Russian teachers and educators, and I’m sure
you have answered many teachers’ questions. What are your impressions? Are they the same
as your expectations were?
Until very recently, I hadn’t really spent a lot of time with
teachers here. I just participated in a conference, and I listened to different people
speak. So I didn’t have a lot of ideas about people in schools. But my impressions are
that there is a tremendous dedication and professionalism in ELT in this country, and
that’s so good to see. I expected there to be a bigger, more pronounced dedication in
the big cities; but I’m finding that teachers, some of whom have never spoken to a
native speaker are just as dedicated. And one of the things that surprised me was that
there were many people who had never spoken to a native speaker and yet their English
language was so good. I mean you wouldn’t expect English with native-like
pronunciation… at least I wouldn’t expect really good language proficiency for people
who have never had access to English. That has been really surprising to me.
Meeting experts at the Ministry of Education
What about teachers’ understanding of the idea of assessment,
testing, evaluation?
This is often a concern for me. I have asked my colleagues from the ELO
(English Language Office) of the US Embassy in Moscow about the level of understanding.
I’m always aware of the fact that sometimes teachers are embarrassed if they don’t
understand the terms. So I try to define things, and I try to repeat definitions, and
I’ve noticed a lot of people were writing things down. But sometimes I get going,
because I really like testing, and you can’t shut me up when I’m speaking about it. So
if someone doesn’t remind me and say: “Hey! What is it? What do you mean by
“transparency”? I don’t think to define it. I hope that everyone got the basic gist
of my presentations. There are some unfamiliar terms, but as I define them they are not
difficult to understand. Assessment is a specific area and you can talk to a room full of
PhDs and they all will learn something. Lots of times if you’ve got a PhD, you go to the
presentation and you think: “This was the waste of my time, there’s nothing new for
me”. With assessment it’s quite different. Even the most basic information is new to
people, so they often feel very good about attending assessment presentations, because
they learn something. I’m lucky in that respect.
Did you feel this desire to learn in Russian audiences?
Certainly…even their coming on Saturday, when it’s so cold outside.
And their questions showed their interest, and also the importance of this knowledge for
them.
The situation in Russia with this new unified exam is rather difficult
for teachers. The exam is to be held in May, and English teachers still don’t have any
information on the tests. There are no variants, no samples of the test. Is such a
situation possible in other countries? What should the teachers do?
I understand the anxiety the teachers feel. If I were a teacher in that
context I would feel nervous. However, what’s happening here is a bit backwards in fact,
but not “backwards” in the sense of stupid. Usually any assessment reform follows
curricular reform. So, what you usually do is you change the curriculum, then you change
the assessment to match the curriculum. That’s the recommended way; although, some
people use assessment reform to bring about changes in the curriculum. It could be
what’s happening in Russia, that people are too entrenched with the current curriculum,
and they need a massive change to the assessment to get people moving to change the
curriculum. What you have here is this classic situation “The tail wagging the dog”.
It’s important to make teachers understand that they shouldn’t regard this unified
exam as being something to dread. It can also bring about a lot of good, positive change.
It’s high-stakes, so there will be a lot of anxiety surrounding it. But also, if it
succeeds, it will be one of the only initiatives I’ve ever heard of to be able to do
this. Even my own country, the US, has not attempted such a reform. So, if it does
succeed, it will be really something to be proud of. And it’s also important to
understand that the first year or so there will be some glitches, some problems…but it
will improve in time. It’s important not to think about “us and them”. That’s what
often happens in testing.
That’s why teachers don’t get involved in testing because they think the testers are
the bad guys. And, in fact, we are not. Good testers want to make sure that they create a
test that is fair for all students. They do the best they can, but ultimately there are
things that are out of their control… especially if government and lawmakers are
involved. They are not educationalists. So, many times they don’t have the best
interests in education in mind when they do things, when they make reforms. It’s
important to know that all testers want to be proud of the tests they write; but sometimes
they do work under a lot of pressure, and there are a lot of constraints imposed upon
them; and what usually happens is that teachers can’t criticise the people who are
really to blame, they focus their criticism and negativity on the people who write the
tests. This basically creates this very antagonistic view with a testing team – the
“us and them”.
It’s very important that if teachers want to express their opinion they need to do so in
a constructive manner.
You know, one of the reasons I want to come to events like these in Russia is that I get
such a happy welcome… everybody is pleased to see me; they are all smiling as I look out
into the audience. Usually when I am at home everybody is attacking me because they
don’t like question 36. What about the other 99 questions on the test that are good?
Nobody comes up and says: “Oh! Question 77 is really excellent!” All you hear is
critique…negative comments. We never hear positivity. So, it’s important, that if you
do want to give criticism, that it be constructive criticism. Not like: “Oh, this is a
horrible question!” Think about ways this question could be made better. Teachers who
want to express their opinion should express them selves in a constructive way, by giving
possible solutions to make the situation better.
Talking with one of the test-writers in St. Petersburg
Do you often have feedback from teachers? Do you take into
consideration teachers’ opinions, when creating your tests?
Of course, of course. Most of the people in my team I’ve gotten
because they’ve stood up and they’ve said: “What a horrible test. Blah, blah,
blah...” And I’ve said: “OK. It’s a horrible test. How do you think I can change
it. Join the testing team and we’ll work together to change it”. So most of my team
were those who expressed their opinion, showed the initiative.
Most of the problem with tests concern the transparency issue. Sometimes teachers don’t
know why things are being done. They are not told; they think everything is secret and
that creates the problem.
I’ve met with the experts and some members of the testing team here and I’ve advised
them that they need to put out practice exams, and they need to use the same rubrics that
will be found on the real exam, because you are going to have wide-scale test anxiety on
the part of the teachers and the students surrounding this exam, and they have to be very
proactive in reducing this anxiety before it upsets the whole nation. There are ways to
get around it. I’m sure that the testing team has the students of Russia as their
primary interest; but their interest is also in getting a test that is going to be fair to
all. But there will be a lot of cooperation needed from teachers in order for this to be
successful.
For example, they send out a script and the instructions must be very explicit, like:
“You can only say this, if a student asks a question”. Teachers need to abide by that,
because if they don’t, there will be stories that the teachers in Khabarovsk (example
only!) were giving directions in Russian while the teachers in Moscow were being very
strict and they would only point to the directions on the test – that’s not fair.
You’re going to have things like this coming out and that will destroy the reputation of
the test.
Do you think it’s important to have independent experts on the test?
I’ve recommended that they have someone external as well. I met with
a group of teachers all based in Moscow, and I assume all the test-writers are based in
St. Petersburg. This smacks of urban control. That’s not what you want for a nation
wide test. My advice will be that if you want the teachers from other regions to buy in,
there needs to be involvement. That means you need to train people in various regions to
write items and all people contribute. All teachers around Russia should have an
opportunity to contribute. Because if you have situations where teachers from
St. Petersburg wrote the test, and if by any chance the students from
St. Petersburg do the best on the test, there will be the wide-spread opinion that
the test is biased, because the writers are teaching their students more about the test
than everywhere else. In order to have reliability you have to have more widespread
involvement. Maybe you need to think about the possibility of having testing units in 3 or
4 different areas. I’ve consulted a lot with Morocco; they have a much smaller country
than Russia and they’ve got a variety of different tests. I think they have six regions
and they have regional testing teams. They have common test specifications (that is the
blueprint of the exam), those are common to all the regions, and then each individual
department or office writes a test to those specifications and that test is administrated
in their region. That is probably how I would have recommended this initiative go.
That’s probably what you will find would be more successful.
It all takes a lot of time. I am involved in a project just like this. I direct the
nation-wide placement test in UAE and I complain, but I only have thirteen thousand
students to test in an area that’s probably not even the population of Moscow; and I
know what a ordeal that is. I can only imagine how difficult it’s going to be.
A big expert in our field, Charles Alderson, had eight years to develop a similar test for
Hungary, and I’m sure he wishes he had a couple of more years. For my initiative, with
eight versions of the test, I’ve been given 4 months, and it’s my full time job, so
you can imagine the difficulty you are facing now with your test.
As far as I know, there should be twenty-four different variants of
the test ready by February or March, and most of the teachers even in big cities, with
their access to the Internet, don’t have any idea of what it is going to be like…not
to mention teachers and students in faraway towns and villages.
I understand what you are saying, but you also must remember that, yes,
probably there is different access to knowledge; but this is what makes the world go
round. Usually city students are better than urban students because of a variety of
different reasons.
You have to write the best test you can, and you can’t write it to target the lower end
of the scale; you must target the higher end of the scale...because you think that urban
students are always better than rural students, but it’s not always the case.
Sometimes you have rural students who far surpass urban students, because they don’t
have as much to do – their focus is on studying. And I think that a good test will not
disadvantage students from any area. But it certainly might make teachers in urban areas
more cognisant of the fact that they can’t sort of kick back and relax if they have to
keep up with their counterparts in outside the cities if they want to do justice to their
students.
Do you agree that the test should not comprise something that is new
to students either in format or in content?
Again we come up to the transparency issue. It is a must that the
testing team puts out a couple of practice exams. You need to make sure, it’s crucial to
have samples of tests out there which teachers can use in class, and they can teach what
the students need to know in order to complete the different question formats. If that
doesn’t occur it will be a disaster. When students encounter new formats on an exam, it
can affect the reliability of the exam scores.
You pay a lot of attention to the psychological aspect in testing.
Taking into consideration this rather difficult situation English teachers in Russia are
facing now, could you find any encouraging and inspirational words for our readers.
OK. This is for the test developers. The most important thing – test
what you teach how you teach it! So, that teachers can teach it, they need to know how
it’s going to be tested. The focus has to be transparency. I don’t care how difficult
it is; there can be no secrets surrounding this test; everybody needs to have open access
to what kinds of formats that will be included; the kinds of themes might be prevalent on
the test. You need to lower the anxiety surrounding this test for it to be successful. And
for teachers: Don’t feel too negatively about the test, because this a great initiative;
it means that Russia is going forward with multiple measures assessment.
You should have a test, but it should be only one of the ways you
assess your students. I hope that you don’t go down the path of getting rid of
everything else, and everybody’s grade will just be what they get on this unified test.
I can’t imagine that that would happen, given the fact that the testing team knows what
they are doing. So don’t be afraid of multiple measures assessment – it can only do
good. It will create a lot of work, so teachers have to bear with it, but once it’s
fully in place it will make your teaching easier.
During the seminar for teachers in St. Petersburg
Interview was taken by Alyona Gromoushkina
Dear Readers! If you have any questions or comments on this relevant topic, please
don’t hesitate to write to our editorial office. We will forward them to Christine
Coombe, who has kindly agreed to answer our readers’ questions on the pages of English.
|