Automated Scoring of Writing Quality

Machines and Human writing

  I am sitting in a lab with students preparing for TOEFL IBT exam. My left ear catches phrases like "In this set of materials...", "The listening passage discusses the difference between to types of bacteria...", while my right ear, to its great surprise, catches the second halves of the same sentences "... the reading passage is a news bulletin on a job announcement, while the listening passage..", " ... the reading passage casts a doubt on the information in the listening passage". The same "automated phrases" are also used in TOEFL writing, as experience has shown me. With a faint smile, I lazily pity the poor person who checks those essays. 

   

  However, a recent  discovery of mine, related to TOEFL and other high-stakes tests is that the essays, written by students, are not only checked by human scorers but also by special automated scoring engines (the  e-rater® in case of TOEFL).  The scores of the human rater and the program are compared and a final score is then assigned.

  The advantages of an automated engine are obvious:

a) Objectivity: A computer program has neither  interests nor judgments. There is no need to worry that it might have a prejudice against you just because you mentioned Justin Beaber as a person you thoroughly admire.

b) Financially economical: Needless to say a computer program is an ideal employee in terms of money. Once it is installed, it only required careful maintenance.The rest is obedience and hard work.

 Unfortunately AES can not be used as a sole evaluating tool of writing (in case of high-stakes exams at least). Although the results of AES have often correlated with human scoring, still it is almost impossible to imagine a computer program justly evaluating the highly complex nature of human writing. Let's have a look at the criteria that the AES take into account while evaluating writing:

• errors in grammar (e.g., subject-verb agreement)
• usage (e.g., preposition selection)
• mechanics (e.g., capitalization) 
• style (e.g., repetitious word use)  
• discourse structure (e.g., presence of a thesis statement, main points) 
• vocabulary usage (e.g., relative sophistication of vocabulary)

 While aspects such as prepositions, agreement between verb and subject and capitalization can be possible handled by an artificial intelligence, more complex parts of human language such as syntax, quality of argumentation, collocation, punctuation, appropriacy of vocabulary,etc seem impossible to asses without human intervention. The quality of argumentation, for example, has little to do with the complexity of vocabulary.

English as a Lingua Franca

Who does English belong to?

     I will start this post with an anecdotal experience. Here's a little piece of conversation I have recently witnessed between two of my students;

 Student A: " I have gone to Kharabakh last summer.  KHARAbakh, no ...KharaBAKH...  What's the right way of saying it?"

 Student B: "And is it right to say "have been" or "went"?" .

 Two questions standing in the queue of my thoughts. But before I can process either of them, it dawns on me that two of my students have just stopped the natural flow of their conversation because of their over-obsession with accuracy. It is one thing that an Armenian has to put his tongue twice between his teeth to pronounce an English word like "thousandth",(or "cenicero" in Spanish")  but why would an Armenian worry so much about the English pronunciation of an Armenian land? Is it KHARAbakh or KharaBAKH...? Who decides this? The English?

  After a brief pause I answered the first question, saying that past simple is preferable when speaking about a specific past event and, turning to the first student, I said smth that an English teacher would not normally say;

  "I think it's up to us to decide how to pronounce Kharabakh".

   Now, as you think of it, the truth is that English has long ceased to be confined to the English of the former British colonies, nor is it now restricted to any geographical location or nationality. The widespread use of English as a lingua franca  (ELF) has led to a change of perspectives as to what should be the model of language in learning English. Traditionally, we followed the model of the native speaker, supposing that the ultimate aim of the language learner is to communicate with the native speaker. However, such a perspective can no longer function. English is now used everyday between non-native speakers as a common tool for communication without any interaction with a native speaker. In fact, many EFL learners never ever meet the "perfect native speaker" in their lives! Keeping this in mind, how reasonable is it to promote the standard English in all circumstances ? Say an Armenian, Russian and a German work in the same office and interact in English, should it trouble them that their intralanguage deviates from the Standard English  as long as the communication is successful?
   Coming back to the question of "KHARAbakh or KharaBAKH" mentioned in the beginning. Not only has English successfully escaped from the British colonies and spread all over the world as a lingua Franca, it has also started to be shaped by the world. Whether English teachers accept it or not, billions of non-native speakers make "mistakes" (deviation from the accurate forms of standard English) every day all over the world. Is it right? Is it wrong? It is a fact. A non-native induced erroneous form "I am loving it" (EFL teachers, put your red pens in your scabbards. I know that "love" is a stative verb) became the slogan for McDonalds. Coincidence or a reflection of a tendency?
  In conclusion, I would like to say that whatever the case is, it's time for EFL teachers to stop blindly following the native model and prioritize the needs of their students in making decisions on what to teach, what to correct and what to leave as it is.
   

Paradigm controversies, contradictions, and emerging confluences

  The vast ocean between theory and practice

    Some articles related to Applied Linguistics make me doubt whether they were written by humans. An article published in 2005 by Guba and Lincoln, called "Paradigm controversies, contradictions, and emerging confluences" is a vivid example. The article raises some controversial issues among the practitioners of new and emergent paradigms, who are looking for key elements that would distinguish their paradigms from those of the others. An average reader, let's say an average language teacher, looking for some guidance for his teaching, will probably need some psychological help after reading this article. Abstract terms, ideas and theory without a hint of its applicability all flood together and leave the teacher clueless as how to improve his teaching the next day, how to make his lessons more effective. While it is clear that such articles might be useful for a narrow circle of researchers that speak the same language in which the article is written, the question is for who, in the long run, is all the research done? If it's ultimately intended for teachers, I am afraid, it will find its readers frustrated and will probably lose them after the first few pages.  
 P.S. I read Loudres Ortega's article called "For What and for Whom is out Research" after writing this post. The author brings up the issue of the "lack of relevance of SLA research for teachers". Research, as author states, is useful when it serves societal needs. Otherwise, it may face the issue of being useless.

Welcome to the Grey world of Applied 
 Linguistics 
Feel the Greyness

  5 multiplied by 5 is 25. Now try and say the opposite. Try to prove that in Europe numbers work in a different way than in Asia or that cultural differences, individual variables, the peculiarities of setting or participants involved may have an influence on a mathematical equation. Your chances, to say the least, are non-existent. Now step out of the black and white universe of natural sciences and go through the threshold of Applied Linguistics.There you are, embrace the Greyness.

      Here's the world where phrases like "take it with a grain of salt", "tentative claims", "non-conclusive data", etc come into play. In fact, they are much more common than words as "proven", "enough evidence", "generalizable", etc. Fortunately, linguistic research tries to shed some light on the overwhelming greyness.

  One issue, for example, that every teacher asks himself/herself at least once in life is how to give feedback.  In what form? How often? Recently I have read a 2009 article on Written Corrective Feedback by Bitchener and Noch. The article, based on a thorough longitudinal experimental study, shows that some  Corrective Feedback (CF) is a must in language teaching but does not distinguish any difference of efficiency between different types of CF (written/oral, direct/indirect). The study, focusing on a 10-month teaching experiment targeting on a single grammatical form (a/the in English), provokes obvious questions on its validity and practicality in day-to-day teaching. Here are a few questions that were raised during our class discussion of the article:

1) Both forms of corrections direct or indirect may lead students to improve their work (correct revision), but do they allow them to retain the information long-term?

2) Was the written corrective feedback the only means of imroving the students' skills for using ''a/the'' during the research period (remember that they had long intervals between the tests)?

3) Is it and if yes, how practical is it to apply WCF on a single-error category throughout a long period of teaching-learning process?

 These are all valid questions and come to show once again that research in Applied linguistics often times fails to bring any clearness and distinction in our understanding of teaching and language learning. Shades of grey, that's all we can hope for.

Cloud computing

Your data everywhere

   Cloud computing, according to this website, is the storing and accessing of applications and computer data often through a Web browser rather than running installed software on your personal computer or office server.

  The real meaning of cloud computing came to me when I had to work on the same data from different computers. I had to export the data via a USB flesh and then I had to keep deleting the old files in both computers and paste the updated files. This doesn't happen when everything is "in the cloud", stored somewhere in the web. As long as you have internet access, your files can be accessed from everywhere, any time. 
    Not only files can be stored online but also software. More and more companies and individual users are now saving valuable space in their PC and are using online software instead. 
  However, since the entire data is "in the cloud" (which, in reality, isn't a cloud at all but a set of mega-powerful servers), questions like "Who is protecting the data?", "Is it safe?", "What if there is a leakage of data?" arise.  Google assures the safety and confidentiality of the data it stores. However, very few would risk giving sensitive data in hands of a third party, irrespective of the company's reputation. 

Google Apps

"Let the robots do the mechanical work and let the human mind shine in the rays of creativity."

 This video, though not groundbreaking by itself, made me think about just how much of my work is done by google. 

 Gmail  diligently sends and receives my messages as well as filters them according to my preferences, Google calendar helps me to save a lot of free space in my memory by scrupulously reminding me abut my tasks and duties, Google Drive makes my documents reachable from every computer on this Earth (provided it has an access to the internet) and helps me to give  feedback to my Ss' writings in no time, Google maps knows my city much better than I do, Google hangout is just the perfect tool for group works, Google search engine is universally acknowledged as the real Superman (if there ever existed one)... Do I need to continue? 

  And the best part of it is that all this is free. And we take it for granted. What we forget sometimes though is that there are many people in the world (teachers in our case) who stay blissfully ignorant of all this. Not that a good teacher can't do her job without google apps. You would never hear me say that. The point is that these G friends (you can call them personal house elves if you wish:)) take away the mundane, the mechanical, the dull part of the job and  create a comfortable a space for  more sophisticated, more creative work for teacher.If you don't use Google products, certainly give them a try. They are real treasures right under your nose!

Academic Analytics

"gut instinct" vs data-driven decisions

AND

Learning that leaves trails

 Suppose you are an EFL teacher and the next agenda in your lesson plan is teaching types of clothes. You have a look at the textbook and you see a list like "vest", "shirt" and "bodywarmer". You prepare a great lesson plan, , go into the classroom, introduce the words and BOOM!: the students find the words confusing. They seem to be ok with the other new words but these devilish three for some reason just don't work! After the lesson you talk with your colleges and  hear smth like this: "Yeah, those three are always a problem". Each of them knows this that but they still follow the book's plan over and over again. The procedure repeats itself, the experience gets reduplicated and the problem persists.

  This story could have been completely different if you have just  looked into another source for guidance. Namely, if you read some research on the topic, you would have found the following:

"It's easier for the human cognitive system to differentiate between two new different items than two similar ones. Therefore, words that are similar  in some aspect (meaning/pronunciation/spelling) shouldn't be introduced together as they present a challenge for memory."

  See how sometimes research beats the "gut instinct" or the "experience"?  This is why we should steer towards a more data-driven decision-making and make a very judicious use of subjective personal experience and instincts. 

  Research is based on data. The more date we have, the more reliable and valid our research is. Now the amount of data we can get in the modern world is simply breathtaking. It has reached to the point of the so-called "Big data". Let's consider the field of education for example. Virtual Learning Environments like Moodle  track every single click that the learners make, thus yielding a complete virtual footprint. We get information about which pages in VLE are never visited by the learners , which of them are most popular, which learners are active and which, on the other hand, are "at-risk" and maybe need guidance. This data is analyzed via Learning Analytics ( the measurement, collection , analyzes of data for the purposes of optimizing learning) and is used to inform changes in the learning and the curriculum. Effective use of readily available data, that's what this is. 

  Now, do you feel a little in a daze? Do you feel like the Earth under your legs is shaking? That's right! The world is changing and moving ahead! Are you coming?