For an entire semester, Georgia Tech computer science professor Ashok Goel
hid the fact that his online teaching assistant “Jill Watson” was a robot.
Goel did not disclose the robot until final grades were handed out. None
of the students figured it out.
Each year, students have an option of selecting an assistant and next year
Goel will do the same, but names will change. Students will have to guess if
they are dealing with a robot or not.
Jill Watson
Please consider Professor reveals to students that his assistant was an AI
all along.
To help with his class this year, a Georgia Tech professor hired Jill
Watson, a teaching assistant unlike any other in the world. Throughout the
semester, she answered questions online for students, relieving the
professor’s overworked teaching staff.
Students were amazed. “I feel like I am part of history because of Jill
and this class!” wrote one in the class’s online forum. “Just when I wanted
to nominate Jill Watson as an outstanding TA in the CIOS survey!” said
another.
Goel and his teaching assistants receive more than 10,000 questions a
semester from students on the course’s online forum. Sometimes the same
questions are asked again and again. Last year he began to wonder if he could
automate the burden of answering so many repetitive questions.
As Goel looked for a technology that could help, he settled on IBM Watson,
which he had used for several other projects. Watson, an artificial
intelligence system, was designed to answer questions, so it seemed like a
strong fit.
To train the system to answer questions correctly, Goel fed it forum posts
from the class’s previous semesters. This gave Jill an extensive background
in common questions and how they should be answered.
Goel tested the system privately for months, having his teaching
assistants examine whether Jill’s answers were correct. Initially the system
struggled with similar questions such as “Where can I find assignment two?”
and “When is assignment two due?” Goel tweaked the software, adding more
layers of decision-making to it. Eventually Jill reached the point where its
answers were good enough.
“I cannot create chaos in my classroom. Jill had to be almost as perfect
as a human TA or I am,” Goel said.
The system is only allowed to answer questions if it calculates that it is
97 percent or more confident in its answer. Goel found that was the threshold
at which he could guarantee the system was accurate.
There are many questions Jill can’t handle. Those questions were reserved
for human teaching assistants.
Goodbye TAs
Jill will not eliminate the need for all TA’s just 97% of them.
The real test comes next year. Students will ask all kings of questions in
attempts to figure out if they are dealing with a robot or not.
“Michelle Tutelage” better be prepared for ridiculous questions, and so
must the real TAs. Precisely consistent answers and tones may help reveal the
robot. I suspect human answers may vary a bit more than robots even for
identical questions.
Perhaps Goel has already figured that out, or perhaps he will make a
program change after reading this.
Secondly, if the same TAs help each year, then all of the TAs’ names must
be disguised. So, why not just randomly assign the TAs?
“A really fun thing in this class has been once students knew about Jill
they were so motivated, so engaged. I’ve never seen this kind of motivation
and engagement,” Goel said. “What a beautiful way of teaching artificial
intelligence.”
This kind of program will eliminate the need for TAs but it will also
lower the cost of education making it more affordable.
Goel is forming a business to bring the chatbot to the wider world of
education. “To me this is a grand challenge,” Goel said. “Education is such a
huge priority for the entire human race.”
Mike “Mish” Shedlock
