|
THE COLLEGE OF NEW |
|
Computer Science |
|
CSC485 03: Special Topics : Open Domain Question Answering Systems |
|
Spring 2003 |
The goal of the information
extraction (IE) is the design of systems that are capable of analyzing only
the text passages which contain relevant (the system given) information.
In addition, such systems do not try a comprehensive analysis of all text
documents but purposely overlook the irrelevant information. Textual Question/Answering
(QA) systems represent the most current trend in the information extraction
from free on-line sources of text. A goal is the construction of systems,
which can identify the answers to a natural-language question from a large quantity of on-line text documents. In contrast
to information retrieval systems which supply a quantity of documents as
a result of a simple, word-based search, QA systems are capable of identifying the exact passages in the text of
relevant documents which represent the concrete answer. In addition, there
are no restrictions concerning the subject of the natural-language questions.
This course will discuss theory and practice of
open domain question answering systems and related bibliographic information.
Topics to be addressed include, but are not limited to: (i) Question answering language resources (LR) and
scientific algorithm developments, (ii) Guidelines, standards, specifications,
models and best practices for question answering LR, (iii) Methods, tools,
and procedures for the acquisition, creation, management, access, distribution,
and use of question answering LR, (iv) LR and evaluation and benchmarking
of question answering systems and algorithms for tasks including (a) Advanced
question analysis, (b) Answer discovery and integration, (c) Answer explanation
and presentation generation, (d) Interactive question answering.
Possible joint products to be created include:
(i) List of existing resources and ones under
development (with planned release dates), (ii) Updates to ARDA Q&A Roadmap
(www-nlpir.nist.gov/projects/duc/papers/qa.Roadmap-paper_v2.doc), (iii) List
of Evaluation methods and benchmarks of question answering systems, (iv)
List of unresolved research problems and/or areas in question answering, (v)
Shared knowledge of research groups and efforts.
Advanced Algorithms (CSC410) + Instructor's Permission.
Dr. Miroslav Martinovic
Brief Biography
Ph.D.
1993
CS Faculty 2000-present, TCNJ
CS/Math Faculty, 1989-2000,
CS/Math Faculty, 1983-1988,
Principal Scientist, 1989-present.
Research Interests
Question-Answering Systems
Natural Language Processing
Information Retrieval
Theory of Gaming
Computer Science Education
Logic Programming
Expert Systems
Sponsors
NSF
DARPA
NIST
Microsoft
E-mail Address :
|
mmmartin@tcnj.edu (click to e-mail) |
Telephone :
|
(609) 771-2789. |
Office :
|
Holman Hall 230. |
Class Time:
|
Lectures |
Monday, Wednesday |
At Holman Hall 128 |
|
Instructor supervised assigned work |
360 minutes at students own schedule. |
at Holman Hall 372 |
Textbooks:
|
Course Main
Text |
|
|
|
Modern Information
Retrieval |
by : R. Baeza, B. Ribeiro, Addison Wesley, 2000. |
ISBN 0-201-39829-X |
|
Additional Texts |
|
|
|
Mathematical
Foundations of Information Retrieval |
by: |
ISBN 0-7923-6861-4 |
Office Hours :
|
Monday |
Tuesday |
Wednesday |
Thursday |
Friday |
|
|
|
|
|
|
|
9-9:30 |
|
9-9:30 |
9-11 |
|
|
|
|
12-2 (by appt.) |
12-3 (by appt.) |
|
Grading Policy:
|
|
CSC485 03 |
Introduction to Corpus-Based Question Answering
|
Week
1and 2 |
|
|
What is corpus-based
Q&A ? Slides (transparencies used in class) Courtesy of : C. Monz and M. de Rijke |
What's in Store for Question Answering ? Ask Jeeves
|
Week
2 and 3 |
|
|
"Take-home" messages
when considering Q&A task Slides (transparencies used in class) Courtesy of : J.B.
Lowe |
Web Information Retrieval : Google's Success
|
Week
4 |
|
|
Paper presentation
and critique. Courtesy of : M.
Henzinger |
Essential Properties of Information
Retrieval : NLP for IR
|
Week
5 |
|
|
Paper presentation
and critique. |
NLP Tools :
Generic Retrieval Systems (SMART System)
|
Week
6 |
|
|
Paper presentation
and critique with a demonstration session. |
NLP Tools :
Part-of-Speech Taggers
|
Week
7 |
|
|
Paper presentation
and critique with a tagger installation and
a demonstration session Paper : Papers/POSTagger/aaai94-tagger.ps
|
NLP Tools :
Parsers (A Parser for English)
|
Week
8 |
|
|
Paper presentation
and critique with a demonstration session Papers : Papers/APParser/manual.ps, Papers/APParser/APParser.htm
|
NLP Tools :
Electronic Lexicons (WordNet)
|
Week
9 |
|
|
Paper presentation
and critique with a demonstration session Documentation
: http://www.cogsci.princeton.edu/~wn/doc.shtml |
Advanced Question Answering : Plenty of Challenges to Go Around
|
Week
10 and 11 |
|
|
AQAINT Program
Slides (transparencies used in class) Courtesy of : ARDA
and J.D. Prange |
Issues, Tasks and Program Structures
to Roadmap Research in Q&A
|
Week
12 and 13 |
|
|
Issues in Q&A
Research Slides (transparencies used in class) Courtesy of : J.Burger, et. al. |
Named Entity Recognition
|
Week
13 |
|
|
Paper presentation
and critique Paper resource directory : ~mmmartin/www/CMSC485/Papers/NER/ |
Anaphora Resolution
|
Week
14 |
|
|
Paper presentation
and critique Resource directory : ~mmmartin/www/CMSC485/Papers/Anaphora/ |
Project Presentations and Demos
|
Week
14 |
|
Topic Paper and Demonstration Materials |
Presenter |
Presentation
|
|
1.
Web Information Retrieval : Google's Success ¤ Paper : Papers/Google/icde.pdf |
R. D. VonGleich |
Week of 02/10 |
|
2
Essential Properties of Information Retrieval : NLP for IR Paper : ¤ Papers/NLPforIR/NLP-IR.pdf |
P. Y. Ng |
Week of 02/17 |
|
3.
SMART System : Paper with a demonstration session |
D. S. Barber |
Week of 02/24 |
|
4.
Eric Brill's Part-of-Speech Tagger : Paper with
the tagger installation and demonstration
Paper : Papers/POSTagger/aaai94-tagger.ps,
Resource directory : ~mmmartin/Information
Retrieval/EricBrill'sTagger/ |
M. D. Scehovic |
Week of 03/03 |
|
5.
Apple Pie Parser for English : Paper with a demonstration session Papers : Papers/APParser/manual.ps, Papers/APParser/APParser.htm
, ¤
Papers/APParser/CBSSemanticParser.pdf, ¤
Papers/APParser/SemanticParsingAIMag.pdf
, |
K. A. Wilson,
Jr. |
Week of 03/17 |
|
6.
WordNet Electronic Lexicon : Paper with a
demonstration session Documentation : http://www.cogsci.princeton.edu/~wn/doc.shtml, Resource directory : ~mmmartin/www/CMSC485/Papers/WordNet/ |
J. N. Hankins |
Week of 03/24 |
|
7.
Named Entity Recognition Paper Resource Directory : Papers/NER/ |
R. J. Wagner |
Week of 04/14 |
|
8.
Anaphora Resolution Paper Resource Directory : Papers/Anaphora/ |
A. Archer Waterman |
Week of 04/14 |
|
8.
Anaphora Resolution Paper Resource Directory : Papers/Anaphora/ |
J. M. Burger |
Week of 04/14 |
|
Topic Paper and Demonstration Materials |
Presenter |
Presentation |
Paper
critique and presentation guidelines
Paper Critique Guidelines
Each critique should be no more than one page long. Less than a page is OK. The purpose of a critique is not to summarize the paper; rather you should choose one or two points about the work that you found interesting.
Examples of questions that you might address are:
Your critique should be typed (single space) and should list the title of the paper and its authors at the top, along with your name.
Avoid unsupported value judgments, like ``I liked...'' or ``I disagreed with...'' If you make judgments of this sort, explain why you liked or disagreed with the point you describe.
Be sure to distinguish comments about the writing of the paper from comment about the technical content of the work.
Paper Presentation Guidelines
Length : class period (60-80 minutes)
Medium : PowerPoint, HTML slides, PDF slides or alike.
Paper Critique Presentation Guidelines
Length : class time (talk of up to 40 minutes to be followed by an up to 40 minutes discussion mediated by the presenter)
Medium : PowerPoint, HTML slides, PDF slides or alike.
Note about how the preparedness for other students presentations affects the grade
(i) All listed papers must be read by every student in class.
(ii) The discussion following the paper presentation and paper critique presentation demonstrates that the student has read the paper.
(iii) Student's involvement and competence
in the discussion from (ii) will directly affect the "Attendance, Class Participation
and Effort"'s 20% of
student's total grade for the entire course.
2003-01-16
CMSC485 03 Special Topics :
Open Domain Question Answering Systems
Spring 2003
Course Project
Due at the beginning of the scheduled presentation, on Monday, April 28 or Wednesday, April 30
Goal for the assignment: to gain a basic experience in the design, implementation, and evaluation of questionanswering (QA) systems. The project is fairly openended. You are a member of a project team of two who is to implement a QA system that will operate in the standard TREC QA framework: the input to the system is a question, the output is a ranked list of five guesses for the answer. No human intervention is allowed in deriving answers.
For the assignment, we are providing a QA corpus that contains a set of questions and the expected answer(s) for each question. Since we can't make available to you the actual 9GB TREC collection used in the TREC QA studies, we will instead provide the top 20 documents retrieved by the Smart IR system (from a similarly large text collection) for each question in the corpus. Answers to each question are to be extracted from these 20 documents. Note that it is possible for some questions that none of the 20 retrieved documents contains the answer.
As noted above, the project is completely openended: you are free to build whatever components you'd like to include in your QA system and are free to use any publicly available software that you wish. You can even share components that you build with others in the class.
The primary caveats are that your system cannot use the answers provided
and must make clear in the writeup what
components you used that you did not write yourself.
Assume that your system has entered the 50byte (short answer) QA track
so all answers should be 10 or fewer words in length. In addition, the output
for each question should be the following:
question# documentid
answertext(for topranked guess)
question# documentid
answertext(for second guess)
question# documentid
answertext(for third guess)
question# documentid
answertext(for fourth guess)
question# documentid
answertext(for fifth guess)
The documentid refers to the document where
the answer string was found. Use "nil" as the answertext
if your system finds no answer for a particular question.
What is provided :
questions.txt: the questions (http://www.tcnj.edu/~mmmartin/CMSC485/Project/questions.txt).
Feel
free to change the format of this file if it makes automatic processing of
the questions easier. Alternatively, you can use the questions as they appear
in
the "answers"
file described below. In either case, you will need to keep around the question
number to include as part of the answers file that your system produces.
answers.txt: all answers found by TREC assessors for
each question (http://www.tcnj.edu/~mmmartin/CMSC485/Project/answers.txt).
The format
of this file should be pretty clear. For each question, the file contains:
(1) one line with the question number, (2) one line with the question, (3)
list of document
id's followed by answer strings, one per line, (4) a blank line separates
the information for each question. Feel free to modify the format of this
file if it's easier for your
system
to process.
top 20 documents retrieved
for each question: A gzippedtar file with
the top 20 documents retrieved for each question by Smart can be downloaded
from
http://www.tcnj.edu/~mmmartin/CMSC485/Project/top-20.tar.gz.
(WinZip should open this file as well.)
Implementation hints :
Start simple!! Select some really really dumb strategy to produce answers for each question
just to make sure that you will have something to evaluate and to turn in.
Only after you can do that should you proceed to something more sophisticated.
It's fine to try a strategy very different from
anything discussed in class. It's even fine if the system that you produce
does terribly in terms of performance. You just need to be able to argue
(in your writeup) why the strategy that
you investigated MIGHT have worked. One possibility is to try using Lemur
(http://www-2.cs.cmu.edu/~lemur/)
or Smart system to implement a passage retrieval strategy for question answering.
Another is to instead focus on one type of question, e.g. "who" questions,
and develop a strategy specifically for that question type.
What to turn in :
1. A description of your QA system. Enough detail should
be provided so that, in theory at least, I could reimplement
it. The description should explain each component in your QA system, the
steps that your system takes to answer a question, any additional online
sources of information used by the system, etc. Make clear which components
of the system you built yourself vs. downloaded from elsewhere vs. got from
another student in the course.
2. The output file of answers produced by your system
for the questions from the development corpus that we provided. The answers
should be in the format described above.
3. An evaluation (e.g. using the mean reciprocal rank
evaluation measure) and analysis of your system's performance on the questions
from the development corpus provided. How well did the system work? What worked?
What didn't work? Can you say anything about which component is strongest/weakest?
4. A detailed walkthrough of what your system did
to handle one question (any one) in the corpus.
5. The output from your system for the question selected
in (4) above. Enough information should be included in the output to convince
me that the system is following the steps
described in (1). It is not necessary to submit your code, but I may ask
to see it in cases where the system description is unclear.
Presentation guidelines :
1. The presentation should be a 35 minutes talk.
2. An additional 5 minutes questions session should follow
the talk.
3. The talk should include a simple demonstration which
is not to exceed 15 minutes in length.