User Guide
Using TedEval
Single Experiment Evaluation:
Usage:
java -jar tedeval.jar -p parse_file -g gold_file -o eval_file -format conll
Pairwise Experiment Evaluation:
Usage:
java -jar tedeval.jar -p1 parse_file1 -g1 gold_file1 -o1 eval_file1 -p2 parse_file2 -g2 gold_file2 -o2 eval_file2 -format conll
Using StatSignTest
Statistical significance testing for a pair of experiments which use the same native gold or the same generalized gold file:
Usage:
java -jar statsigntest.jar -i eval_file1.ted -i eval_file2.ted -o results.statsign -n 9999
Command Line Options
TedEval
Usage:
java -jar tedeval.jar -h for more help and options
help ( -h) : Show options
version ( -v) : Show version
-----------------------------------------------------------------------------
Single experiment evaluation:
gold_file (-g) : path to gold-parses file (single experiment)
parsed_file (-p) : path to parse-hypotheses (single experiment)
output_file (-o) : path to result file (single experiment), default file name is eval
Pairwise Experiment evaluation:
first_gold_file (-g1) : path to gold-parses file (the first of two experiments)
first_parsed_file (-p1) : path to parse-hypotheses (the first of two experiments)
second_gold_file (-g2) : path to gold-parses file (the second of two experiments)
second_parsed_file (-p2) : path to parse-hypotheses (the second of two experiments)
first_result_file (-o1) : path to result file (the first of two experiments), default file name is eval1
second_result_file (-o2) : path to result file (the second of two experiments), default file name is eval2
-----------------------------------------------------------------------------
file_format (-format) : the default is labeled ptb-like bracketed format (ignoring anything after the dash)
-format conll the conll-x format
-format bracketed labeled ptb-like bracketed format (empty elements not allowed, ignoring anything after the dash)
-----------------------------------------------------------------------------
labeling_flag (-unlabeled ) : use the unlabeled measure instead of the default labeled one
avg_format_flag (-micro ) : use the micro average instead of the default macro average one
-----------------------------------------------------------------------------
normalization_factor (-norm) :
StatSignTest
Usage:
java -jar statsigntest.jar -h for more help and options
help (-h) : Show options
version (-v) : Show version
-----------------------------------------------------------------------------
ted_file_1 (-i) : Path to the first output-file from tedeval
ted_file_2 (-i) : Path to the second output-file from tedeval
output_file (-o) : path to result file
iterations (-n) : number of iterations (default: 10 000)
-----------------------------------------------------------------------------