Taja Kuzman
commited on
Update README.md
Browse files
README.md
CHANGED
|
@@ -31,6 +31,20 @@ and 0.720 in macro-F1 on a Croatian test set (440 instances from ParlaMint-HR 4.
|
|
| 31 |
An additional evaluation on smaller samples from Czech ParlaMint-CZ, Bulgarian ParlaMint-BG and Ukrainian ParlaMint-UA datasets shows
|
| 32 |
that the model achieves macro-F1 scores of 0.736, 0.75 and 0.805 on these three test datasets, respectively.
|
| 33 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
|
| 35 |
## Use
|
| 36 |
|
|
|
|
| 31 |
An additional evaluation on smaller samples from Czech ParlaMint-CZ, Bulgarian ParlaMint-BG and Ukrainian ParlaMint-UA datasets shows
|
| 32 |
that the model achieves macro-F1 scores of 0.736, 0.75 and 0.805 on these three test datasets, respectively.
|
| 33 |
|
| 34 |
+
For end use scenarios, we recommend filtering out predictions based on the model's prediction confidence.
|
| 35 |
+
|
| 36 |
+
When the model was applied to the ParlaMint datasets, we annotated instances that were predicted with confidence below 0.60 as "Mix".
|
| 37 |
+
|
| 38 |
+
With this approach, we annotate as Mix:
|
| 39 |
+
- 8.6% of instances in the English test set
|
| 40 |
+
- 11.4% of instances in the Croatian test set
|
| 41 |
+
|
| 42 |
+
Performance of the model on the remaining instances (all instances not annotated as "Mix"):
|
| 43 |
+
|
| 44 |
+
| | micro-F1 | macro-F1 | accuracy |
|
| 45 |
+
|:---|-----------:|-----------:|-----------:|
|
| 46 |
+
| EN | 0.838 | 0.838 | 0.838 |
|
| 47 |
+
| HR | 0.724 | 0.726 | 0.724 |
|
| 48 |
|
| 49 |
## Use
|
| 50 |
|