Skip to content

Latest commit

 

History

History
113 lines (88 loc) · 6.13 KB

METRICS.md

File metadata and controls

113 lines (88 loc) · 6.13 KB

1. Russian Drug Reaction Corpus (RuDReC)

Llama2 7B with LoRA

Drugname Drugclass Drugform DI ADR Finding overall
precision 0.793722 0.5 0.636905 0.408304 0.333333 0.166667 0.551905
recall 0.59596 0.0625 0.426295 0.27896 0.00421941 0.150685 0.305011
f1 0.680769 0.111111 0.51074 0.331461 0.00833333 0.158273 0.392891

Mistral 7B with LoRA

Base instruction text

Drugname Drugclass Drugform DI ADR Finding overall
precision 0.910959 0.988889 0.935223 0.655172 0.662857 0.380952 0.785942
recall 0.895623 0.927083 0.920319 0.628842 0.489451 0.219178 0.714597
f1 0.903226 0.956989 0.927711 0.641737 0.563107 0.278261 0.748574

Extended instruction text

Drugname Drugclass Drugform DI ADR Finding overall
precision 0.905085 0.924731 0.936255 0.686567 0.633508 0.351852 0.780715
recall 0.89899 0.895833 0.936255 0.652482 0.510549 0.260274 0.729121
f1 0.902027 0.910053 0.936255 0.669091 0.565421 0.299213 0.754037

rubert-tiny2 29.4M (encoder)

Drugname Drugclass Drugform DI ADR Finding overall
precision 0.774481 0.884211 0.926923 0.533589 0.368771 0.529412 0.642717
recall 0.884746 0.884211 0.964 0.65721 0.468354 0.123288 0.716679
f1 0.825949 0.884211 0.945098 0.588983 0.412639 0.2 0.677686

2. NEREL-BIO (Nested Named Entities)

  • Displayed only most frequency entities

Exact match

ANATOMY CHEM DATE DISO LABPROC MEDPROC NUMBER PERCENT PERSON PHYS overall
precision 0.642762 0.737569 0.631579 0.700925 0.516854 0.550909 0.712803 0.862348 0.763938 0.409692 0.628516
recall 0.736682 0.719677 0.597015 0.73176 0.383333 0.693364 0.778828 0.832031 0.86443 0.508197 0.634225
f1 0.686525 0.728513 0.613811 0.71601 0.440191 0.613982 0.744354 0.846918 0.811083 0.453659 0.631358

Partial match

ANATOMY CHEM DATE DISO LABPROC MEDPROC NUMBER PERCENT PERSON PHYS overall
precision 0.678503 0.730126 0.856693 0.725 0.467033 0.528908 0.725676 0.828467 0.752621 0.390723 0.627512
recall 0.797069 0.75705 0.846034 0.767003 0.388128 0.688982 0.817352 0.856604 0.837806 0.56865 0.65154
f1 0.733022 0.743344 0.85133 0.74541 0.42394 0.598425 0.76879 0.842301 0.792932 0.463187 0.6393

3. CoNLL-2003

Mistral 7B with LoRA

Default insturct-ner target format (exact match)

{'PER': ['Nadim Ladki'], 'ORG': [], 'LOC': [], 'MISC': []}

Base instruction text

PER ORG LOC MISC overall
precision 0.974953 0.889528 0.944994 0.791785 0.9173
recall 0.962894 0.930765 0.916667 0.796296 0.919086
f1 0.968886 0.909679 0.930615 0.794034 0.918192

Extended instruction text

PER ORG LOC MISC overall
precision 0.971535 0.906509 0.934218 0.795297 0.918924
recall 0.970934 0.922336 0.928058 0.819088 0.925106
f1 0.971234 0.914354 0.931128 0.807018 0.922005

Splitted by words target format (partial match)

split_entities=True (instruction_ner/metric.py)
{'PER': ['Nadim', 'Ladki'], 'ORG': [], 'LOC': [], 'MISC': []}
PER ORG LOC MISC overall
precision 0.983333 0.899354 0.940926 0.782744 0.923367
recall 0.978723 0.948718 0.918442 0.820261 0.937253
f1 0.981023 0.923377 0.929548 0.801064 0.930258

4. MultiCoNER II (2023)

  • English (test)
  • Shuffled with seed 42
  • First 10k test samples (due to inferece time)

The fine to coarse level mapping of the tags (link)

Mistral 7B with LoRA

Coarse tagset

LOC CW GRP PER PROD MED overall
precision 0.691605 0.748318 0.792315 0.921085 0.647929 0.622877 0.793725
recall 0.764024 0.763423 0.735144 0.928661 0.568339 0.620767 0.796922
f1 0.726013 0.755795 0.76266 0.924858 0.60553 0.62182 0.79532

Fine tagset

overall
precision 0.624569
recall 0.621516
f1 0.623039