一、面向低资源和增量类型的命名实体识别挑战赛简介
使用无所不能的PaddleNLP写个比赛基线,第一次提交,分数虽然比较低,但是还凑合,主要是给的初赛数据集覆盖范围小,太小了。
竞赛地址:
1.数据简介
本赛题采用的数据聚焦装备领域,主要从以下三个方面的来源收集整理得到,具有一定的权威性和领域价值:
- 开源资讯:对国内外主流新闻网站、百度百科、维基百科、武器大全等开源资讯网站进行数据收集,优先收集中文,并将外文数据进行翻译后获得情报数据;
- 智库报告:从智库网站中获取含有装备情报信息的论文以及报告;
- 内部成果:通过国内军工企业、研究院所、国内综合图书馆、数字图书馆、军工院所图书馆等内部网站获取成果相关的文件进行分析和整理。
本赛题从上述来源收集到充足原始无标注数据后,先结合人工排查和关键字匹配等自动化方法过滤偏离主题、不真实和有偏见的数据;随后清洗无效和非法字符并过滤篇幅较长以及不含领域实体的文本;其次采用参考权威装备标准与论著制定的标签体系对文本进行标注,并采用相关领域以往研究成果中的模型对数据进行预打标;最终统计筛选出类型分布符合任务需求的样本作为原始数据集。
2.数据说明
• 初赛数据说明
该赛题数据集共计约6000条样本,包含以下9种实体类型:飞行器, 单兵武器, 炸弹, 装甲车辆, 火炮, 导弹, 舰船舰艇, 太空装备, 其他武器装备。参考低资源学习领域的任务设置,为每种类型从原始数据集中采样50个左右样本案例,形成共97条标注样本的训练集(每一条样本可能包含多个实体和实体类型),其余样本均用于测试集。所有数据文件编码均为UTF-8。
文件类型 | 文件名 | 文件说明 |
---|---|---|
训练集 | ner_train.json | 97条已标注样本,每个样本对应内容为:样本id(sample_id),原始文本(text)和标注实体列表(annotations),列表中每个元素对应一个实体,包括类型(type)、文本(text)、跨度起始位置(start)和结束位置(end) |
测试集 | ner_test.json | 5920条未标注样本,每个样本对应内容为:样本id(sample_id)和原始文本(text) |
二、数据处理
1.数据查看
!ls data/data218296/
ner_test.json ner_train.json
- 查看可知,需要提取9种实体类型:飞行器, 单兵武器, 炸弹, 装甲车辆, 火炮, 导弹, 舰船舰艇, 太空装备, 其他武器装备
- 目前训练集97条标注样本
2.数据集格式转换&& 数据集划分
主要是:
- 转换格式
一般使用docano进行数据标注,完毕进行格式转换。这里我直接处理文件格式为我所需要的二个是。 - 分割训练集和测试机
按照 8:2比例进行数据切分
%cd ~
import json
import csv
from pprint import pprint
import random
# 读取 JSON 文件
with open('data/data218296/ner_train.json', 'r', encoding='utf-8') as f:
data = json.load(f)
print(len(data))
data_len=len(data)
random.shuffle(data)
train_data=data[:int(data_len*0.75)]
dev_data=data[int(data_len*0.75):]
/home/aistudio
97
%cd ~
import json
import csv
from pprint import pprint
import random
# schema
key_words = "飞行器, 单兵武器, 炸弹, 装甲车辆, 火炮, 导弹, 舰船舰艇, 太空装备, 其他武器装备".split(", ")
# 数据集格式转换
# 并根据8:2比例分割为train和dev
def convert(source_data, key_word):
convert_target = []
for item in source_data:
# 单条记录
result_list = []
# 标注格式化
for item2 in item["annotations"]:
result_temp = dict()
if item2['type'] == key_word:
# 构造结果列表
result_temp['text'] = item2['text']
result_temp['start'] = item2['start']
result_temp['end'] = item2['end']
result_list.append(result_temp)
# 构造单条数据
temp = dict()
temp['content'] = item['text']
temp['result_list'] = result_list
temp['prompt'] = key_word
# 加入列表
convert_target.append(temp)
return convert_target
def convert_main(data,key_words):
result=[]
for key_word in key_words:
temp_list = convert(data, key_word)
result=result+temp_list
random.shuffle(result)
return result
# 转换后总列表
train_data_convert = convert_main(train_data,key_words)
dev_data_convert = convert_main(dev_data,key_words)
# 将JSON数据转换为CSV格式
with open('train.txt', 'w', encoding="utf-8") as f:
for item in train_data_convert:
f.write(json.dumps(item, ensure_ascii=False) + 'n')
with open('dev.txt', 'w', encoding="utf-8") as f:
for item in dev_data_convert:
f.write(json.dumps(item, ensure_ascii=False) + 'n')
/home/aistudio
三、训练训练
1.环境设置
主要是下载并安装PaddleNLP
# git 下载PaddleNLP
!git clone https://gitee.com/paddlepaddle/PaddleNLP.git --depth=1
正克隆到 'PaddleNLP'...
remote: Enumerating objects: 5825, done.[K
remote: Counting objects: 100% (5825/5825), done.[K
remote: Compressing objects: 100% (4099/4099), done.[K
remote: Total 5825 (delta 2254), reused 3581 (delta 1437), pack-reused 0[K
接收对象中: 100% (5825/5825), 22.98 MiB | 1.19 MiB/s, 完成.
处理 delta 中: 100% (2254/2254), 完成.
检查连接... 完成。
# 安装升级PaddleNLP
%cd ~/PaddleNLP
!pip install -U -e ./
from IPython.display import clear_output
clear_output() # 清理很长的内容
2.模型微调
推荐使用 Trainer API 对模型进行微调。只需输入模型、数据集等就可以使用 Trainer API 高效快速地进行预训练、微调和模型压缩等任务,可以一键启动多卡训练、混合精度训练、梯度累积、断点重启、日志显示等功能,Trainer API 还针对训练过程的通用训练配置做了封装,比如:优化器、学习率调度等。
可配置参数说明:
model_name_or_path
:必须,进行 few shot 训练使用的预训练模型。可选择的有 “uie-base”、 “uie-medium”, “uie-mini”, “uie-micro”, “uie-nano”, “uie-m-base”, “uie-m-large”。multilingual
:是否是跨语言模型,用 “uie-m-base”, “uie-m-large” 等模型进微调得到的模型也是多语言模型,需要设置为 True;默认为 False。output_dir
:必须,模型训练或压缩后保存的模型目录;默认为None
。device
: 训练设备,可选择 ‘cpu’、’gpu’ 、’npu’其中的一种;默认为 GPU 训练。per_device_train_batch_size
:训练集训练过程批处理大小,请结合显存情况进行调整,若出现显存不足,请适当调低这一参数;默认为 32。per_device_eval_batch_size
:开发集评测过程批处理大小,请结合显存情况进行调整,若出现显存不足,请适当调低这一参数;默认为 32。learning_rate
:训练最大学习率,UIE 推荐设置为 1e-5;默认值为3e-5。num_train_epochs
: 训练轮次,使用早停法时可以选择 100;默认为10。logging_steps
: 训练过程中日志打印的间隔 steps 数,默认100。save_steps
: 训练过程中保存模型 checkpoint 的间隔 steps 数,默认100。seed
:全局随机种子,默认为 42。weight_decay
:除了所有 bias 和 LayerNorm 权重之外,应用于所有层的权重衰减数值。可选;默认为 0.0;do_train
:是否进行微调训练,设置该参数表示进行微调训练,默认不设置。do_eval
:是否进行评估,设置该参数表示进行评估。
该示例代码中由于设置了参数 --do_eval
,因此在训练完会自动进行评估。
%cd ~/PaddleNLP/model_zoo/uie/
# !python -u -m paddle.distributed.launch --gpus "0,1,2,3" finetune.py
!python finetune.py
--device gpu
--logging_steps 10
--save_steps 50
--eval_steps 50
--seed 1000
--model_name_or_path uie-medium
--output_dir ./checkpoint/model_best
--train_path ~/train.txt
--dev_path ~/dev.txt
--max_seq_length 512
--per_device_eval_batch_size 16
--per_device_train_batch_size 16
--num_train_epochs 32
--learning_rate 1e-5
--label_names "start_positions" "end_positions"
--do_train
--do_eval
--do_export
--export_model_dir ./checkpoint/model_best
--overwrite_output_dir
--disable_tqdm True
--metric_for_best_model eval_f1
--load_best_model_at_end True
--save_total_limit 1
/home/aistudio/PaddleNLP/model_zoo/uie
[33m[2023-06-19 16:20:21,620] [ WARNING][0m - evaluation_strategy reset to IntervalStrategy.STEPS for do_eval is True. you can also set evaluation_strategy='epoch'.[0m[32m[2023-06-19 16:20:21,620] [ INFO][0m - The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).[0m[32m[2023-06-19 16:20:21,621] [ INFO][0m - ============================================================[0m[32m[2023-06-19 16:20:21,621] [ INFO][0m - Model Configuration Arguments [0m[32m[2023-06-19 16:20:21,621] [ INFO][0m - paddle commit id :3fa7a736e32508e797616b6344d97814c37d3ff8[0m[32m[2023-06-19 16:20:21,621] [ INFO][0m - export_model_dir :./checkpoint/model_best[0m[32m[2023-06-19 16:20:21,621] [ INFO][0m - model_name_or_path :uie-medium[0m[32m[2023-06-19 16:20:21,621] [ INFO][0m - multilingual :False[0m[32m[2023-06-19 16:20:21,621] [ INFO][0m - [0m[32m[2023-06-19 16:20:21,621] [ INFO][0m - ============================================================[0m[32m[2023-06-19 16:20:21,621] [ INFO][0m - Data Configuration Arguments [0m[32m[2023-06-19 16:20:21,621] [ INFO][0m - paddle commit id :3fa7a736e32508e797616b6344d97814c37d3ff8[0m[32m[2023-06-19 16:20:21,621] [ INFO][0m - dev_path :/home/aistudio/dev.txt[0m[32m[2023-06-19 16:20:21,621] [ INFO][0m - dynamic_max_length :None[0m[32m[2023-06-19 16:20:21,621] [ INFO][0m - max_seq_length :512[0m[32m[2023-06-19 16:20:21,622] [ INFO][0m - train_path :/home/aistudio/train.txt[0m[32m[2023-06-19 16:20:21,622] [ INFO][0m - [0m[33m[2023-06-19 16:20:21,622] [ WARNING][0m - Process rank: -1, device: gpu, world_size: 1, distributed training: False, 16-bits training: False[0m[32m[2023-06-19 16:20:21,622] [ INFO][0m - We are using <class 'paddlenlp.transformers.ernie.tokenizer.ErnieTokenizer'> to load 'uie-medium'.[0m[32m[2023-06-19 16:20:21,622] [ INFO][0m - Downloading https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_medium_zh_vocab.txt and saved to /home/aistudio/.paddlenlp/models/uie-medium[0m[32m[2023-06-19 16:20:21,752] [ INFO][0m - Downloading ernie_3.0_medium_zh_vocab.txt from https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_medium_zh_vocab.txt[0m100%|█████████████████████████████████████████| 182k/182k [00:00<00:00, 889kB/s]
[32m[2023-06-19 16:20:22,092] [ INFO][0m - tokenizer config file saved in /home/aistudio/.paddlenlp/models/uie-medium/tokenizer_config.json[0m[32m[2023-06-19 16:20:22,092] [ INFO][0m - Special tokens file saved in /home/aistudio/.paddlenlp/models/uie-medium/special_tokens_map.json[0m[32m[2023-06-19 16:20:22,093] [ INFO][0m - Model config ErnieConfig { "attention_probs_dropout_prob": 0.1, "enable_recompute": false, "fuse": false, "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "initializer_range": 0.02, "intermediate_size": 3072, "layer_norm_eps": 1e-12, "max_position_embeddings": 2048, "model_type": "ernie", "num_attention_heads": 12, "num_hidden_layers": 6, "pad_token_id": 0, "paddlenlp_version": null, "pool_act": "tanh", "task_id": 0, "task_type_vocab_size": 16, "type_vocab_size": 4, "use_task_id": true, "vocab_size": 40000}[0m[32m[2023-06-19 16:20:22,094] [ INFO][0m - Configuration saved in /home/aistudio/.paddlenlp/models/uie-medium/config.json[0m[32m[2023-06-19 16:20:22,095] [ INFO][0m - Downloading uie_medium.pdparams from https://bj.bcebos.com/paddlenlp/models/transformers/uie/uie_medium.pdparams[0m100%|████████████████████████████████████████| 288M/288M [02:40<00:00, 1.88MB/s]
W0619 16:23:05.005470 1653 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
W0619 16:23:05.010092 1653 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
[32m[2023-06-19 16:23:05,669] [ INFO][0m - All model checkpoint weights were used when initializing UIE.[0m[32m[2023-06-19 16:23:05,669] [ INFO][0m - All the weights of UIE were initialized from the model checkpoint at uie-medium.If your task is similar to the task the model of the checkpoint was trained on, you can already use UIE for predictions without further training.[0m
四、模型评估
1.评估模型
可配置参数说明:
model_path
: 进行评估的模型文件夹路径,路径下需包含模型权重文件model_state.pdparams
及配置文件model_config.json
。test_path
: 进行评估的测试集文件。batch_size
: 批处理大小,请结合机器情况进行调整,默认为16。max_seq_len
: 文本最大切分长度,输入超过最大长度时会对输入文本进行自动切分,默认为512。debug
: 是否开启debug模式对每个正例类别分别进行评估,该模式仅用于模型调试,默认关闭。multilingual
: 是否是跨语言模型,默认关闭。schema_lang
: 选择schema的语言,可选有ch
和en
。默认为ch
,英文数据集请选择en
。
通过运行以下命令进行模型评估:
%cd ~/PaddleNLP/model_zoo/uie/
!python evaluate.py
--model_path ./checkpoint0.67789894/model_best
--test_path ~/dev.txt
--batch_size 16
--max_seq_len 512
# --model_path ./checkpoint/model_best
/home/aistudio/PaddleNLP/model_zoo/uie
[32m[2023-06-03 19:58:07,106] [ INFO][0m - We are using <class 'paddlenlp.transformers.ernie.tokenizer.ErnieTokenizer'> to load './checkpoint0.67789894/model_best'.[0m
[32m[2023-06-03 19:58:07,132] [ INFO][0m - loading configuration file ./checkpoint0.67789894/model_best/config.json[0m
[32m[2023-06-03 19:58:07,133] [ INFO][0m - Model config ErnieConfig {
"architectures": [
"UIE"
],
"attention_probs_dropout_prob": 0.1,
"dtype": "float32",
"enable_recompute": false,
"fuse": false,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"layer_norm_eps": 1e-12,
"max_position_embeddings": 2048,
"model_type": "ernie",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"pad_token_id": 0,
"paddlenlp_version": null,
"pool_act": "tanh",
"task_id": 0,
"task_type_vocab_size": 3,
"type_vocab_size": 4,
"use_task_id": true,
"vocab_size": 40000
}
[0m
W0603 19:58:09.367168 9655 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
W0603 19:58:09.370771 9655 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
[32m[2023-06-03 19:58:10,168] [ INFO][0m - All model checkpoint weights were used when initializing UIE.
[0m
[32m[2023-06-03 19:58:10,169] [ INFO][0m - All the weights of UIE were initialized from the model checkpoint at ./checkpoint0.67789894/model_best.
If your task is similar to the task the model of the checkpoint was trained on, you can already use UIE for predictions without further training.[0m
[32m[2023-06-03 19:58:15,447] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 19:58:15,447] [ INFO][0m - Class Name: all_classes[0m
[32m[2023-06-03 19:58:15,447] [ INFO][0m - Evaluation Precision: 0.92920 | Recall: 0.86066 | F1: 0.89362[0m
[0m
%cd ~/PaddleNLP/model_zoo/uie/
!python evaluate.py
--model_path ./checkpoint/model_best
--test_path ~/dev.txt
--batch_size 16
--max_seq_len 512
/home/aistudio/PaddleNLP/model_zoo/uie
[32m[2023-06-03 19:58:59,933] [ INFO][0m - We are using <class 'paddlenlp.transformers.ernie.tokenizer.ErnieTokenizer'> to load './checkpoint/model_best'.[0m
[32m[2023-06-03 19:58:59,958] [ INFO][0m - loading configuration file ./checkpoint/model_best/config.json[0m
[32m[2023-06-03 19:58:59,960] [ INFO][0m - Model config ErnieConfig {
"architectures": [
"UIE"
],
"attention_probs_dropout_prob": 0.1,
"dtype": "float32",
"enable_recompute": false,
"fuse": false,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"layer_norm_eps": 1e-12,
"max_position_embeddings": 2048,
"model_type": "ernie",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"pad_token_id": 0,
"paddlenlp_version": null,
"pool_act": "tanh",
"task_id": 0,
"task_type_vocab_size": 3,
"type_vocab_size": 4,
"use_task_id": true,
"vocab_size": 40000
}
[0m
W0603 19:59:02.195425 9944 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
W0603 19:59:02.199051 9944 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
[32m[2023-06-03 19:59:02,995] [ INFO][0m - All model checkpoint weights were used when initializing UIE.
[0m
[32m[2023-06-03 19:59:02,996] [ INFO][0m - All the weights of UIE were initialized from the model checkpoint at ./checkpoint/model_best.
If your task is similar to the task the model of the checkpoint was trained on, you can already use UIE for predictions without further training.[0m
[32m[2023-06-03 19:59:08,312] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 19:59:08,312] [ INFO][0m - Class Name: all_classes[0m
[32m[2023-06-03 19:59:08,312] [ INFO][0m - Evaluation Precision: 0.89922 | Recall: 0.95082 | F1: 0.92430[0m
[0m
2.debug模式评估模型
可开启debug
模式对每个正例类别分别进行评估,该模式仅用于模型调试:
%cd ~/PaddleNLP/model_zoo/uie/
!python evaluate.py
--model_path ./checkpoint0.67789894/model_best
--test_path ~/dev.txt
--debug
# --model_path ./checkpoint/model_best
/home/aistudio/PaddleNLP/model_zoo/uie
[32m[2023-06-03 19:59:51,590] [ INFO][0m - We are using <class 'paddlenlp.transformers.ernie.tokenizer.ErnieTokenizer'> to load './checkpoint0.67789894/model_best'.[0m
[32m[2023-06-03 19:59:51,617] [ INFO][0m - loading configuration file ./checkpoint0.67789894/model_best/config.json[0m
[32m[2023-06-03 19:59:51,618] [ INFO][0m - Model config ErnieConfig {
"architectures": [
"UIE"
],
"attention_probs_dropout_prob": 0.1,
"dtype": "float32",
"enable_recompute": false,
"fuse": false,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"layer_norm_eps": 1e-12,
"max_position_embeddings": 2048,
"model_type": "ernie",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"pad_token_id": 0,
"paddlenlp_version": null,
"pool_act": "tanh",
"task_id": 0,
"task_type_vocab_size": 3,
"type_vocab_size": 4,
"use_task_id": true,
"vocab_size": 40000
}
[0m
W0603 19:59:53.878067 10161 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
W0603 19:59:53.881646 10161 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
[32m[2023-06-03 19:59:54,673] [ INFO][0m - All model checkpoint weights were used when initializing UIE.
[0m
[32m[2023-06-03 19:59:54,673] [ INFO][0m - All the weights of UIE were initialized from the model checkpoint at ./checkpoint0.67789894/model_best.
If your task is similar to the task the model of the checkpoint was trained on, you can already use UIE for predictions without further training.[0m
[32m[2023-06-03 19:59:55,584] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 19:59:55,585] [ INFO][0m - Class Name: 炸弹[0m
[32m[2023-06-03 19:59:55,585] [ INFO][0m - Evaluation Precision: 0.86667 | Recall: 0.86667 | F1: 0.86667[0m
[32m[2023-06-03 19:59:55,660] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 19:59:55,660] [ INFO][0m - Class Name: 装甲车辆[0m
[32m[2023-06-03 19:59:55,660] [ INFO][0m - Evaluation Precision: 1.00000 | Recall: 1.00000 | F1: 1.00000[0m
[32m[2023-06-03 19:59:55,809] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 19:59:55,809] [ INFO][0m - Class Name: 火炮[0m
[32m[2023-06-03 19:59:55,809] [ INFO][0m - Evaluation Precision: 0.84615 | Recall: 0.73333 | F1: 0.78571[0m
[32m[2023-06-03 19:59:55,903] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 19:59:55,904] [ INFO][0m - Class Name: 舰船舰艇[0m
[32m[2023-06-03 19:59:55,904] [ INFO][0m - Evaluation Precision: 1.00000 | Recall: 1.00000 | F1: 1.00000[0m
[32m[2023-06-03 19:59:55,966] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 19:59:55,966] [ INFO][0m - Class Name: 飞行器[0m
[32m[2023-06-03 19:59:55,966] [ INFO][0m - Evaluation Precision: 1.00000 | Recall: 1.00000 | F1: 1.00000[0m
[32m[2023-06-03 19:59:56,072] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 19:59:56,072] [ INFO][0m - Class Name: 单兵武器[0m
[32m[2023-06-03 19:59:56,072] [ INFO][0m - Evaluation Precision: 0.90000 | Recall: 0.75000 | F1: 0.81818[0m
[32m[2023-06-03 19:59:56,169] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 19:59:56,170] [ INFO][0m - Class Name: 太空装备[0m
[32m[2023-06-03 19:59:56,170] [ INFO][0m - Evaluation Precision: 0.95833 | Recall: 0.95833 | F1: 0.95833[0m
[32m[2023-06-03 19:59:56,227] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 19:59:56,228] [ INFO][0m - Class Name: 导弹[0m
[32m[2023-06-03 19:59:56,228] [ INFO][0m - Evaluation Precision: 1.00000 | Recall: 1.00000 | F1: 1.00000[0m
[32m[2023-06-03 19:59:56,308] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 19:59:56,308] [ INFO][0m - Class Name: 其他武器装备[0m
[32m[2023-06-03 19:59:56,308] [ INFO][0m - Evaluation Precision: 1.00000 | Recall: 0.50000 | F1: 0.66667[0m
[0m
%cd ~/PaddleNLP/model_zoo/uie/
!python evaluate.py
--model_path ./checkpoint/model_best
--test_path ~/dev.txt
--debug
/home/aistudio/PaddleNLP/model_zoo/uie
[32m[2023-06-03 20:00:12,238] [ INFO][0m - We are using <class 'paddlenlp.transformers.ernie.tokenizer.ErnieTokenizer'> to load './checkpoint/model_best'.[0m
[32m[2023-06-03 20:00:12,263] [ INFO][0m - loading configuration file ./checkpoint/model_best/config.json[0m
[32m[2023-06-03 20:00:12,265] [ INFO][0m - Model config ErnieConfig {
"architectures": [
"UIE"
],
"attention_probs_dropout_prob": 0.1,
"dtype": "float32",
"enable_recompute": false,
"fuse": false,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"layer_norm_eps": 1e-12,
"max_position_embeddings": 2048,
"model_type": "ernie",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"pad_token_id": 0,
"paddlenlp_version": null,
"pool_act": "tanh",
"task_id": 0,
"task_type_vocab_size": 3,
"type_vocab_size": 4,
"use_task_id": true,
"vocab_size": 40000
}
[0m
W0603 20:00:14.519615 10292 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
W0603 20:00:14.523311 10292 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
[32m[2023-06-03 20:00:15,331] [ INFO][0m - All model checkpoint weights were used when initializing UIE.
[0m
[32m[2023-06-03 20:00:15,332] [ INFO][0m - All the weights of UIE were initialized from the model checkpoint at ./checkpoint/model_best.
If your task is similar to the task the model of the checkpoint was trained on, you can already use UIE for predictions without further training.[0m
[32m[2023-06-03 20:00:16,208] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 20:00:16,208] [ INFO][0m - Class Name: 炸弹[0m
[32m[2023-06-03 20:00:16,208] [ INFO][0m - Evaluation Precision: 0.93750 | Recall: 1.00000 | F1: 0.96774[0m
[32m[2023-06-03 20:00:16,281] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 20:00:16,281] [ INFO][0m - Class Name: 装甲车辆[0m
[32m[2023-06-03 20:00:16,281] [ INFO][0m - Evaluation Precision: 1.00000 | Recall: 1.00000 | F1: 1.00000[0m
[32m[2023-06-03 20:00:16,425] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 20:00:16,426] [ INFO][0m - Class Name: 火炮[0m
[32m[2023-06-03 20:00:16,426] [ INFO][0m - Evaluation Precision: 0.92308 | Recall: 0.80000 | F1: 0.85714[0m
[32m[2023-06-03 20:00:16,519] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 20:00:16,519] [ INFO][0m - Class Name: 舰船舰艇[0m
[32m[2023-06-03 20:00:16,519] [ INFO][0m - Evaluation Precision: 1.00000 | Recall: 1.00000 | F1: 1.00000[0m
[32m[2023-06-03 20:00:16,579] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 20:00:16,579] [ INFO][0m - Class Name: 飞行器[0m
[32m[2023-06-03 20:00:16,579] [ INFO][0m - Evaluation Precision: 1.00000 | Recall: 1.00000 | F1: 1.00000[0m
[32m[2023-06-03 20:00:16,681] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 20:00:16,681] [ INFO][0m - Class Name: 单兵武器[0m
[32m[2023-06-03 20:00:16,681] [ INFO][0m - Evaluation Precision: 0.91667 | Recall: 0.91667 | F1: 0.91667[0m
[32m[2023-06-03 20:00:16,775] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 20:00:16,775] [ INFO][0m - Class Name: 太空装备[0m
[32m[2023-06-03 20:00:16,775] [ INFO][0m - Evaluation Precision: 0.88889 | Recall: 1.00000 | F1: 0.94118[0m
[32m[2023-06-03 20:00:16,830] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 20:00:16,830] [ INFO][0m - Class Name: 导弹[0m
[32m[2023-06-03 20:00:16,830] [ INFO][0m - Evaluation Precision: 0.75000 | Recall: 1.00000 | F1: 0.85714[0m
[32m[2023-06-03 20:00:16,907] [ INFO][0m - -----------------------------[0m
[32m[2023-06-03 20:00:16,908] [ INFO][0m - Class Name: 其他武器装备[0m
[32m[2023-06-03 20:00:16,908] [ INFO][0m - Evaluation Precision: 0.80000 | Recall: 0.85714 | F1: 0.82759[0m
[0m
五、预测
1.读取test数据集
%cd ~/PaddleNLP/model_zoo/uie/
import json
import csv
from pprint import pprint
# 读取 JSON 文件
with open('https://b2.7b2.com/home/aistudio/data/data218296/ner_test.json', 'r', encoding='utf-8') as f:
test_data = json.load(f)
print(f"数据集长度:{len(test_data)}")
print("查看数据样例:")
pprint(test_data[0])
/home/aistudio/PaddleNLP/model_zoo/uie
数据集长度:5920
查看数据样例:
{'sample_id': 0,
'text': '第五艘西班牙海军F-100级护卫舰即将装备集成通信控制系统。该系统由葡萄牙EID公司生产。该系统已经用于巴西海军的圣保罗航母,荷兰海军的四艘荷兰级海上巡逻舰和四艘西班牙海军BAM近海巡逻舰。F-105护卫舰于2009年初铺设龙骨。该舰预计2010年建造完成,2012年夏交付。'}
2.设定抽取目标 && 定制化模型权重路径
from pprint import pprint
from paddlenlp import Taskflow
schema = "飞行器, 单兵武器, 炸弹, 装甲车辆, 火炮, 导弹, 舰船舰艇, 太空装备, 其他武器装备".split(", ")
# 设定抽取目标和定制化模型权重路径
# PaddleNLP/model_zoo/uie/checkpoint0.67789894
my_ie = Taskflow("information_extraction", schema=schema, task_path='./checkpoint0.67789894/model_best')
# my_ie = Taskflow("information_extraction", schema=schema, task_path='./checkpoint/model_best')
[2023-06-03 20:04:27,292] [ INFO] - loading configuration file ./checkpoint0.67789894/model_best/config.json
[2023-06-03 20:04:27,297] [ INFO] - Model config ErnieConfig {
"architectures": [
"UIE"
],
"attention_probs_dropout_prob": 0.1,
"dtype": "float32",
"enable_recompute": false,
"fuse": false,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"layer_norm_eps": 1e-12,
"max_position_embeddings": 2048,
"model_type": "ernie",
"num_attention_heads": 12,
"num_hidden_layers": 12,
"pad_token_id": 0,
"paddlenlp_version": null,
"pool_act": "tanh",
"task_id": 0,
"task_type_vocab_size": 3,
"type_vocab_size": 4,
"use_task_id": true,
"vocab_size": 40000
}
W0603 20:04:27.792821 9916 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
W0603 20:04:27.796425 9916 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
[2023-06-03 20:04:28,581] [ INFO] - All model checkpoint weights were used when initializing UIE.
[2023-06-03 20:04:28,584] [ INFO] - All the weights of UIE were initialized from the model checkpoint at ./checkpoint0.67789894/model_best.
If your task is similar to the task the model of the checkpoint was trained on, you can already use UIE for predictions without further training.
[2023-06-03 20:04:28,590] [ INFO] - Converting to the inference model cost a little time.
[2023-06-03 20:04:42,631] [ INFO] - The inference model save in the path:./checkpoint0.67789894/model_best/static/inference
[2023-06-03 20:04:44,793] [ INFO] - We are using <class 'paddlenlp.transformers.ernie.tokenizer.ErnieTokenizer'> to load './checkpoint0.67789894/model_best'.
3.预测
list_text=[]
for item in test_data:
list_text.append(item['text'])
%%time
results=my_ie(list_text)
CPU times: user 16min 20s, sys: 2min 27s, total: 18min 47s
Wall time: 18min 45s
print(len(results))
5920
print(results[0])
{'舰船舰艇': [{'text': 'F-105护卫舰', 'start': 95, 'end': 103, 'probability': 0.9999301440846722}, {'text': 'F-100级护卫舰', 'start': 8, 'end': 17, 'probability': 0.999897838723939}, {'text': 'BAM近海巡逻舰', 'start': 86, 'end': 94, 'probability': 0.9974057948851112}, {'text': '荷兰级海上巡逻舰', 'start': 70, 'end': 78, 'probability': 0.9998418145644905}, {'text': '圣保罗航母', 'start': 57, 'end': 62, 'probability': 0.9989470402669269}]}
with open('https://b2.7b2.com/home/aistudio/result_list.json','w', encoding="utf-8") as f:
json.dump(results, f, ensure_ascii=False)
results_list=[]
for i in range(len(test_data)):
for key, item in results[i].items():
for ii in range(len(item)):
temp_result=dict()
temp_result['sample_id']=test_data[i]['sample_id']
temp_result['text']=item[ii]['text']
temp_result['type']=key
temp_result['start']=item[ii]['start']
temp_result['end']=item[ii]['end']
results_list.append(temp_result)
with open('https://b2.7b2.com/home/aistudio/result.json','w', encoding="utf-8") as f:
json.dump(results_list, f,indent=4, ensure_ascii=False)
results=[]
for i in range(len(test_data)):
uie_result=my_ie(test_data[i]['text'])
# pprint(uie_result)
for key, item in uie_result[0].items():
for ii in range(len(item)):
temp_result=dict()
temp_result['sample_id']=test_data[i]['sample_id']
temp_result['text']=item[ii]['text']
temp_result['type']=key
temp_result['start']=item[ii]['start']
temp_result['end']=item[ii]['end']
results.append(temp_result)
print(len(results))
4.保存结果
with open('result.json','w', encoding="utf-8") as f:
json.dump(results, f,indent=4, ensure_ascii=False)