人工智能大模型预训练微调LoRARLHF强化学习分布式训练【免费下载链接】PaddleNLPEasy-to-use and powerful LLM and SLM library with awesome model zoo.项目地址https://gitcode.com/gh_mirrors/pa/PaddleNLP点击查看免费下载BigBird 是一类基于块稀疏注意力Block Sparse Attention的长文本 Transformer 模型能够在序列长度数千乃至上万时把注意力复杂度从 $O(n^2)$ 降低到 $O(n)$非常适合文档级理解、抽取式问答、长文档分类等场景。本文以 PaddleNLP 仓库中的 modeling 模块 为主体对应的 API 文档入口为 paddlenlp.transformers.bigbird.modeling.rst通过 Sphinxautomodule指令自动收集该模块的成员与文档字符串系统讲解其架构设计、配置体系、核心模型与全部下游任务头并结合源码给出可直接运行的示例帮助你快速掌握在 PaddleNLP 中使用 BigBird 的完整路径。读完本文你将掌握BigBird 三种注意力机制全局块、窗口块、随机块在源码中的落地方式BigBirdConfig每一个配置项的语义与默认值BigBirdModel的前向数据流与掩码处理逻辑以及面向序列分类、问答、Token 分类、多项选择、MLM、因果 LM 与预训练等任务的九类模型头的用法。一、BigBird 核心机制三种块稀疏注意力BigBird 论文Zaheer 等2020的核心思想是在保持近似全量注意力表达能力的同时只让每个 Token 关注三类位置从而把注意力矩阵从稠密的 $n \times n$ 变成稀疏结构全局块Global Blocks序列中的少量块如开头几个块被所有 Token 关注同时它们也关注所有 Token承担全局信息收集与分发的角色窗口块Window Blocks每个 Token 只关注相邻窗口内的块捕捉局部上下文随机块Random Blocks每个 Token 再随机关注少量块保证信息在长距离上的流通。在 PaddleNLP 中这三种机制通过三个配置项直接暴露给用户window_size一个窗口内包含的块数、num_global_blocks每条序列的全局块数量、num_rand_blocks每行随机块的个数它们共同作用于 MultiHeadAttention 的attention_typebigbird分支。每个块的大小由block_size控制序列按块粒度组织而非按 Token 粒度组织这正是复杂度从 $O(n^2)$ 降为 $O(n)$ 的关键。二、模块总览文件结构与导出符号BigBird 相关实现位于 paddlenlp/transformers/bigbird/ 目录共四个文件文件职责configuration.pyBigBirdConfig配置类、预训练初始化配置与权重资源映射modeling.py全部模型结构主干编码器、池化器、嵌入层与各类任务头1706 行tokenizer.py分词器支持直接返回paddle.Tensor输入init.py模块导出modeling.py 显式导出了以下 11 个符号其中 9 个是可直接实例化的模型/组件类BigBirdModel主干模型输出原始 hidden-statesBigBirdPretrainedModel预训练模型抽象基类负责权重下载与加载BigBirdForPretraining预训练任务模型MLM NSPBigBirdPretrainingCriterion预训练损失函数BigBirdPretrainingHeads预训练头掩码语言建模 句间关系BigBirdForSequenceClassification序列分类/回归如 GLUEBigBirdForQuestionAnswering抽取式问答如 SQuADBigBirdForTokenClassificationToken 级分类如 NERBigBirdForMultipleChoice多项选择如 SWAG/RocStoriesBigBirdForMaskedLM掩码语言建模BigBirdForCausalLM因果语言建模。模块还定义了BIG_BIRD_PRETRAINED_MODEL_ARCHIVE_LISTmodeling.py L59-L63列出兼容的官方预训练权重google/bigbird-roberta-base、google/bigbird-roberta-large、google/bigbird-base-trivia-itc。三、配置体系BigBirdConfig 全参数解析所有模型类均以 BigBirdConfig 为唯一构造参数。它继承自PretrainedConfig并定义了model_type big_bird。为了兼容不同命名习惯配置类内置了attribute_mapconfiguration.py L120-L126将 PaddleNLP 风格字段映射到通用命名num_classes - num_labels、nhead - num_attention_heads、num_layers - num_hidden_layers、dim_feedforward - intermediate_size、d_model - hidden_size。3.1 完整参数表默认值以源码为准参数默认值说明vocab_size50358词表大小决定 token embedding 矩阵行数hidden_size768编码器与池化层维度num_hidden_layers12Transformer 编码器层数num_attention_heads12每层注意力头数intermediate_size3072前馈网络中间维度hidden_actgelu_new编码器激活函数gelu/relu/selu/gelu_newhidden_dropout_prob0.1全连接层与嵌入层 dropoutattention_probs_dropout_prob0.1注意力概率 dropoutmax_position_embeddings4096支持的最大序列长度BigBird 长文本能力的直接体现type_vocab_size2token_type_ids 词表大小initializer_range0.02权重初始化标准差layer_norm_eps1e-12LayerNorm epsilonuse_cacheTrue是否返回缓存仅 decoder 场景相关pad_token_id/bos_token_id/eos_token_id/sep_token_id0 / 1 / 2 / 66特殊 Token idattention_typebigbirdbigbird块稀疏注意力或original_full原始 $O(n^2)$ 注意力use_biasTrueQ/K/V 是否使用偏置rescale_embeddingsFalse是否以hidden_size ** 0.5缩放嵌入block_size1每个块的大小仅在attention_type bigbird时有效num_random_blocks3每个 query 关注的随机块数量dropout0.1分类头 dropoutpadding_idx0嵌入层 padding 索引attn_dropout0.1多头注意力 dropoutact_dropoutNone前馈激活 dropout为 None 时回退到dropoutnormalize_beforeFalse是否采用 pre-LayerNorm 结构weight_attr/bias_attrNone线性层参数初始化属性window_size3一个窗口内的块数num_global_blocks2每条序列的全局块数num_rand_blocks2每行随机块数seedNone生成随机块 id 的随机种子activationreluMLM 预测头激活函数注意与hidden_act是两个独立字段embedding_weightsNone预训练嵌入权重用于 MLM 头权重共享/复用3.2 预训练初始化配置与权重映射configuration.py L23-L45 中的BIGBIRD_PRETRAINED_INIT_CONFIGURATION给出了bigbird-base-uncased的完整结构参数12 层、50358 词表、12 头、hidden 768、max_position_embeddings4096、block_size16、window_size3、num_global_blocks2、num_rand_blocks3、initializer_range0.02。注意这里与BigBirdConfig.__init__的默认值如block_size1、num_rand_blocks2并不完全相同——实际加载预训练权重时from_pretrained会以预训练配置覆盖默认值。BIGBIRD_PRETRAINED_RESOURCE_FILES_MAPconfiguration.py L47-L51将bigbird-base-uncased的模型权重指向 PaddleNLP 官方静态资源.pdparams文件这是from_pretrained自动下载的依据。目前资源映射中已登记的预训练权重为bigbird-base-uncased一个条目。四、主干模型 BigBirdModel结构与数据流BigBirdModel 用register_base_model装饰被注册为 BigBird 系列的基础模型是全部任务头的公共底座。其内部由四个子模块组成L366-L378self.embeddings BigBirdEmbeddings(config) encoder_layer TransformerEncoderLayer(config) self.encoder TransformerEncoder(encoder_layer, config.num_layers) self.pooler BigBirdPooler(config.hidden_size)4.1 嵌入层 BigBirdEmbeddingsBigBirdEmbeddings 由词嵌入、位置嵌入、Token 类型嵌入三部分相加构成word_embeddingsvocab_size × hidden_size带padding_idx、position_embeddingsmax_position_embeddings × hidden_size、token_type_embeddingstype_vocab_size × hidden_size最后经过hidden_dropout_prob的 dropout。当未显式传入position_ids时源码L246-L250会用paddle.cumsum在 batch 内自动生成从 0 开始的递增位置编号。4.2 编码器层 TransformerEncoderLayer 与 TransformerEncoder每个 TransformerEncoderLayer 包含一个MultiHeadAttention通过attention_type、block_size、window_size、num_global_blocks、num_rand_blocks、seed六个参数直接实例化 BigBird 块稀疏注意力L84-L96两层前馈Linear(hidden_size - dim_feedforward)与Linear(dim_feedforward - hidden_size)中间夹激活函数getattr(F, config.activation)和 dropout两个 LayerNormepsilon 1e-12与两组残差连接。normalize_before控制归一化位置为 True 时采用 pre-LayerNorm先归一化再进子层为 False 时采用 post-LayerNorm残差后归一化对应前向代码 L110-L126。该层输出一个BigBirdEncoderLayerOutput数据类包含src编码后的序列与attn_output重排为[B, nhead, T, -1]的注意力输出。TransformerEncoder 用LayerList堆叠num_layers层首层复用传入的 layer 实例其余层用type(encoder_layer)(encoder_layer.config)复制并在最外层套一个 LayerNorm。前向时逐层传递每层专属的随机掩码索引rand_mask_idx_list[i]L172-L187并可选择收集all_hidden_states与all_attentions。4.3 池化器 BigBirdPoolerBigBirdPooler 采用 BERT 风格取序列首 Tokenhidden_states[:, 0]经过一个Linear(hidden_size, hidden_size)与Tanh激活得到[batch_size, hidden_size]的池化输出供分类类任务头使用。4.4 前向流程与掩码处理BigBirdModel.forward接受input_ids、token_type_ids、attention_mask、rand_mask_idx_list、inputs_embeds、output_hidden_states、output_attentions、return_dict八个参数L396-L405其中input_ids与inputs_embeds二选一同时传入会抛异常。掩码处理集中在_process_maskL380-L394默认以pad_token_id的位置构造[B, T]的注意力掩码再分别 unsqueeze 成[B, 1, T, 1]的query_mask和[B, 1, 1, T]的key_mask并做1 - mask翻转供块稀疏注意力内部使用。前向的关键一步是随机掩码索引的生成L500-L512即使调用方没有传rand_mask_idx_list模型也会根据num_layers/seq_len/nhead/block_size/window_size/num_global_blocks/num_rand_blocks/seed自动调用create_bigbird_rand_mask_idx_list生成并转换为paddle.Tensor因此普通调用可以完全忽略该参数。return_dictFalse默认时返回(sequence_output, pooled_output)二元组为 True 时返回BaseModelOutputWithPoolingAndCrossAttentions携带last_hidden_state、pooler_output、hidden_states、attentions字段L528-L533。4.5 权重初始化BigBirdPretrainedModel._init_weights 用均值为 0、标准差为initializer_range默认 0.02的正态分布初始化Linear/Embedding权重并强制 LayerNorm 的 epsilon 为 1e-12。该基类同时声明base_model_prefix bigbird、config_class BigBirdConfig接入 PaddleNLP 统一的PretrainedModel下载/加载体系。五、随机掩码索引块稀疏注意力的调度表随机块注意力需要一个“每一层、每个注意力头、每一行 query 该关注哪些块”的索引表。这个逻辑不在 modeling 模块内而是由 attention_utils.py 的 create_bigbird_rand_mask_idx_list 提供create_bigbird_rand_mask_idx_list( num_layers, query_length, key_length, num_heads, block_size, window_size, num_global_blocks, num_rand_blocks, seed )它按层循环调用create_bigbird_rand_mask_idx生成每一层的随机块索引返回一个长度为num_layers的列表。在 BigBirdModel 前向 中该列表被逐层转成paddle.Tensor并传给TransformerEncoder在 BigBirdModel 文档字符串示例 中也可以看到调用方手动构造该列表后传入的写法。随机种子由config[seed]控制设置后即可保证结果可复现。六、九类任务头从预训练到下游任务的完整覆盖所有任务头均以BigBirdModel为底座self.bigbird BigBirdModel(config)在其输出之上叠加一层或多层线性头并统一支持return_dict与标签入参计算损失。下面逐一说明其结构与源码要点。6.1 BigBirdForSequenceClassification序列分类/回归在池化输出上叠加Dropout Linear(hidden_size, num_labels)modeling.py L554-L560。损失类型由num_labels与标签数据类型自动推断L658-L678num_labels 1回归任务使用MSELossnum_labels 1且标签为整型单标签分类使用CrossEntropyLoss其余情况多标签分类使用BCEWithLogitsLoss。return_dictTrue时返回SequenceClassifierOutput。6.2 BigBirdForQuestionAnswering抽取式问答在序列输出上叠加Linear(hidden_size, 2)输出经transpose与unstack拆成start_logits与end_logits两个[batch_size, seq_len]张量modeling.py L1210-L1212。计算损失时会先把超出序列长度的起止位置 clip 到ignored_index再分别用CrossEntropyLoss(ignore_indexignored_index)计算起止损失并取平均L1215-L1229。类还提供了静态方法prepare_question_mask用于构造问题区域的掩码。官方预训练权重列表中的google/bigbird-base-trivia-itc即面向此类任务。6.3 BigBirdForTokenClassificationToken 级分类在序列输出上叠加Dropout Linear(hidden_size, num_labels)对每个 Token 位置独立分类modeling.py L1265-L1270标签形状为[batch_size, sequence_length]损失为CrossEntropyLoss适合 NER 等任务。6.4 BigBirdForMultipleChoice多项选择输入形状为[batch_size, num_choice, sequence_length]前向时先 reshape 压平为[batch_size*num_choice, sequence_length]统一过主干再在池化输出上接Linear(hidden_size, 1)得到每个选项的得分最终 reshape 回[batch_size, num_choice]modeling.py L1455-L1497配合CrossEntropyLoss计算损失。6.5 BigBirdForMaskedLM掩码语言建模BigBirdForMaskedLM 在主干之上叠加BigBirdLMPredictionHead构造时调用self.tie_weights()将 LM 头解码权重与输入嵌入绑定权重共享。损失在labels ! None时以CrossEntropyLoss计算标签中-100的位置将被忽略。6.6 BigBirdForCausalLM因果语言建模BigBirdForCausalLM 与 MLM 版本结构相同共享 LM 头区别在于训练目标计算损失时进行标准的 next-token 平移——预测分数取[:, :-1, :]标签取[:, 1:]L1688-L1695。6.7 预训练三件套Heads、ForPretraining、CriterionBigBirdPretrainingHeads 组合了BigBirdLMPredictionHeadLinear(hidden_size, hidden_size) 激活 LayerNorm后与解码权重做转置矩阵乘见 L692-L719与一个Linear(hidden_size, 2)的句间关系头输出(prediction_scores, seq_relationship_score)BigBirdForPretraining 将两者接到主干上labels与next_sentence_label同时提供时总损失为 MLM 损失与 NSP 损失之和L927-L933BigBirdPretrainingCriterion 是一个独立的paddle.nn.Layer损失模块按masked_lm_weights对掩码 Token 损失做加权归一化分母加 1e-5 防止除零L1071-L1075并通过use_nsp开关控制是否计入 NSP 损失为 False 时 NSP 损失缩放为 0L1076-L1080。七、可运行示例从零加载 BigBird 模型以下示例均出自 modeling 模块的文档字符串按原样整理以保证可直接运行需要已安装paddlepaddle与paddlenlp首次运行会自动下载bigbird-base-uncased权重。7.1 主干模型推理含手动构造随机掩码索引import paddle from paddlenlp.transformers import BigBirdModel, BigBirdTokenizer from paddlenlp.transformers import create_bigbird_rand_mask_idx_list tokenizer BigBirdTokenizer.from_pretrained(bigbird-base-uncased) model BigBirdModel.from_pretrained(bigbird-base-uncased) config model.config max_seq_len 512 input_ids tokenizer.convert_tokens_to_ids( tokenizer( This is a docudrama story on the Lindy Chamberlain case and a look at its impact on Australian society It especially looks at the problem of innuendo gossip and expectation when dealing with reallife dramasbr br One issue the story deals with is the way it is expected people will all give the same emotional response to similar situations Not everyone goes into wild melodramatic hysterics to every major crisis Just because the characters in the movies and on TV act in a certain way is no reason to expect real people to do so )) input_ids.extend([0] * (max_seq_len - len(input_ids))) seq_len len(input_ids) input_ids paddle.to_tensor([input_ids]) rand_mask_idx_list create_bigbird_rand_mask_idx_list( config[num_layers], seq_len, seq_len, config[nhead], config[block_size], config[window_size], config[num_global_blocks], config[num_rand_blocks], config[seed]) rand_mask_idx_list [ paddle.to_tensor(rand_mask_idx) for rand_mask_idx in rand_mask_idx_list ] output model(input_ids, rand_mask_idx_listrand_mask_idx_list)说明这里手动构造的rand_mask_idx_list与模型内部自动生成的结果等价如果不传该参数模型也会按相同逻辑自动生成。文本被 padding 到 512 后再送入可直观感受 BigBird 对超长序列的处理方式。7.2 序列分类import paddle from paddlenlp.transformers import BigBirdForSequenceClassification, BigBirdTokenizer from paddlenlp.transformers import create_bigbird_rand_mask_idx_list tokenizer BigBirdTokenizer.from_pretrained(bigbird-base-uncased) model BigBirdForSequenceClassification.from_pretrained(bigbird-base-uncased) config model.bigbird.config max_seq_len 512 input_ids tokenizer.convert_tokens_to_ids( tokenizer( This is a docudrama story on the Lindy Chamberlain case and a look at its impact on Australian society It especially looks at the problem of innuendo gossip and expectation when dealing with reallife dramasbr br One issue the story deals with is the way it is expected people will all give the same emotional response to similar situations Not everyone goes into wild melodramatic hysterics to every major crisis Just because the characters in the movies and on TV act in a certain way is no reason to expect real people to do so )) input_ids.extend([0] * (max_seq_len - len(input_ids))) seq_len len(input_ids) input_ids paddle.to_tensor([input_ids]) rand_mask_idx_list create_bigbird_rand_mask_idx_list( config[num_layers], seq_len, seq_len, config[nhead], config[block_size], config[window_size], config[num_global_blocks], config[num_rand_blocks], config[seed]) rand_mask_idx_list [ paddle.to_tensor(rand_mask_idx) for rand_mask_idx in rand_mask_idx_list ] output model(input_ids, rand_mask_idx_listrand_mask_idx_list) print(output)7.3 预训练MLM NSP推理import paddle from paddlenlp.transformers import BigBirdForPretraining, BigBirdTokenizer from paddlenlp.transformers import create_bigbird_rand_mask_idx_list tokenizer BigBirdTokenizer.from_pretrained(bigbird-base-uncased) model BigBirdForPretraining.from_pretrained(bigbird-base-uncased) config model.bigbird.config max_seq_len 512 input_ids, masked_lm_positions, masked_lm_ids, masked_lm_weights tokenizer.encode( This is a docudrama story on the Lindy Chamberlain case and a look at its impact on Australian society It especially looks at the problem of innuendo gossip and expectation when dealing with reallife dramasbr br One issue the story deals with is the way it is expected people will all give the same emotional response to similar situations Not everyone goes into wild melodramatic hysterics to every major crisis Just because the characters in the movies and on TV act in a certain way is no reason to expect real people to do so, max_seq_lenmax_seq_len) seq_len len(input_ids) input_ids paddle.to_tensor([input_ids]) rand_mask_idx_list create_bigbird_rand_mask_idx_list( config[num_layers], seq_len, seq_len, config[nhead], config[block_size], config[window_size], config[num_global_blocks], config[num_rand_blocks], config[seed]) rand_mask_idx_list [ paddle.to_tensor(rand_mask_idx) for rand_mask_idx in rand_mask_idx_list ] output model(input_ids, rand_mask_idx_listrand_mask_idx_list) print(output)这里的tokenizer.encode(...)会返回掩码位置、掩码后的原始词 id 与权重BigBirdForPretraining支持传入masked_positions只对掩码 Token 输出预测分数。7.4 预训练损失计算BigBirdPretrainingCriterionimport numpy as np import paddle from paddlenlp.transformers import BigBirdForPretraining, BigBirdTokenizer, BigBirdPretrainingCriterion from paddlenlp.transformers import create_bigbird_rand_mask_idx_list tokenizer BigBirdTokenizer.from_pretrained(bigbird-base-uncased) model BigBirdForPretraining.from_pretrained(bigbird-base-uncased) config model.bigbird.config criterion BigBirdPretrainingCriterion(config[vocab_size], False) max_seq_len 512 max_pred_length 75 input_ids, masked_lm_positions, masked_lm_ids, masked_lm_weights tokenizer.encode( This is a docudrama story on the Lindy Chamberlain case and a look at its impact on Australian society It especially looks at the problem of innuendo gossip and expectation when dealing with reallife dramasbr br One issue the story deals with is the way it is expected people will all give the same emotional response to similar situations Not everyone goes into wild melodramatic hysterics to every major crisis Just because the characters in the movies and on TV act in a certain way is no reason to expect real people to do so, max_seq_lenmax_seq_len, max_pred_lenmax_pred_length) seq_len len(input_ids) masked_lm_positions_tmp np.full(seq_len, 0, dtypenp.int32) masked_lm_ids_tmp np.full([seq_len, 1], -1, dtypenp.int64) masked_lm_weights_tmp np.full([seq_len], 0, dtypefloat32) mask_token_num 0 for i, x in enumerate([input_ids]): for j, pos in enumerate(masked_lm_positions): masked_lm_positions_tmp[mask_token_num] i * seq_len pos masked_lm_ids_tmp[mask_token_num] masked_lm_ids[j] masked_lm_weights_tmp[mask_token_num] masked_lm_weights[j] masked_lm_positions masked_lm_positions_tmp masked_lm_ids masked_lm_ids_tmp masked_lm_weights masked_lm_weights_tmp input_ids paddle.to_tensor([input_ids]) masked_lm_positions paddle.to_tensor(masked_lm_positions) masked_lm_ids paddle.to_tensor(masked_lm_ids, dtypeint64) masked_lm_weights paddle.to_tensor(masked_lm_weights) masked_lm_scale 1.0 next_sentence_labels paddle.zeros(shape(1, 1), dtypeint64) rand_mask_idx_list create_bigbird_rand_mask_idx_list( config[num_layers], seq_len, seq_len, config[nhead], config[block_size], config[window_size], config[num_global_blocks], config[num_rand_blocks], config[seed]) rand_mask_idx_list [ paddle.to_tensor(rand_mask_idx) for rand_mask_idx in rand_mask_idx_list ] prediction_scores, seq_relationship_score model( input_ids, rand_mask_idx_listrand_mask_idx_list, masked_positionsmasked_lm_positions) loss criterion(prediction_scores, seq_relationship_score, masked_lm_ids, next_sentence_labels, masked_lm_scale, masked_lm_weights) print(loss)这个示例完整展示了 PaddleNLP 预训练数据的组织方式masked_lm_positions记录掩码 Token 的全局位置masked_lm_ids为标签masked_lm_weights用于加权next_sentence_labels为 NSP 标签use_nspFalse时 NSP 损失不参与梯度。7.5 抽取式问答与 Token 分类极简调用import paddle from paddlenlp.transformers.bigbird.modeling import BigBirdForQuestionAnswering from paddlenlp.transformers.bigbird.tokenizer import BigBirdTokenizer tokenizer BigBirdTokenizer.from_pretrained(bigbird-base-uncased) model BigBirdForQuestionAnswering.from_pretrained(bigbird-base-uncased) inputs tokenizer(Welcome to use PaddlePaddle and PaddleNLP!, return_tensorspd) inputs {k: paddle.to_tensor([v]) for (k, v) in inputs.items()} outputs model(**inputs) start_logits outputs[0] end_logits outputs[1]Token 分类模型BigBirdForTokenClassification的调用方式与之完全相同只需把BigBirdForQuestionAnswering换成BigBirdForTokenClassification即可拿到[batch_size, seq_len, num_labels]的 logits。八、源码阅读导航进一步探索的入口如果你希望深入理解 BigBird 的实现细节推荐按以下路径阅读源码配置与权重configuration.py预训练初始化配置、资源映射、configuration.pyBigBirdConfig类定义主干模型modeling.pyBigBirdModel、modeling.pyTransformerEncoderLayer、modeling.pyTransformerEncoder块稀疏注意力底层attention_utils.pycreate_bigbird_rand_mask_idx_list与其中的MultiHeadAttention各类任务头modeling.py序列分类、modeling.py问答、modeling.pyToken 分类、modeling.py多项选择、modeling.pyMLM、modeling.py因果 LM预训练组件modeling.pyBigBirdForPretraining、modeling.pyBigBirdPretrainingCriterion。九、总结PaddleNLP 的 BigBird 建模模块以一份BigBirdConfig配置驱动全部结构block_size、window_size、num_global_blocks、num_rand_blocks四个参数精确控制块稀疏注意力的形状attention_type提供退化为原始全注意力的开关max_position_embeddings4096的默认值使其天然适配长文本任务。以BigBirdModel为底座模块通过统一的“主干 任务头 可选损失”模式覆盖了序列分类、抽取式问答、Token 分类、多项选择、掩码语言建模、因果语言建模与预训练共七类任务且全部任务头均支持return_dict结构化输出与标签驱动的损失计算配合from_pretrained一键加载官方权重无论是研究块稀疏注意力机制还是落地长文本下游任务都可以直接基于这份源码快速上手。赞分享人工智能大模型预训练微调LoRARLHF强化学习分布式训练【免费下载链接】PaddleNLPEasy-to-use and powerful LLM and SLM library with awesome model zoo.项目地址https://gitcode.com/gh_mirrors/pa/PaddleNLP点击查看免费下载相关推荐PaddleNLP BigBird 模型实战指南从块稀疏注意力原理到预训练权重使用PaddleNLP BigBird 模型实战指南从块稀疏注意力原理到预训练权重使用 本篇技术指南围绕 PaddleNLP 中 BigBird 模型的支持情况展人工智能大模型预训练微调LoRARLHF强化学习分布式训练模型推理服务推理引擎模型量化模型压缩本地部署NLPPaddleNLP 中的 BigBird 模型稀疏注意力大模型的使用与原理详解PaddleNLP 中的 BigBird 模型稀疏注意力大模型的使用与原理详解 BigBird 是 Google Research 提出的稀疏注意力 Tran人工智能大模型预训练微调LoRARLHF强化学习分布式训练模型推理服务推理引擎模型量化模型压缩本地部署NLPTransformers 中的 BigBird面向超长序列的稀疏注意力模型完全指南Transformers 中的 BigBird面向超长序列的稀疏注意力模型完全指南 导读 BigBird 是 Hugging Face Transformer人工智能深度学习机器学习预训练微调NLP计算机视觉语音多模态上一篇claude-seo 的 Sitemap 架构实战XML 站点地图校验、质量门禁与规模化生成指南下一篇kkFileView 在线打开 CAD 图纸3 步上手手绘批注创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考