任务调度后端消息队列【免费下载链接】celeryDistributed Task Queue (development branch)项目地址https://gitcode.com/gh_mirrors/ce/celery点击查看免费下载本篇技术指南以 Celery 仓库中的celery upgrade命令celery/bin/upgrade.py为核心讲解如何将 Celery 3.x 时代的旧式大写配置如BROKER_URL、CELERY_ALWAYS_EAGER一键迁移为 4.x 及以后版本的全小写新命名如broker_url、task_always_eager并覆盖 Django 项目的CELERY_命名空间改造。读完本文你将掌握celery upgrade settings的完整用法、三个选项--django/--compat/--no-backup的适用场景、备份与回滚机制以及新旧设置名背后的映射表与源码实现原理从而安全、无痛地完成项目配置升级。为什么需要升级设置名4.0 起命名规范全面变更Celery 4.0 对配置系统做了一次为美而战in pursuit of beauty的大规模改名所有设置名改为全小写并对前缀进行了系统性梳理。这一变更完全向后兼容——旧的大写名称仍然可用但官方强烈建议尽快迁移。仓库文档 docs/history/whatsnew-4.0.rst 明确指出迁移工作可以交给celery upgrade settings命令自动完成。核心变更可归纳为三条规则大小写规范化所有设置名由全大写改为全小写例如BROKER_URL→broker_url。前缀重构celerybeat_前缀更名为beat_如CELERYBEAT_SCHEDULE→beat_scheduleceleryd_前缀更名为worker_如CELERYD_CONCURRENCY→worker_concurrency去掉celery_前缀任务相关设置改挂task_前缀worker 相关设置改挂worker_前缀如CELERY_ALWAYS_EAGER→task_always_eager、CELERY_TASK_SERIALIZER→task_serializer。个别特殊改名少量设置不仅改前缀还改了语义名称例如CELERY_MAX_CACHED_RESULTS→result_cache_max、CELERY_RESULT_DBURI→ 直接改用result_backend详见下文特殊重命名表。命令基本用法一条命令完成就地迁移celery upgrade是 Celery CLI 中用于在版本之间执行升级的命令组目前包含settings一个子命令其职责在源码 docstring 中写明为Migrate settings from Celery 3.x to Celery 4.xcelery/bin/upgrade.py。基本调用方式针对普通 Python 项目的配置模块例如proj/celeryconfig.py$ celery upgrade settings proj/settings.py执行过程分三步对应 celery/bin/upgrade.py 的实现读取文件_slurp以 UTF-8 编码逐行读入目标文件全部内容逐行替换对每一行调用_to_new_key做新旧键名映射替换得到(did_change, line_contents)元组列表写回与备份只要有任何一行发生了变更默认先在原文件旁生成备份文件如proj/settings.py.orig再把替换后的内容写回原文件。命令结束时会在标准输出打印结果有变更时输出Changes to your setting have been made!无需变更时输出Does not seem to require any changes :-)。备份机制.orig文件与回滚celery upgrade settings默认会进行原地修改 备份。备份逻辑位于_backupcelery/bin/upgrade.py它把原文件完整复制为追加.orig后缀的新文件例如proj/settings.py→proj/settings.py.orig并打印writing backup to proj/settings.py.orig...。settings命令文档docs/history/whatsnew-4.0.rst也明确承诺命令会就地修改你的模块为新的小写名称并保存一份.orig备份。这意味着升级后可随时用备份文件对比检查或回滚。三个选项详解--django、--compat、--no-backup在 celery/bin/upgrade.py 中settings子命令声明了三个互不冲突的选项选项类型作用--djangoflag以 Django 项目模式升级所有设置补上CELERY_前缀同时转为新命名--compatflag保持向后兼容同样为设置名补上CELERY_前缀但不限定 Django 场景--no-backupflag不生成.orig备份文件直接覆盖原文件这三个选项的生效逻辑非常简洁celery/bin/upgrade.pykeyfilter _compat_key if django or compat else pass1默认两者均不指定时keyfilter为恒等函数pass1仅做大小写/前缀规范化即BROKER_URL→broker_url指定--django或--compat时keyfilter切换为_compat_keycelery/bin/upgrade.py它会在原有基础上再给每个设置名补上CELERY_前缀若本身不以CELERY开头得到CELERY_BROKER_URL这种 Django 风格的大写命名。因此三条命令产生三种结果风格# 普通模式全小写新命名 $ celery upgrade settings proj/settings.py # BROKER_URL - broker_url # Django/兼容模式大写 CELERY_ 前缀 $ celery upgrade settings proj/settings.py --django # BROKER_URL - CELERY_BROKER_URL # 不备份直接覆盖 $ celery upgrade settings proj/settings.py --no-backupDjango 项目迁移--django与CELERY_命名空间对于从 Djangosettings.py加载 Celery 配置的项目官方推荐使用CELERY_前缀把 Celery 配置与 Django 自身及其他应用的配置隔离。仓库的 Django 示例项目即采用这一模式见 examples/django/proj/celery.py 与 examples/django/proj/settings.py# proj/celery.py app.config_from_object(django.conf:settings, namespaceCELERY)# proj/settings.py —— 升级后的目标形态 CELERY_BROKER_URL redis://localhost:6379/0 CELERY_TASK_ALWAYS_EAGER True CELERY_WORKER_CONCURRENCY 4迁移旧 Django 项目分两步docs/userguide/configuration.rst 与 docs/history/whatsnew-4.0.rst第一步升级设置文件。对旧的 Django 配置执行$ celery upgrade settings proj/settings.py --django--django会为原本没有前缀的设置补上CELERY_。例如旧的BROKER_URL应写成CELERY_BROKER_URL——这一转换完全由命令自动完成无需手工改动。第二步在celery.py中显式声明命名空间。在proj/celery.py里通过app.config_from_object(django.conf:settings, namespaceCELERY)让应用从带前缀的 Django 设置中读取配置。完整的 Django 集成示例可参考 docs/django/first-steps-with-django.rst。新旧设置名对照表常见设置对照旧 → 新下表完整摘录自 docs/userguide/configuration.rst 的官方对照表覆盖 broker、result backend、任务与 worker 等核心配置域旧设置名3.x新设置名4.xCELERY_ACCEPT_CONTENTaccept_contentCELERY_ENABLE_UTCenable_utcCELERY_IMPORTSimportsCELERY_INCLUDEincludeCELERY_TIMEZONEtimezoneCELERYBEAT_MAX_LOOP_INTERVALbeat_max_loop_intervalCELERYBEAT_SCHEDULEbeat_scheduleCELERYBEAT_SCHEDULERbeat_schedulerCELERYBEAT_SCHEDULE_FILENAMEbeat_schedule_filenameCELERYBEAT_SYNC_EVERYbeat_sync_everyBROKER_URLbroker_urlBROKER_TRANSPORTbroker_transportBROKER_TRANSPORT_OPTIONSbroker_transport_optionsBROKER_CONNECTION_TIMEOUTbroker_connection_timeoutBROKER_CONNECTION_RETRYbroker_connection_retryBROKER_CONNECTION_MAX_RETRIESbroker_connection_max_retriesBROKER_FAILOVER_STRATEGYbroker_failover_strategyBROKER_HEARTBEATbroker_heartbeatBROKER_LOGIN_METHODbroker_login_methodBROKER_NATIVE_DELAYED_DELIVERY_QUEUE_TYPEbroker_native_delayed_delivery_queue_typeBROKER_POOL_ACQUIRE_TIMEOUTbroker_pool_acquire_timeoutBROKER_POOL_LIMITbroker_pool_limitBROKER_USE_SSLbroker_use_sslCELERY_CACHE_BACKENDcache_backendCELERY_CACHE_BACKEND_OPTIONScache_backend_optionsCASSANDRA_COLUMN_FAMILYcassandra_tableCASSANDRA_ENTRY_TTLcassandra_entry_ttlCASSANDRA_KEYSPACEcassandra_keyspaceCASSANDRA_PORTcassandra_portCASSANDRA_READ_CONSISTENCYcassandra_read_consistencyCASSANDRA_SERVERScassandra_serversCASSANDRA_WRITE_CONSISTENCYcassandra_write_consistencyCASSANDRA_OPTIONScassandra_optionsS3_ACCESS_KEY_IDs3_access_key_idS3_SECRET_ACCESS_KEYs3_secret_access_keyS3_BUCKETs3_bucketS3_BASE_PATHs3_base_pathS3_ENDPOINT_URLs3_endpoint_urlS3_REGIONs3_regionCELERY_COUCHBASE_BACKEND_SETTINGScouchbase_backend_settingsCELERY_ARANGODB_BACKEND_SETTINGSarangodb_backend_settingsCELERY_MONGODB_BACKEND_SETTINGSmongodb_backend_settingsCELERY_EVENT_QUEUE_EXPIRESevent_queue_expiresCELERY_EVENT_QUEUE_TTLevent_queue_ttlCELERY_EVENT_QUEUE_DURABLEevent_queue_durableCELERY_EVENT_QUEUE_EXCLUSIVEevent_queue_exclusiveCELERY_EVENT_QUEUE_PREFIXevent_queue_prefixCELERY_EVENT_SERIALIZERevent_serializerCELERY_REDIS_DBredis_dbCELERY_REDIS_HOSTredis_hostCELERY_REDIS_MAX_CONNECTIONSredis_max_connectionsCELERY_REDIS_USERNAMEredis_usernameCELERY_REDIS_PASSWORDredis_passwordCELERY_REDIS_PORTredis_portCELERY_REDIS_BACKEND_USE_SSLredis_backend_use_sslCELERY_REDIS_BACKEND_CREDENTIAL_PROVIDERredis_backend_credential_providerCELERY_RESULT_BACKENDresult_backendCELERY_MAX_CACHED_RESULTSresult_cache_maxCELERY_RESULT_COMPRESSIONresult_compressionCELERY_RESULT_EXCHANGEresult_exchangeCELERY_RESULT_EXCHANGE_TYPEresult_exchange_typeCELERY_RESULT_EXPIRESresult_expiresCELERY_RESULT_PERSISTENTresult_persistentCELERY_RESULT_SERIALIZERresult_serializerCELERY_RESULT_DBURI改用result_backendCELERY_RESULT_ENGINE_OPTIONSdatabase_engine_options[...]_DB_SHORT_LIVED_SESSIONSdatabase_short_lived_sessionsCELERY_RESULT_DB_TABLE_NAMESdatabase_db_namesCELERY_SECURITY_CERTIFICATEsecurity_certificateCELERY_SECURITY_CERT_STOREsecurity_cert_storeCELERY_SECURITY_KEYsecurity_keyCELERY_SECURITY_KEY_PASSWORDsecurity_key_passwordCELERY_ACKS_LATEtask_acks_lateCELERY_ACKS_ON_FAILURE_OR_TIMEOUTtask_acks_on_failure_or_timeoutCELERY_TASK_ALWAYS_EAGERtask_always_eagerCELERY_ANNOTATIONStask_annotationsCELERY_MESSAGE_COMPRESSIONtask_compressionCELERY_CREATE_MISSING_QUEUEStask_create_missing_queuesCELERY_CREATE_MISSING_QUEUE_TYPEtask_create_missing_queue_typeCELERY_CREATE_MISSING_QUEUE_EXCHANGE_TYPEtask_create_missing_queue_exchange_typeCELERY_DEFAULT_DELIVERY_MODEtask_default_delivery_modeCELERY_DEFAULT_EXCHANGEtask_default_exchangeCELERY_DEFAULT_EXCHANGE_TYPEtask_default_exchange_typeCELERY_DEFAULT_QUEUEtask_default_queueCELERY_DEFAULT_QUEUE_TYPEtask_default_queue_typeCELERY_DEFAULT_RATE_LIMITtask_default_rate_limitCELERY_DEFAULT_ROUTING_KEYtask_default_routing_keyCELERY_EAGER_PROPAGATEStask_eager_propagatesCELERY_IGNORE_RESULTtask_ignore_resultCELERY_PUBLISH_RETRYtask_publish_retryCELERY_PUBLISH_RETRY_POLICYtask_publish_retry_policyCELERY_QUEUEStask_queuesCELERY_ROUTEStask_routesCELERY_SEND_SENT_EVENTtask_send_sent_eventCELERY_TASK_SERIALIZERtask_serializerCELERYD_SOFT_TIME_LIMITtask_soft_time_limitCELERY_TASK_TRACK_STARTEDtask_track_startedCELERY_TASK_REJECT_ON_WORKER_LOSTtask_reject_on_worker_lostCELERYD_TIME_LIMITtask_time_limitCELERY_ALLOW_ERROR_CB_ON_CHORD_HEADERtask_allow_error_cb_on_chord_headerCELERYD_AGENTworker_agentCELERYD_AUTOSCALERworker_autoscalerCELERYD_CONCURRENCYworker_concurrencyCELERYD_CONSUMERworker_consumerCELERY_WORKER_DIRECTworker_directCELERY_DISABLE_RATE_LIMITSworker_disable_rate_limitsCELERY_ENABLE_REMOTE_CONTROLworker_enable_remote_controlCELERYD_HIJACK_ROOT_LOGGERworker_hijack_root_loggerCELERYD_LOG_COLORworker_log_colorCELERY_WORKER_LOG_FORMATworker_log_formatCELERYD_WORKER_LOST_WAITworker_lost_waitCELERYD_MAX_TASKS_PER_CHILDworker_max_tasks_per_childCELERYD_POOLworker_poolCELERYD_POOL_PUTLOCKSworker_pool_putlocksCELERYD_POOL_RESTARTSworker_pool_restartsCELERYD_PREFETCH_MULTIPLIERworker_prefetch_multiplierCELERYD_ETA_TASK_LIMITworker_eta_task_limitCELERYD_ENABLE_PREFETCH_COUNT_REDUCTIONworker_enable_prefetch_count_reductionCELERYD_REDIRECT_STDOUTSworker_redirect_stdoutsCELERYD_REDIRECT_STDOUTS_LEVELworker_redirect_stdouts_levelCELERY_SEND_EVENTSworker_send_task_eventsCELERYD_STATE_DBworker_state_dbCELERY_WORKER_TASK_LOG_FORMATworker_task_log_formatCELERYD_TIMERworker_timerCELERYD_TIMER_PRECISIONworker_timer_precisionCELERYD_DETECT_QUORUM_QUEUESworker_detect_quorum_queues特殊重命名语义变化除前缀统一外个别设置在改名时还调整了语义或归属摘录自 docs/history/whatsnew-4.0.rst旧设置名3.x新设置名4.xCELERY_MAX_CACHED_RESULTSresult_cache_maxCELERY_MESSAGE_COMPRESSIONresult_compression/task_compressionCELERY_TASK_RESULT_EXPIRESresult_expiresCELERY_RESULT_DBURIresult_backendCELERY_RESULT_ENGINE_OPTIONSdatabase_engine_options-*-_DB_SHORT_LIVED_SESSIONSdatabase_short_lived_sessionsCELERY_RESULT_DB_TABLE_NAMESdatabase_db_namesCELERY_ACKS_LATEtask_acks_lateCELERY_ALWAYS_EAGERtask_always_eagerCELERY_ANNOTATIONStask_annotationsCELERY_CREATE_MISSING_QUEUEStask_create_missing_queuesCELERY_DEFAULT_DELIVERY_MODEtask_default_delivery_modeCELERY_DEFAULT_EXCHANGEtask_default_exchangeCELERY_DEFAULT_EXCHANGE_TYPEtask_default_exchange_typeCELERY_DEFAULT_QUEUEtask_default_queueCELERY_DEFAULT_RATE_LIMITtask_default_rate_limitCELERY_DEFAULT_ROUTING_KEYtask_default_routing_key--_EAGER_PROPAGATES_EXCEPTIONStask_eager_propagatesCELERY_IGNORE_RESULTtask_ignore_resultCELERY_TASK_PUBLISH_RETRYtask_publish_retryCELERY_TASK_PUBLISH_RETRY_POLICYtask_publish_retry_policyCELERY_QUEUEStask_queuesCELERY_ROUTEStask_routesCELERY_SEND_TASK_SENT_EVENTtask_send_sent_eventCELERY_TASK_SERIALIZERtask_serializerCELERYD_TASK_SOFT_TIME_LIMITtask_soft_time_limit源码原理_TO_NEW_KEY映射表如何驱动替换celery upgrade settings之所以能自动完成新旧键名替换底层依赖一张由celery/app/defaults.py在模块加载时生成的映射表_TO_NEW_KEY。映射表的生成链路如下celery/app/defaults.pyNAMESPACES以嵌套字典定义全部配置项每个配置项是Option对象带旧名的选项通过old{...}参数声明其历史名称例如send_task_events声明了old{celery_send_events}celery/app/defaults.py_to_compat遍历所有配置项若选项声明了opt.old则把旧键映射到新键否则把同名键大写后映射到自身从而保证新旧名称一一对应flatten(NAMESPACES, keyfilter_to_compat)生成(old_key, new_key, opt)三元组列表最终产出三个字典_TO_NEW_KEY旧键 → 新键upgrade settings命令正是消费这张表_TO_OLD_KEY新键 → 旧键用于反向兼容_OLD_DEFAULTS旧键 → 默认值。替换逻辑位于_to_new_keycelery/bin/upgrade.py它有一个值得注意的细节——按旧键名长度降序匹配for old_key in reversed(sorted(source, keylambda x: len(x))): new_line line.replace(old_key, keyfilter(source[old_key])) if line ! new_line and CELERY_CELERY not in new_line: return 1, new_line # only one match per line. return 0, line源码注释解释了原因避免broker_transport抢先匹配并覆盖broker_transport_options这类前缀包含关系。长键先替换可保证子键不被父键误伤同时每行只做一次替换即返回并且用CELERY_CELERY not in new_line防止重复加前缀产生CELERY_CELERY_...这类错误键名。单元测试 t/unit/app/test_defaults.py 对映射表的自洽性做了严格校验DEFAULTS中不含任何大写键_OLD_DEFAULTS中不含任何小写键新旧两套命名互不混杂_TO_NEW_KEY的每个键都属于_OLD_SETTING_KEYS_TO_OLD_KEY的每个键都属于SETTING_KEYS且映射后的值大小写方向正确。这套测试保证了两张映射表始终闭合、可双向查找是celery upgrade settings可靠性的根基。注意事项与限制新旧命名不可混用loader 会自动检测配置使用的是新格式还是旧格式并据此解析但这意味着你不允许在同一个配置里混用新旧设置名——除非你为两个替代名都提供了值docs/history/whatsnew-4.0.rst。迁移时应一次性完成。--django会补前缀该选项会给原本没有前缀的设置统一加上CELERY_例如BROKER_URL变为CELERY_BROKER_URL。如果目标项目本就不打算走 DjangonamespaceCELERY路线应使用默认模式而非--django。--compat与--django行为等价从源码看两者都只是把keyfilter切换为_compat_key区别在于语义场景——--compat面向希望保留大写风格的非 Django 项目。备份是默认行为除非显式传入--no-backup命令总会生成.orig备份若对替换结果不满意可用备份文件恢复。文件必须可写且存在命令以 UTF-8 读写文件源码中_slurp留有 TODO 注释尚未专门处理文件不存在的情况celery/bin/upgrade.py因此请确认目标文件路径正确。小结celery upgrade settings是 Celery 从 3.x 走向 4.x 命名体系时提供的自动化迁移工具它的核心价值在于以 celery/app/defaults.py 中的_TO_NEW_KEY映射表为唯一事实来源逐行扫描并就地重写配置文件同时默认保留.orig备份兼顾了正确性与安全性。对普通项目使用默认模式即可获得全小写新命名对 Django 项目则配合--django与namespaceCELERY两步走即可平滑过渡到官方推荐的前缀化配置风格。如需完整的设置项语义说明可继续查阅 docs/userguide/configuration.rst 中的 Configuration Directives 章节该命令的 API 文档位于 docs/reference/celery.bin.upgrade.rst。赞分享任务调度后端消息队列【免费下载链接】celeryDistributed Task Queue (development branch)项目地址https://gitcode.com/gh_mirrors/ce/celery点击查看免费下载相关推荐Wazuh 4.x 到 5.x Coordinator 迁移指南HAProxy 与 dataplaneapi 配置升级实战Wazuh 4.x 到 5.x Coordinator 迁移指南HAProxy 与 dataplaneapi 配置升级实战 导读 本文档面向在 Wazuh 分网络安全IDS日志分析应用安全漏洞扫描GraphQL CLI 4.x迁移指南从3.x到4.x的无缝升级策略GraphQL CLI 4.x迁移指南从3.x到4.x的无缝升级策略 你是否正在使用GraphQL CLI 3.x版本面对4.x的重大架构调整感到无从下手gh-pages故障排除大全解决常见部署错误的10个方法gh pages故障排除大全解决常见部署错误的10个方法 gh pages是GitHub Pages部署的终极工具但部署过程中可能会遇到各种问题。本指南将为开发工具上一篇如何快速搭建奈雪の茶风格小程序nxdc-milktea前端模板完整指南下一篇Video2X 实用入门指南视频超分辨率与帧率插值从入门到上手创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考