figmar/deepdata
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(deepdata): 引入修改密码、文件预览组件与清洗算子增强，deepflow 改名为 deep-flow

Browse Source 
本次同步 09-12 至 09-28 的代码变更，主要内容：

新增功能
- 修改密码：后端新增 ChangePasswordRequest DTO 与UserController/UserService
  对应接口，前端新增 ChangePasswordDialog 并在 Header 接入入口
- 文件预览组件重构：file-preview 目录拆分出 VideoPreview（支持 blobUrl 直传）、
  ImagePreview、AudioPreview，index.tsx 统一调度各类型预览器
- 新增 my_ops 算子模块：markitdown_parser、mineru_pdf_parser、text_quality_evaluator
- 新增 tools/minio-file-server MinIO 文件服务

功能增强
- 数据清洗：cleaning_task_service/scheduler/routes 重构，
  data_juicer_executor 与 wrapper 增强，OperatorTable/LogsTable/TaskDetail 同步调整
- 数据标注：annotation sync.py、project.py 接口与schema 更新
- 数据采集、算子市场、首页与菜单路由相关页面调整
- DataJuicer 算子接入（useOperatorOperations、dj_ops_schema）

结构调整
- deepflow/ 更名为 deep-flow/（419 个文件重命名）
- 新增 helm 部署编排、GitHub Actions 之外的自建镜像构建脚本

未纳入本次提交
- scripts-backup/、runtime/python-executor-backup/、components_backup/（备份目录）
- *.backup*、__pycache__ 产物
- zh_core_web_sm-3.8.0-py3-none-any.whl（46MB 构建产物）
- DataX/（内嵌独立仓库，远程亦未收录其源码）
This commit is contained in:
沃巴蒂
2026-10-03 22:44:56 +08:00
parent cbb1048e18
commit ff22ed3a8b
1156 changed files with 129912 additions and 21716 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/ray/log/export_output_videos_meta.jsonl_time_20260920155425.txt
+12
View File
@@ -0,0 +1,12 @@
2026-09-20 15:54:25.045 | INFO | data_juicer.config.config:init_setup_from_cfg:682 - dataset_path config is set and a valid local path
2026-09-20 15:54:25.045 | DEBUG | data_juicer.utils.lazy_loader:_load:406 - Loading ray...
2026-09-20 15:54:25.047 | DEBUG | data_juicer.config.config:timing_context:43 - Initializing setup from config took 0.02 seconds
2026-09-20 15:54:25.286 | DEBUG | data_juicer.config.config:timing_context:43 - Updating operator process took 0.24 seconds
2026-09-20 15:54:25.286 | INFO | data_juicer.config.config:config_backup:998 - Back up the input config file [D:\xu_projects\for_mage\llm\repos\deepdata\test\ray\video_pipeline_ray.yaml] into the work_dir [D:\xu_projects\for_mage\llm\repos\deepdata\test\ray]
2026-09-20 15:54:25.300 | INFO | data_juicer.config.config:display_config:1019 - Configuration table:
2026-09-20 15:54:25.421 | DEBUG | data_juicer.config.config:timing_context:43 - Total config initialization time took 1.39 seconds
2026-09-20 15:55:14.068 | INFO | data_juicer.core.executor.default_executor:__init__:54 - Using cache compression method: [None]
2026-09-20 15:55:14.068 | INFO | data_juicer.core.executor.default_executor:__init__:58 - Setting up dataset builder...
2026-09-20 15:55:14.069 | INFO | data_juicer.core.data.dataset_builder:__init__:48 - found dataset_path setting: D:\xu_projects\for_mage\llm\repos\deepdata\test\ray\input_videos_meta.jsonl
2026-09-20 15:55:14.070 | INFO | data_juicer.core.data.load_strategy:get_strategy_class:84 - Getting strategy class for exec: default, data_type: huggingface, data_source: None
2026-09-20 15:55:14.070 | WARNING | data_juicer.core.data.load_strategy:get_strategy_class:129 - No matching strategy found for combination exec: default, data_type: huggingface, data_source: *
test/ray/log/export_output_videos_meta.jsonl_time_20260920155425_DEBUG.txt
+4
View File
@@ -0,0 +1,4 @@
{"text": "2026-09-20 15:54:25.045 | DEBUG | data_juicer.utils.lazy_loader:406 - Loading ray...\n", "record": {"elapsed": {"repr": "0:00:42.969853", "seconds": 42.969853}, "exception": null, "extra": {}, "file": {"name": "lazy_loader.py", "path": "d:\\miniconda3\\envs\\python3.11\\Lib\\site-packages\\data_juicer\\utils\\lazy_loader.py"}, "function": "_load", "level": {"icon": "🐞", "name": "DEBUG", "no": 10}, "line": 406, "message": "Loading ray...", "module": "lazy_loader", "name": "data_juicer.utils.lazy_loader", "process": {"id": 3412, "name": "MainProcess"}, "thread": {"id": 24364, "name": "MainThread"}, "time": {"repr": "2026-09-20 15:54:25.045540+08:00", "timestamp": 1789890865.04554}}}
{"text": "2026-09-20 15:54:25.047 | DEBUG | data_juicer.config.config:43 - Initializing setup from config took 0.02 seconds\n", "record": {"elapsed": {"repr": "0:00:42.971854", "seconds": 42.971854}, "exception": null, "extra": {}, "file": {"name": "config.py", "path": "d:\\miniconda3\\envs\\python3.11\\Lib\\site-packages\\data_juicer\\config\\config.py"}, "function": "timing_context", "level": {"icon": "🐞", "name": "DEBUG", "no": 10}, "line": 43, "message": "Initializing setup from config took 0.02 seconds", "module": "config", "name": "data_juicer.config.config", "process": {"id": 3412, "name": "MainProcess"}, "thread": {"id": 24364, "name": "MainThread"}, "time": {"repr": "2026-09-20 15:54:25.047541+08:00", "timestamp": 1789890865.047541}}}
{"text": "2026-09-20 15:54:25.286 | DEBUG | data_juicer.config.config:43 - Updating operator process took 0.24 seconds\n", "record": {"elapsed": {"repr": "0:00:43.210449", "seconds": 43.210449}, "exception": null, "extra": {}, "file": {"name": "config.py", "path": "d:\\miniconda3\\envs\\python3.11\\Lib\\site-packages\\data_juicer\\config\\config.py"}, "function": "timing_context", "level": {"icon": "🐞", "name": "DEBUG", "no": 10}, "line": 43, "message": "Updating operator process took 0.24 seconds", "module": "config", "name": "data_juicer.config.config", "process": {"id": 3412, "name": "MainProcess"}, "thread": {"id": 24364, "name": "MainThread"}, "time": {"repr": "2026-09-20 15:54:25.286136+08:00", "timestamp": 1789890865.286136}}}
{"text": "2026-09-20 15:54:25.421 | DEBUG | data_juicer.config.config:43 - Total config initialization time took 1.39 seconds\n", "record": {"elapsed": {"repr": "0:00:43.345850", "seconds": 43.34585}, "exception": null, "extra": {}, "file": {"name": "config.py", "path": "d:\\miniconda3\\envs\\python3.11\\Lib\\site-packages\\data_juicer\\config\\config.py"}, "function": "timing_context", "level": {"icon": "🐞", "name": "DEBUG", "no": 10}, "line": 43, "message": "Total config initialization time took 1.39 seconds", "module": "config", "name": "data_juicer.config.config", "process": {"id": 3412, "name": "MainProcess"}, "thread": {"id": 24364, "name": "MainThread"}, "time": {"repr": "2026-09-20 15:54:25.421537+08:00", "timestamp": 1789890865.421537}}}
test/ray/log/export_output_videos_meta.jsonl_time_20260920155425_ERROR.txt
View File
test/ray/log/export_output_videos_meta.jsonl_time_20260920155425_WARNING.txt
+1
View File
@@ -0,0 +1 @@
{"text": "2026-09-20 15:55:14.070 | WARNING | data_juicer.core.data.load_strategy:129 - No matching strategy found for combination exec: default, data_type: huggingface, data_source: *\n", "record": {"elapsed": {"repr": "0:01:31.994509", "seconds": 91.994509}, "exception": null, "extra": {}, "file": {"name": "load_strategy.py", "path": "d:\\miniconda3\\envs\\python3.11\\Lib\\site-packages\\data_juicer\\core\\data\\load_strategy.py"}, "function": "get_strategy_class", "level": {"icon": "⚠️", "name": "WARNING", "no": 30}, "line": 129, "message": "No matching strategy found for combination exec: default, data_type: huggingface, data_source: *", "module": "load_strategy", "name": "data_juicer.core.data.load_strategy", "process": {"id": 3412, "name": "MainProcess"}, "thread": {"id": 24364, "name": "MainThread"}, "time": {"repr": "2026-09-20 15:55:14.070196+08:00", "timestamp": 1789890914.070196}}}