沃巴蒂
ff22ed3a8b
本次同步 09-12 至 09-28 的代码变更，主要内容： 新增功能 - 修改密码：后端新增 ChangePasswordRequest DTO 与UserController/UserService 对应接口，前端新增 ChangePasswordDialog 并在 Header 接入入口 - 文件预览组件重构：file-preview 目录拆分出 VideoPreview（支持 blobUrl 直传）、 ImagePreview、AudioPreview，index.tsx 统一调度各类型预览器 - 新增 my_ops 算子模块：markitdown_parser、mineru_pdf_parser、text_quality_evaluator - 新增 tools/minio-file-server MinIO 文件服务 功能增强 - 数据清洗：cleaning_task_service/scheduler/routes 重构， data_juicer_executor 与 wrapper 增强，OperatorTable/LogsTable/TaskDetail 同步调整 - 数据标注：annotation sync.py、project.py 接口与schema 更新 - 数据采集、算子市场、首页与菜单路由相关页面调整 - DataJuicer 算子接入（useOperatorOperations、dj_ops_schema） 结构调整 - deepflow/ 更名为 deep-flow/（419 个文件重命名） - 新增 helm 部署编排、GitHub Actions 之外的自建镜像构建脚本 未纳入本次提交 - scripts-backup/、runtime/python-executor-backup/、components_backup/（备份目录） - *.backup*、__pycache__ 产物 - zh_core_web_sm-3.8.0-py3-none-any.whl（46MB 构建产物） - DataX/（内嵌独立仓库，远程亦未收录其源码）
59 lines
1.5 KiB
YAML
59 lines
1.5 KiB
YAML
project_name: t-final
|
|
dataset_path: "/home/apesea/xuhq/codes/DeepData/test/ray/input"
|
|
executor_type: ray
|
|
ray_address: 127.0.0.1:6388
|
|
export_type: jsonl
|
|
use_cache: false
|
|
export_path: "/home/apesea/xuhq/codes/DeepData/test/ray/out_final"
|
|
skip_op_error: false
|
|
video_key: video_path
|
|
open_monitor: false
|
|
text_keys: ['text']
|
|
np: 1
|
|
keep_stats_in_res_ds: True # ← 让 video_audio_tags 等 meta 字段导出
|
|
|
|
process:
|
|
- video_extract_frames_mapper:
|
|
frame_num: 5
|
|
frame_sampling_method: uniform
|
|
video_key: video_path
|
|
output_format: path
|
|
frame_dir: "./frames"
|
|
num_proc: 1
|
|
|
|
- video_tagging_from_audio_mapper:
|
|
video_key: video_path
|
|
num_proc: 1
|
|
hf_ast: 'MIT/ast-finetuned-audioset-10-10-0.4593'
|
|
|
|
- video_duration_filter:
|
|
video_key: video_path
|
|
min_duration: 3
|
|
max_duration: 60
|
|
num_proc: 1
|
|
|
|
- video_aesthetics_filter:
|
|
video_key: video_path
|
|
min_score: 0.4
|
|
num_proc: 1
|
|
|
|
- video_captioning_from_frames_mapper:
|
|
video_key: video_path
|
|
hf_img2seq: Salesforce/blip-image-captioning-base # ~990MB，稳
|
|
frame_sampling_method: uniform
|
|
frame_num: 3
|
|
keep_original_sample: false
|
|
caption_num: 1
|
|
keep_candidate_mode: random_any
|
|
num_proc: 1
|
|
|
|
- video_frames_text_similarity_filter:
|
|
video_key: video_path
|
|
min_score: 0.25
|
|
num_proc: 1
|
|
|
|
- ray_video_deduplicator: # ← 不是 ray_video_deduplicator
|
|
video_key: video_path
|
|
backend: ray_actor
|
|
|