feat(deepdata): 引入修改密码、文件预览组件与清洗算子增强，deepflow 改名为 deep-flow
本次同步 09-12 至 09-28 的代码变更，主要内容： 新增功能 - 修改密码：后端新增 ChangePasswordRequest DTO 与UserController/UserService 对应接口，前端新增 ChangePasswordDialog 并在 Header 接入入口 - 文件预览组件重构：file-preview 目录拆分出 VideoPreview（支持 blobUrl 直传）、 ImagePreview、AudioPreview，index.tsx 统一调度各类型预览器 - 新增 my_ops 算子模块：markitdown_parser、mineru_pdf_parser、text_quality_evaluator - 新增 tools/minio-file-server MinIO 文件服务 功能增强 - 数据清洗：cleaning_task_service/scheduler/routes 重构， data_juicer_executor 与 wrapper 增强，OperatorTable/LogsTable/TaskDetail 同步调整 - 数据标注：annotation sync.py、project.py 接口与schema 更新 - 数据采集、算子市场、首页与菜单路由相关页面调整 - DataJuicer 算子接入（useOperatorOperations、dj_ops_schema） 结构调整 - deepflow/ 更名为 deep-flow/（419 个文件重命名） - 新增 helm 部署编排、GitHub Actions 之外的自建镜像构建脚本 未纳入本次提交 - scripts-backup/、runtime/python-executor-backup/、components_backup/（备份目录） - *.backup*、__pycache__ 产物 - zh_core_web_sm-3.8.0-py3-none-any.whl（46MB 构建产物） - DataX/（内嵌独立仓库，远程亦未收录其源码）
This commit is contained in:
沃巴蒂
@@ -27,7 +27,7 @@ LABEL_STUDIO_BASE_URL=http://localhost:30001
|
||||
|
||||
LABEL_STUDIO_USER_TOKEN="demo_dev_token"
|
||||
|
||||
DEEPDATA_JWT_ENABLE=false
|
||||
DEEPDATA_jwt_enable=false
|
||||
|
||||
# Milvus settings (Vector Database)
|
||||
# Development: use localhost
|
||||
|
||||
+107
-107
@@ -1,107 +1,107 @@
|
||||
# DeepData Python Service (DeepData)
|
||||
|
||||
这是 DeepData 的 Python 服务，负责DeepData的数据合成、数据标注、数据评估等功能。
|
||||
|
||||
## 简要说明
|
||||
|
||||
- 框架：FastAPI
|
||||
- 异步数据库/ORM：SQLAlchemy (async)
|
||||
- 数据库迁移：Alembic
|
||||
- 运行器：uvicorn
|
||||
|
||||
## 快速开始（开发）
|
||||
|
||||
### 前置条件
|
||||
|
||||
- Python 3.11+
|
||||
- poetry 包管理器
|
||||
|
||||
1. 克隆仓库
|
||||
```bash
|
||||
git clone https://gitea.apescale.com/figmar/deepdata.git
|
||||
```
|
||||
2. 进入项目目录
|
||||
```bash
|
||||
cd runtime/deepdata-python
|
||||
```
|
||||
|
||||
3. 安装依赖
|
||||
由于项目使用poetry管理依赖，你可以使用以下命令安装：：
|
||||
git clone https://gitea.apescale.com/figmar/deepdata.git
|
||||
poetry install
|
||||
```
|
||||
或者直接使用pip安装（如果poetry不可用）：
|
||||
|
||||
git clone https://gitea.apescale.com/figmar/deepdata.git
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
4. 配置环境变量
|
||||
复制环境变量示例文件并配置：
|
||||
|
||||
git clone https://gitea.apescale.com/figmar/deepdata.git
|
||||
cp .env.example .env
|
||||
```
|
||||
编辑.env文件，设置必要的环境变量，如数据库连接、Label Studio配置等。
|
||||
|
||||
5. 数据库迁移（开发环境）：
|
||||
|
||||
git clone https://gitea.apescale.com/figmar/deepdata.git
|
||||
alembic upgrade head
|
||||
```
|
||||
|
||||
6. 启动开发服务器（示例与常用参数）：
|
||||
|
||||
- 本地开发（默认 host/port，自动重载）：
|
||||
|
||||
git clone https://gitea.apescale.com/figmar/deepdata.git
|
||||
set -a && source .env && set +a && poetry run uvicorn app.main:app --port 18000 --log-level debug --reload
|
||||
```
|
||||
或者
|
||||
git clone https://gitea.apescale.com/figmar/deepdata.git
|
||||
poetry run python -m app.main
|
||||
```
|
||||
|
||||
- 指定主机与端口并打开调试日志：
|
||||
|
||||
git clone https://gitea.apescale.com/figmar/deepdata.git
|
||||
uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload --log-level debug
|
||||
```
|
||||
|
||||
- 在生产环境使用多个 worker（不使用 --reload）：
|
||||
|
||||
git clone https://gitea.apescale.com/figmar/deepdata.git
|
||||
uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 4 --log-level info --proxy-headers
|
||||
```
|
||||
|
||||
- 使用环境变量启动（示例）：
|
||||
|
||||
git clone https://gitea.apescale.com/figmar/deepdata.git
|
||||
HOST=0.0.0.0 PORT=8000 uvicorn app.main:app --reload
|
||||
```
|
||||
|
||||
注意：
|
||||
|
||||
- `--reload` 仅用于开发，会监视文件变化并重启进程；不要在生产中使用。
|
||||
- `--workers` 提供并发处理能力，但会增加内存占用；生产时通常配合进程管理或容器编排（Kubernetes）使用。
|
||||
- 若需要完整的生产部署建议使用 ASGI 服务器（如 gunicorn + uvicorn workers / 或直接使用 uvicorn 在容器中配合进程管理）。
|
||||
|
||||
访问 API 文档：
|
||||
|
||||
- Swagger UI: http://127.0.0.1:8000/docs
|
||||
- ReDoc: http://127.0.0.1:8000/redoc （推荐使用）
|
||||
|
||||
## 开发新功能
|
||||
- 安装开发依赖：
|
||||
|
||||
git clone https://gitea.apescale.com/figmar/deepdata.git
|
||||
poetry add xxx
|
||||
```
|
||||
|
||||
|
||||
## 使用（简要）
|
||||
|
||||
- 所有 API 路径以 `/api` 前缀注册（见 `app/main.py` 中 `app.include_router(api_router, prefix="/api")`）。
|
||||
- 根路径 `/` 返回服务信息和文档链接。
|
||||
|
||||
更多细节请查看 `doc/usage.md`（接口使用）和 `doc/development.md`（开发说明）。
|
||||
# DeepData Python Service (DeepData)
|
||||
|
||||
这是 DeepData 的 Python 服务，负责DeepData的数据合成、数据标注、数据评估等功能。
|
||||
|
||||
## 简要说明
|
||||
|
||||
- 框架：FastAPI
|
||||
- 异步数据库/ORM：SQLAlchemy (async)
|
||||
- 数据库迁移：Alembic
|
||||
- 运行器：uvicorn
|
||||
|
||||
## 快速开始（开发）
|
||||
|
||||
### 前置条件
|
||||
|
||||
- Python 3.11+
|
||||
- poetry 包管理器
|
||||
|
||||
1. 克隆仓库
|
||||
```bash
|
||||
git clone git@github.com:ModelEngine-Group/DeepData.git
|
||||
```
|
||||
2. 进入项目目录
|
||||
```bash
|
||||
cd runtime/deepdata-python
|
||||
```
|
||||
|
||||
3. 安装依赖
|
||||
由于项目使用poetry管理依赖，你可以使用以下命令安装：：
|
||||
```bash
|
||||
poetry install
|
||||
```
|
||||
或者直接使用pip安装（如果poetry不可用）：
|
||||
|
||||
```bash
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
4. 配置环境变量
|
||||
复制环境变量示例文件并配置：
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
编辑.env文件，设置必要的环境变量，如数据库连接、Label Studio配置等。
|
||||
|
||||
5. 数据库迁移（开发环境）：
|
||||
|
||||
```bash
|
||||
alembic upgrade head
|
||||
```
|
||||
|
||||
6. 启动开发服务器（示例与常用参数）：
|
||||
|
||||
- 本地开发（默认 host/port，自动重载）：
|
||||
|
||||
```bash
|
||||
set -a && source .env && set +a && poetry run uvicorn app.main:app --port 18000 --log-level debug --reload
|
||||
```
|
||||
或者
|
||||
```bash
|
||||
poetry run python -m app.main
|
||||
```
|
||||
|
||||
- 指定主机与端口并打开调试日志：
|
||||
|
||||
```bash
|
||||
uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload --log-level debug
|
||||
```
|
||||
|
||||
- 在生产环境使用多个 worker（不使用 --reload）：
|
||||
|
||||
```bash
|
||||
uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 4 --log-level info --proxy-headers
|
||||
```
|
||||
|
||||
- 使用环境变量启动（示例）：
|
||||
|
||||
```bash
|
||||
HOST=0.0.0.0 PORT=8000 uvicorn app.main:app --reload
|
||||
```
|
||||
|
||||
注意：
|
||||
|
||||
- `--reload` 仅用于开发，会监视文件变化并重启进程；不要在生产中使用。
|
||||
- `--workers` 提供并发处理能力，但会增加内存占用；生产时通常配合进程管理或容器编排（Kubernetes）使用。
|
||||
- 若需要完整的生产部署建议使用 ASGI 服务器（如 gunicorn + uvicorn workers / 或直接使用 uvicorn 在容器中配合进程管理）。
|
||||
|
||||
访问 API 文档：
|
||||
|
||||
- Swagger UI: http://127.0.0.1:8000/docs
|
||||
- ReDoc: http://127.0.0.1:8000/redoc （推荐使用）
|
||||
|
||||
## 开发新功能
|
||||
- 安装开发依赖：
|
||||
|
||||
```bash
|
||||
poetry add xxx
|
||||
```
|
||||
|
||||
|
||||
## 使用（简要）
|
||||
|
||||
- 所有 API 路径以 `/api` 前缀注册（见 `app/main.py` 中 `app.include_router(api_router, prefix="/api")`）。
|
||||
- 根路径 `/` 返回服务信息和文档链接。
|
||||
|
||||
更多细节请查看 `doc/usage.md`（接口使用）和 `doc/development.md`（开发说明）。
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
from pydantic_settings import BaseSettings
|
||||
from pydantic_settings import BaseSettings
|
||||
from pydantic import model_validator
|
||||
from typing import Optional
|
||||
|
||||
@@ -83,7 +83,7 @@ class Settings(BaseSettings):
|
||||
# DeepData
|
||||
dm_file_path_prefix: str = "/dataset" # DM存储文件夹前缀
|
||||
|
||||
deepdata_jwt_enable: bool = False
|
||||
DEEPDATA_jwt_enable: bool = False
|
||||
|
||||
# Milvus 配置
|
||||
milvus_uri: str = "http://milvus:19530"
|
||||
|
||||
@@ -80,9 +80,13 @@ class BaseError(Exception):
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
"""将异常转换为响应字典"""
|
||||
msg = self.message
|
||||
if self.data and isinstance(self.data, str):
|
||||
msg = f"{msg}，detail：{self.data}"
|
||||
|
||||
return {
|
||||
"code": self.code,
|
||||
"message": self.message,
|
||||
"message": msg, #self.message,
|
||||
"data": self.data
|
||||
}
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import logging
|
||||
import logging
|
||||
import re
|
||||
import sys
|
||||
from logging.handlers import RotatingFileHandler
|
||||
@@ -49,13 +49,13 @@ def _rotated_log_namer(default_name: str) -> str:
|
||||
|
||||
def _remove_managed_handlers(logger: logging.Logger) -> None:
|
||||
for handler in list(logger.handlers):
|
||||
if getattr(handler, "_deepdata_managed", False):
|
||||
if getattr(handler, "_DeepData_managed", False):
|
||||
logger.removeHandler(handler)
|
||||
handler.close()
|
||||
|
||||
|
||||
def _mark_managed(handler: logging.Handler) -> logging.Handler:
|
||||
setattr(handler, "_deepdata_managed", True)
|
||||
setattr(handler, "_DeepData_managed", True)
|
||||
return handler
|
||||
|
||||
|
||||
|
||||
@@ -5,6 +5,7 @@ from app.db.models.base_entity import BaseEntity, Base
|
||||
class CleaningTask(BaseEntity):
|
||||
"""Data cleaning task entity"""
|
||||
__tablename__ = "t_clean_task"
|
||||
__ignore_data_scope__ = True
|
||||
|
||||
id = Column(String(36), primary_key=True, comment="Task ID")
|
||||
name = Column(String(255), nullable=False, comment="Task name")
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""
|
||||
"""
|
||||
Application middleware.
|
||||
|
||||
Note: Exception handling has been moved to app.core.exception.ExceptionHandlingMiddleware
|
||||
@@ -26,7 +26,7 @@ class UserContextMiddleware(BaseHTTPMiddleware):
|
||||
|
||||
def __init__(self, app):
|
||||
super().__init__(app)
|
||||
self.jwt_enable = settings.deepdata_jwt_enable
|
||||
self.jwt_enable = settings.DEEPDATA_jwt_enable
|
||||
|
||||
async def dispatch(self, request: Request, call_next):
|
||||
user: Optional[str] = request.headers.get("User")
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
from typing import Optional
|
||||
from typing import Optional
|
||||
import math
|
||||
import uuid
|
||||
|
||||
@@ -47,7 +47,7 @@ async def login_label_studio_project(
|
||||
- 不负责跳转
|
||||
"""
|
||||
user = request.headers.get("User") if request else None
|
||||
if settings.deepdata_jwt_enable and (not user or not user.strip()):
|
||||
if settings.DEEPDATA_jwt_enable and (not user or not user.strip()):
|
||||
logger.warning(
|
||||
f"Unauthorized access attempt to annotation login: mapping_id={mapping_id}"
|
||||
)
|
||||
@@ -113,7 +113,7 @@ async def login_label_studio_project(
|
||||
"""登录 Label Studio 项目 — 需要认证且校验项目归属"""
|
||||
# 检查用户认证（仅在 JWT 启用时强制要求）
|
||||
user = request.headers.get("User") if request else None
|
||||
if settings.deepdata_jwt_enable and (not user or not user.strip()):
|
||||
if settings.DEEPDATA_jwt_enable and (not user or not user.strip()):
|
||||
logger.warning(f"Unauthorized access attempt to annotation login: mapping_id={mapping_id}")
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
from typing import Literal, List, Dict, Any, Optional
|
||||
from typing import Literal, List, Dict, Any, Optional
|
||||
from datetime import datetime
|
||||
|
||||
from pydantic import Field, ConfigDict
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
from typing import Optional, List, Dict, Any, Tuple, Set
|
||||
from typing import Optional, List, Dict, Any, Tuple, Set
|
||||
import json
|
||||
import os
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -42,6 +42,7 @@ class RuntimeClient:
|
||||
logger.error(f"Unexpected error stopping task {task_id}: {e}")
|
||||
return False
|
||||
|
||||
'''
|
||||
async def get_task_status(self, task_id: str) -> Optional[dict]:
|
||||
"""Get task status from runtime"""
|
||||
try:
|
||||
@@ -55,6 +56,48 @@ class RuntimeClient:
|
||||
except Exception as e:
|
||||
logger.error(f"Unexpected error getting task status {task_id}: {e}")
|
||||
return None
|
||||
'''
|
||||
|
||||
async def get_task_status(self, task_id: str) -> Optional[dict]:
|
||||
"""Get task status from runtime"""
|
||||
url = f"{self.base_url}/api/task/{task_id}/status"
|
||||
try:
|
||||
logger.info(f"[RuntimeClient] Requesting status for task {task_id} from {url}")
|
||||
response = await self.client.get(url)
|
||||
|
||||
# [调试] 打印 HTTP 状态码
|
||||
logger.info(f"[RuntimeClient] Response status: {response.status_code}")
|
||||
|
||||
# 尝试读取响应体，无论状态码是多少
|
||||
try:
|
||||
resp_json = response.json()
|
||||
logger.info(f"[RuntimeClient] Response body: {resp_json}")
|
||||
|
||||
# 如果 HTTP 状态码不是 2xx，我们也打印出来，但不抛异常（视具体业务逻辑而定）
|
||||
# 这里保持原逻辑：raise_for_status
|
||||
response.raise_for_status()
|
||||
return resp_json
|
||||
except httpx.HTTPError as e:
|
||||
logger.error(f"[RuntimeClient] HTTP Error: {e}")
|
||||
logger.debug(f"[RuntimeClient] Response text: {response.text}")
|
||||
return None
|
||||
except json.JSONDecodeError as e:
|
||||
logger.error(f"[RuntimeClient] JSON Decode Error: {e}")
|
||||
logger.debug(f"[RuntimeClient] Raw response: {response.text}")
|
||||
return None
|
||||
|
||||
except httpx.ConnectError as e:
|
||||
# [调试] 区分连接超时和连接被拒绝
|
||||
logger.error(f"[RuntimeClient] Connect Error: {e}")
|
||||
return None
|
||||
except httpx.TimeoutException as e:
|
||||
# [调试] 明确记录超时
|
||||
logger.error(f"[RuntimeClient] Timeout waiting for status of {task_id}: {e}")
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.error(f"[RuntimeClient] Unexpected error: {e}")
|
||||
return None
|
||||
|
||||
|
||||
async def close(self):
|
||||
"""Close HTTP client"""
|
||||
|
||||
@@ -21,7 +21,7 @@ class OperatorInstanceDto(BaseResponseModel):
|
||||
inputs: Optional[str] = Field(None, description="Input types: text/image/audio/video/multimodal")
|
||||
outputs: Optional[str] = Field(None, description="Output types: text/image/audio/video/multimodal")
|
||||
categories: Optional[List[str]] = Field(None, description="Category IDs")
|
||||
settings: Optional[str] = Field(None, description="算子设置（JSON）")
|
||||
settings: Optional[Any] = Field(None, description="算子设置（JSON）")
|
||||
overrides: Dict[str, Any] = Field(default_factory=dict, description="Operator parameter overrides")
|
||||
|
||||
|
||||
|
||||
@@ -6,12 +6,10 @@ from app.module.cleaning.runtime_client import RuntimeClient
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
# 模块级单例：进程内所有请求共享同一个调度器
|
||||
# 模块级单例
|
||||
_scheduler: "CleaningTaskScheduler | None" = None
|
||||
|
||||
|
||||
def get_scheduler() -> "CleaningTaskScheduler":
|
||||
"""获取全局调度器单例，不存在则创建"""
|
||||
global _scheduler
|
||||
if _scheduler is None:
|
||||
_scheduler = CleaningTaskScheduler(
|
||||
@@ -20,23 +18,22 @@ def get_scheduler() -> "CleaningTaskScheduler":
|
||||
)
|
||||
return _scheduler
|
||||
|
||||
|
||||
class CleaningTaskScheduler:
|
||||
"""Scheduler for executing cleaning tasks"""
|
||||
|
||||
def __init__(self, task_repo: CleaningTaskRepository, runtime_client: RuntimeClient):
|
||||
self.task_repo = task_repo
|
||||
self.runtime_client = runtime_client
|
||||
self._polling_task_ids: set[str] = set() # 待轮询的任务 ID 集合
|
||||
self._polling_task_ids: set[str] = set()
|
||||
self._polling_started: bool = False
|
||||
self._poll_failure_count: dict[str, int] = {} # runtime 不可达连续失败计数
|
||||
self._MAX_POLL_FAILURES = 1 # 一次超时(60s) → 标记 FAILED
|
||||
self._poll_failure_count: dict[str, int] = {}
|
||||
self._not_found_counts: dict[str, int] = {} # 🔴 新增：记录 NOT_FOUND 次数
|
||||
self._MAX_POLL_FAILURES = 1
|
||||
|
||||
async def execute_task(self, db: AsyncSession, task_id: str, retry_count: int) -> bool:
|
||||
"""Execute cleaning task"""
|
||||
from app.module.cleaning.schema import CleaningTaskDto, CleaningTaskStatus
|
||||
from datetime import datetime
|
||||
|
||||
logger.info(f"🚀 [Scheduler] 任务 {task_id} 开始执行...")
|
||||
|
||||
task = CleaningTaskDto()
|
||||
task.id = task_id
|
||||
task.status = CleaningTaskStatus.RUNNING
|
||||
@@ -53,108 +50,130 @@ class CleaningTaskScheduler:
|
||||
task.status = CleaningTaskStatus.FAILED
|
||||
task.finished_at = datetime.now()
|
||||
await self.task_repo.update_task(db, task)
|
||||
|
||||
return submitted
|
||||
|
||||
async def stop_task(self, db: AsyncSession, task_id: str) -> bool:
|
||||
"""Stop cleaning task"""
|
||||
from app.module.cleaning.schema import CleaningTaskDto, CleaningTaskStatus
|
||||
|
||||
self._polling_task_ids.discard(task_id)
|
||||
self._poll_failure_count.pop(task_id, None)
|
||||
self._not_found_counts.pop(task_id, None) # 🔴 清理计数器
|
||||
|
||||
await self.runtime_client.stop_task(task_id)
|
||||
|
||||
task = CleaningTaskDto()
|
||||
task.id = task_id
|
||||
task.status = CleaningTaskStatus.STOPPED
|
||||
|
||||
await self.task_repo.update_task(db, task)
|
||||
return True
|
||||
|
||||
async def startup(self):
|
||||
"""进程启动时调用：恢复未完成任务的轮询 + 启动全局轮询协程"""
|
||||
from app.module.cleaning.schema import CleaningTaskStatus
|
||||
from app.db.session import AsyncSessionLocal
|
||||
|
||||
# 从数据库恢复所有 RUNNING 状态的任务
|
||||
try:
|
||||
async with AsyncSessionLocal() as db:
|
||||
tasks = await self.task_repo.find_tasks(db, status=CleaningTaskStatus.RUNNING)
|
||||
for task in tasks:
|
||||
logger.info(f"[Polling] Recovered RUNNING task from DB: {task.id}")
|
||||
logger.info(f"[Polling] Recovered RUNNING task: {task.id}")
|
||||
self._polling_task_ids.add(task.id)
|
||||
except Exception as e:
|
||||
logger.error(f"[Polling] Failed to recover tasks from DB: {e}")
|
||||
logger.error(f"[Polling] Failed to recover tasks: {e}")
|
||||
|
||||
# 启动全局轮询协程
|
||||
if not self._polling_started:
|
||||
self._polling_started = True
|
||||
asyncio.create_task(self._poll_all_tasks())
|
||||
logger.info("[Polling] Global polling loop started")
|
||||
|
||||
async def _poll_all_tasks(self):
|
||||
"""全局轮询协程：每 2 秒轮询所有 RUNNING 任务的 runtime 状态"""
|
||||
from app.module.cleaning.schema import CleaningTaskDto, CleaningTaskStatus
|
||||
from app.db.session import AsyncSessionLocal
|
||||
from datetime import datetime
|
||||
|
||||
logger.info("[Polling] Global status polling loop started")
|
||||
terminal_statuses = {"completed", "failed", "cancelled", "stopped", "not_found"}
|
||||
logger.info("[Polling] Global polling loop started")
|
||||
terminal_statuses = {"completed", "failed", "cancelled", "stopped"}
|
||||
|
||||
while True:
|
||||
task_ids = list(self._polling_task_ids)
|
||||
if not task_ids:
|
||||
# 没有待轮询任务，休眠后继续等
|
||||
await asyncio.sleep(2)
|
||||
continue
|
||||
|
||||
for task_id in task_ids:
|
||||
try:
|
||||
status_data = await self.runtime_client.get_task_status(task_id)
|
||||
|
||||
if status_data is None:
|
||||
# runtime 不可达，累计失败计数
|
||||
count = self._poll_failure_count.get(task_id, 0) + 1
|
||||
self._poll_failure_count[task_id] = count
|
||||
logger.warning(
|
||||
f"[Polling] Task {task_id} unreachable ({count}/{self._MAX_POLL_FAILURES})"
|
||||
)
|
||||
if count >= self._MAX_POLL_FAILURES:
|
||||
logger.error(
|
||||
f"[Polling] Task {task_id} marked FAILED: "
|
||||
f"runtime unreachable after {count} attempts"
|
||||
)
|
||||
logger.error(f"[Polling] Task {task_id} FAILED (runtime unreachable)")
|
||||
async with AsyncSessionLocal() as db:
|
||||
task_dto = CleaningTaskDto()
|
||||
task_dto.id = task_id
|
||||
task_dto.status = CleaningTaskStatus.FAILED
|
||||
task_dto.finished_at = datetime.now()
|
||||
await self.task_repo.update_task(db, task_dto)
|
||||
t = CleaningTaskDto()
|
||||
t.id = task_id
|
||||
t.status = CleaningTaskStatus.FAILED
|
||||
t.finished_at = datetime.now()
|
||||
await self.task_repo.update_task(db, t)
|
||||
await db.commit()
|
||||
self._polling_task_ids.discard(task_id)
|
||||
self._poll_failure_count.pop(task_id, None)
|
||||
continue
|
||||
|
||||
# runtime 可达，重置失败计数
|
||||
self._poll_failure_count.pop(task_id, None)
|
||||
|
||||
current_status = (status_data.get("status", "") or "").lower()
|
||||
logger.debug(f"[Polling] Task {task_id} status: {current_status}")
|
||||
|
||||
# 🔴 核心修复：处理 NOT_FOUND
|
||||
if current_status == 'not_found':
|
||||
cnt = self._not_found_counts.get(task_id, 0) + 1
|
||||
self._not_found_counts[task_id] = cnt
|
||||
|
||||
# 阶段 1：启动容忍期 (10秒内认为是在初始化)
|
||||
if cnt <= 5:
|
||||
continue # 继续轮询，不报错
|
||||
|
||||
# 阶段 2：检查数据库状态 (判断是真丢失还是已完成)
|
||||
try:
|
||||
async with AsyncSessionLocal() as check_db:
|
||||
db_task = await self.task_repo.find_task_by_id(check_db, task_id)
|
||||
db_status = db_task.status if db_task else "UNKNOWN"
|
||||
|
||||
# ✅ 成功收尾：如果数据库已经是终态，说明任务其实跑完了
|
||||
# 此时 Runtime 清理了内存状态返回 NOT_FOUND，我们应停止轮询
|
||||
if db_status in ["COMPLETED", "FAILED", "STOPPED"]:
|
||||
logger.info(f"[Scheduler] 任务 {task_id} 已结束 (DB: {db_status})，停止轮询。")
|
||||
self._polling_task_ids.discard(task_id)
|
||||
self._not_found_counts.pop(task_id, None)
|
||||
continue
|
||||
|
||||
# ✅ 失败收尾：DB 还是 RUNNING，但 Runtime 彻底找不到它了 (>60秒)
|
||||
if cnt > 30:
|
||||
logger.error(f"[Scheduler] 任务 {task_id} 在 Runtime 丢失，标记 FAILED。")
|
||||
t = CleaningTaskDto()
|
||||
t.id = task_id
|
||||
t.status = CleaningTaskStatus.FAILED
|
||||
t.finished_at = datetime.now()
|
||||
await self.task_repo.update_task(check_db, t)
|
||||
await check_db.commit()
|
||||
|
||||
self._polling_task_ids.discard(task_id)
|
||||
self._not_found_counts.pop(task_id, None)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
continue # 继续等待
|
||||
|
||||
# 如果找到了，重置计数器
|
||||
self._not_found_counts.pop(task_id, None)
|
||||
|
||||
# ==========================================
|
||||
|
||||
if current_status in terminal_statuses:
|
||||
logger.info(f"[Scheduler] 任务 {task_id} 结束: {current_status}")
|
||||
async with AsyncSessionLocal() as db:
|
||||
task = CleaningTaskDto()
|
||||
task.id = task_id
|
||||
if current_status == "completed":
|
||||
task.status = CleaningTaskStatus.COMPLETED
|
||||
else:
|
||||
task.status = CleaningTaskStatus.FAILED
|
||||
task.finished_at = datetime.now()
|
||||
await self.task_repo.update_task(db, task)
|
||||
t = CleaningTaskDto()
|
||||
t.id = task_id
|
||||
t.status = CleaningTaskStatus.COMPLETED if current_status == "completed" else CleaningTaskStatus.FAILED
|
||||
t.finished_at = datetime.now()
|
||||
await self.task_repo.update_task(db, t)
|
||||
await db.commit()
|
||||
logger.info(
|
||||
f"[Polling] Task {task_id} finished: {current_status}"
|
||||
)
|
||||
self._polling_task_ids.discard(task_id)
|
||||
self._poll_failure_count.pop(task_id, None)
|
||||
|
||||
|
||||
@@ -131,6 +131,7 @@ class CleaningTaskService:
|
||||
"""Count cleaning tasks using SQL COUNT"""
|
||||
return await self.task_repo.count_tasks(db, status, keyword)
|
||||
|
||||
'''
|
||||
async def get_task(self, db: AsyncSession, task_id: str) -> CleaningTaskDto:
|
||||
"""Get task by ID"""
|
||||
task = await self.task_repo.find_task_by_id(db, task_id)
|
||||
@@ -168,6 +169,64 @@ class CleaningTaskService:
|
||||
task.instance.append(OperatorInstanceDto(id=inst.operator_id))
|
||||
|
||||
return task
|
||||
'''
|
||||
|
||||
async def get_task(self, db: AsyncSession, task_id: str) -> CleaningTaskDto:
|
||||
"""Get task by ID"""
|
||||
task = await self.task_repo.find_task_by_id(db, task_id)
|
||||
if not task:
|
||||
raise BusinessError(ErrorCodes.CLEANING_TASK_NOT_FOUND, task_id)
|
||||
|
||||
await self._set_process(db, task)
|
||||
|
||||
instances = await self.operator_instance_repo.find_operator_by_instance_id(
|
||||
db, task_id
|
||||
)
|
||||
|
||||
# Batch query operators
|
||||
all_operators = await self.operator_service.get_operators(
|
||||
db=db, page=0, size=1000, categories=[], keyword=None, is_star=None
|
||||
)
|
||||
operator_map = {op.id: op for op in all_operators}
|
||||
|
||||
task.instance = []
|
||||
for inst in instances:
|
||||
operator = operator_map.get(inst.operator_id)
|
||||
if operator:
|
||||
# 创建 DTO 并填充基本信息
|
||||
operator_dto = OperatorInstanceDto(
|
||||
id=operator.id,
|
||||
name=operator.name,
|
||||
description=operator.description,
|
||||
inputs=operator.inputs,
|
||||
outputs=operator.outputs,
|
||||
settings=json.loads(operator.settings) if operator.settings else {},
|
||||
categories=operator.categories,
|
||||
)
|
||||
|
||||
# 🔴 核心修复：解析并应用 overrides
|
||||
if inst.settings_override:
|
||||
try:
|
||||
operator_dto.overrides = json.loads(inst.settings_override)
|
||||
except (json.JSONDecodeError, TypeError) as e:
|
||||
logger.error(
|
||||
f"Failed to parse settings_override for {inst.operator_id}: {e}"
|
||||
)
|
||||
|
||||
task.instance.append(operator_dto)
|
||||
else:
|
||||
# 容错：如果找不到算子元数据
|
||||
operator_dto = OperatorInstanceDto(id=inst.operator_id)
|
||||
if inst.settings_override:
|
||||
try:
|
||||
operator_dto.overrides = json.loads(inst.settings_override)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
pass
|
||||
task.instance.append(operator_dto)
|
||||
|
||||
return task
|
||||
|
||||
|
||||
|
||||
async def create_task(
|
||||
self, db: AsyncSession, request: CreateCleaningTaskRequest
|
||||
@@ -355,6 +414,7 @@ class CleaningTaskService:
|
||||
logger.error(f"Failed to parse runtime config: {e}")
|
||||
return {}
|
||||
|
||||
'''
|
||||
async def scan_dataset(
|
||||
self,
|
||||
db: AsyncSession,
|
||||
@@ -399,6 +459,72 @@ class CleaningTaskService:
|
||||
"metadata": metadata_dict,
|
||||
}
|
||||
f.write(json.dumps(file_info, ensure_ascii=False) + "\n")
|
||||
'''
|
||||
|
||||
async def scan_dataset(
|
||||
self,
|
||||
db: AsyncSession,
|
||||
task_id: str,
|
||||
src_dataset_id: str,
|
||||
succeed_files: Set[str] | None = None,
|
||||
) -> None:
|
||||
"""Scan source dataset and create dataset.jsonl"""
|
||||
target_file_path = Path(f"{FLOW_PATH}/{task_id}/dataset.jsonl")
|
||||
target_file_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger.info(f"🔍 [Scan] 开始扫描数据集 {src_dataset_id} ...")
|
||||
|
||||
query = text("""
|
||||
SELECT id, file_name, file_path, file_type, file_size, metadata
|
||||
FROM t_dm_dataset_files
|
||||
WHERE dataset_id = :dataset_id
|
||||
ORDER BY created_at
|
||||
""")
|
||||
|
||||
result = await db.execute(query, {"dataset_id": src_dataset_id})
|
||||
files = result.fetchall()
|
||||
|
||||
logger.info(f"🔥 [Scan] 任务所属数据集: {src_dataset_id}")
|
||||
logger.info(f"🔥 [Scan] 数据库中文件数: {len(files)}") # 🔴 加这行
|
||||
|
||||
total_found = len(files)
|
||||
skipped_count = 0
|
||||
written_count = 0
|
||||
if total_found == 0:
|
||||
logger.warning("⚠️ [Scan] 数据库中没有找到文件！请检查数据集是否正确上传了文件。")
|
||||
|
||||
with open(target_file_path, "w", encoding="utf-8") as f:
|
||||
for file in files:
|
||||
# 核心逻辑：如果文件已处理过，则跳过
|
||||
if succeed_files and file.id in succeed_files:
|
||||
skipped_count += 1
|
||||
continue
|
||||
|
||||
written_count += 1
|
||||
metadata_dict = {}
|
||||
if file.metadata:
|
||||
try:
|
||||
parsed = json.loads(file.metadata)
|
||||
if isinstance(parsed, dict):
|
||||
metadata_dict = parsed
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
pass
|
||||
|
||||
file_info = {
|
||||
"fileId": file.id,
|
||||
"fileName": file.file_name,
|
||||
"filePath": file.file_path,
|
||||
"fileType": file.file_type,
|
||||
"fileSize": file.file_size,
|
||||
"metadata": metadata_dict,
|
||||
}
|
||||
f.write(json.dumps(file_info, ensure_ascii=False) + "\n")
|
||||
|
||||
logger.info(f"✅ [Scan] 扫描完成: 共 {total_found} 个 | 跳过 {skipped_count} 个 | 写入 {written_count} 个。")
|
||||
|
||||
if written_count == 0 and total_found > 0:
|
||||
logger.warning("⚠️ [Scan] 所有文件都被跳过（因为之前已处理成功）。如果修改了参数，请清空历史结果或添加新文件。")
|
||||
|
||||
|
||||
async def get_task_results(
|
||||
self, db: AsyncSession, task_id: str
|
||||
@@ -406,17 +532,44 @@ class CleaningTaskService:
|
||||
"""Get task results"""
|
||||
return await self.result_repo.find_by_instance_id(db, task_id)
|
||||
|
||||
def _resolve_log_path(self, task_id: str, retry_count: int) -> Path | None:
|
||||
"""
|
||||
解析日志路径（严格模式）：
|
||||
只查找特定的 output.log 或 output.log.N。
|
||||
如果文件不存在，返回 None，让调用者（前端或路由）处理等待逻辑。
|
||||
"""
|
||||
safe_task_id = _sanitize_task_id(task_id)
|
||||
flow_root = Path(FLOW_PATH).resolve()
|
||||
task_dir = flow_root / safe_task_id
|
||||
|
||||
# 构建目标文件名
|
||||
target_name = "output.log" if retry_count == 0 else f"output.log.{retry_count}"
|
||||
target_path = task_dir / target_name
|
||||
|
||||
# 严格检查：存在则返回，不存在则返回 None
|
||||
if target_path.exists():
|
||||
return target_path
|
||||
|
||||
return None
|
||||
|
||||
async def get_task_log(
|
||||
self, db: AsyncSession, task_id: str, retry_count: int
|
||||
) -> List[CleaningTaskLog]:
|
||||
"""Get task log"""
|
||||
"""Get task log (严格匹配模式，禁止兜底旧日志)"""
|
||||
safe_task_id = _sanitize_task_id(task_id)
|
||||
safe_retry_count = _sanitize_retry_count(retry_count)
|
||||
|
||||
flow_root = Path(FLOW_PATH).resolve()
|
||||
log_path = flow_root / safe_task_id / "output.log"
|
||||
if safe_retry_count > 0:
|
||||
log_path = flow_root / safe_task_id / f"output.log.{safe_retry_count}"
|
||||
task_dir = flow_root / safe_task_id
|
||||
|
||||
# 1. 严格定位指定的日志文件路径
|
||||
log_path = task_dir / ("output.log" if safe_retry_count == 0 else f"output.log.{safe_retry_count}")
|
||||
|
||||
# 🔴 核心修改：移除 glob 兜底逻辑
|
||||
# 如果文件不存在，说明 Runtime 尚未生成或正在初始化写入。
|
||||
# 直接返回空列表，由前端根据 TaskStatus 显示 Loading/等待状态。
|
||||
if not log_path.exists():
|
||||
return []
|
||||
|
||||
# 防止路径穿越：规范化后校验仍在 FLOW_PATH 下
|
||||
resolved_log_path = log_path.resolve()
|
||||
@@ -424,6 +577,7 @@ class CleaningTaskService:
|
||||
logger.warning(f"Path traversal attempt detected: task_id={task_id}")
|
||||
return []
|
||||
|
||||
# 二次校验（安全冗余）
|
||||
if not resolved_log_path.exists():
|
||||
return []
|
||||
|
||||
@@ -444,6 +598,7 @@ class CleaningTaskService:
|
||||
|
||||
return logs
|
||||
|
||||
|
||||
def _get_log_level(
|
||||
self, line: str, default_level: str, std_pattern, ex_pattern
|
||||
) -> str:
|
||||
@@ -503,9 +658,16 @@ class CleaningTaskService:
|
||||
|
||||
async def execute_task(self, db: AsyncSession, task_id: str) -> bool:
|
||||
"""Execute task"""
|
||||
logger.info(f"🚀 [Service] 任务 {task_id} 开始执行...")
|
||||
|
||||
# 目前是全量扫描源数据集，并不考虑已经处理过的文件
|
||||
succeed_set = None
|
||||
'''
|
||||
succeeded = await self.result_repo.find_by_instance_id(db, task_id, "COMPLETED")
|
||||
succeed_set = {res.src_file_id for res in succeeded}
|
||||
|
||||
logger.info(f"📊 [Execute] 发现 {len(succeed_set)} 个历史成功文件，将采用增量模式（跳过已处理文件）。")
|
||||
'''
|
||||
|
||||
task = await self.task_repo.find_task_by_id(db, task_id)
|
||||
if not task:
|
||||
raise BusinessError(ErrorCodes.CLEANING_TASK_NOT_FOUND, task_id)
|
||||
@@ -516,13 +678,16 @@ class CleaningTaskService:
|
||||
task.file_count = src_dataset.fileCount
|
||||
await self.task_repo.update_task(db, task)
|
||||
|
||||
await self.scan_dataset(db, task_id, task.src_dataset_id, succeed_set)
|
||||
await self.scan_dataset(db, task_id, task.src_dataset_id, succeed_files=succeed_set)
|
||||
await self.result_repo.delete_by_instance_id(db, task_id, "FAILED")
|
||||
|
||||
try:
|
||||
return await self.scheduler.execute_task(
|
||||
logger.info(f"[Service] Calling scheduler.execute_task for {task_id}, retry: {(task.retry_count or 0) + 1}")
|
||||
result = await self.scheduler.execute_task(
|
||||
db, task_id, (task.retry_count or 0) + 1
|
||||
)
|
||||
logger.info(f"[Service] Scheduler returned result for {task_id}: {result}")
|
||||
return result
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"execute_task failed, task_id={task_id}, retry_count={(task.retry_count or 0) + 1}: {e}"
|
||||
@@ -576,3 +741,109 @@ class CleaningTaskService:
|
||||
result.append(operator_dto)
|
||||
|
||||
return result
|
||||
|
||||
async def update_task_config(self, db: AsyncSession, task_id: str, data: dict) -> None:
|
||||
"""
|
||||
更新任务配置（全量更新算子列表和参数）
|
||||
"""
|
||||
# 1. 检查任务是否存在
|
||||
task = await self.task_repo.find_task_by_id(db, task_id)
|
||||
if not task:
|
||||
raise BusinessError(ErrorCodes.CLEANING_TASK_NOT_FOUND, task_id)
|
||||
|
||||
# 2. 运行中的任务不可修改
|
||||
if task.status == CleaningTaskStatus.RUNNING:
|
||||
raise BusinessError(
|
||||
ErrorCodes.CLEANING_TASK_STATUS_INVALID,
|
||||
"任务正在运行中，无法修改配置。请先停止任务。"
|
||||
)
|
||||
|
||||
# 3. 更新任务基本信息 (名称、描述)
|
||||
if "name" in data:
|
||||
task.name = data["name"]
|
||||
if "description" in data:
|
||||
task.description = data["description"]
|
||||
|
||||
# 🔴 修复：删除 task.updated_at = datetime.now()
|
||||
# 因为 CleaningTaskDto 中没有 updated_at 字段
|
||||
|
||||
await self.task_repo.update_task(db, task)
|
||||
|
||||
# 4. 更新算子配置 (先删后插)
|
||||
instances_data = data.get("instance", [])
|
||||
|
||||
# 4a. 删除旧的算子关联
|
||||
await self.operator_instance_repo.delete_by_instance_id(db, task_id)
|
||||
|
||||
if instances_data:
|
||||
# 4b. 准备写入数据
|
||||
wrapped_instances = []
|
||||
|
||||
for d in instances_data:
|
||||
op_id = d.get("id")
|
||||
if not op_id:
|
||||
logger.warning(f"算子 ID 缺失，跳过")
|
||||
continue
|
||||
|
||||
# 确保 overrides 不为 None
|
||||
overrides = d.get("overrides")
|
||||
if overrides is None:
|
||||
overrides = {}
|
||||
|
||||
# 创建一个简单的对象适配 Repository 的 insert_instance 方法
|
||||
class InstanceWrapper:
|
||||
def __init__(self, oid, over):
|
||||
self.id = oid
|
||||
self.overrides = over
|
||||
|
||||
wrapped_instances.append(InstanceWrapper(op_id, overrides))
|
||||
|
||||
# 写入数据库
|
||||
if wrapped_instances:
|
||||
await self.operator_instance_repo.insert_instance(db, task_id, wrapped_instances)
|
||||
|
||||
# 5. 重新生成 Process YAML 文件
|
||||
try:
|
||||
all_operators = await self.operator_service.get_operators(
|
||||
db=db, page=0, size=1000, categories=[], keyword=None, is_star=None
|
||||
)
|
||||
op_map = {op.id: op for op in all_operators}
|
||||
|
||||
enriched_instances = []
|
||||
for d in instances_data:
|
||||
op_id = d.get("id")
|
||||
if not op_id: continue
|
||||
|
||||
op = op_map.get(op_id)
|
||||
if op:
|
||||
enriched_instances.append(
|
||||
OperatorInstanceDto(
|
||||
id=op_id,
|
||||
name=d.get("name", op.name),
|
||||
inputs=op.inputs,
|
||||
outputs=op.outputs,
|
||||
settings=op.settings,
|
||||
overrides=d.get("overrides", {}),
|
||||
categories=op.categories,
|
||||
)
|
||||
)
|
||||
else:
|
||||
# 容错：如果找不到元数据，使用基础信息
|
||||
enriched_instances.append(
|
||||
OperatorInstanceDto(
|
||||
id=op_id,
|
||||
name=d.get("name", "Unknown Operator"),
|
||||
inputs="",
|
||||
outputs="",
|
||||
overrides=d.get("overrides", {}),
|
||||
)
|
||||
)
|
||||
|
||||
executor_type = self.validator.check_and_get_executor_type(enriched_instances)
|
||||
|
||||
# 复用已有的 prepare_task 生成 YAML
|
||||
await self.prepare_task(task.dest_dataset_id, task_id, enriched_instances, op_map, executor_type)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"生成 YAML 配置失败: {e}", exc_info=True)
|
||||
raise BusinessError(ErrorCodes.CLEANING_FILE_SYSTEM_ERROR, f"生成配置文件失败: {str(e)}")
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import re
|
||||
import re
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
|
||||
@@ -1,124 +1,114 @@
|
||||
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from fastapi.responses import FileResponse
|
||||
from sqlalchemy import select, func
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.db.models.data_collection import TaskExecution
|
||||
from app.db.session import get_db
|
||||
from app.module.collection.schema.collection import TaskExecutionBase, converter_execution_to_response
|
||||
from app.module.shared.schema import StandardResponse, PaginatedData
|
||||
|
||||
router = APIRouter(
|
||||
prefix="/executions",
|
||||
tags=["data-collection/executions"],
|
||||
)
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
@router.get("", response_model=StandardResponse[PaginatedData[TaskExecutionBase]])
|
||||
async def list_executions(
|
||||
page: int = 1,
|
||||
size: int = 20,
|
||||
task_id: Optional[str] = Query(None, description="Task ID"),
|
||||
task_name: Optional[str] = Query(None, description="Fuzzy search by task name"),
|
||||
status: Optional[str] = Query(None, description="Filter by execution status"),
|
||||
start_time: Optional[datetime] = Query(None, description="Start time range from (started_at >= start_time)"),
|
||||
end_time: Optional[datetime] = Query(None, description="Start time range to (started_at <= end_time)"),
|
||||
db: AsyncSession = Depends(get_db)
|
||||
):
|
||||
"""分页查询归集任务执行记录"""
|
||||
try:
|
||||
query = select(TaskExecution)
|
||||
|
||||
if task_id:
|
||||
query = query.where(TaskExecution.task_id == task_id)
|
||||
|
||||
if task_name:
|
||||
query = query.where(TaskExecution.task_name.ilike(f"%{task_name}%"))
|
||||
|
||||
if status:
|
||||
query = query.where(TaskExecution.status == status)
|
||||
|
||||
if start_time:
|
||||
query = query.where(TaskExecution.started_at >= start_time)
|
||||
|
||||
if end_time:
|
||||
query = query.where(TaskExecution.started_at <= end_time)
|
||||
|
||||
count_query = select(func.count()).select_from(query.subquery())
|
||||
total = (await db.execute(count_query)).scalar_one()
|
||||
|
||||
offset = (page - 1) * size
|
||||
executions = (await db.execute(
|
||||
query.order_by(TaskExecution.created_at.desc())
|
||||
.offset(offset)
|
||||
.limit(size)
|
||||
)).scalars().all()
|
||||
|
||||
items = [converter_execution_to_response(exe) for exe in executions]
|
||||
total_pages = math.ceil(total / size) if total > 0 else 0
|
||||
|
||||
return StandardResponse(
|
||||
code="0",
|
||||
message="Success",
|
||||
data=PaginatedData(
|
||||
content=items,
|
||||
total_elements=total,
|
||||
total_pages=total_pages,
|
||||
page=page,
|
||||
size=size,
|
||||
)
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to list task executions: {str(e)}", e)
|
||||
raise HTTPException(status_code=500, detail="Internal server error")
|
||||
|
||||
|
||||
@router.get("/{execution_id}/log")
|
||||
async def get_execution_log(
|
||||
execution_id: str,
|
||||
db: AsyncSession = Depends(get_db)
|
||||
):
|
||||
"""Get log file content for execution record"""
|
||||
try:
|
||||
execution = await db.get(TaskExecution, execution_id)
|
||||
if not execution:
|
||||
raise HTTPException(status_code=404, detail="Execution record not found")
|
||||
|
||||
log_path = getattr(execution, "log_path", None)
|
||||
if not log_path:
|
||||
raise HTTPException(status_code=404, detail="Log path not found")
|
||||
|
||||
path = Path(str(log_path))
|
||||
if not path.is_absolute():
|
||||
path = Path(os.getcwd()) / path
|
||||
path = path.resolve()
|
||||
|
||||
if not path.exists() or not path.is_file():
|
||||
raise HTTPException(status_code=404, detail="Log file not found")
|
||||
|
||||
filename = path.name
|
||||
headers = {
|
||||
"Content-Disposition": f'inline; filename={filename}'
|
||||
}
|
||||
return FileResponse(
|
||||
path=str(path),
|
||||
media_type="text/plain; charset=utf-8",
|
||||
filename=filename,
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get execution log: {str(e)}", e)
|
||||
raise HTTPException(status_code=500, detail="Internal server error")
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from fastapi.responses import FileResponse, PlainTextResponse # 👈 核心修复：确保导入了 PlainTextResponse
|
||||
from sqlalchemy import select, func
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.db.models.data_collection import TaskExecution
|
||||
from app.db.session import get_db
|
||||
from app.module.collection.schema.collection import TaskExecutionBase, converter_execution_to_response
|
||||
from app.module.shared.schema import StandardResponse, PaginatedData
|
||||
|
||||
router = APIRouter(
|
||||
prefix="/executions",
|
||||
tags=["data-collection/executions"],
|
||||
)
|
||||
logger = get_logger(__name__)
|
||||
|
||||
@router.get("", response_model=StandardResponse[PaginatedData[TaskExecutionBase]])
|
||||
async def list_executions(
|
||||
page: int = 1,
|
||||
size: int = 20,
|
||||
task_id: Optional[str] = Query(None, description="Task ID"),
|
||||
task_name: Optional[str] = Query(None, description="Fuzzy search by task name"),
|
||||
status: Optional[str] = Query(None, description="Filter by execution status"),
|
||||
start_time: Optional[datetime] = Query(None, description="Start time range from (started_at >= start_time)"),
|
||||
end_time: Optional[datetime] = Query(None, description="Start time range to (started_at <= end_time)"),
|
||||
db: AsyncSession = Depends(get_db)
|
||||
):
|
||||
"""分页查询归集任务执行记录"""
|
||||
try:
|
||||
query = select(TaskExecution)
|
||||
if task_id:
|
||||
query = query.where(TaskExecution.task_id == task_id)
|
||||
if task_name:
|
||||
query = query.where(TaskExecution.task_name.ilike(f"%{task_name}%"))
|
||||
if status:
|
||||
query = query.where(TaskExecution.status == status)
|
||||
if start_time:
|
||||
query = query.where(TaskExecution.started_at >= start_time)
|
||||
if end_time:
|
||||
query = query.where(TaskExecution.started_at <= end_time)
|
||||
|
||||
count_query = select(func.count()).select_from(query.subquery())
|
||||
total = (await db.execute(count_query)).scalar_one()
|
||||
offset = (page - 1) * size
|
||||
executions = (await db.execute(
|
||||
query.order_by(TaskExecution.created_at.desc()).offset(offset).limit(size)
|
||||
)).scalars().all()
|
||||
|
||||
items = [converter_execution_to_response(exe) for exe in executions]
|
||||
total_pages = math.ceil(total / size) if total > 0 else 0
|
||||
|
||||
return StandardResponse(
|
||||
code="0", message="Success",
|
||||
data=PaginatedData(content=items, total_elements=total, total_pages=total_pages, page=page, size=size)
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to list task executions: {str(e)}", e)
|
||||
raise HTTPException(status_code=500, detail="Internal server error")
|
||||
|
||||
@router.get("/{execution_id}/log")
|
||||
async def get_execution_log(
|
||||
execution_id: str,
|
||||
db: AsyncSession = Depends(get_db)
|
||||
):
|
||||
"""Get log file content for execution record"""
|
||||
try:
|
||||
execution = await db.get(TaskExecution, execution_id)
|
||||
if not execution:
|
||||
raise HTTPException(status_code=404, detail="Execution record not found")
|
||||
|
||||
log_path = getattr(execution, "log_path", None)
|
||||
if not log_path:
|
||||
raise HTTPException(status_code=404, detail="Log path not found")
|
||||
|
||||
path = Path(str(log_path))
|
||||
if not path.is_absolute():
|
||||
path = Path(os.getcwd()) / path
|
||||
path = path.resolve()
|
||||
|
||||
# ✅ 核心修复：文件尚未生成时，返回友好提示（HTTP 200），而不是崩溃（HTTP 500）
|
||||
if not path.exists() or not path.is_file():
|
||||
return PlainTextResponse(
|
||||
content="⏳ 日志文件尚未生成。任务正在排队或 Runtime 初始化中，请稍后刷新查看...",
|
||||
media_type="text/plain; charset=utf-8"
|
||||
)
|
||||
|
||||
filename = path.name
|
||||
headers = {
|
||||
"Content-Disposition": f'inline; filename="{filename}"'
|
||||
}
|
||||
return FileResponse(
|
||||
path=str(path),
|
||||
media_type="text/plain; charset=utf-8",
|
||||
filename=filename,
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
# 🛡️ 防御性编程：万一还有其他错误，返回文本提示，不让浏览器报 500 白屏
|
||||
logger.error(f"Failed to get execution log: {str(e)}", e)
|
||||
return PlainTextResponse(
|
||||
content=f"Error reading log: {str(e)}",
|
||||
media_type="text/plain; charset=utf-8",
|
||||
status_code=500
|
||||
)
|
||||
|
||||
@@ -1,281 +1,281 @@
|
||||
import os
|
||||
import yaml
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.db.models import Dataset
|
||||
# ✅ 确保引入 TaskExecution
|
||||
from app.db.models.data_collection import CollectionTask, CollectionTemplate, TaskExecution
|
||||
from app.db.session import AsyncSessionLocal
|
||||
from app.module.collection.client.datax_client import DataxClient
|
||||
from app.module.collection.schema.collection import SyncMode, create_execute_record
|
||||
from app.module.collection.schedule import validate_schedule_expression
|
||||
from app.module.dataset.service.service import Service
|
||||
from app.module.shared.schema import TaskStatus, NodeType, EdgeType
|
||||
from app.module.shared.common.lineage import LineageService
|
||||
from app.db.models.base_entity import LineageNode, LineageEdge
|
||||
# ✅ 确保引入 RuntimeClient
|
||||
from app.module.shared.runtime_client import RuntimeClient
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
# 执行模式开关：'local' (默认) | 'runtime'
|
||||
COLLECTION_EXECUTION_MODE = os.getenv("COLLECTION_EXECUTION_MODE", "local")
|
||||
|
||||
class _TempExecution:
|
||||
"""用于生成配置时的临时占位符"""
|
||||
def __init__(self, task_id):
|
||||
self.id = task_id
|
||||
|
||||
class CollectionTaskService:
|
||||
def __init__(self, db: AsyncSession):
|
||||
self.db = db
|
||||
|
||||
async def create_task(self, task: CollectionTask, dataset: Dataset) -> CollectionTask:
|
||||
self.db.add(task)
|
||||
if task.sync_mode == SyncMode.ONCE:
|
||||
task.status = TaskStatus.RUNNING.name
|
||||
await self.db.commit()
|
||||
asyncio.create_task(CollectionTaskService.run_async(task.id, dataset.id if dataset else None))
|
||||
elif task.sync_mode == SyncMode.SCHEDULED:
|
||||
if not task.schedule_expression:
|
||||
raise ValueError("schedule_expression is required for scheduled tasks")
|
||||
validate_schedule_expression(task.schedule_expression)
|
||||
return task
|
||||
|
||||
@staticmethod
|
||||
async def run_async(task_id: str, dataset_id: str = None):
|
||||
"""任务执行入口"""
|
||||
logger.info(f"Start to execute task {task_id}")
|
||||
|
||||
# 1. 获取任务基础信息
|
||||
async with AsyncSessionLocal() as session:
|
||||
task_result = await session.execute(select(CollectionTask).where(CollectionTask.id == task_id))
|
||||
task = task_result.scalar_one_or_none()
|
||||
if not task:
|
||||
logger.error(f"task {task_id} not exist")
|
||||
return
|
||||
|
||||
template_result = await session.execute(select(CollectionTemplate).where(CollectionTemplate.id == task.template_id))
|
||||
template = template_result.scalar_one_or_none()
|
||||
if not template:
|
||||
logger.error(f"template {task.template_name} not exist")
|
||||
return
|
||||
|
||||
# 2. 创建执行记录 (此时 task_execution.log_path 已经根据 schema 规则被设置好)
|
||||
task_execution = create_execute_record(task)
|
||||
session.add(task_execution)
|
||||
await session.commit()
|
||||
await session.refresh(task_execution)
|
||||
exec_id = task_execution.id
|
||||
|
||||
# 3. 根据模式分发
|
||||
try:
|
||||
if COLLECTION_EXECUTION_MODE == "runtime":
|
||||
# ✅ 关键修改：将 task_execution 对象传入，以便提取 execution_id
|
||||
await CollectionTaskService._execute_via_runtime(
|
||||
task_id, dataset_id, task, template, task_execution
|
||||
)
|
||||
|
||||
# 提交成功后更新状态为 SUBMITTED
|
||||
async with AsyncSessionLocal() as update_session:
|
||||
exec_record = await update_session.get(TaskExecution, exec_id)
|
||||
if exec_record:
|
||||
exec_record.status = "SUBMITTED"
|
||||
await update_session.commit()
|
||||
|
||||
logger.info(f"Task {task_id} submitted to Runtime. Starting background poller.")
|
||||
# 启动后台轮询
|
||||
asyncio.create_task(
|
||||
CollectionTaskService._poll_runtime_status(task_id, dataset_id, exec_id)
|
||||
)
|
||||
else:
|
||||
# Local 模式
|
||||
await CollectionTaskService._execute_locally(task_id, dataset_id, task, template, exec_id)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Task execution failed for {task_id}: {e}")
|
||||
async with AsyncSessionLocal() as fail_session:
|
||||
exec_record = await fail_session.get(TaskExecution, exec_id)
|
||||
if exec_record:
|
||||
exec_record.status = TaskStatus.FAILED.name
|
||||
exec_record.error_message = str(e)
|
||||
await fail_session.commit()
|
||||
|
||||
# ==========================================
|
||||
# 🟢 模式 A：本地执行逻辑
|
||||
# ==========================================
|
||||
@staticmethod
|
||||
async def _execute_locally(task_id, dataset_id, task, template, exec_id):
|
||||
"""本地直接运行 DataX"""
|
||||
logger.info(f"[Local Mode] Executing DataX for task {task_id}")
|
||||
|
||||
async with AsyncSessionLocal() as session:
|
||||
exec_record = await session.get(TaskExecution, exec_id)
|
||||
# DataX 运行是阻塞的，放在线程池中运行
|
||||
datax_client = DataxClient(task=task, execution=exec_record, template=template)
|
||||
await asyncio.to_thread(datax_client.run_datax_job)
|
||||
|
||||
# 同步 Task 状态
|
||||
task.status = exec_record.status
|
||||
session.add(task)
|
||||
|
||||
if exec_record.status == TaskStatus.COMPLETED.name:
|
||||
await CollectionTaskService._finalize_task(session, task_id, dataset_id, task)
|
||||
|
||||
await session.commit()
|
||||
|
||||
# ==========================================
|
||||
# 🔵 模式 B：Runtime -> Ray 执行逻辑
|
||||
# ==========================================
|
||||
@staticmethod
|
||||
async def _execute_via_runtime(task_id, dataset_id, task, template, task_execution):
|
||||
"""
|
||||
生成配置并提交给 Runtime。
|
||||
不需要 session，因为 task_execution 对象已包含所有必要信息。
|
||||
"""
|
||||
logger.info(f"[Runtime Mode] Submitting task {task_id} to deepdata-runtime")
|
||||
|
||||
# ✅ 关键步骤：提取 execution_id，这是日志文件名的关键
|
||||
execution_id = task_execution.id
|
||||
log_path = task_execution.log_path
|
||||
|
||||
# 1. 生成 DataX JSON 配置
|
||||
datax_client = DataxClient(task=task, execution=_TempExecution(task_id), template=template)
|
||||
datax_json_path = datax_client.generate_config()
|
||||
|
||||
# 2. 构造 process.yaml 指令文件
|
||||
flow_dir = Path(f"/flow/{task_id}")
|
||||
flow_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
process_cfg = {
|
||||
"executor_type": "deepdata",
|
||||
"instance_id": task_id,
|
||||
"execution_id": execution_id,
|
||||
"log_path": log_path,
|
||||
"dataset_id": dataset_id,
|
||||
"cli_config": {
|
||||
"command": "python",
|
||||
"args": ["/opt/datax/bin/datax.py", datax_json_path],
|
||||
"timeout": 7200
|
||||
}
|
||||
}
|
||||
|
||||
# 写入指令文件
|
||||
with open(flow_dir / "process.yaml", "w", encoding="utf-8") as f:
|
||||
yaml.safe_dump(process_cfg, f)
|
||||
|
||||
# 3. 提交给 Runtime
|
||||
runtime_client = RuntimeClient()
|
||||
success = await runtime_client.submit_task(task_id)
|
||||
if not success:
|
||||
raise RuntimeError("Runtime submission returned False")
|
||||
|
||||
@staticmethod
|
||||
async def _poll_runtime_status(task_id, dataset_id, exec_id):
|
||||
"""后台轮询 Runtime 状态"""
|
||||
runtime_client = RuntimeClient()
|
||||
|
||||
for _ in range(2000):
|
||||
try:
|
||||
status_info = await runtime_client.get_task_status(task_id)
|
||||
|
||||
if status_info:
|
||||
current_status = (status_info.get("status") or "").upper()
|
||||
|
||||
async with AsyncSessionLocal() as session:
|
||||
exec_record = await session.get(TaskExecution, exec_id)
|
||||
if not exec_record: return
|
||||
|
||||
task_result = await session.execute(select(CollectionTask).where(CollectionTask.id == task_id))
|
||||
task = task_result.scalar_one_or_none()
|
||||
|
||||
if current_status in ["COMPLETED", "SUCCEEDED", "SUCCESS"]:
|
||||
exec_record.status = TaskStatus.COMPLETED.name
|
||||
await session.commit()
|
||||
logger.info(f"Task {task_id} completed in Runtime. Finalizing.")
|
||||
if task:
|
||||
await CollectionTaskService._finalize_task(session, task_id, dataset_id, task)
|
||||
return
|
||||
|
||||
elif current_status in ["FAILED", "ERROR"]:
|
||||
exec_record.status = TaskStatus.FAILED.name
|
||||
exec_record.error_message = status_info.get("error", "Runtime task failed")
|
||||
await session.commit()
|
||||
logger.error(f"Task {task_id} failed in Runtime: {exec_record.error_message}")
|
||||
return
|
||||
|
||||
await asyncio.sleep(5)
|
||||
except Exception as e:
|
||||
logger.warning(f"Polling error for {task_id}: {e}")
|
||||
await asyncio.sleep(5)
|
||||
|
||||
# ==========================================
|
||||
# 🛠️ 公共收尾逻辑
|
||||
# ==========================================
|
||||
@staticmethod
|
||||
async def _finalize_task(session, task_id, dataset_id, task):
|
||||
"""任务执行成功后的后续处理"""
|
||||
if not dataset_id: return
|
||||
|
||||
# 🐛 修复：对于 S3/远端写入器，本地 target_path 可能不存在
|
||||
# 应当检查 task.target_path 是否为本地路径，或者跳过本地扫描逻辑
|
||||
target_path = Path(task.target_path)
|
||||
if target_path.exists() and target_path.is_dir():
|
||||
service = Service(db=session)
|
||||
source_paths = []
|
||||
for file_path in target_path.rglob('*'):
|
||||
if file_path.is_file():
|
||||
source_paths.append(str(file_path.absolute()))
|
||||
|
||||
if source_paths:
|
||||
await service.add_files_to_dataset(dataset_id=dataset_id, source_paths=source_paths)
|
||||
await CollectionTaskService._add_dataset_to_graph(session, dataset_id, task)
|
||||
else:
|
||||
logger.warning(f"[Finalize] Target path {target_path} does not exist locally. "
|
||||
f"Skipping dataset import. (This is expected for s3writer/nfswriter)")
|
||||
|
||||
@staticmethod
|
||||
async def _add_dataset_to_graph(session, dataset_id, task):
|
||||
"""将数据集加入血缘图"""
|
||||
try:
|
||||
dataset = await session.get(Dataset, dataset_id)
|
||||
if not dataset:
|
||||
logger.warning(f"dataset {dataset_id} not found when building lineage graph")
|
||||
return
|
||||
|
||||
dataset_node = LineageNode(
|
||||
id=dataset.id,
|
||||
node_type=NodeType.DATASET.value,
|
||||
name=dataset.name,
|
||||
description=dataset.description
|
||||
)
|
||||
|
||||
collection_node = LineageNode(
|
||||
id=task.id,
|
||||
node_type=NodeType.DATASOURCE.value,
|
||||
name=task.name,
|
||||
description=task.description
|
||||
)
|
||||
|
||||
collection_edge = LineageEdge(
|
||||
process_id=task.id,
|
||||
name=task.name,
|
||||
edge_type=EdgeType.DATA_COLLECTION.value,
|
||||
description=dataset.description,
|
||||
from_node_id=task.id,
|
||||
to_node_id=dataset.id
|
||||
)
|
||||
|
||||
lineage_service = LineageService(db=session)
|
||||
await lineage_service.generate_graph(collection_node, collection_edge, dataset_node)
|
||||
await session.commit()
|
||||
|
||||
except Exception as exc:
|
||||
logger.error(f"Failed to add dataset lineage graph: {exc}")
|
||||
await session.rollback()
|
||||
import os
|
||||
import yaml
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.db.models import Dataset
|
||||
# ✅ 确保引入 TaskExecution
|
||||
from app.db.models.data_collection import CollectionTask, CollectionTemplate, TaskExecution
|
||||
from app.db.session import AsyncSessionLocal
|
||||
from app.module.collection.client.datax_client import DataxClient
|
||||
from app.module.collection.schema.collection import SyncMode, create_execute_record
|
||||
from app.module.collection.schedule import validate_schedule_expression
|
||||
from app.module.dataset.service.service import Service
|
||||
from app.module.shared.schema import TaskStatus, NodeType, EdgeType
|
||||
from app.module.shared.common.lineage import LineageService
|
||||
from app.db.models.base_entity import LineageNode, LineageEdge
|
||||
# ✅ 确保引入 RuntimeClient
|
||||
from app.module.shared.runtime_client import RuntimeClient
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
# 执行模式开关：'local' (默认) | 'runtime'
|
||||
COLLECTION_EXECUTION_MODE = os.getenv("COLLECTION_EXECUTION_MODE", "local")
|
||||
|
||||
class _TempExecution:
|
||||
"""用于生成配置时的临时占位符"""
|
||||
def __init__(self, task_id):
|
||||
self.id = task_id
|
||||
|
||||
class CollectionTaskService:
|
||||
def __init__(self, db: AsyncSession):
|
||||
self.db = db
|
||||
|
||||
async def create_task(self, task: CollectionTask, dataset: Dataset) -> CollectionTask:
|
||||
self.db.add(task)
|
||||
if task.sync_mode == SyncMode.ONCE:
|
||||
task.status = TaskStatus.RUNNING.name
|
||||
await self.db.commit()
|
||||
asyncio.create_task(CollectionTaskService.run_async(task.id, dataset.id if dataset else None))
|
||||
elif task.sync_mode == SyncMode.SCHEDULED:
|
||||
if not task.schedule_expression:
|
||||
raise ValueError("schedule_expression is required for scheduled tasks")
|
||||
validate_schedule_expression(task.schedule_expression)
|
||||
return task
|
||||
|
||||
@staticmethod
|
||||
async def run_async(task_id: str, dataset_id: str = None):
|
||||
"""任务执行入口"""
|
||||
logger.info(f"Start to execute task {task_id}")
|
||||
|
||||
# 1. 获取任务基础信息
|
||||
async with AsyncSessionLocal() as session:
|
||||
task_result = await session.execute(select(CollectionTask).where(CollectionTask.id == task_id))
|
||||
task = task_result.scalar_one_or_none()
|
||||
if not task:
|
||||
logger.error(f"task {task_id} not exist")
|
||||
return
|
||||
|
||||
template_result = await session.execute(select(CollectionTemplate).where(CollectionTemplate.id == task.template_id))
|
||||
template = template_result.scalar_one_or_none()
|
||||
if not template:
|
||||
logger.error(f"template {task.template_name} not exist")
|
||||
return
|
||||
|
||||
# 2. 创建执行记录 (此时 task_execution.log_path 已经根据 schema 规则被设置好)
|
||||
task_execution = create_execute_record(task)
|
||||
session.add(task_execution)
|
||||
await session.commit()
|
||||
await session.refresh(task_execution)
|
||||
exec_id = task_execution.id
|
||||
|
||||
# 3. 根据模式分发
|
||||
try:
|
||||
if COLLECTION_EXECUTION_MODE == "runtime":
|
||||
# ✅ 关键修改：将 task_execution 对象传入，以便提取 execution_id
|
||||
await CollectionTaskService._execute_via_runtime(
|
||||
task_id, dataset_id, task, template, task_execution
|
||||
)
|
||||
|
||||
# 提交成功后更新状态为 SUBMITTED
|
||||
async with AsyncSessionLocal() as update_session:
|
||||
exec_record = await update_session.get(TaskExecution, exec_id)
|
||||
if exec_record:
|
||||
exec_record.status = "SUBMITTED"
|
||||
await update_session.commit()
|
||||
|
||||
logger.info(f"Task {task_id} submitted to Runtime. Starting background poller.")
|
||||
# 启动后台轮询
|
||||
asyncio.create_task(
|
||||
CollectionTaskService._poll_runtime_status(task_id, dataset_id, exec_id)
|
||||
)
|
||||
else:
|
||||
# Local 模式
|
||||
await CollectionTaskService._execute_locally(task_id, dataset_id, task, template, exec_id)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Task execution failed for {task_id}: {e}")
|
||||
async with AsyncSessionLocal() as fail_session:
|
||||
exec_record = await fail_session.get(TaskExecution, exec_id)
|
||||
if exec_record:
|
||||
exec_record.status = TaskStatus.FAILED.name
|
||||
exec_record.error_message = str(e)
|
||||
await fail_session.commit()
|
||||
|
||||
# ==========================================
|
||||
# 🟢 模式 A：本地执行逻辑
|
||||
# ==========================================
|
||||
@staticmethod
|
||||
async def _execute_locally(task_id, dataset_id, task, template, exec_id):
|
||||
"""本地直接运行 DataX"""
|
||||
logger.info(f"[Local Mode] Executing DataX for task {task_id}")
|
||||
|
||||
async with AsyncSessionLocal() as session:
|
||||
exec_record = await session.get(TaskExecution, exec_id)
|
||||
# DataX 运行是阻塞的，放在线程池中运行
|
||||
datax_client = DataxClient(task=task, execution=exec_record, template=template)
|
||||
await asyncio.to_thread(datax_client.run_datax_job)
|
||||
|
||||
# 同步 Task 状态
|
||||
task.status = exec_record.status
|
||||
session.add(task)
|
||||
|
||||
if exec_record.status == TaskStatus.COMPLETED.name:
|
||||
await CollectionTaskService._finalize_task(session, task_id, dataset_id, task)
|
||||
|
||||
await session.commit()
|
||||
|
||||
# ==========================================
|
||||
# 🔵 模式 B：Runtime -> Ray 执行逻辑
|
||||
# ==========================================
|
||||
@staticmethod
|
||||
async def _execute_via_runtime(task_id, dataset_id, task, template, task_execution):
|
||||
"""
|
||||
生成配置并提交给 Runtime。
|
||||
不需要 session，因为 task_execution 对象已包含所有必要信息。
|
||||
"""
|
||||
logger.info(f"[Runtime Mode] Submitting task {task_id} to deepdata-runtime")
|
||||
|
||||
# ✅ 关键步骤：提取 execution_id，这是日志文件名的关键
|
||||
execution_id = task_execution.id
|
||||
log_path = task_execution.log_path
|
||||
|
||||
# 1. 生成 DataX JSON 配置
|
||||
datax_client = DataxClient(task=task, execution=_TempExecution(task_id), template=template)
|
||||
datax_json_path = datax_client.generate_config()
|
||||
|
||||
# 2. 构造 process.yaml 指令文件
|
||||
flow_dir = Path(f"/flow/{task_id}")
|
||||
flow_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
process_cfg = {
|
||||
"executor_type": "deepdata",
|
||||
"instance_id": task_id,
|
||||
"execution_id": execution_id,
|
||||
"log_path": log_path,
|
||||
"dataset_id": dataset_id,
|
||||
"cli_config": {
|
||||
"command": "python",
|
||||
"args": ["/opt/datax/bin/datax.py", datax_json_path],
|
||||
"timeout": 7200
|
||||
}
|
||||
}
|
||||
|
||||
# 写入指令文件
|
||||
with open(flow_dir / "process.yaml", "w", encoding="utf-8") as f:
|
||||
yaml.safe_dump(process_cfg, f)
|
||||
|
||||
# 3. 提交给 Runtime
|
||||
runtime_client = RuntimeClient()
|
||||
success = await runtime_client.submit_task(task_id)
|
||||
if not success:
|
||||
raise RuntimeError("Runtime submission returned False")
|
||||
|
||||
@staticmethod
|
||||
async def _poll_runtime_status(task_id, dataset_id, exec_id):
|
||||
"""后台轮询 Runtime 状态"""
|
||||
runtime_client = RuntimeClient()
|
||||
|
||||
for _ in range(2000):
|
||||
try:
|
||||
status_info = await runtime_client.get_task_status(task_id)
|
||||
|
||||
if status_info:
|
||||
current_status = (status_info.get("status") or "").upper()
|
||||
|
||||
async with AsyncSessionLocal() as session:
|
||||
exec_record = await session.get(TaskExecution, exec_id)
|
||||
if not exec_record: return
|
||||
|
||||
task_result = await session.execute(select(CollectionTask).where(CollectionTask.id == task_id))
|
||||
task = task_result.scalar_one_or_none()
|
||||
|
||||
if current_status in ["COMPLETED", "SUCCEEDED", "SUCCESS"]:
|
||||
exec_record.status = TaskStatus.COMPLETED.name
|
||||
await session.commit()
|
||||
logger.info(f"Task {task_id} completed in Runtime. Finalizing.")
|
||||
if task:
|
||||
await CollectionTaskService._finalize_task(session, task_id, dataset_id, task)
|
||||
return
|
||||
|
||||
elif current_status in ["FAILED", "ERROR"]:
|
||||
exec_record.status = TaskStatus.FAILED.name
|
||||
exec_record.error_message = status_info.get("error", "Runtime task failed")
|
||||
await session.commit()
|
||||
logger.error(f"Task {task_id} failed in Runtime: {exec_record.error_message}")
|
||||
return
|
||||
|
||||
await asyncio.sleep(5)
|
||||
except Exception as e:
|
||||
logger.warning(f"Polling error for {task_id}: {e}")
|
||||
await asyncio.sleep(5)
|
||||
|
||||
# ==========================================
|
||||
# 🛠️ 公共收尾逻辑
|
||||
# ==========================================
|
||||
@staticmethod
|
||||
async def _finalize_task(session, task_id, dataset_id, task):
|
||||
"""任务执行成功后的后续处理"""
|
||||
if not dataset_id: return
|
||||
|
||||
# 🐛 修复：对于 S3/远端写入器，本地 target_path 可能不存在
|
||||
# 应当检查 task.target_path 是否为本地路径，或者跳过本地扫描逻辑
|
||||
target_path = Path(task.target_path)
|
||||
if target_path.exists() and target_path.is_dir():
|
||||
service = Service(db=session)
|
||||
source_paths = []
|
||||
for file_path in target_path.rglob('*'):
|
||||
if file_path.is_file():
|
||||
source_paths.append(str(file_path.absolute()))
|
||||
|
||||
if source_paths:
|
||||
await service.add_files_to_dataset(dataset_id=dataset_id, source_paths=source_paths)
|
||||
await CollectionTaskService._add_dataset_to_graph(session, dataset_id, task)
|
||||
else:
|
||||
logger.warning(f"[Finalize] Target path {target_path} does not exist locally. "
|
||||
f"Skipping dataset import. (This is expected for s3writer/nfswriter)")
|
||||
|
||||
@staticmethod
|
||||
async def _add_dataset_to_graph(session, dataset_id, task):
|
||||
"""将数据集加入血缘图"""
|
||||
try:
|
||||
dataset = await session.get(Dataset, dataset_id)
|
||||
if not dataset:
|
||||
logger.warning(f"dataset {dataset_id} not found when building lineage graph")
|
||||
return
|
||||
|
||||
dataset_node = LineageNode(
|
||||
id=dataset.id,
|
||||
node_type=NodeType.DATASET.value,
|
||||
name=dataset.name,
|
||||
description=dataset.description
|
||||
)
|
||||
|
||||
collection_node = LineageNode(
|
||||
id=task.id,
|
||||
node_type=NodeType.DATASOURCE.value,
|
||||
name=task.name,
|
||||
description=task.description
|
||||
)
|
||||
|
||||
collection_edge = LineageEdge(
|
||||
process_id=task.id,
|
||||
name=task.name,
|
||||
edge_type=EdgeType.DATA_COLLECTION.value,
|
||||
description=dataset.description,
|
||||
from_node_id=task.id,
|
||||
to_node_id=dataset.id
|
||||
)
|
||||
|
||||
lineage_service = LineageService(db=session)
|
||||
await lineage_service.generate_graph(collection_node, collection_edge, dataset_node)
|
||||
await session.commit()
|
||||
|
||||
except Exception as exc:
|
||||
logger.error(f"Failed to add dataset lineage graph: {exc}")
|
||||
await session.rollback()
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""
|
||||
"""
|
||||
Operator Market Constants
|
||||
算子市场常量定义
|
||||
"""
|
||||
|
||||
@@ -539,7 +539,8 @@ class OperatorService:
|
||||
setting_type = setting.get("type")
|
||||
|
||||
match setting_type:
|
||||
case "slider" | "switch" | "select" | "input" | "radio":
|
||||
# xuhuanqing
|
||||
case "slider" | "switch" | "select" | "input" | "radio" | "textarea":
|
||||
setting["defaultVal"] = value
|
||||
case "checkbox":
|
||||
setting["defaultVal"] = self._convert_to_list_string(value)
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
from __future__ import annotations
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
@@ -10,9 +10,9 @@ from app.core.config import settings
|
||||
from app.core.logging import setup_logging
|
||||
|
||||
|
||||
def _remove_deepdata_handlers(logger: logging.Logger) -> None:
|
||||
def _remove_DeepData_handlers(logger: logging.Logger) -> None:
|
||||
for handler in list(logger.handlers):
|
||||
if getattr(handler, "_deepdata_managed", False):
|
||||
if getattr(handler, "_DeepData_managed", False):
|
||||
logger.removeHandler(handler)
|
||||
handler.close()
|
||||
|
||||
@@ -40,7 +40,7 @@ def test_setup_logging_uses_rotating_file_handler_without_duplicates(
|
||||
file_handlers = [
|
||||
handler
|
||||
for handler in root_logger.handlers
|
||||
if getattr(handler, "_deepdata_managed", False)
|
||||
if getattr(handler, "_DeepData_managed", False)
|
||||
and isinstance(handler, RotatingFileHandler)
|
||||
]
|
||||
assert len(file_handlers) == 1
|
||||
@@ -50,11 +50,11 @@ def test_setup_logging_uses_rotating_file_handler_without_duplicates(
|
||||
"python-backend.1.log"
|
||||
)
|
||||
finally:
|
||||
_remove_deepdata_handlers(root_logger)
|
||||
_remove_deepdata_handlers(logging.getLogger("uvicorn"))
|
||||
_remove_deepdata_handlers(logging.getLogger("uvicorn.access"))
|
||||
_remove_deepdata_handlers(logging.getLogger("uvicorn.error"))
|
||||
_remove_deepdata_handlers(logging.getLogger("sqlalchemy.engine"))
|
||||
_remove_DeepData_handlers(root_logger)
|
||||
_remove_DeepData_handlers(logging.getLogger("uvicorn"))
|
||||
_remove_DeepData_handlers(logging.getLogger("uvicorn.access"))
|
||||
_remove_DeepData_handlers(logging.getLogger("uvicorn.error"))
|
||||
_remove_DeepData_handlers(logging.getLogger("sqlalchemy.engine"))
|
||||
|
||||
|
||||
def test_copytruncate_log_rotation_script_rotates_only_log_like_files(tmp_path: Path) -> None:
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import pytest
|
||||
import pytest
|
||||
|
||||
from app.core.exception import BusinessError
|
||||
from app.module.cleaning.schema.cleaning import OperatorInstanceDto
|
||||
@@ -39,7 +39,7 @@ def test_check_and_get_executor_type_raises_when_mixed_categories() -> None:
|
||||
CleaningTaskValidator.check_and_get_executor_type(instances)
|
||||
|
||||
|
||||
def test_check_and_get_executor_type_defaults_to_deepdata() -> None:
|
||||
def test_check_and_get_executor_type_defaults_to_DeepData() -> None:
|
||||
instances = [_op("a", None, None, None)]
|
||||
|
||||
executor = CleaningTaskValidator.check_and_get_executor_type(instances)
|
||||
|
||||
Reference in New Issue
Block a user