沃巴蒂
ff22ed3a8b
本次同步 09-12 至 09-28 的代码变更，主要内容： 新增功能 - 修改密码：后端新增 ChangePasswordRequest DTO 与UserController/UserService 对应接口，前端新增 ChangePasswordDialog 并在 Header 接入入口 - 文件预览组件重构：file-preview 目录拆分出 VideoPreview（支持 blobUrl 直传）、 ImagePreview、AudioPreview，index.tsx 统一调度各类型预览器 - 新增 my_ops 算子模块：markitdown_parser、mineru_pdf_parser、text_quality_evaluator - 新增 tools/minio-file-server MinIO 文件服务 功能增强 - 数据清洗：cleaning_task_service/scheduler/routes 重构， data_juicer_executor 与 wrapper 增强，OperatorTable/LogsTable/TaskDetail 同步调整 - 数据标注：annotation sync.py、project.py 接口与schema 更新 - 数据采集、算子市场、首页与菜单路由相关页面调整 - DataJuicer 算子接入（useOperatorOperations、dj_ops_schema） 结构调整 - deepflow/ 更名为 deep-flow/（419 个文件重命名） - 新增 helm 部署编排、GitHub Actions 之外的自建镜像构建脚本 未纳入本次提交 - scripts-backup/、runtime/python-executor-backup/、components_backup/（备份目录） - *.backup*、__pycache__ 产物 - zh_core_web_sm-3.8.0-py3-none-any.whl（46MB 构建产物） - DataX/（内嵌独立仓库，远程亦未收录其源码）
147 lines
4.5 KiB
Markdown
147 lines
4.5 KiB
Markdown
# DeepData Runtime
|
|
|
|
## Overview
|
|
|
|
DeepData Runtime provides core functionality for data processing, operator execution, and data collection, built on Python 3.12+ and the FastAPI framework.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
runtime/
|
|
├── deepdata-python/ # FastAPI backend service (port 18000)
|
|
├── python-executor/ # Ray distributed executor
|
|
├── ops/ # Operator ecosystem
|
|
├── datax/ # DataX data read/write framework
|
|
└── deep-flow/ # DeepFlow service
|
|
```
|
|
|
|
## Components
|
|
|
|
### 1. deepdata-python (FastAPI Backend)
|
|
**Port**: 18000
|
|
|
|
Core Python service providing:
|
|
- **Data Synthesis**: QA generation, document processing
|
|
- **Data Annotation**: Label Studio integration, auto-annotation
|
|
- **Data Evaluation**: Model evaluation, quality checks
|
|
- **Data Cleaning**: Data cleaning pipelines
|
|
- **Operator Marketplace**: Operator management, upload
|
|
- **RAG Indexing**: Vector indexing, knowledge base management
|
|
- **Data Collection**: Scheduled tasks, data source integration
|
|
|
|
**Technology Stack**:
|
|
- FastAPI 0.124+
|
|
- SQLAlchemy 2.0+ (async)
|
|
- Pydantic 2.12+
|
|
- PostgreSQL (via asyncpg)
|
|
- Milvus (via pymilvus)
|
|
- APScheduler (scheduled tasks)
|
|
|
|
### 2. python-executor (Ray Executor)
|
|
Ray distributed execution framework responsible for:
|
|
- **Operator Execution**: Execute data processing operators
|
|
- **Task Scheduling**: Async task management
|
|
- **Distributed Computing**: Multi-node parallel processing
|
|
|
|
**Technology Stack**:
|
|
- Ray 2.7.0
|
|
- FastAPI (executor API)
|
|
- Data-Juicer (data processing)
|
|
|
|
### 3. ops (Operator Ecosystem)
|
|
Operator ecosystem including:
|
|
- **filter**: Data filtering (deduplication, sensitive content, quality filtering)
|
|
- **mapper**: Data transformation (cleaning, normalization)
|
|
- **slicer**: Data slicing (text splitting, slide extraction)
|
|
- **formatter**: Format conversion (PDF → text, slide → JSON)
|
|
- **llms**: LLM operators (quality evaluation, condition checking)
|
|
- **annotation**: Annotation operators (object detection, segmentation)
|
|
|
|
**See**: `runtime/ops/README.md` for operator development guide
|
|
|
|
### 4. datax (DataX Framework)
|
|
DataX data read/write framework supporting multiple data sources:
|
|
- **Readers**: MySQL, PostgreSQL, Oracle, MongoDB, Elasticsearch, HDFS, S3, NFS, GlusterFS, API, etc.
|
|
- **Writers**: Same as above, supports writing to targets
|
|
|
|
**Technology Stack**: Java (Maven build)
|
|
|
|
### 5. deep-flow (DeepFlow Service)
|
|
DeepFlow service (see `conf.yaml` for configuration).
|
|
|
|
## Quick Start
|
|
|
|
### Prerequisites
|
|
- Python 3.12+
|
|
- Poetry (for deepdata-python)
|
|
- Ray 2.7.0+ (for python-executor)
|
|
|
|
### Run deepdata-python
|
|
```bash
|
|
cd runtime/deepdata-python
|
|
poetry install
|
|
poetry run uvicorn app.main:app --reload --port 18000
|
|
```
|
|
|
|
### Run python-executor
|
|
```bash
|
|
cd runtime/python-executor
|
|
poetry install
|
|
ray start --head
|
|
```
|
|
|
|
## Development
|
|
|
|
### deepdata-python Module Structure
|
|
```
|
|
app/
|
|
├── core/ # Logging, exception, config
|
|
├── db/
|
|
│ ├── models/ # SQLAlchemy models
|
|
│ └── session.py # Async session
|
|
├── module/
|
|
│ ├── annotation/ # Label Studio integration
|
|
│ ├── collection/ # Data collection
|
|
│ ├── cleaning/ # Data cleaning
|
|
│ ├── dataset/ # Dataset management
|
|
│ ├── evaluation/ # Model evaluation
|
|
│ ├── generation/ # QA synthesis
|
|
│ ├── operator/ # Operator marketplace
|
|
│ ├── rag/ # RAG indexing
|
|
│ └── shared/ # Shared schemas
|
|
└── main.py # FastAPI entry
|
|
```
|
|
|
|
### Code Conventions
|
|
- **Routes**: `APIRouter` in `interface/*.py`
|
|
- **Dependency Injection**: `Depends(get_db)` for session
|
|
- **Error Handling**: `raise BusinessError(ErrorCodes.XXX, context)`
|
|
- **Transactions**: `async with transaction(db):`
|
|
- **Models**: Extend `BaseEntity` (audit fields auto-filled)
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
cd runtime/deepdata-python
|
|
poetry run pytest
|
|
```
|
|
|
|
## Configuration
|
|
|
|
### Environment Variables
|
|
- `DATABASE_URL`: PostgreSQL connection string
|
|
- `LABEL_STUDIO_BASE_URL`: Label Studio URL
|
|
- `RAY_ENABLED`: Enable Ray executor
|
|
- `RAY_ADDRESS`: Ray cluster address
|
|
|
|
## Documentation
|
|
|
|
- **API Docs**: http://localhost:18000/redoc
|
|
- **Operator Guide**: See `runtime/ops/README.md` for operator development
|
|
|
|
## Related Links
|
|
|
|
- [FastAPI Documentation](https://fastapi.tiangolo.com/)
|
|
- [Ray Documentation](https://docs.ray.io/)
|
|
- [SQLAlchemy Documentation](https://docs.sqlalchemy.org/)
|