沃巴蒂
ff22ed3a8b
本次同步 09-12 至 09-28 的代码变更，主要内容： 新增功能 - 修改密码：后端新增 ChangePasswordRequest DTO 与UserController/UserService 对应接口，前端新增 ChangePasswordDialog 并在 Header 接入入口 - 文件预览组件重构：file-preview 目录拆分出 VideoPreview（支持 blobUrl 直传）、 ImagePreview、AudioPreview，index.tsx 统一调度各类型预览器 - 新增 my_ops 算子模块：markitdown_parser、mineru_pdf_parser、text_quality_evaluator - 新增 tools/minio-file-server MinIO 文件服务 功能增强 - 数据清洗：cleaning_task_service/scheduler/routes 重构， data_juicer_executor 与 wrapper 增强，OperatorTable/LogsTable/TaskDetail 同步调整 - 数据标注：annotation sync.py、project.py 接口与schema 更新 - 数据采集、算子市场、首页与菜单路由相关页面调整 - DataJuicer 算子接入（useOperatorOperations、dj_ops_schema） 结构调整 - deepflow/ 更名为 deep-flow/（419 个文件重命名） - 新增 helm 部署编排、GitHub Actions 之外的自建镜像构建脚本 未纳入本次提交 - scripts-backup/、runtime/python-executor-backup/、components_backup/（备份目录） - *.backup*、__pycache__ 产物 - zh_core_web_sm-3.8.0-py3-none-any.whl（46MB 构建产物） - DataX/（内嵌独立仓库，远程亦未收录其源码）
DeepData Backend
Overview
DeepData Backend is a microservices architecture based on Spring Boot 3.5 + Java 21, providing core functions such as data management, RAG indexing, and API gateway.
Architecture
backend/
├── api-gateway/ # API Gateway + Authentication
├── services/
│ ├── data-management-service/ # Dataset management
│ ├── rag-indexer-service/ # RAG indexing
│ └── main-application/ # Main application entry
└── shared/
├── domain-common/ # DDD building blocks, exception handling
└── security-common/ # JWT utilities
Services
| Service | Port | Description |
|---|---|---|
| main-application | 8080 | Main application, includes data management, data cleaning, operator marketplace modules |
| api-gateway | 8080 | API Gateway, route forwarding and authentication |
Technology Stack
- Framework: Spring Boot 3.5.6, Spring Cloud 2025.0.0
- Language: Java 21
- Database: PostgreSQL 8.0.33 + MyBatis-Plus 3.5.14
- Cache: Redis 3.2.0
- Vector DB: Milvus (via SDK 2.6.6)
- Documentation: SpringDoc OpenAPI 2.2.0
- Build: Maven
Dependencies
External Services
- PostgreSQL:
deepdata-database:5432 - Redis:
deepdata-redis:6379 - Milvus: Vector database (RAG indexing)
Shared Libraries
- domain-common: Business exceptions, system parameters, domain entity base classes
- security-common: JWT utilities, auth helpers
Quick Start
Prerequisites
- JDK 21+
- Maven 3.8+
- PostgreSQL 12+
- Redis 6+
Build
cd backend
mvn clean install
Run Main Application
cd backend/services/main-application
mvn spring-boot:run
Run API Gateway
cd backend/api-gateway
mvn spring-boot:run
Development
Module Structure (DDD)
com.deepdata.{module}/
├── interfaces/
│ ├── rest/ # Controllers
│ ├── dto/ # Request/Response DTOs
│ ├── converter/ # MapStruct converters
│ └── validation/ # Custom validators
├── application/ # Application services
├── domain/
│ ├── model/ # Entities
│ └── repository/ # Repository interfaces
└── infrastructure/
├── persistence/ # Repository implementations
├── client/ # External API clients
└── config/ # Service configuration
Code Conventions
- Entities: Extend
BaseEntity<ID>, use@TableName("t_*") - Controllers:
@RestController+@RequiredArgsConstructor - Services:
@Service+@Transactional - Error Handling:
throw BusinessException.of(ErrorCode.XXX) - MapStruct:
@Mapper(componentModel = "spring")
Testing
# Run all tests
mvn test
# Run specific test
mvn test -Dtest=ClassName#methodName
# Run specific module tests
mvn -pl services/data-management-service -am test
Configuration
Environment Variables
DB_USERNAME: Database usernameDB_PASSWORD: Database passwordREDIS_PASSWORD: Redis passwordJWT_SECRET: JWT secret key
Profiles
application.yml: Default configurationapplication-dev.yml: Development overrides
Documentation
- API Docs: http://localhost:8080/api/swagger-ui.html
- AGENTS.md: See
backend/shared/AGENTS.mdfor shared libraries documentation - Service Docs: See individual service READMEs