沃巴蒂
ff22ed3a8b
本次同步 09-12 至 09-28 的代码变更，主要内容： 新增功能 - 修改密码：后端新增 ChangePasswordRequest DTO 与UserController/UserService 对应接口，前端新增 ChangePasswordDialog 并在 Header 接入入口 - 文件预览组件重构：file-preview 目录拆分出 VideoPreview（支持 blobUrl 直传）、 ImagePreview、AudioPreview，index.tsx 统一调度各类型预览器 - 新增 my_ops 算子模块：markitdown_parser、mineru_pdf_parser、text_quality_evaluator - 新增 tools/minio-file-server MinIO 文件服务 功能增强 - 数据清洗：cleaning_task_service/scheduler/routes 重构， data_juicer_executor 与 wrapper 增强，OperatorTable/LogsTable/TaskDetail 同步调整 - 数据标注：annotation sync.py、project.py 接口与schema 更新 - 数据采集、算子市场、首页与菜单路由相关页面调整 - DataJuicer 算子接入（useOperatorOperations、dj_ops_schema） 结构调整 - deepflow/ 更名为 deep-flow/（419 个文件重命名） - 新增 helm 部署编排、GitHub Actions 之外的自建镜像构建脚本 未纳入本次提交 - scripts-backup/、runtime/python-executor-backup/、components_backup/（备份目录） - *.backup*、__pycache__ 产物 - zh_core_web_sm-3.8.0-py3-none-any.whl（46MB 构建产物） - DataX/（内嵌独立仓库，远程亦未收录其源码）
182 lines
7.1 KiB
Markdown
182 lines
7.1 KiB
Markdown
# DEVELOPMENT GUIDE for DeepData
|
|
|
|
This document provides a comprehensive development guide for DeepData, a polyglot, microservices-based project consisting of Java, Python, and React components. It describes how to set up, build, test, run, and contribute in a local Docker Compose-based environment, without exposing secrets.
|
|
|
|
<!-- NOTE: This file is intended as a high-level guide. Do not duplicate content from component READMEs; reference them where appropriate. -->
|
|
|
|
## Overview
|
|
|
|
DeepData is composed of multiple services (Java backend, Python runtime, and React frontend) coordinated via Docker Compose for local development. The guide below covers prerequisites, quick-start steps, project structure, development workflow, environment configuration, testing, debugging, common issues, documentation, contribution workflow, and licensing.
|
|
|
|
Refer to the component READMEs for detailed implementation notes:
|
|
- Backend: backend/README.md
|
|
- Runtime: runtime/deepdata-python/README.md
|
|
- Frontend: frontend/README.md
|
|
|
|
For code style guidelines, see AGENTS.md in the repository root.
|
|
|
|
## Prerequisites
|
|
|
|
- Java Development: JDK 21 and Maven
|
|
- Python: Python 3.12 and Poetry
|
|
- Node.js: Node.js 18
|
|
- Docker and Docker Compose
|
|
- Optional: Make (for convenience)
|
|
|
|
Notes:
|
|
- Ensure Java and Python environments are on the system PATH where applicable.
|
|
- Docker Compose will orchestrate the local development stack.
|
|
|
|
## Quick Start
|
|
|
|
1) Clone the repository and install dependencies:
|
|
- git clone https://github.com/your-org/datemate.git
|
|
- cd datemate
|
|
- (Optional) Create and activate a Python virtual environment if not using Poetry-managed envs.
|
|
- Build dependencies per component as described below.
|
|
|
|
2) Start the local stack with Docker Compose:
|
|
- docker compose up -d
|
|
- This brings up the Java backend, Python runtime, and React frontend services along with any required databases and caches as defined in the docker-compose.yml.
|
|
|
|
3) Start individual components (if you prefer not to use the Docker stack):
|
|
- Java backend
|
|
- mvn -f backend/pom.xml -DskipTests package
|
|
- Run the main application (path may vary): java -jar backend/main-application/target/*.jar
|
|
- Python runtime
|
|
- cd runtime/deepdata-python
|
|
- poetry install
|
|
- uvicorn app.main:app --reload --port 18000 --host 0.0.0.0
|
|
- React frontend
|
|
- cd frontend
|
|
- npm ci
|
|
- npm run dev
|
|
|
|
4) Stop the stack:
|
|
- docker compose down
|
|
|
|
> Tip: In a team setting, prefer Docker Compose for consistency across development environments.
|
|
|
|
## Project Structure
|
|
|
|
- backend/
|
|
- frontend/
|
|
- runtime/
|
|
- deployment/
|
|
- docs/
|
|
- AGENTS.md (code style guidelines)
|
|
- docker/ (docker-related tooling)
|
|
- .env* files (per-component configurations, see Environment Configuration section)
|
|
|
|
This is a polyglot project with the following language footprints:
|
|
- Java for the backend services under backend/
|
|
- Python for the runtime under runtime/deepdata-python/
|
|
- React/TypeScript for the frontend under frontend/
|
|
|
|
## Development Workflow
|
|
|
|
Language-specific workflows:
|
|
|
|
- Java (Backend)
|
|
- Build: mvn -f backend/pom.xml -DskipTests package
|
|
- Test: mvn -f backend/pom.xml test
|
|
- Run: mvn -f backend/pom.xml -Dexec.mainClass=... spring-boot:run (or run the packaged jar)
|
|
- Python (Runtime)
|
|
- Install: cd runtime/deepdata-python && poetry install
|
|
- Test: pytest
|
|
- Run: uvicorn app.main:app --reload --port 18000 --host 0.0.0.0
|
|
- Frontend (React)
|
|
- Install: cd frontend && npm ci
|
|
- Test: No frontend tests configured
|
|
- Build: npm run build
|
|
- Run: npm run dev
|
|
|
|
General tips:
|
|
- Use Docker Compose for a repeatable local stack.
|
|
- Run linters and tests before creating PRs.
|
|
- Keep dependencies in sync across environments.
|
|
|
|
## Environment Configuration
|
|
|
|
Each component can have its own environment file(s). Do not commit secrets. Use sample/.env.example files as references when available.
|
|
|
|
- Backend
|
|
- Path: backend/.env (example keys below)
|
|
- Typical keys: DB_URL, DB_USER, DB_PASSWORD, JWT_SECRET, REDIS_URL, CLOUD_STORAGE_ENDPOINT
|
|
- Runtime (Python)
|
|
- Path: runtime/deepdata-python/.env
|
|
- Typical keys: DATABASE_URL, RAY_ADDRESS, CELERY_BROKER_URL, APP_SETTINGS
|
|
- Frontend
|
|
- Path: frontend/.env
|
|
- Typical keys: VITE_API_BASE_URL, VITE_DEFAULT_LOCALE, NODE_ENV
|
|
|
|
Notes:
|
|
- Copy the corresponding .env.example to .env and fill in values as needed.
|
|
- Do not commit .env files containing secrets.
|
|
|
|
## Testing
|
|
|
|
- Java: JUnit 5 tests run via Maven (mvn test).
|
|
- Python: pytest in runtime/deepdata-python/test or relevant tests.
|
|
- Frontend: No frontend tests configured in this repo.
|
|
|
|
## Code Style
|
|
|
|
Code style follows the repository-wide guidelines described in AGENTS.md. See:
|
|
- AGENTS.md (root): Code style guidelines for all languages.
|
|
- Java: Follow Java conventions in backend/ and accordance with project conventions.
|
|
- Python: Follow PEP 8 and project-specific conventions in runtime/deepdata-python.
|
|
- React: Follow the frontend conventions in frontend/ (TypeScript/TSX).
|
|
|
|
Link to guidelines: AGENTS.md
|
|
|
|
## Debugging
|
|
|
|
- Java (Backend): Enable JPDA debugging by starting the JVM with a debug port and attach a debugger.
|
|
- Example (local): export JAVA_TOOL_OPTIONS='-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005' && java -jar path/to/app.jar
|
|
- Attach with IDE on port 5005 after launch.
|
|
- Python (Runtime): Run with debugpy listening on port 5678 to attach from IDEs.
|
|
- Example: cd runtime/deepdata-python && poetry install
|
|
python -m debugpy --listen 5678 --wait-for-client -m uvicorn app.main:app --reload --port 18000 --host 0.0.0.0
|
|
- Frontend (React): Use Node inspector to debug front-end code in dev server.
|
|
- Example: npm run dev -- --inspect-brk=9229
|
|
|
|
Tips: Use your preferred IDEs (IntelliJ/VSCode/WebStorm) to attach to the running processes on their respective ports.
|
|
|
|
## Common Issues
|
|
|
|
- Port conflicts: Check which process is using a port with lsof -i TCP:<PORT> or ss -ltnp. Stop or reconfigure conflicting services.
|
|
- Database connection errors: Ensure .env contains correct DATABASE_URL and credentials; ensure the database service is up in Docker Compose.
|
|
- Ray cluster issues (Python runtime): Ensure Ray is started and accessible at the configured RAY_ADDRESS; check logs for worker failures and bootstrap status.
|
|
|
|
## Documentation
|
|
|
|
Component READMEs provide detailed usage and design decisions. See:
|
|
- backend/README.md
|
|
- runtime/deepdata-python/README.md
|
|
- frontend/README.md
|
|
- deployment/README.md
|
|
|
|
## Contributing
|
|
|
|
Contributions follow a PR workflow:
|
|
- Create a feature/bugfix branch from main (e.g., feature/new-action)
|
|
- Implement changes with tests where applicable
|
|
- Run unit tests for the changed components
|
|
- Open a PR with a clear description of the changes and the rationale
|
|
- Ensure CI checks pass (build, unit tests, lint)
|
|
- Obtain reviews and address feedback
|
|
- Merge to main after approval
|
|
|
|
## License
|
|
|
|
Apache 2.0
|
|
|
|
---
|
|
|
|
References:
|
|
- AGENTS.md for code style guidelines: AGENTS.md
|
|
- Java dependencies: backend/pom.xml
|
|
- Node dependencies: frontend/package.json
|
|
- Python dependencies: runtime/deepdata-python/pyproject.toml
|