figmar/deepdata
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(deepdata): 引入修改密码、文件预览组件与清洗算子增强，deepflow 改名为 deep-flow

Browse Source 
本次同步 09-12 至 09-28 的代码变更，主要内容：

新增功能
- 修改密码：后端新增 ChangePasswordRequest DTO 与UserController/UserService
  对应接口，前端新增 ChangePasswordDialog 并在 Header 接入入口
- 文件预览组件重构：file-preview 目录拆分出 VideoPreview（支持 blobUrl 直传）、
  ImagePreview、AudioPreview，index.tsx 统一调度各类型预览器
- 新增 my_ops 算子模块：markitdown_parser、mineru_pdf_parser、text_quality_evaluator
- 新增 tools/minio-file-server MinIO 文件服务

功能增强
- 数据清洗：cleaning_task_service/scheduler/routes 重构，
  data_juicer_executor 与 wrapper 增强，OperatorTable/LogsTable/TaskDetail 同步调整
- 数据标注：annotation sync.py、project.py 接口与schema 更新
- 数据采集、算子市场、首页与菜单路由相关页面调整
- DataJuicer 算子接入（useOperatorOperations、dj_ops_schema）

结构调整
- deepflow/ 更名为 deep-flow/（419 个文件重命名）
- 新增 helm 部署编排、GitHub Actions 之外的自建镜像构建脚本

未纳入本次提交
- scripts-backup/、runtime/python-executor-backup/、components_backup/（备份目录）
- *.backup*、__pycache__ 产物
- zh_core_web_sm-3.8.0-py3-none-any.whl（46MB 构建产物）
- DataX/（内嵌独立仓库，远程亦未收录其源码）
This commit is contained in:
沃巴蒂
2026-10-03 22:44:56 +08:00
parent cbb1048e18
commit ff22ed3a8b
1156 changed files with 129912 additions and 21716 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+194 -193
View File
@@ -1,193 +1,194 @@
# DeepData All-in-One Data Work Platform
<div align="center">
![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)
![Backend: Spring Boot 3](https://img.shields.io/badge/backend-Spring%20Boot%203-blue)
![Frontend: React](https://img.shields.io/badge/frontend-React%2018-61DAFB)
![Python: 3.12](https://img.shields.io/badge/python-3.12+-blue)
**DeepData is an enterprise-level data processing platform for model fine-tuning and RAG retrieval, supporting core
functions such as data collection, data management, operator marketplace, data cleaning, data synthesis, data
annotation, data evaluation, and knowledge generation.**
[简体中文](./README-zh.md) | [English](./README.md)
If you like this project, please give it a Star⭐️!
</div>
## 🌟 Core Features
- **Core Modules**: Data Collection, Data Management, Operator Marketplace, Data Cleaning, Data Synthesis, Data
Annotation, Data Evaluation, Knowledge Generation.
- **Visual Orchestration**: Drag-and-drop data processing workflow design.
- **Operator Ecosystem**: Rich built-in operators and support for custom operators.
## 🚀 Quick Start
### Prerequisites
- Git (for pulling source code)
- Make (for building and installing)
- Docker (for building images and deploying services)
- Docker-Compose (for service deployment - Docker method)
- Kubernetes (for service deployment - k8s method)
- Helm (for service deployment - k8s method)
- **K8s deployment additionally requires**: [Sealed Secrets Controller](https://github.com/bitnami-labs/sealed-secrets) (for encrypted secret management)
### Secret Management (K8s deployment only)
DeepData K8s deployment uses **Bitnami Sealed Secrets** to manage sensitive configuration such as database passwords and JWT secrets. All secrets are stored in encrypted form in Git (`deployment/kubernetes/sealed-secrets/`) and automatically decrypted by the Sealed Secrets Controller in the cluster at deploy time.
**Online environment - install Sealed Secrets Controller:**
```bash
# Install via Helm (recommended)
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system
# Verify installation
kubectl get pods -n kube-system | grep sealed-secrets
```
**Air-gapped / offline environment:**
1. Download the Sealed Secrets image on an internet-connected machine:
```bash
# Download controller image (~60MB)
docker pull bitnami/sealed-secrets-controller:latest
docker save bitnami/sealed-secrets-controller:latest -o sealed-secrets-controller.tar
# Download kubeseal CLI (for updating secrets)
# macOS:
brew install kubeseal
# Linux:
wget https://github.com/bitnami-labs/sealed-secrets/releases/latest/download/kubeseal-linux-amd64
```
2. Transfer the image to your offline registry, then install via Helm with the custom image reference.
**Updating secrets:**
```bash
# When passwords change, re-encrypt with kubeseal
echo -n "new-password" | kubeseal --raw --name deepdata-conf --namespace deepdata --scope namespace-wide
```
> Note: Docker deployments do not require Sealed Secrets — secrets are managed via the `.env` file (excluded from Git via `.gitignore`).
### Docker Quick deploy
```shell
make build # Build all images from this repository
make install # Deploy base services (deepdata + milvus)
```
> Images are built from source in this repository — no external registry is required.
### Clone the Code
```bash
git clone https://gitea.apescale.com/figmar/deepdata.git
cd deepdata
```
### Deploy the basic services
```bash
make install
```
This project supports deployment via two methods: docker-compose and helm. After executing the command, please enter the corresponding number for the deployment method. The command echo is as follows:
```shell
Choose a deployment method:
1. Docker/Docker-Compose
2. Kubernetes/Helm
Enter choice:
```
If the machine you are using does not have make installed, please run the following command to deploy it:
```bash
docker compose -f deployment/docker/deepdata/docker-compose.yml --profile milvus up -d
```
Once the container is running, access http://localhost:30000 in a browser to view the front-end interface.
To list all available Make targets, flags and help text, run:
```bash
make help
```
If you are in an offline environment, you can run the following command to download all dependent images:
```bash
make download
```
### Deploy Label Studio as an annotation tool
```bash
make install-label-studio
```
### Build and deploy Mineru Enhanced PDF Processing
```bash
make build-mineru
make install-mineru
```
### Deploy the DeepFlow service
```bash
make install-deepflow
```
### Local Development and Deployment
After modifying the local code, please execute the following commands to build the image and deploy using the local image.
```bash
make build
make install dev=true
```
### Uninstall
```bash
make uninstall
```
When running make uninstall, the installer will prompt once whether to delete volumes; that single choice is applied to all components. The uninstall order is: milvus -> label-studio -> deepdata, which ensures the deepdata network is removed cleanly after services that use it have stopped.
## 📚 Documentation
### Core Documentation
- **[DEVELOPMENT.md](./DEVELOPMENT.md)** - Local development environment setup and workflow
- **[AGENTS.md](./AGENTS.md)** - AI assistant guidelines and code style
### Backend Documentation
- **[backend/README.md](./backend/README.md)** - Backend architecture, services, and technology stack
- **[backend/api-gateway/README.md](./backend/api-gateway/README.md)** - API Gateway configuration and routing
- **[backend/services/main-application/README.md](./backend/services/main-application/README.md)** - Main application modules
- **[backend/shared/README.md](./backend/shared/README.md)** - Shared libraries (domain-common, security-common)
### Runtime Documentation
- **[runtime/README.md](./runtime/README.md)** - Runtime architecture and components
- **[runtime/deepdata-python/README.md](./runtime/deepdata-python/README.md)** - FastAPI backend service
- **[runtime/python-executor/README.md](./runtime/python-executor/README.md)** - Ray executor framework
- **[runtime/ops/README.md](./runtime/ops/README.md)** - Operator ecosystem
- **[runtime/datax/README.md](./runtime/datax/README.md)** - DataX data framework
- **[runtime/deepflow/README.md](./runtime/deepflow/README.md)** - DeepFlow LLM service
### Frontend Documentation
- **[frontend/README.md](./frontend/README.md)** - React frontend application
## 🤝 Contribution Guidelines
Thank you for your interest in this project! We warmly welcome contributions from the community. Whether it's submitting
bug reports, suggesting new features, or directly participating in code development, all forms of help make a project
better.
• 📮 [GitHub Issues](../../issues): Submit bugs or feature suggestions.
• 🔧 [GitHub Pull Requests](../../pulls): Contribute code improvements.
## 📄 License
DeepData is open source under the [MIT](LICENSE) license. You are free to use, modify, and distribute the code of this
project in compliance with the license terms.
# DeepData All-in-One Data Work Platform
<div align="center">
[![Backend CI](https://github.com/ModelEngine-Group/DeepData/actions/workflows/docker-image-backend.yml/badge.svg)](https://github.com/ModelEngine-Group/DeepData/actions/workflows/docker-image-backend.yml)
[![Frontend CI](https://github.com/ModelEngine-Group/DeepData/actions/workflows/docker-image-frontend.yml/badge.svg)](https://github.com/ModelEngine-Group/DeepData/actions/workflows/docker-image-frontend.yml)
![GitHub Stars](https://img.shields.io/github/stars/ModelEngine-Group/DeepData)
![GitHub Forks](https://img.shields.io/github/forks/ModelEngine-Group/DeepData)
![GitHub Issues](https://img.shields.io/github/issues/ModelEngine-Group/DeepData)
![GitHub License](https://img.shields.io/github/license/ModelEngine-Group/deepdata-docs)
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/ModelEngine-Group/DeepData)
**DeepData is an enterprise-level data processing platform for model fine-tuning and RAG retrieval, supporting core
functions such as data collection, data management, operator marketplace, data cleaning, data synthesis, data
annotation, data evaluation, and knowledge generation.**
[简体中文](./README-zh.md) | [English](./README.md)
If you like this project, please give it a Star⭐️!
</div>
## 🌟 Core Features
- **Core Modules**: Data Collection, Data Management, Operator Marketplace, Data Cleaning, Data Synthesis, Data
Annotation, Data Evaluation, Knowledge Generation.
- **Visual Orchestration**: Drag-and-drop data processing workflow design.
- **Operator Ecosystem**: Rich built-in operators and support for custom operators.
## 🚀 Quick Start
### Prerequisites
- Git (for pulling source code)
- Make (for building and installing)
- Docker (for building images and deploying services)
- Docker-Compose (for service deployment - Docker method)
- Kubernetes (for service deployment - k8s method)
- Helm (for service deployment - k8s method)
- **K8s deployment additionally requires**: [Sealed Secrets Controller](https://github.com/bitnami-labs/sealed-secrets) (for encrypted secret management)
### Secret Management (K8s deployment only)
DeepData K8s deployment uses **Bitnami Sealed Secrets** to manage sensitive configuration such as database passwords and JWT secrets. All secrets are stored in encrypted form in Git (`deployment/kubernetes/sealed-secrets/`) and automatically decrypted by the Sealed Secrets Controller in the cluster at deploy time.
**Online environment - install Sealed Secrets Controller:**
```bash
# Install via Helm (recommended)
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system
# Verify installation
kubectl get pods -n kube-system | grep sealed-secrets
```
**Air-gapped / offline environment:**
1. Download the Sealed Secrets image on an internet-connected machine:
```bash
# Download controller image (~60MB)
docker pull bitnami/sealed-secrets-controller:latest
docker save bitnami/sealed-secrets-controller:latest -o sealed-secrets-controller.tar
# Download kubeseal CLI (for updating secrets)
# macOS:
brew install kubeseal
# Linux:
wget https://github.com/bitnami-labs/sealed-secrets/releases/latest/download/kubeseal-linux-amd64
```
2. Transfer the image to your offline registry, then install via Helm with the custom image reference.
**Updating secrets:**
```bash
# When passwords change, re-encrypt with kubeseal
echo -n "new-password" | kubeseal --raw --name deepdata-conf --namespace deepdata --scope namespace-wide
```
> Note: Docker deployments do not require Sealed Secrets — secrets are managed via the `.env` file (excluded from Git via `.gitignore`).
### Docker Quick deploy
```shell
wget -qO docker-compose.yml https://raw.githubusercontent.com/ModelEngine-Group/DeepData/refs/heads/main/deployment/docker/deepdata/docker-compose.yml \
&& REGISTRY=ghcr.io/modelengine-group/ docker compose up -d
```
### Clone the Code
```bash
git clone git@github.com:ModelEngine-Group/DeepData.git
cd DeepData
```
### Deploy the basic services
```bash
make install
```
This project supports deployment via two methods: docker-compose and helm. After executing the command, please enter the corresponding number for the deployment method. The command echo is as follows:
```shell
Choose a deployment method:
1. Docker/Docker-Compose
2. Kubernetes/Helm
Enter choice:
```
If the machine you are using does not have make installed, please run the following command to deploy it:
```bash
REGISTRY=ghcr.io/modelengine-group/ docker compose -f deployment/docker/deepdata/docker-compose.yml --profile milvus up -d
```
Once the container is running, access http://localhost:30000 in a browser to view the front-end interface.
To list all available Make targets, flags and help text, run:
```bash
make help
```
If you are in an offline environment, you can run the following command to download all dependent images:
```bash
make download
```
### Deploy Label Studio as an annotation tool
```bash
make install-label-studio
```
### Build and deploy Mineru Enhanced PDF Processing
```bash
make build-mineru
make install-mineru
```
### Deploy the DeepFlow service
```bash
make install-deep-flow
```
### Local Development and Deployment
After modifying the local code, please execute the following commands to build the image and deploy using the local image.
```bash
make build
make install dev=true
```
### Uninstall
```bash
make uninstall
```
When running make uninstall, the installer will prompt once whether to delete volumes; that single choice is applied to all components. The uninstall order is: milvus -> label-studio -> deepdata, which ensures the deepdata network is removed cleanly after services that use it have stopped.
## 📚 Documentation
### Core Documentation
- **[DEVELOPMENT.md](./DEVELOPMENT.md)** - Local development environment setup and workflow
- **[AGENTS.md](./AGENTS.md)** - AI assistant guidelines and code style
### Backend Documentation
- **[backend/README.md](./backend/README.md)** - Backend architecture, services, and technology stack
- **[backend/api-gateway/README.md](./backend/api-gateway/README.md)** - API Gateway configuration and routing
- **[backend/services/main-application/README.md](./backend/services/main-application/README.md)** - Main application modules
- **[backend/shared/README.md](./backend/shared/README.md)** - Shared libraries (domain-common, security-common)
### Runtime Documentation
- **[runtime/README.md](./runtime/README.md)** - Runtime architecture and components
- **[runtime/deepdata-python/README.md](./runtime/deepdata-python/README.md)** - FastAPI backend service
- **[runtime/python-executor/README.md](./runtime/python-executor/README.md)** - Ray executor framework
- **[runtime/ops/README.md](./runtime/ops/README.md)** - Operator ecosystem
- **[runtime/datax/README.md](./runtime/datax/README.md)** - DataX data framework
- **[runtime/deep-flow/README.md](./runtime/deep-flow/README.md)** - DeepFlow LLM service
### Frontend Documentation
- **[frontend/README.md](./frontend/README.md)** - React frontend application
## 🤝 Contribution Guidelines
Thank you for your interest in this project! We warmly welcome contributions from the community. Whether it's submitting
bug reports, suggesting new features, or directly participating in code development, all forms of help make a project
better.
• 📮 [GitHub Issues](../../issues): Submit bugs or feature suggestions.
• 🔧 [GitHub Pull Requests](../../pulls): Contribute code improvements.
## 📄 License
DeepData is open source under the [MIT](LICENSE) license. You are free to use, modify, and distribute the code of this
project in compliance with the license terms.