194 lines
7.1 KiB
Markdown
194 lines
7.1 KiB
Markdown
# DeepData All-in-One Data Work Platform
|
|
|
|
<div align="center">
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**DeepData is an enterprise-level data processing platform for model fine-tuning and RAG retrieval, supporting core
|
|
functions such as data collection, data management, operator marketplace, data cleaning, data synthesis, data
|
|
annotation, data evaluation, and knowledge generation.**
|
|
|
|
[简体中文](./README-zh.md) | [English](./README.md)
|
|
|
|
If you like this project, please give it a Star⭐️!
|
|
|
|
</div>
|
|
|
|
## 🌟 Core Features
|
|
|
|
- **Core Modules**: Data Collection, Data Management, Operator Marketplace, Data Cleaning, Data Synthesis, Data
|
|
Annotation, Data Evaluation, Knowledge Generation.
|
|
- **Visual Orchestration**: Drag-and-drop data processing workflow design.
|
|
- **Operator Ecosystem**: Rich built-in operators and support for custom operators.
|
|
|
|
## 🚀 Quick Start
|
|
|
|
### Prerequisites
|
|
|
|
- Git (for pulling source code)
|
|
- Make (for building and installing)
|
|
- Docker (for building images and deploying services)
|
|
- Docker-Compose (for service deployment - Docker method)
|
|
- Kubernetes (for service deployment - k8s method)
|
|
- Helm (for service deployment - k8s method)
|
|
- **K8s deployment additionally requires**: [Sealed Secrets Controller](https://github.com/bitnami-labs/sealed-secrets) (for encrypted secret management)
|
|
|
|
### Secret Management (K8s deployment only)
|
|
|
|
DeepData K8s deployment uses **Bitnami Sealed Secrets** to manage sensitive configuration such as database passwords and JWT secrets. All secrets are stored in encrypted form in Git (`deployment/kubernetes/sealed-secrets/`) and automatically decrypted by the Sealed Secrets Controller in the cluster at deploy time.
|
|
|
|
**Online environment - install Sealed Secrets Controller:**
|
|
|
|
```bash
|
|
# Install via Helm (recommended)
|
|
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
|
|
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system
|
|
|
|
# Verify installation
|
|
kubectl get pods -n kube-system | grep sealed-secrets
|
|
```
|
|
|
|
**Air-gapped / offline environment:**
|
|
|
|
1. Download the Sealed Secrets image on an internet-connected machine:
|
|
```bash
|
|
# Download controller image (~60MB)
|
|
docker pull bitnami/sealed-secrets-controller:latest
|
|
docker save bitnami/sealed-secrets-controller:latest -o sealed-secrets-controller.tar
|
|
|
|
# Download kubeseal CLI (for updating secrets)
|
|
# macOS:
|
|
brew install kubeseal
|
|
# Linux:
|
|
wget https://github.com/bitnami-labs/sealed-secrets/releases/latest/download/kubeseal-linux-amd64
|
|
```
|
|
|
|
2. Transfer the image to your offline registry, then install via Helm with the custom image reference.
|
|
|
|
**Updating secrets:**
|
|
|
|
```bash
|
|
# When passwords change, re-encrypt with kubeseal
|
|
echo -n "new-password" | kubeseal --raw --name deepdata-conf --namespace deepdata --scope namespace-wide
|
|
```
|
|
|
|
> Note: Docker deployments do not require Sealed Secrets — secrets are managed via the `.env` file (excluded from Git via `.gitignore`).
|
|
|
|
### Docker Quick deploy
|
|
```shell
|
|
make build # Build all images from this repository
|
|
make install # Deploy base services (deepdata + milvus)
|
|
```
|
|
|
|
> Images are built from source in this repository — no external registry is required.
|
|
|
|
### Clone the Code
|
|
|
|
```bash
|
|
git clone https://gitea.apescale.com/figmar/deepdata.git
|
|
cd deepdata
|
|
```
|
|
|
|
### Deploy the basic services
|
|
|
|
```bash
|
|
make install
|
|
```
|
|
|
|
This project supports deployment via two methods: docker-compose and helm. After executing the command, please enter the corresponding number for the deployment method. The command echo is as follows:
|
|
```shell
|
|
Choose a deployment method:
|
|
1. Docker/Docker-Compose
|
|
2. Kubernetes/Helm
|
|
Enter choice:
|
|
```
|
|
|
|
If the machine you are using does not have make installed, please run the following command to deploy it:
|
|
```bash
|
|
docker compose -f deployment/docker/deepdata/docker-compose.yml --profile milvus up -d
|
|
```
|
|
|
|
Once the container is running, access http://localhost:30000 in a browser to view the front-end interface.
|
|
|
|
To list all available Make targets, flags and help text, run:
|
|
|
|
```bash
|
|
make help
|
|
```
|
|
|
|
If you are in an offline environment, you can run the following command to download all dependent images:
|
|
```bash
|
|
make download
|
|
```
|
|
|
|
### Deploy Label Studio as an annotation tool
|
|
```bash
|
|
make install-label-studio
|
|
```
|
|
|
|
### Build and deploy Mineru Enhanced PDF Processing
|
|
```bash
|
|
make build-mineru
|
|
make install-mineru
|
|
```
|
|
|
|
### Deploy the DeepFlow service
|
|
```bash
|
|
make install-deepflow
|
|
```
|
|
|
|
### Local Development and Deployment
|
|
After modifying the local code, please execute the following commands to build the image and deploy using the local image.
|
|
```bash
|
|
make build
|
|
make install dev=true
|
|
```
|
|
|
|
### Uninstall
|
|
```bash
|
|
make uninstall
|
|
```
|
|
|
|
When running make uninstall, the installer will prompt once whether to delete volumes; that single choice is applied to all components. The uninstall order is: milvus -> label-studio -> deepdata, which ensures the deepdata network is removed cleanly after services that use it have stopped.
|
|
|
|
## 📚 Documentation
|
|
|
|
### Core Documentation
|
|
- **[DEVELOPMENT.md](./DEVELOPMENT.md)** - Local development environment setup and workflow
|
|
- **[AGENTS.md](./AGENTS.md)** - AI assistant guidelines and code style
|
|
|
|
### Backend Documentation
|
|
- **[backend/README.md](./backend/README.md)** - Backend architecture, services, and technology stack
|
|
- **[backend/api-gateway/README.md](./backend/api-gateway/README.md)** - API Gateway configuration and routing
|
|
- **[backend/services/main-application/README.md](./backend/services/main-application/README.md)** - Main application modules
|
|
- **[backend/shared/README.md](./backend/shared/README.md)** - Shared libraries (domain-common, security-common)
|
|
|
|
### Runtime Documentation
|
|
- **[runtime/README.md](./runtime/README.md)** - Runtime architecture and components
|
|
- **[runtime/deepdata-python/README.md](./runtime/deepdata-python/README.md)** - FastAPI backend service
|
|
- **[runtime/python-executor/README.md](./runtime/python-executor/README.md)** - Ray executor framework
|
|
- **[runtime/ops/README.md](./runtime/ops/README.md)** - Operator ecosystem
|
|
- **[runtime/datax/README.md](./runtime/datax/README.md)** - DataX data framework
|
|
- **[runtime/deepflow/README.md](./runtime/deepflow/README.md)** - DeepFlow LLM service
|
|
|
|
### Frontend Documentation
|
|
- **[frontend/README.md](./frontend/README.md)** - React frontend application
|
|
|
|
## 🤝 Contribution Guidelines
|
|
|
|
Thank you for your interest in this project! We warmly welcome contributions from the community. Whether it's submitting
|
|
bug reports, suggesting new features, or directly participating in code development, all forms of help make a project
|
|
better.
|
|
|
|
• 📮 [GitHub Issues](../../issues): Submit bugs or feature suggestions.
|
|
|
|
• 🔧 [GitHub Pull Requests](../../pulls): Contribute code improvements.
|
|
|
|
## 📄 License
|
|
|
|
DeepData is open source under the [MIT](LICENSE) license. You are free to use, modify, and distribute the code of this
|
|
project in compliance with the license terms.
|