figmar/deepdata
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
Files
deepdata-20260928
deepdata/scripts/images/runtime-ray/Dockerfile
T
沃巴蒂 ff22ed3a8b feat(deepdata): 引入修改密码、文件预览组件与清洗算子增强，deepflow 改名为 deep-flow 
本次同步 09-12 至 09-28 的代码变更，主要内容：

新增功能
- 修改密码：后端新增 ChangePasswordRequest DTO 与UserController/UserService
  对应接口，前端新增 ChangePasswordDialog 并在 Header 接入入口
- 文件预览组件重构：file-preview 目录拆分出 VideoPreview（支持 blobUrl 直传）、
  ImagePreview、AudioPreview，index.tsx 统一调度各类型预览器
- 新增 my_ops 算子模块：markitdown_parser、mineru_pdf_parser、text_quality_evaluator
- 新增 tools/minio-file-server MinIO 文件服务

功能增强
- 数据清洗：cleaning_task_service/scheduler/routes 重构，
  data_juicer_executor 与 wrapper 增强，OperatorTable/LogsTable/TaskDetail 同步调整
- 数据标注：annotation sync.py、project.py 接口与schema 更新
- 数据采集、算子市场、首页与菜单路由相关页面调整
- DataJuicer 算子接入（useOperatorOperations、dj_ops_schema）

结构调整
- deepflow/ 更名为 deep-flow/（419 个文件重命名）
- 新增 helm 部署编排、GitHub Actions 之外的自建镜像构建脚本

未纳入本次提交
- scripts-backup/、runtime/python-executor-backup/、components_backup/（备份目录）
- *.backup*、__pycache__ 产物
- zh_core_web_sm-3.8.0-py3-none-any.whl（46MB 构建产物）
- DataX/（内嵌独立仓库，远程亦未收录其源码）
2026-10-03 22:44:56 +08:00

127 lines
5.3 KiB
Docker
Raw Permalink Blame History

# ==============================================================================
# Stage 1: DataX Builder
# ==============================================================================
FROM maven:3-eclipse-temurin-8-alpine AS datax-builder
# 安装 git
# 仅安装必要的 git
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories
RUN apk update && apk add --no-cache git
COPY DataX/ /DataX/
COPY runtime/datax/ DataX/
# 创建阿里云 settings.xml
RUN mkdir -p /root/.m2 && \
cat > /root/.m2/settings.xml <<EOF
<settings>
<mirrors>
<mirror>
<id>aliyunmaven</id>
<mirrorOf>central</mirrorOf>
<name>Aliyun Maven</name>
<url>https://maven.aliyun.com/repository/public</url>
</mirror>
</mirrors>
</settings>
EOF
# 设置工作目录
WORKDIR /DataX
RUN find /DataX -type f -name "DataBaseType.java" -exec sed -i "s/com.mysql.jdbc.Driver/com.mysql.cj.jdbc.Driver/g" {} + || echo "⚠️ MySQL driver file not found, skipping." && \
sed -i 's/<maven-assembly-plugin.version>.*<\/maven-assembly-plugin.version>/<maven-assembly-plugin.version>2.6<\/maven-assembly-plugin.version>/' /DataX/pom.xml || echo "⚠️ Maven assembly version tag not found, skipping."
# 编译 DataX (精确挂载 repository 目录，不覆盖 settings.xml)
RUN --mount=type=cache,target=/root/.m2/repository \
mvn -s /root/.m2/settings.xml clean package assembly:assembly -Dmaven.test.skip=true
# ==============================================================================
# Stage 2: Ray Worker Runtime (保留原有的 AI 环境，增加 Java 支持)
# ==============================================================================
FROM ghcr.io/astral-sh/uv:python3.11-bookworm
# 1. 安装系统依赖
RUN sed -i 's/deb.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list.d/debian.sources && \
sed -i 's/security.debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list.d/debian.sources && \
apt-get update && \
apt-get install -y --no-install-recommends \
libgl1 libglib2.0-0 vim libmagic1 dos2unix swig \
poppler-utils tesseract-ocr openjdk-17-jre-headless \
&& rm -rf /var/lib/apt/lists/*
# 2. 下载 AI 模型 (保持不变)
RUN mkdir -p /home/models \
&& wget -q --show-progress https://paddle-model-ecology.bj.bcebos.com/paddlex/official_inference_model/paddle3.0.0/PP-LCNet_x1_0_doc_ori_infer.tar \
&& tar -xf PP-LCNet_x1_0_doc_ori_infer.tar -C /home/models \
&& rm -f PP_*.tar
# 3. 复制业务代码 (保持不变)
COPY runtime/python-executor /opt/runtime
COPY runtime/ops /opt/runtime/deepdata/ops
COPY runtime/ops/user /opt/runtime/user
COPY scripts/images/runtime-ray/start.sh /opt/runtime/start.sh
COPY scripts/images/common/log-rotate-copytruncate.sh /usr/local/bin/log-rotate-copytruncate.sh
# 4. 复制编译好的 DataX (从 builder 阶段复制)
# ✅ 新增: 将编译好的 datax 复制到 /opt/datax
COPY --from=datax-builder /DataX/target/datax/datax /opt/datax
# ✅ 新增: 修复 MySQL 驱动路径 (参照 backend-python)
RUN cp /opt/datax/plugin/reader/mysqlreader/libs/mysql* /opt/datax/plugin/reader/starrocksreader/libs/ || true
# 5. 环境变量配置
ENV PYTHONPATH=/opt/runtime/deepdata/
ENV UV_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple
ENV UV_EXTRA_INDEX_URL=https://download.pytorch.org/whl/cpu
ENV UV_INDEX_STRATEGY=unsafe-best-match
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
# ✅ 新增: 设置 JAVA_HOME，确保 datax.py 能正确调用 Java
ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
ENV PATH="$PATH:$JAVA_HOME/bin"
WORKDIR /opt/runtime
COPY scripts/images/runtime-ray/zh_core_web_sm-3.8.0-py3-none-any.whl /tmp/
COPY scripts/images/runtime-ray/nltk_data /root/nltk_data
ENV UV_LINK_MODE=copy
# 6. 安装 Python 依赖
RUN --mount=type=cache,target=/root/.cache/uv \
uv pip install -e . --system \
&& uv pip install -r /opt/runtime/deepdata/ops/pyproject.toml --system \
&& uv pip install "py-data-juicer[generic,nlp]==1.4.4" --system \
&& uv pip install "ray[default]==2.47.1" --system \
&& uv pip install "pandas>=2.0.0" --system \
&& uv pip install "pyarrow>=14.0.0" --system \
&& uv pip install "sqlalchemy>=2.0.0" --system \
&& uv pip install "loguru>=0.7.0" --system \
&& uv pip install "jsonargparse>=4.27.0" --system \
&& uv pip install "pyyaml>=6.0" --system \
&& uv pip install "pypdf>=3.17.0" --system \
&& uv pip install "python-docx>=1.1.0" --system \
&& uv pip install "python-pptx>=0.6.23" --system \
&& uv pip install "setuptools<75.0" --system \
&& uv pip install \
"opentelemetry-semantic-conventions>=0.48b0" \
"opentelemetry-exporter-prometheus>=0.48b0" \
--system \
&& uv pip install pyarrow pandas spacy nltk --system \
&& uv pip install /tmp/zh_core_web_sm-3.8.0-py3-none-any.whl --system \
&& echo "/usr/local/lib/ops/site-packages" > /usr/local/lib/python3.11/site-packages/ops.pth
# 7. 初始化与权限
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
&& chmod +x /opt/runtime/start.sh \
&& chmod +x /usr/local/bin/log-rotate-copytruncate.sh \
&& dos2unix /opt/runtime/start.sh
EXPOSE 8081
ENTRYPOINT ["/opt/runtime/start.sh"]
Reference in New Issue View Git Blame Copy Permalink