deepdata/DEVELOPMENT.md

# DEVELOPMENT GUIDE for DeepData

This document provides a comprehensive development guide for DeepData, a polyglot, microservices-based project consisting of Java, Python, and React components. It describes how to set up, build, test, run, and contribute in a local Docker Compose-based environment, without exposing secrets.

<!-- NOTE: This file is intended as a high-level guide. Do not duplicate content from component READMEs; reference them where appropriate. -->

## Overview

DeepData is composed of multiple services (Java backend, Python runtime, and React frontend) coordinated via Docker Compose for local development. The guide below covers prerequisites, quick-start steps, project structure, development workflow, environment configuration, testing, debugging, common issues, documentation, contribution workflow, and licensing.

Refer to the component READMEs for detailed implementation notes:
- Backend: backend/README.md
- Runtime: runtime/deepdata-python/README.md
- Frontend: frontend/README.md

For code style guidelines, see AGENTS.md in the repository root.

## Prerequisites

- Java Development: JDK 21 and Maven
- Python: Python 3.12 and Poetry
- Node.js: Node.js 18
- Docker and Docker Compose
- Optional: Make (for convenience)

Notes:
- Ensure Java and Python environments are on the system PATH where applicable.
- Docker Compose will orchestrate the local development stack.

## Quick Start

1) Clone the repository and install dependencies:
- git clone https://gitea.apescale.com/figmar/deepdata.git
- cd datemate
- (Optional) Create and activate a Python virtual environment if not using Poetry-managed envs.
- Build dependencies per component as described below.

2) Start the local stack with Docker Compose:
- docker compose up -d
- This brings up the Java backend, Python runtime, and React frontend services along with any required databases and caches as defined in the docker-compose.yml.

3) Start individual components (if you prefer not to use the Docker stack):
- Java backend
  - mvn -f backend/pom.xml -DskipTests package
  - Run the main application (path may vary): java -jar backend/main-application/target/*.jar
- Python runtime
  - cd runtime/deepdata-python
  - poetry install
  - uvicorn app.main:app --reload --port 18000 --host 0.0.0.0
- React frontend
  - cd frontend
  - npm ci
  - npm run dev

4) Stop the stack:
- docker compose down

> Tip: In a team setting, prefer Docker Compose for consistency across development environments.

## Project Structure

- backend/
- frontend/
- runtime/
- deployment/
- docs/
- AGENTS.md (code style guidelines)
- docker/ (docker-related tooling)
- .env* files (per-component configurations, see Environment Configuration section)

This is a polyglot project with the following language footprints:
- Java for the backend services under backend/
- Python for the runtime under runtime/deepdata-python/
- React/TypeScript for the frontend under frontend/

## Development Workflow

Language-specific workflows:

- Java (Backend)
  - Build: mvn -f backend/pom.xml -DskipTests package
  - Test: mvn -f backend/pom.xml test
  - Run: mvn -f backend/pom.xml -Dexec.mainClass=... spring-boot:run (or run the packaged jar)
- Python (Runtime)
  - Install: cd runtime/deepdata-python && poetry install
  - Test: pytest
  - Run: uvicorn app.main:app --reload --port 18000 --host 0.0.0.0
- Frontend (React)
  - Install: cd frontend && npm ci
  - Test: No frontend tests configured
  - Build: npm run build
  - Run: npm run dev

General tips:
- Use Docker Compose for a repeatable local stack.
- Run linters and tests before creating PRs.
- Keep dependencies in sync across environments.

## Environment Configuration

Each component can have its own environment file(s). Do not commit secrets. Use sample/.env.example files as references when available.

- Backend
  - Path: backend/.env (example keys below)
  - Typical keys: DB_URL, DB_USER, DB_PASSWORD, JWT_SECRET, REDIS_URL, CLOUD_STORAGE_ENDPOINT
- Runtime (Python)
  - Path: runtime/deepdata-python/.env
  - Typical keys: DATABASE_URL, RAY_ADDRESS, CELERY_BROKER_URL, APP_SETTINGS
- Frontend
  - Path: frontend/.env
  - Typical keys: VITE_API_BASE_URL, VITE_DEFAULT_LOCALE, NODE_ENV

Notes:
- Copy the corresponding .env.example to .env and fill in values as needed.
- Do not commit .env files containing secrets.

## Testing

- Java: JUnit 5 tests run via Maven (mvn test).
- Python: pytest in runtime/deepdata-python/test or relevant tests.
- Frontend: No frontend tests configured in this repo.

## Code Style

Code style follows the repository-wide guidelines described in AGENTS.md. See:
- AGENTS.md (root): Code style guidelines for all languages.
- Java: Follow Java conventions in backend/ and accordance with project conventions.
- Python: Follow PEP 8 and project-specific conventions in runtime/deepdata-python.
- React: Follow the frontend conventions in frontend/ (TypeScript/TSX).

Link to guidelines: AGENTS.md

## Debugging

- Java (Backend): Enable JPDA debugging by starting the JVM with a debug port and attach a debugger.
  - Example (local): export JAVA_TOOL_OPTIONS='-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005' && java -jar path/to/app.jar
  - Attach with IDE on port 5005 after launch.
- Python (Runtime): Run with debugpy listening on port 5678 to attach from IDEs.
  - Example: cd runtime/deepdata-python && poetry install
    python -m debugpy --listen 5678 --wait-for-client -m uvicorn app.main:app --reload --port 18000 --host 0.0.0.0
- Frontend (React): Use Node inspector to debug front-end code in dev server.
  - Example: npm run dev -- --inspect-brk=9229

Tips: Use your preferred IDEs (IntelliJ/VSCode/WebStorm) to attach to the running processes on their respective ports.

## Common Issues

- Port conflicts: Check which process is using a port with lsof -i TCP:<PORT> or ss -ltnp. Stop or reconfigure conflicting services.
- Database connection errors: Ensure .env contains correct DATABASE_URL and credentials; ensure the database service is up in Docker Compose.
- Ray cluster issues (Python runtime): Ensure Ray is started and accessible at the configured RAY_ADDRESS; check logs for worker failures and bootstrap status.

## Documentation

Component READMEs provide detailed usage and design decisions. See:
- backend/README.md
- runtime/deepdata-python/README.md
- frontend/README.md
- deployment/README.md

## Contributing

Contributions follow a PR workflow:
- Create a feature/bugfix branch from main (e.g., feature/new-action)
- Implement changes with tests where applicable
- Run unit tests for the changed components
- Open a PR with a clear description of the changes and the rationale
- Ensure CI checks pass (build, unit tests, lint)
- Obtain reviews and address feedback
- Merge to main after approval

## License

Apache 2.0

---

References:
- AGENTS.md for code style guidelines: AGENTS.md
- Java dependencies: backend/pom.xml
- Node dependencies: frontend/package.json
- Python dependencies: runtime/deepdata-python/pyproject.toml