diff --git a/README.md b/README.md new file mode 100644 index 0000000..4ad54f2 --- /dev/null +++ b/README.md @@ -0,0 +1,209 @@ +# 3D Model Generator + +A local desktop tool for turning short object ideas into image prompts, generated concept images, Replicate Hunyuan 3D models, and optional HDRI scene lighting. + +The app runs as a `pywebview` desktop window. It uses Ollama for prompt generation, local Diffusers or Replicate for image generation, Replicate for image-to-3D conversion, and Blender for viewing generated `.glb` files. + +## Features + +- Generate Stable Diffusion-style prompts from a short object name. +- Convert prompts into editable JSON generation settings. +- Generate image batches with either: + - local Diffusers using a local DreamShaper model + - Replicate API using one of three configured image models +- Generate 3D `.glb` models from selected images with `tencent/hunyuan-3d-3.1`. +- Generate an equirectangular HDRI panorama and open the model in Blender. +- Store image metadata in sidecar JSON files so prompts can be reused later. +- Browse generated 2D images and generated 3D models in the built-in gallery. + +## Requirements + +- macOS +- Python 3.11 +- Ollama running locally +- An Ollama model named `mistral:latest` +- Optional but recommended: a Replicate API token +- Optional for local image generation: local Diffusers model files +- Optional for HDRI: Topaz Photo AI CLI and local HDRI Diffusers model files +- Optional for 3D viewing: Blender installed at `/Applications/Blender.app` + +The launcher creates a local virtual environment in `.venv/` and installs Python dependencies from `requirements.txt`. + +## Quick Start + +```bash +./run.sh +``` + +The script will: + +1. Create `.venv/` if it does not exist. +2. Activate the virtual environment. +3. Install dependencies. +4. Run `python main.py`. + +If you want to force a specific Python executable: + +```bash +PYTHON=/path/to/python3.11 ./run.sh +``` + +## Ollama Setup + +The app expects Ollama at: + +```text +http://localhost:11434/api/generate +``` + +and uses: + +```text +mistral:latest +``` + +Install and run the model with: + +```bash +ollama pull mistral +ollama serve +``` + +## Replicate Setup + +Open Settings in the app and paste your Replicate API token, or set it in your shell: + +```bash +export REPLICATE_API_TOKEN="r8_..." +./run.sh +``` + +The token is saved outside the repository at: + +```text +~/Library/Application Support/3d-model-generator/settings.json +``` + +Do not commit this file. + +## Image Generation Backends + +Settings lets you choose between: + +- `Local (Diffusers)` +- `Replicate API` + +The generated prompt JSON stays the same for both modes. Replicate mode maps the same positive prompt, negative prompt, dimensions/aspect ratio, steps where supported, and batch count to each model's API. + +Configured Replicate image models: + +| Model | Use | Listed price | +| --- | --- | --- | +| `black-forest-labs/flux-schnell` | Cheap drafts | $3 / 1000 output images | +| `google/imagen-4-fast` | Balanced default | $0.02 / output image | +| `bytedance/seedream-4` | Higher quality | $0.03 / output image | + +Prices are listed by Replicate and may change. + +## 3D Generation + +Right-click a generated image and choose: + +- `Generate 3D Model` +- `Generate 3D Model + HDRI` + +3D conversion uses: + +```text +tencent/hunyuan-3d-3.1 +``` + +Current listed Replicate price in the app: + +```text +$0.16 / unit +``` + +The script waits up to 10 minutes for Replicate predictions before timing out. + +## Local Model Paths + +Some local paths are currently hard-coded and should be edited for your machine. + +In `main.py`: + +```python +MODEL_PATH = "/Volumes/SD/ML-Models/diffusers/dreamshaper_8_diffusers" +``` + +In `generate_equirect.py`: + +```python +model_path = "/Volumes/SD/ML-Models/diffusers/hdri-panorama-v1-diffusers" +topaz_cli = "/Applications/Topaz Photo AI.app/Contents/MacOS/Topaz Photo AI" +``` + +In `main.py`, external app paths are also hard-coded: + +```python +"/Applications/Blender.app/Contents/MacOS/Blender" +"Adobe Photoshop 2024" +``` + +If you only use Replicate image generation and 3D generation, the local DreamShaper path is not needed for image generation. + +## Output Files + +Generated files are written under: + +```text +web/output/ +``` + +Typical outputs: + +- `.png` generated image +- `.png.json` prompt metadata sidecar +- `.glb` generated 3D model +- `_hdri_seamless.png` generated HDRI image + +`web/output/` is ignored by Git. + +## Project Structure + +```text +main.py Desktop app, UI API, prompt flow, image generation +image_to_3d.py Replicate Hunyuan 3D conversion +generate_equirect.py HDRI generation and seam fixing +scene_setup.py Blender scene setup for generated models +web/index.html App UI +requirements.txt Python dependencies +run.sh Virtualenv setup and app launcher +``` + +## Git And Secrets + +The repository ignores local output, model weights, virtual environments, Hugging Face cache files, and local secret/config files. + +Before publishing, it is still worth running: + +```bash +git status --short --ignored +git ls-files -co --exclude-standard +``` + +Do not commit: + +- `.env` +- `settings.json` +- Replicate API tokens +- model weights +- generated media +- `web/output/` + +## Notes + +- `numpy<2` is pinned because some installed ML/runtime dependencies may still be incompatible with NumPy 2.x. +- The app currently targets macOS paths and behavior. +- HDRI generation depends on local models and Topaz Photo AI CLI. +- Replicate image generation and Hunyuan 3D generation spend Replicate credits.