Python Dependency Management for AI Projects: pipx + Poetry

A typical generative-AI project pulls in torch, transformers, a vector store, and a web framework — each with its own opinion about which version of everything else it needs. Install a CLI tool globally with pip and it’ll happily clobber a dependency your project pins. The fix is to stop using one tool for two jobs: pipx for isolated global apps, Poetry for per-project dependencies. Used together they keep your AI environment reproducible instead of a slow accretion of version conflicts.

What are pipx and poetry?

pipx is a tool that lets you install and run Python applications in isolated environments. Think of it as npm install -g for Python, but with better isolation. Poetry, on the other hand, is a dependency management and packaging tool that makes it easy to manage project dependencies and build packages.

Setting Up Your Environment

1. Installing pipx

First, let’s install pipx. It’s recommended to use pip to install pipx globally:

python -m pip install --user pipx
python -m pipx ensurepath

2. Installing poetry using pipx

Now that we have pipx, we can use it to install poetry in an isolated environment:

pipx install poetry

Creating a New GenAI Project

1. Project Initialization

Let’s create a new project:

poetry new genai-project
cd genai-project

This creates a basic project structure:

genai-project/
├── pyproject.toml
├── README.md
├── genai_project/
│   └── __init__.py
└── tests/
    └── __init__.py

2. Configuring poetry

Let’s modify the pyproject.toml file for our GenAI project:

[tool.poetry]
name = "genai-project"
version = "0.1.0"
description = "A generative AI project using modern Python tools"
authors = ["Your Name <your.email@example.com>"]

[tool.poetry.dependencies]
python = "^3.9"
torch = "^2.0.0"
transformers = "^4.30.0"
datasets = "^2.12.0"
accelerate = "^0.20.0"

[tool.poetry.group.dev.dependencies]
pytest = "^7.3.1"
black = "^23.3.0"
isort = "^5.12.0"
flake8 = "^6.0.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

3. Installing Dependencies

Install the project dependencies:

poetry install

Working with Virtual Environments

1. Activating the Environment

Poetry automatically creates and manages virtual environments. To activate it:

poetry shell

2. Running Scripts

You can run Python scripts in your project using:

poetry run python your_script.py

Best Practices for GenAI Projects

1. Managing GPU Dependencies

For GPU support, you might need to install PyTorch with CUDA. Modify your pyproject.toml:

[tool.poetry.dependencies]
torch = { version = "^2.0.0", source = "pytorch" }

[[tool.poetry.source]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu117"
priority = "explicit"

2. Dependency Groups

Organize dependencies into groups for better management:

[tool.poetry.group.training]
optional = true
dependencies = {accelerate = "^0.20.0", wandb = "^0.15.0"}

[tool.poetry.group.inference]
optional = true
dependencies = {onnxruntime-gpu = "^1.15.0"}

Install specific groups:

poetry install --with training

3. Version Control

Add these entries to your .gitignore:

.venv/
dist/
__pycache__/
*.pyc
.pytest_cache/

Common Workflows

1. Adding New Dependencies

poetry add transformers datasets

2. Updating Dependencies

poetry update

3. Exporting Requirements

For environments that don’t use poetry:

poetry export -f requirements.txt --output requirements.txt

Troubleshooting

1. GPU Dependencies

If you encounter GPU-related issues:

• Ensure CUDA is properly installed
• Match PyTorch version with your CUDA version
• Use nvidia-smi to verify GPU availability

2. Memory Issues

For large models:

• Use poetry config virtualenvs.in-project true to create the virtual environment in your project directory
• Consider using poetry run python -m pytest instead of pytest directly

Conclusion

Using pipx and poetry together provides a robust foundation for GenAI projects. The isolation provided by pipx ensures that poetry itself doesn’t interfere with other Python tools, while poetry’s dependency management makes it easy to handle complex AI library requirements.

Remember to:

• Always use poetry for dependency management
• Keep your pyproject.toml updated
• Commit both pyproject.toml and poetry.lock to version control
• Use dependency groups to organize optional dependencies

This setup will help you maintain a clean, reproducible environment for your GenAI projects, making it easier to collaborate and deploy your models.