Python Project Structure: Best Practices

Welcome to Python Project Structure! Think of this as architectural blueprints for your code. A well-structured project is maintainable, scalable, and easy for others to understand.

Why Project Structure Matters

Good structure provides:

• Clarity - Easy to find and understand code
• Maintainability - Simple to modify and extend
• Collaboration - Multiple developers can work effectively
• Testing - Easy to write and run tests
• Deployment - Clear separation of concerns

Basic Project Structure

Simple Script Project

my_script/
├── script.py          # Main script
├── requirements.txt   # Dependencies
└── README.md         # Documentation

Small Package Project

my_package/
├── my_package/
│   ├── __init__.py
│   ├── core.py
│   └── utils.py
├── tests/
│   ├── __init__.py
│   └── test_core.py
├── setup.py
├── requirements.txt
├── README.md
└── LICENSE

Large Application Project

my_app/
├── src/
│   └── my_app/
│       ├── __init__.py
│       ├── core/
│       │   ├── __init__.py
│       │   ├── models.py
│       │   └── services.py
│       ├── api/
│       │   ├── __init__.py
│       │   ├── routes.py
│       │   └── handlers.py
│       └── utils/
│           ├── __init__.py
│           └── helpers.py
├── tests/
│   ├── __init__.py
│   ├── unit/
│   ├── integration/
│   └── fixtures/
├── docs/
├── scripts/
├── requirements/
│   ├── base.txt
│   ├── dev.txt
│   └── prod.txt
├── setup.py
├── pyproject.toml
├── README.md
├── LICENSE
└── .gitignore

The src/ Layout Pattern

For larger projects, use the src/ layout:

my_project/
├── src/
│   └── my_package/
│       ├── __init__.py
│       ├── module1.py
│       └── subpackage/
│           ├── __init__.py
│           └── module2.py
├── tests/
├── docs/
├── scripts/
├── requirements.txt
├── setup.py
└── README.md

Benefits:

• Clear separation between code and project files
• No import path issues
• Clean for development and packaging

Package Organization Patterns

By Layer (Web Applications)

my_webapp/
├── src/
│   └── myapp/
│       ├── __init__.py
│       ├── config.py          # Configuration
│       ├── models/            # Data models
│       │   ├── __init__.py
│       │   ├── user.py
│       │   └── post.py
│       ├── views/             # Presentation layer
│       │   ├── __init__.py
│       │   ├── user_views.py
│       │   └── post_views.py
│       ├── controllers/       # Business logic
│       │   ├── __init__.py
│       │   ├── user_controller.py
│       │   └── post_controller.py
│       ├── services/          # External services
│       │   ├── __init__.py
│       │   ├── email_service.py
│       │   └── payment_service.py
│       ├── utils/             # Utilities
│       │   ├── __init__.py
│       │   └── helpers.py
│       └── db/                # Database
│           ├── __init__.py
│           └── connection.py

By Feature (Domain-Driven Design)

ecommerce/
├── src/
│   └── ecommerce/
│       ├── __init__.py
│       ├── products/
│       │   ├── __init__.py
│       │   ├── models.py
│       │   ├── views.py
│       │   ├── controllers.py
│       │   └── tests/
│       ├── orders/
│       │   ├── __init__.py
│       │   ├── models.py
│       │   ├── views.py
│       │   ├── controllers.py
│       │   └── tests/
│       ├── users/
│       │   ├── __init__.py
│       │   ├── models.py
│       │   ├── views.py
│       │   ├── controllers.py
│       │   └── tests/
│       └── shared/
│           ├── __init__.py
│           ├── utils.py
│           └── exceptions.py

By Component Type

data_pipeline/
├── src/
│   └── pipeline/
│       ├── __init__.py
│       ├── extractors/
│       │   ├── __init__.py
│       │   ├── csv_extractor.py
│       │   ├── api_extractor.py
│       │   └── database_extractor.py
│       ├── transformers/
│       │   ├── __init__.py
│       │   ├── data_cleaner.py
│       │   ├── normalizer.py
│       │   └── aggregator.py
│       ├── loaders/
│       │   ├── __init__.py
│       │   ├── database_loader.py
│       │   ├── file_loader.py
│       │   └── api_loader.py
│       └── utils/
│           ├── __init__.py
│           └── validation.py

Configuration Files

setup.py - Traditional Packaging

from setuptools import setup, find_packages

setup(
    name="my_package",
    version="1.0.0",
    author="Your Name",
    author_email="your.email@example.com",
    description="A short description",
    long_description=open('README.md').read(),
    long_description_content_type="text/markdown",
    url="https://github.com/yourusername/my_package",
    packages=find_packages(where="src"),
    package_dir={"": "src"},
    classifiers=[
        "Development Status :: 3 - Alpha",
        "Intended Audience :: Developers",
        "License :: OSI Approved :: MIT License",
        "Programming Language :: Python :: 3",
        "Programming Language :: Python :: 3.8",
        "Programming Language :: Python :: 3.9",
    ],
    python_requires=">=3.8",
    install_requires=[
        "requests>=2.25.0",
        "pandas>=1.3.0",
    ],
    extras_require={
        "dev": ["pytest>=6.0", "black", "flake8"],
        "docs": ["sphinx>=4.0"],
    },
    entry_points={
        "console_scripts": [
            "my_command=my_package.cli:main",
        ],
    },
)

pyproject.toml - Modern Packaging (Python 3.8+)

[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "my_package"
version = "1.0.0"
description = "A short description"
readme = "README.md"
license = {file = "LICENSE"}
requires-python = ">=3.8"
authors = [
    {name = "Your Name", email = "your.email@example.com"},
]
classifiers = [
    "Development Status :: 3 - Alpha",
    "Intended Audience :: Developers",
    "License :: OSI Approved :: MIT License",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
]
dependencies = [
    "requests>=2.25.0",
    "pandas>=1.3.0",
]

[project.optional-dependencies]
dev = ["pytest>=6.0", "black", "flake8"]
docs = ["sphinx>=4.0"]

[project.scripts]
my_command = "my_package.cli:main"

[tool.setuptools.packages.find]
where = ["src"]

[tool.black]
line-length = 88
target-version = ['py38']

[tool.pytest.ini_options]
minversion = "6.0"
addopts = "-ra -q"
testpaths = ["tests"]

requirements.txt - Dependencies

# Core dependencies
requests==2.28.1
pandas==1.5.2
numpy==1.23.4

# Development dependencies
pytest==7.1.3
black==22.10.0
flake8==5.0.4
mypy==0.991

# Optional dependencies
matplotlib==3.6.2
seaborn==0.12.1

MANIFEST.in - Include Extra Files

include README.md
include LICENSE
include requirements.txt
recursive-include docs *.md
recursive-include tests *.py
global-exclude *.pyc
global-exclude __pycache__

Testing Structure

Basic Test Structure

tests/
├── __init__.py
├── test_module1.py
├── test_module2.py
└── fixtures/
    ├── __init__.py
    └── test_data.json

Advanced Test Structure

tests/
├── __init__.py
├── unit/
│   ├── __init__.py
│   ├── test_models.py
│   └── test_utils.py
├── integration/
│   ├── __init__.py
│   ├── test_api.py
│   └── test_database.py
├── e2e/
│   ├── __init__.py
│   └── test_user_flow.py
├── fixtures/
│   ├── __init__.py
│   ├── sample_data.json
│   └── mock_responses.py
└── conftest.py

conftest.py - Pytest Configuration

import pytest
import os
import sys

# Add src to path for imports
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))

@pytest.fixture
def sample_data():
    return {"key": "value"}

@pytest.fixture(scope="session")
def db_connection():
    # Setup database connection
    connection = create_test_db()
    yield connection
    # Teardown
    connection.close()

Documentation Structure

docs/
├── index.md
├── installation.md
├── usage.md
├── api/
│   ├── models.md
│   └── utils.md
├── examples/
│   ├── basic_usage.py
│   └── advanced_usage.py
└── images/
    └── diagram.png

Scripts and Tools

Development Scripts

scripts/
├── setup_dev.py      # Development environment setup
├── run_tests.py      # Test runner
├── build_docs.py     # Documentation builder
├── deploy.py         # Deployment script
└── format_code.py    # Code formatting

Command-Line Interface

my_package/
├── __init__.py
├── cli.py            # Command-line interface
└── commands/
    ├── __init__.py
    ├── init.py
    ├── build.py
    └── deploy.py

# cli.py
import click
from .commands.init import init
from .commands.build import build

@click.group()
def cli():
    """My Package CLI"""
    pass

cli.add_command(init)
cli.add_command(build)

if __name__ == "__main__":
    cli()

Environment Management

.env Files

# .env
DATABASE_URL=postgresql://localhost/mydb
SECRET_KEY=your-secret-key-here
DEBUG=True

# .env.prod
DATABASE_URL=postgresql://prod-server/prod-db
SECRET_KEY=production-secret-key
DEBUG=False

Environment Configuration

# config.py
import os
from dotenv import load_dotenv

load_dotenv()

class Config:
    DATABASE_URL = os.getenv('DATABASE_URL', 'sqlite:///default.db')
    SECRET_KEY = os.getenv('SECRET_KEY', 'default-secret')
    DEBUG = os.getenv('DEBUG', 'False').lower() == 'true'

config = Config()

Version Control

.gitignore for Python

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg

# PyInstaller
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/

# Virtual environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# IDEs
.vscode/
.idea/
*.swp
*.swo

# OS
.DS_Store
Thumbs.db

Practical Examples

Example 1: Web API Project

web_api/
├── src/
│   └── myapi/
│       ├── __init__.py
│       ├── app.py
│       ├── config.py
│       ├── models/
│       │   ├── __init__.py
│       │   ├── user.py
│       │   └── post.py
│       ├── routes/
│       │   ├── __init__.py
│       │   ├── users.py
│       │   └── posts.py
│       ├── services/
│       │   ├── __init__.py
│       │   ├── auth.py
│       │   └── email.py
│       └── utils/
│           ├── __init__.py
│           └── validation.py
├── tests/
│   ├── __init__.py
│   ├── unit/
│   ├── integration/
│   └── fixtures/
├── docs/
├── scripts/
├── requirements/
├── pyproject.toml
├── README.md
└── .gitignore

Example 2: Data Science Project

data_science/
├── src/
│   └── analysis/
│       ├── __init__.py
│       ├── data/
│       │   ├── __init__.py
│       │   ├── loaders.py
│       │   └── processors.py
│       ├── models/
│       │   ├── __init__.py
│       │   ├── train.py
│       │   └── predict.py
│       ├── visualization/
│       │   ├── __init__.py
│       │   └── plots.py
│       └── utils/
│           ├── __init__.py
│           └── metrics.py
├── notebooks/
│   ├── exploratory_analysis.ipynb
│   └── model_training.ipynb
├── data/
│   ├── raw/
│   ├── processed/
│   └── models/
├── tests/
├── docs/
├── requirements.txt
├── pyproject.toml
├── README.md
└── .gitignore

Example 3: CLI Tool Project

cli_tool/
├── src/
│   └── mytool/
│       ├── __init__.py
│       ├── cli.py
│       ├── commands/
│       │   ├── __init__.py
│       │   ├── init.py
│       │   ├── build.py
│       │   └── deploy.py
│       ├── core/
│       │   ├── __init__.py
│       │   ├── processor.py
│       │   └── validator.py
│       └── utils/
│           ├── __init__.py
│           └── helpers.py
├── tests/
├── docs/
├── scripts/
├── pyproject.toml
├── README.md
└── .gitignore

Best Practices

1. Use the src/ Layout

# Good
my_project/
├── src/
│   └── my_package/
└── tests/

# Avoid
my_project/
├── my_package/
└── tests/

2. Separate Concerns

# Good: Clear separation
models/     # Data structures
views/      # Presentation
controllers/# Business logic
services/   # External integrations

# Bad: Mixed concerns
utils/      # Everything dumped here

3. Use Descriptive Names

# Good
user_management/
data_processing/
payment_gateway/

# Bad
stuff/
utils/
misc/

4. Keep Tests Close to Code

# Good
src/my_package/feature.py
tests/test_feature.py

# Bad
src/my_package/feature.py
tests/unit/test_feature.py  # Too deep

5. Document Everything

# README.md structure
# Project Title
# Description
# Installation
# Usage
# API Reference
# Contributing
# License

6. Use Configuration Files

# pyproject.toml for modern Python
# setup.py for legacy support
# requirements.txt for dependencies
# MANIFEST.in for extra files

Common Project Templates

Flask Web App

flask_app/
├── app/
│   ├── __init__.py
│   ├── routes.py
│   ├── models.py
│   └── templates/
├── tests/
├── requirements.txt
├── config.py
└── run.py

Django Project

django_project/
├── manage.py
├── project_name/
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── apps/
│   ├── users/
│   └── blog/
└── static/

FastAPI Project

fastapi_app/
├── app/
│   ├── __init__.py
│   ├── main.py
│   ├── dependencies.py
│   ├── routers/
│   └── models/
├── tests/
├── requirements.txt
└── Dockerfile

Practice Exercises

Exercise 1: Project Template

Create a project template for:

• A web API (FastAPI/Flask)
• A data science project
• A CLI tool
• A machine learning package

Exercise 2: Refactor Existing Code

Take a single-file script and refactor it into a proper package structure with:

• Separate modules for different concerns
• Tests
• Documentation
• Configuration management

Exercise 3: Multi-Environment Setup

Create a project that supports:

• Development environment
• Testing environment
• Production environment
• Different configuration files
• Environment-specific dependencies

Exercise 4: Package Distribution

Create a complete package that can be:

• Installed with pip
• Uploaded to PyPI
• Has proper metadata
• Includes command-line scripts
• Has comprehensive tests

Summary

Project structure is the foundation of maintainable code:

Key Principles:

• Separation of concerns - Different responsibilities in different modules
• Clear hierarchy - Logical organization with packages and subpackages
• Testing integration - Tests alongside code
• Documentation - Clear README and API docs
• Configuration management - Environment-specific settings

Essential Files:

• pyproject.toml or setup.py - Package configuration
• requirements.txt - Dependencies
• README.md - Project documentation
• .gitignore - Version control exclusions
• tests/ - Test suite

Best Practices:

• Use src/ layout for packages
• Group by feature, not by type
• Keep tests close to code
• Document everything
• Use configuration files for settings

Next: Distribution - creating installable Python packages! 📦