Agent Skills: Folder Organization Best Practices

Folder Organization Best Practices

Expert guidance for organizing project directories, establishing file naming conventions, and maintaining clean, navigable project structures for research and development work.

When to Use This Skill

• Setting up new projects
• Reorganizing existing projects
• Establishing team conventions
• Creating reproducible research structures
• Managing data-intensive projects

Core Principles

1. Predictability - Standard locations for common file types
2. Scalability - Structure grows gracefully with project
3. Discoverability - Easy for others (and future you) to navigate
4. Separation of Concerns - Code, data, documentation, outputs separated
5. Version Control Friendly - Large/generated files excluded appropriately

Standard Project Structure

Research/Analysis Projects

project-name/
├── README.md                 # Project overview and getting started
├── .gitignore               # Exclude data, outputs, env files
├── environment.yml          # Conda environment (or requirements.txt)
├── data/                    # Input data (often gitignored)
│   ├── raw/                # Original, immutable data
│   ├── processed/          # Cleaned, transformed data
│   └── external/           # Third-party data
├── notebooks/               # Jupyter notebooks for exploration
│   ├── 01-exploration.ipynb
│   ├── 02-analysis.ipynb
│   └── figures/            # Notebook-generated figures
├── src/                     # Source code (reusable modules)
│   ├── __init__.py
│   ├── data_processing.py
│   ├── analysis.py
│   └── visualization.py
├── scripts/                 # Standalone scripts and workflows
│   ├── download_data.sh
│   └── run_pipeline.py
├── tests/                   # Unit tests
│   └── test_analysis.py
├── docs/                    # Documentation
│   ├── methods.md
│   └── references.md
├── results/                 # Analysis outputs (gitignored)
│   ├── figures/
│   ├── tables/
│   └── models/
└── config/                  # Configuration files
    └── analysis_config.yaml

Development Projects

project-name/
├── README.md
├── .gitignore
├── setup.py                 # Package configuration
├── requirements.txt         # or pyproject.toml
├── src/
│   └── package_name/
│       ├── __init__.py
│       ├── core.py
│       └── utils.py
├── tests/
│   ├── test_core.py
│   └── test_utils.py
├── docs/
│   ├── api.md
│   └── usage.md
├── examples/                # Example usage
│   └── example_workflow.py
└── .github/                 # CI/CD workflows
    └── workflows/
        └── tests.yml

Bioinformatics/Workflow Projects

project-name/
├── README.md
├── data/
│   ├── raw/                # Raw sequencing data
│   ├── reference/          # Reference genomes, annotations
│   └── processed/          # Workflow outputs
├── workflows/               # Galaxy .ga or Snakemake files
│   ├── preprocessing.ga
│   └── assembly.ga
├── config/
│   ├── workflow_params.yaml
│   └── sample_sheet.tsv
├── scripts/                # Helper scripts
│   ├── submit_workflow.py
│   └── quality_check.py
├── results/                # Final outputs
│   ├── figures/
│   ├── tables/
│   └── reports/
└── logs/                   # Workflow execution logs

File Naming Conventions

General Rules

1. Use lowercase with hyphens or underscores

   • ✅ data-analysis.py or data_analysis.py
   • ❌ DataAnalysis.py or data analysis.py

2. Be descriptive but concise

   • ✅ process-telomere-data.py
   • ❌ script.py or process_all_the_telomere_sequencing_data_from_experiments.py

3. Use consistent separators

   • Choose either hyphens or underscores and stick with it
   • Convention: hyphens for file names, underscores for Python modules

4. Include version/date for important outputs

   • ✅ report-2026-01-23.pdf or model-v2.pkl
   • ❌ report-final-final-v3.pdf

Numbered Sequences

For sequential files (notebooks, scripts), use zero-padded numbers:

notebooks/
├── 01-data-exploration.ipynb
├── 02-quality-control.ipynb
├── 03-statistical-analysis.ipynb
└── 04-visualization.ipynb

Data Files

Include metadata in filename when possible:

data/raw/
├── sample-A_hifi_reads_2026-01-15.fastq.gz
├── sample-B_hifi_reads_2026-01-15.fastq.gz
└── reference_genome_v3.fasta

Directory Management Best Practices

What to Version Control

DO commit:

• Source code
• Documentation
• Configuration files
• Small test datasets (<1MB)
• Requirements/environment files
• README files

DON'T commit:

• Large data files (use .gitignore)
• Generated outputs
• Environment directories (venv/, conda-env/)
• Logs
• Temporary files
• API keys/secrets

.gitignore Template

# Python
__pycache__/
*.py[cod]
*$py.class
.venv/
venv/
*.egg-info/

# Jupyter
.ipynb_checkpoints/
*.ipynb_checkpoints

# Data
data/raw/
data/processed/
*.fastq.gz
*.bam
*.vcf.gz

# Outputs
results/
outputs/
*.png
*.pdf
*.html

# Logs
logs/
*.log

# Environment
.env
environment.local.yml

# OS
.DS_Store
Thumbs.db

Data Organization

Raw Data is Sacred

• Never modify raw data - Always keep originals untouched
• Store in data/raw/ and make it read-only if possible
• Document data provenance (where it came from, when downloaded)

Processed Data Hierarchy

data/
├── raw/                    # Original, immutable
├── interim/                # Intermediate processing steps
├── processed/              # Final, analysis-ready data
└── external/               # Third-party data

Documentation Standards

README.md Essentials

Every project should have a README with:

# Project Name

Brief description

## Installation

How to set up the environment

## Usage

How to run the analysis/code

## Project Structure

Brief overview of directories

## Data

Where data lives and how to access it

## Results

Where to find outputs

Code Documentation

• Docstrings for all functions/classes
• Comments for complex logic
• CHANGELOG.md for tracking changes
• TODO.md for tracking work (gitignored or removed before merge)

Common Anti-Patterns to Avoid

❌ Flat structure with everything in root

project/
├── script1.py
├── script2.py
├── data.csv
├── output1.png
├── output2.png
└── final_really_final_v3.xlsx

❌ Ambiguous naming

notebooks/
├── notebook1.ipynb
├── test.ipynb
├── analysis.ipynb
└── analysis_new.ipynb

❌ Mixed concerns

project/
├── src/
│   ├── analysis.py
│   ├── data.csv          # Data in source code directory
│   └── figure1.png       # Output in source code directory

Cleanup and Maintenance

Regular Maintenance Tasks

1. Archive old branches - Delete merged feature branches
2. Clean temp files - Remove TODO.md, NOTES.md from completed work
3. Update documentation - Keep README current with changes
4. Review .gitignore - Ensure large files aren't tracked
5. Organize notebooks - Rename/renumber as project evolves

End-of-Project Checklist

• [ ] README complete and accurate
• [ ] Code documented
• [ ] Tests passing
• [ ] Large files gitignored
• [ ] Working files removed (TODO.md, scratch notebooks)
• [ ] Final outputs in results/
• [ ] Environment files current
• [ ] License added (if applicable)

Integration with Other Skills

This skill works well with:

• python-environment - Environment setup and management
• claude-collaboration - Team workflow best practices
• jupyter-notebook-analysis - Notebook organization standards

Templates and Tools

Quick Project Setup

# Create standard research project structure
mkdir -p data/{raw,processed,external} notebooks scripts src tests docs results config
touch README.md .gitignore environment.yml

Cookiecutter Templates

Consider using cookiecutter for standardized project templates:

• cookiecutter-data-science - Data science projects
• cookiecutter-research - Research projects
• Custom team templates

References and Resources

• Cookiecutter Data Science
• A Quick Guide to Organizing Computational Biology Projects
• Good Enough Practices in Scientific Computing