Explain Dynamic Test Grouping System

This document describes the dynamic test grouping system that automatically optimizes test execution in CI by balancing test groups based on execution time and test count.

Overview

The dynamic test grouping system replaces hardcoded test groups with an intelligent algorithm that:

• Automatically discovers all test files in the repository
• Uses a greedy algorithm to balance test execution times across groups
• Caches results to avoid unnecessary regeneration
• Learns from execution data to improve future groupings
• Targets 10-13 minutes execution time per group

Architecture

Core Components

1. Dynamic Test Grouper (scripts/dynamic_test_grouper.py)

   • Main script implementing the grouping algorithm
   • Handles caching, timing collection, and group generation

2. Makefile Integration

   • Dynamic targets that replace hardcoded test groups
   • Automatic cache invalidation and regeneration

3. CI Workflow Integration

   • GitHub Actions steps for cache management
   • Automatic timing data collection
   • Performance monitoring and optimization

4. Management Scripts

   • Update script for CI environments
   • Performance analysis and reporting

Algorithm Details

The system uses a greedy bin-packing algorithm:

1. Discover all test files automatically
2. Load historical timing data (if available)
3. Sort tests by execution time (descending)
4. For each test:
   - Find group with minimum total time
   - Assign test to that group
   - Update group's total time
5. Cache results for future use

File Structure

.test_groups_cache/           # Cache directory (git-ignored)
├── unit_groups.json         # Cached unit test groups
├── unit_groups.mk          # Makefile format for unit groups
├── unit_timings.json       # Historical timing data for unit tests
├── integration_groups.json # Cached integration test groups
├── integration_groups.mk  # Makefile format for integration groups
└── integration_timings.json # Historical timing data for integration tests

scripts/
└── dynamic_test_grouper.py  # Main grouping algorithm

.github/
├── scripts/
│   └── update_test_groups.sh # CI update script
└── workflows/
    ├── pull_request.yml      # Updated CI workflow
    └── test-groups-management.yml # Management workflow

Usage

Manual Group Generation

# Generate unit test groups (6 groups)
python3 scripts/dynamic_test_grouper.py --type=unit --groups=6

# Generate integration test groups (9 groups)  
python3 scripts/dynamic_test_grouper.py --type=integration --groups=9

# Force regeneration (ignore cache)
python3 scripts/dynamic_test_grouper.py --type=unit --groups=6 --force

# Update timing data from JUnit results
python3 scripts/dynamic_test_grouper.py --type=unit --update-timings --junit-dir=tests/unit/outputs

Makefile Targets

# Run a specific test group (uses dynamic groups)
make unit_test_group TEST_GROUP=1
make integration_test_group TEST_GROUP=3

# Force regenerate all groups
make regenerate_test_groups

# Clean cache
make clean_test_groups

CI Integration

The system automatically:

1. Checks for new tests before each CI run
2. Updates groups if tests have been added/removed
3. Collects timing data after test execution
4. Improves groupings over time

Configuration

Target Execution Time

The default target is 10-13 minutes per group. To modify:

# In scripts/dynamic_test_grouper.py
MAX_GROUP_TIME = 13 * 60  # 13 minutes in seconds

Default Test Time

For new tests without timing data:

# In scripts/dynamic_test_grouper.py
DEFAULT_TEST_TIME = 30    # 30 seconds

Number of Groups

Configured in CI workflow and Makefile:

• Unit tests: 6 groups
• Integration tests: 9 groups

To change, update:

• .github/workflows/pull_request.yml (matrix strategy)
• Makefile calls to the grouper script

Monitoring & Analysis

Performance Reports

The system generates statistics showing:

# Group Statistics:
# Group 1: 12 tests, 8m 45s
# Group 2: 10 tests, 9m 12s
# Group 3: 11 tests, 8m 56s
# ...
# Max group time: 9.2m, Min: 8.7m, Avg: 9.0m

Automated Monitoring

• Weekly optimization runs via GitHub Actions
• Performance reports generated as artifacts
• Automatic issue creation for balance problems

Manual Analysis

# View current group statistics
./.github/scripts/update_test_groups.sh --stats

# Force regeneration with analysis
make regenerate_test_groups

Troubleshooting

Cache Issues

If groups seem incorrect:

# Clear cache and regenerate
make clean_test_groups
make regenerate_test_groups

Missing Timing Data

For new repositories or after major changes:

# The system will use default times initially
# After first CI run, timing data will be collected automatically

Group Imbalance

If groups exceed target time:

1. Increase number of groups in CI configuration
2. Identify slow tests for optimization
3. Use the management workflow for analysis

CI Integration Issues

Check that:

1. Cache keys in CI workflow match current setup
2. Python 3 is available in the CI environment
3. File permissions allow script execution

Cache Invalidation

The cache is invalidated when:

1. New tests are added or existing tests removed
2. Script is modified (dynamic_test_grouper.py)
3. Force regeneration is requested
4. Cache files are corrupted or missing

Performance Benefits

Before (Hardcoded Groups)

• Manual group assignment
• Unbalanced execution times
• New tests forgotten in CI
• No optimization over time

After (Dynamic Groups)

• Automatic test discovery
• Optimal load balancing
• Self-improving over time
• Target execution time control
• Zero maintenance required

Development

Adding New Features

To extend the system:

1. Modify the Python script for algorithm changes
2. Update CI workflows for new caching strategies
3. Test with various scenarios before deployment

Algorithm Improvements

Possible enhancements:

• Machine learning for better time prediction
• Dependency-aware grouping for related tests
• Dynamic group sizing based on available runners
• Test prioritization for faster feedback

Limitations

1. Initial runs use estimated times (30s default)
2. Perl dependency for the grouping script
3. Cache storage consumes some CI storage
4. JUnit XML format required for timing collection

References

• Bin Packing Algorithms
• GitHub Actions Caching
• JUnit XML Format