Upgrade Handlers

Overview

Upgrade handlers enable coordinated on-chain upgrades across all validators at specific block heights via governance proposals. They provide a mechanism for chains to perform data migrations, parameter updates, and module upgrades in a deterministic way.

Upgrade handlers are critical for maintaining consensus during chain upgrades. All validators must run the same upgrade logic at the same height.

When to Use Upgrade Handlers

Upgrade handlers are required when:

Breaking state changes: Modifying storage formats or data structures
Module migrations: Updating module versions or parameters
Protocol upgrades: Implementing new features that require state transitions
Data migrations: Moving data between different storage locations

Basic Structure

Registering Upgrade Handlers

Upgrade handlers are registered in your app’s RegisterUpgradeHandlers() method:

Show View upgrade handler registration example

// app/upgrades.go
package app

import (
    "context"

    upgradetypes "cosmossdk.io/x/upgrade/types"
    sdk "github.com/cosmos/cosmos-sdk/types"
    "github.com/cosmos/cosmos-sdk/types/module"
)

func (app *App) RegisterUpgradeHandlers() {
    app.UpgradeKeeper.SetUpgradeHandler(
        "v1.0.0", // upgrade name (must match governance proposal)
        func(ctx context.Context, plan upgradetypes.Plan, fromVM module.VersionMap) (module.VersionMap, error) {
            sdkCtx := sdk.UnwrapSDKContext(ctx)
            sdkCtx.Logger().Info("Starting upgrade", "name", plan.Name)

            // Run module migrations
            migrations, err := app.ModuleManager.RunMigrations(ctx, app.configurator, fromVM)
            if err != nil {
                return nil, err
            }

            // Add custom migration logic here

            sdkCtx.Logger().Info("Upgrade complete", "name", plan.Name)
            return migrations, nil
        },
    )
}

Module Version Management

The upgrade handler receives and returns a module.VersionMap that tracks module versions:

// fromVM contains the module versions before the upgrade
// The returned VersionMap contains the new versions after migration
migrations, err := app.ModuleManager.RunMigrations(ctx, app.configurator, fromVM)

Organizing Upgrade Code

For better maintainability, organize upgrades in separate packages:

Directory Structure

app/
├── upgrades/
│   ├── v1_0_0/
│   │   ├── constants.go    # Upgrade name and configuration
│   │   ├── handler.go      # Main upgrade handler
│   │   └── migrations.go   # Migration logic
│   ├── v1_1_0/
│   │   ├── constants.go
│   │   ├── handler.go
│   │   └── migrations.go
│   └── types.go            # Shared types
└── upgrades.go             # RegisterUpgradeHandlers

constants.go

Show View upgrade constants definition

// app/upgrades/v1_0_0/constants.go
package v1_0_0

import (
    storetypes "cosmossdk.io/store/types"
    "github.com/yourchain/app/upgrades"
)

const UpgradeName = "v1.0.0"

var Upgrade = upgrades.Upgrade{
    UpgradeName:          UpgradeName,
    CreateUpgradeHandler: CreateUpgradeHandler,
    StoreUpgrades: storetypes.StoreUpgrades{
        Added:   []string{}, // New modules
        Deleted: []string{}, // Removed modules
    },
}

handler.go

Show View upgrade handler implementation

// app/upgrades/v1_0_0/handler.go
package v1_0_0

import (
    "context"

    storetypes "cosmossdk.io/store/types"
    upgradetypes "cosmossdk.io/x/upgrade/types"
    sdk "github.com/cosmos/cosmos-sdk/types"
    "github.com/cosmos/cosmos-sdk/types/module"
)

func CreateUpgradeHandler(
    mm *module.Manager,
    configurator module.Configurator,
    keepers *upgrades.UpgradeKeepers,
    storeKeys map[string]*storetypes.KVStoreKey,
) upgradetypes.UpgradeHandler {
    return func(c context.Context, plan upgradetypes.Plan, vm module.VersionMap) (module.VersionMap, error) {
        ctx := sdk.UnwrapSDKContext(c)

        // Run module migrations
        vm, err := mm.RunMigrations(c, configurator, vm)
        if err != nil {
            return nil, err
        }

        // Custom migrations
        if err := runCustomMigrations(ctx, keepers, storeKeys); err != nil {
            return nil, err
        }

        return vm, nil
    }
}

Migration Patterns

Parameter Migrations

Migrating module parameters to new formats:

func migrateParams(ctx sdk.Context, keeper paramskeeper.Keeper) error {
    // Get old params
    var oldParams v1.Params
    keeper.GetParamSet(ctx, &oldParams)

    // Convert to new format
    newParams := v2.Params{
        Field1: oldParams.Field1,
        Field2: convertField(oldParams.Field2),
        // New field with default value
        Field3: "default",
    }

    // Set new params
    keeper.SetParams(ctx, newParams)
    return nil
}

State Migrations

Moving data between different storage locations:

func migrateState(ctx sdk.Context, storeKey storetypes.StoreKey) error {
    store := ctx.KVStore(storeKey)

    // Iterate over old storage
    iterator := storetypes.KVStorePrefixIterator(store, oldPrefix)
    defer iterator.Close()

    for ; iterator.Valid(); iterator.Next() {
        oldKey := iterator.Key()
        value := iterator.Value()

        // Transform key/value if needed
        newKey := transformKey(oldKey)
        newValue := transformValue(value)

        // Write to new location
        store.Set(newKey, newValue)

        // Delete old entry
        store.Delete(oldKey)
    }

    return nil
}

Module Addition/Removal

Adding or removing modules during upgrade:

// In constants.go
var Upgrade = upgrades.Upgrade{
    UpgradeName: "v2.0.0",
    CreateUpgradeHandler: CreateUpgradeHandler,
    StoreUpgrades: storetypes.StoreUpgrades{
        Added:   []string{"newmodule"},
        Deleted: []string{"oldmodule"},
    },
}

// In handler.go
func CreateUpgradeHandler(...) upgradetypes.UpgradeHandler {
    return func(c context.Context, plan upgradetypes.Plan, vm module.VersionMap) (module.VersionMap, error) {
        // Delete old module version
        delete(vm, "oldmodule")

        // Initialize new module
        if err := newModuleKeeper.InitGenesis(ctx, defaultGenesis); err != nil {
            return nil, err
        }

        // Run migrations
        return mm.RunMigrations(c, configurator, vm)
    }
}

Best Practices

Always test upgrade handlers thoroughly on testnets before mainnet deployment.

Idempotency

Make migrations idempotent when possible:

func migrateSomething(ctx sdk.Context, store sdk.KVStore) error {
    // Check if migration already done
    if store.Has(migrationCompleteKey) {
        ctx.Logger().Info("Migration already completed, skipping")
        return nil
    }

    // Perform migration
    // ...

    // Mark as complete
    store.Set(migrationCompleteKey, []byte{1})
    return nil
}

Error Handling

Use comprehensive error handling and logging:

func migrate(ctx sdk.Context, keeper Keeper) error {
    ctx.Logger().Info("Starting migration", "module", "mymodule")

    count := 0
    iterator := keeper.IterateAllRecords(ctx)
    defer iterator.Close()

    for ; iterator.Valid(); iterator.Next() {
        if err := processRecord(iterator.Key(), iterator.Value()); err != nil {
            ctx.Logger().Error("Failed to migrate record",
                "key", iterator.Key(),
                "error", err,
            )
            return fmt.Errorf("migration failed at record %d: %w", count, err)
        }
        count++

        // Log progress for long migrations
        if count%1000 == 0 {
            ctx.Logger().Info("Migration progress", "processed", count)
        }
    }

    ctx.Logger().Info("Migration complete", "total_migrated", count)
    return nil
}

Testing

Create comprehensive tests for upgrade handlers:

func TestUpgradeHandler(t *testing.T) {
    app := setupApp(t)
    ctx := app.NewContext(false, tmproto.Header{Height: 1})

    // Setup pre-upgrade state
    setupOldState(t, ctx, app)

    // Run upgrade handler
    _, err := v1_0_0.CreateUpgradeHandler(
        app.ModuleManager,
        app.configurator,
        &upgrades.UpgradeKeepers{
            // ... keepers
        },
        app.keys,
    )(ctx, upgradetypes.Plan{Name: "v1.0.0"}, app.ModuleManager.GetVersionMap())

    require.NoError(t, err)

    // Verify post-upgrade state
    verifyNewState(t, ctx, app)
}

Upgrade Process

Create Upgrade Proposal

Submit a governance proposal with the upgrade details:

mantrachaind tx gov submit-proposal software-upgrade v1.0.0 \
    --title "Upgrade to v1.0.0" \
    --description "Upgrade description" \
    --upgrade-height 1000000 \
    --from validator \
    --deposit 10000000stake

Vote on Proposal

Validators and delegators vote on the upgrade:

mantrachaind tx gov vote 1 yes --from validator

Prepare Binary

Build and distribute the new binary with the upgrade handler:

# Build new binary
make build

# Test upgrade on local network
./scripts/test-upgrade.sh

# Distribute to validators
# Use Cosmovisor for automated upgrades

Monitor Upgrade

Watch logs during the upgrade height:

# Monitor upgrade logs
tail -f ~/.mantrachaind/logs/upgrade.log

# Verify upgrade success
mantrachaind query upgrade applied v1.0.0

Cosmos EVM Specific Migrations

For Cosmos EVM chains, specific migrations include:

ERC20 Precompiles Migration: Required for v0.3.x to v0.4.0
Fee Market Parameters: Updating EIP-1559 parameters
Custom Precompiles: Registering new precompiled contracts
EVM State: Migrating account balances or contract storage

Troubleshooting

Consensus Failure

Symptom: Chain halts with consensus failure at upgrade height Causes:

Validators running different binary versions
Upgrade handler not registered
Non-deterministic migration logic

Solution:

Ensure all validators have the same binary
Verify upgrade handler is registered
Review migration logic for non-determinism

Upgrade Panic

Symptom: Node panics during upgrade Causes:

Unhandled error in migration
Missing required state
Invalid type assertions

Solution:

Add comprehensive error handling
Validate state before migration
Use safe type conversions

State Corruption

Symptom: Invalid state after upgrade Causes:

Partial migration completion
Incorrect data transformation
Missing cleanup of old data

Solution:

Make migrations atomic
Thoroughly test transformations
Ensure old data is properly cleaned up

About

Build

Learn

Migrations

Overview

When to Use Upgrade Handlers

Basic Structure

Registering Upgrade Handlers

Module Version Management

Organizing Upgrade Code

Directory Structure

constants.go

handler.go

Migration Patterns

Parameter Migrations

State Migrations

Module Addition/Removal

Best Practices

Idempotency

Error Handling

Testing

Upgrade Process

Create Upgrade Proposal

Vote on Proposal

Prepare Binary

Monitor Upgrade

Cosmos EVM Specific Migrations

Troubleshooting

Consensus Failure

Upgrade Panic

State Corruption

References

​Overview

​When to Use Upgrade Handlers

​Basic Structure

​Registering Upgrade Handlers

​Module Version Management

​Organizing Upgrade Code

​Directory Structure

​constants.go

​handler.go

​Migration Patterns

​Parameter Migrations

​State Migrations

​Module Addition/Removal

​Best Practices

​Idempotency

​Error Handling

​Testing

​Upgrade Process

​Create Upgrade Proposal

​Vote on Proposal

​Prepare Binary

​Monitor Upgrade

​Cosmos EVM Specific Migrations

​Troubleshooting

​Consensus Failure

​Upgrade Panic

​State Corruption

​References

Overview

When to Use Upgrade Handlers

Basic Structure

Registering Upgrade Handlers

Module Version Management

Organizing Upgrade Code

Directory Structure

constants.go

handler.go

Migration Patterns

Parameter Migrations

State Migrations

Module Addition/Removal

Best Practices

Idempotency

Error Handling

Testing

Upgrade Process

Create Upgrade Proposal

Vote on Proposal

Prepare Binary

Monitor Upgrade

Cosmos EVM Specific Migrations

Troubleshooting

Consensus Failure

Upgrade Panic

State Corruption

References