Error Handling Migration Skill

Error Handling Migration

Create next-version services with standard gRPC Status error handling.

Purpose

Migrate services from CustomRPC implementations using Failure entity to standard gRPC error handling. This ensures:

Proper error visibility in distributed tracing
Standard gRPC Status codes
RFC-33 compliance

Skill Contents

Sections

Purpose
Prerequisites
Determine the Next Version Number
Key Principle: Create New Versioned Services (Not Versioned Methods)
Key Principle: Separate Proto Files for New Version Definitions
Before and After
Standard gRPC Error Codes
Proto Package Naming for NEW Services
Migration Steps
Migration Checklist
Common Pitfalls
Consumer Migration Guide
References

Available Resources

📚 references/ - Detailed documentation

ERROR PATTERNS

Prerequisites

The grpc-compliance-validate-repository command must be available for the migration checklist validation step.

Cloud agents: Pre-installed (no action needed).

Local setup:

export HOMEBREW_GITHUB_API_TOKEN=your-token
brew tap bitsoex/homebrew-bitso
brew install bitso-grpc-linter

Verify: grpc-compliance-validate-repository --help

See ../grpc-services-rfc-33/references/installation.md for details.

Determine the Next Version Number

Before starting, check what versioned services already exist to pick the correct next version suffix.

How to determine V{N}:

Search the proto files for existing versioned services (e.g., AccountServiceV2, AccountServiceV3)
Use the next available number: if V2 exists, create V3; if no versioned service exists, create V2
Apply the same suffix consistently to the service, response messages, proto file name, and handler class

Example: If AccountService and AccountServiceV2 already exist, create AccountServiceV3 — along with AccountResponseV3, account_service_v3.proto, and AccountServiceV3Handler.

Throughout this document, V{N} means the next available version number. Examples use V2 as a concrete illustration — substitute the actual version for your project.

Key Principle: Create New Versioned Services (Not Versioned Methods)

Why a new versioned service?

Cleaner separation between old and new error handling patterns
Easier for consumers to migrate (just switch stub)
Better service discovery (V{N} appears as separate service in gRPC tools)
Method names remain clean without version suffix
Simpler to eventually deprecate and remove old services

Key Principle: Separate Proto Files for New Version Definitions

Always place the new versioned service and its specific message definitions in a NEW proto file, separate from the previous version's definitions.

Why separate files?

Clean removal: deleting the old version is just deleting its file — no surgical edits inside a shared file
Proper package assignment: the new version can use a project-specific package even if the old version uses a generic one (e.g., protos.model)
Independent evolution: the new version can be modified without touching the old version at all
Clearer ownership: each file has a single purpose (one version)

What goes in the new version's file:

New versioned service definition
Version-specific response messages (ones without Failure/oneof)

What stays in the old version's file:

Old service definition (marked deprecated)
Old version-specific response messages (marked deprecated)
Shared types (request messages, payload types, enums) reused by the new version

The new version's file imports the old version's file (or its model file) to reference shared types.

Before and After

Note: Examples below use V2 as a concrete illustration. If V2 already exists in your project, substitute V3 (or the appropriate next version) throughout.

Before (CustomRPC - DO NOT USE)

service TransferService {
  rpc Transfer(TransferRequest) returns (TransferResponse);
}

message TransferResponse {
  oneof result {
    Transfer transfer = 1;
    Failure failure = 2;  // DO NOT USE
  }
}

// Old pattern
if (invalidAmount) {
  return TransferResponse.newBuilder()
    .setFailure(Failure.newBuilder()
      .setCode("INVALID_AMOUNT")
      .setMessage("Amount must be positive")
      .build())
    .build();
}

After (Standard gRPC - REQUIRED)

File: transfer_service.proto (old version — deprecated, kept for backwards compatibility)

syntax = "proto3";
package com.bitso.transfer;

/**
 * @deprecated since v2.0.0, forRemoval (planned for v3.0.0).
 * Use TransferServiceV2 instead.
 */
service TransferService {
  option deprecated = true;

  // @replacedBy: TransferServiceV2.Transfer
  rpc Transfer(TransferRequest) returns (TransferResponse) {
    option deprecated = true;
  }
}

// Deprecated since v2.0.0. Planned for removal in v3.0.0.
// Use TransferResponseV2 instead.
message TransferResponse {
  option deprecated = true;

  oneof result {
    Transfer transfer = 1;
    Failure failure = 2;
  }
}

// Shared types — NOT deprecated, reused by V2
message TransferRequest { ... }
message Transfer { ... }

File: transfer_service_v2.proto (new version — separate file)

syntax = "proto3";
package com.bitso.transfer.v2;

option java_package = "com.bitso.transfer.v2";
option java_outer_classname = "TransferServiceV2Proto";

import "transfer_service.proto";

// V2 service with standard gRPC error handling
service TransferServiceV2 {
  rpc Transfer(com.bitso.transfer.TransferRequest) returns (TransferResponseV2);
}

message TransferResponseV2 {
  com.bitso.transfer.Transfer transfer = 1;  // No oneof, no Failure
}

// Business error details
enum TransferErrorCode {
  INVALID_AMOUNT = 0;
  INSUFFICIENT_FUNDS = 1;
}

message TransferError {
  TransferErrorCode code = 1;
  string message = 2;
}

// New versioned service handler
public class TransferServiceV2Handler extends TransferServiceV2Grpc.TransferServiceV2ImplBase {

  @Override
  public void transfer(TransferRequest request, StreamObserver<TransferResponseV2> responseObserver) {
    if (invalidAmount) {
      TransferError error = TransferError.newBuilder()
        .setCode(TransferErrorCode.INVALID_AMOUNT)
        .setMessage("Amount must be positive")
        .build();

      // Use google.rpc.Status (not io.grpc.Status)
      Status status = Status.newBuilder()
        .setCode(Code.FAILED_PRECONDITION.getNumber())
        .setMessage("Business constraint violation")
        .addDetails(Any.pack(error))
        .build();

      responseObserver.onError(StatusProto.toStatusRuntimeException(status));
      return;
    }

    // Success case
    TransferResponseV2 response = TransferResponseV2.newBuilder()
      .setTransfer(transfer)
      .build();
    responseObserver.onNext(response);
    responseObserver.onCompleted();
  }
}

Standard gRPC Error Codes

Use ONLY these error codes at Bitso:

| Error Code | When to Use | |------------|-------------| | INTERNAL | Infrastructure errors, dependency unavailable, system failures | | UNKNOWN | Unknown errors from downstream services | | FAILED_PRECONDITION | Business constraint violations, business logic errors |

Key Principle: gRPC errors are TECHNICAL. Business error details go in trailers/details using google.rpc.Status.

Proto Package Naming for NEW Services

⚠️ CRITICAL: Package naming differs for existing vs NEW protos

For EXISTING protos being moved (contract distribution only):

Keep the existing package declaration unchanged for backwards compatibility
Example: If original uses package protos.model;, keep it

For NEW protos (new versioned services, new messages, new enums):

DO NOT use generic package protos.model;
Use project-specific package following Java conventions:
- Pattern: com.bitso.{service-name}.{version} or com.bitso.{service-name}.v{N}
- Example: com.bitso.iba.rate.v2, com.bitso.transfer.v3
The java_package option should align with the actual package structure

Determine project package convention by:

Check existing Java packages in the codebase (e.g., com.bitso.iba.grpc.service)
Use the same root package for new proto contracts
Look for patterns in java_package options in existing protos

// ❌ WRONG - Generic package for NEW versioned services
syntax = "proto3";
package protos.model;
option java_package = "com.bitso.iba.model";

// ✅ CORRECT - Project-specific package for NEW versioned services
syntax = "proto3";
package com.bitso.iba.rate.v2;
option java_package = "com.bitso.iba.rate.v2";

Proto file location must match package:

package com.bitso.iba.rate.v2; → src/main/resources/com/bitso/iba/rate/v2/iba_rate.proto

Migration Steps

Step 0: Create FailureHelper Utility Class (If Not Exists)

Before implementing the new versioned services, create a centralized FailureHelper in the grpc module:

package com.bitso.{service}.grpc.util;

import com.bitso.commons.protobuf.DataCommonsProto;
import io.grpc.Metadata;
import io.grpc.Status;
import io.grpc.StatusRuntimeException;
import io.grpc.protobuf.lite.ProtoLiteUtils;

/**
 * Utility class for consistent gRPC error handling.
 * Use this for all new versioned service error responses.
 */
public class FailureHelper {

    private static final Metadata.Key<DataCommonsProto.Failure> FAILURE_DETAILS_KEY =
        Metadata.Key.of(
            "bitso-failure-detail-bin",
            ProtoLiteUtils.metadataMarshaller(DataCommonsProto.Failure.getDefaultInstance()));

    /**
     * Creates a StatusRuntimeException with failure details in metadata.
     *
     * @param code The gRPC status code (INTERNAL, FAILED_PRECONDITION, UNKNOWN)
     * @param failure The failure details to include in metadata
     * @return StatusRuntimeException ready to be thrown
     */
    public static StatusRuntimeException createStatusRuntimeException(
            Status.Code code, DataCommonsProto.Failure failure) {
        Metadata metadata = new Metadata();
        metadata.put(FAILURE_DETAILS_KEY, failure);
        return code.toStatus().withDescription(failure.getCode()).asRuntimeException(metadata);
    }

    /**
     * Extracts failure details from a throwable (typically StatusRuntimeException).
     *
     * @param throwable The exception to extract failure from
     * @return The Failure details, or empty Failure if not present
     */
    public static DataCommonsProto.Failure extractFailure(Throwable throwable) {
        if (throwable instanceof StatusRuntimeException statusRuntimeException) {
            if (statusRuntimeException.getTrailers() != null
                    && statusRuntimeException.getTrailers().get(FAILURE_DETAILS_KEY) != null) {
                return statusRuntimeException.getTrailers().get(FAILURE_DETAILS_KEY);
            }
        }
        return DataCommonsProto.Failure.newBuilder().build();
    }
}

Location: {grpc-module}/src/main/java/com/bitso/{service}/grpc/util/FailureHelper.java

Step 1: Determine Version Number

Check existing proto files for versioned services:

If no versioned service exists → use V2
If ServiceV2 exists → use V3
If ServiceV3 exists → use V4
And so on

Step 2: Create New Versioned Service in a Separate Proto File

Create a new proto file (e.g., account_service_v{N}.proto). Import the existing proto to reuse shared request/payload types.

// File: account_service_v2.proto (NEW file — separate from previous version)
syntax = "proto3";
package com.bitso.account.v2;

option java_package = "com.bitso.account.v2";
option java_outer_classname = "AccountServiceV2Proto";

import "account_service.proto";

service AccountServiceV2 {
  rpc CreateAccount(original.package.CreateAccountRequest) returns (CreateAccountResponseV2);
  rpc GetAccount(original.package.GetAccountRequest) returns (GetAccountResponseV2);
}

message CreateAccountResponseV2 {
  original.package.Account account = 1;  // No oneof, no Failure
}

Do NOT add the new versioned service or its response messages to an existing proto file. Shared types (request messages, payload types, enums) remain in the old file and are imported by the new one.

Step 3: Deprecate Previous Service

Mark the previous service with both @replacedBy comments and protobuf option deprecated = true. The comments provide human-readable migration guidance; the options generate @Deprecated annotations in Java/Kotlin/Go stubs, giving gRPC clients compile-time deprecation warnings.

/**
 * @deprecated since v{CURRENT}, forRemoval (planned for v{NEXT_MAJOR}).
 * Use AccountServiceV{N} instead.
 */
service AccountService {
  option deprecated = true;

  // @replacedBy: AccountServiceV{N}.CreateAccount
  rpc CreateAccount(CreateAccountRequest) returns (CreateAccountResponse) {
    option deprecated = true;
  }
}

// Deprecated since v{CURRENT}. Planned for removal in v{NEXT_MAJOR}.
// Use CreateAccountResponseV{N} instead.
message CreateAccountResponse {
  option deprecated = true;

  oneof result {
    Account account = 1;
    Failure failure = 2;
  }
}

Deprecation comment format:

Service javadoc: @deprecated since v{CURRENT}, forRemoval (planned for v{NEXT_MAJOR}).
Message/RPC line comments: Deprecated since v{CURRENT}. Planned for removal in v{NEXT_MAJOR}.
{CURRENT} = the version where the new service is introduced (from gradle.properties)
{NEXT_MAJOR} = the next MAJOR version bump, when the old service will be deleted

What to deprecate:

| Element | When to deprecate | |---------|-------------------| | service | Always - the entire service is superseded by V{N} | | rpc methods | Always - each method has a V{N} replacement | | Request/response messages | Only if V{N} uses new message types (e.g., flattened contract) | | Shared types (enums, nested messages) | Never - if reused by V{N} service, keep them non-deprecated |

Step 4: Compile Protos

./gradlew generateProto
# OR
./gradlew :module-protos-generated:generateProto

Step 5: Implement Java Handler

Create V{N} handler class extending generated base class. See references/ERROR_PATTERNS.md for implementation patterns.

Step 6: Register New Versioned Service

Step 7: MANDATORY - Bump MAJOR Version in Protobuf Module

This step is NON-NEGOTIABLE. Adding versioned services is a BREAKING CHANGE.

Update the version in the protobuf module's gradle.properties file:

# File: {proto-module}/gradle.properties
# Before: version=1.2.3
# After:  version=2.0.0 (BREAKING CHANGE - MAJOR bump required)

# {proto-module}/gradle.properties
version=2.0.0

Why MAJOR version bump is mandatory:

External consumers depend on the published proto-only module
Adding V2 services changes the error-handling contract
Consumers must update their code to handle StatusRuntimeException instead of Failure entity
Deprecating existing services signals a migration requirement
Semantic versioning REQUIRES major bump for breaking changes

Failure to bump MAJOR version will cause:

Consumers unaware they need to migrate to new error-handling
Silent compatibility issues when consumers upgrade
Production incidents due to unexpected error-handling behavior
RFC-33 compliance violations

Migration Checklist

Pre-Implementation

[ ] Determine the correct version number (check for existing V2, V3, etc.)
[ ] Create FailureHelper utility class in grpc module if not exists
[ ] Determine correct package naming for project (check existing Java packages)

Proto Changes

[ ] Create a NEW proto file (e.g., service_v{N}.proto) — do NOT add to an existing file
[ ] Import the previous version's proto file to reuse shared types
[ ] Create new versioned service (e.g., AccountServiceV{N})
[ ] Use project-specific package for NEW protos (NOT protos.model)
[ ] Move methods to new versioned service (no version suffix on method names)
[ ] Annotate previous methods with @replacedBy: ServiceNameV{N}.methodName
[ ] Add option deprecated = true to old service, RPC methods, and superseded messages
[ ] Add deprecation timeline comments (since v{CURRENT}, forRemoval planned for v{NEXT_MAJOR})
[ ] Remove Failure entity from new versioned response messages
[ ] Remove oneof patterns in new versioned service responses
[ ] Create new response messages without Failure (e.g., AccountResponseV{N})
[ ] Define error code enumerations for all business errors
[ ] Place proto files in correct directory matching package structure

Implementation

[ ] Compile protos to generate Java stubs: ./gradlew generateProto
[ ] Implement Java handlers for new versioned service
[ ] Use FailureHelper utility for error responses (not inline implementations)
[ ] Implement standard gRPC Status codes (INTERNAL, UNKNOWN, FAILED_PRECONDITION)
[ ] Use google.rpc.Status with addDetails() for business errors

Version Management (MANDATORY)

[ ] CRITICAL: Bump MAJOR version in protobuf module's gradle.properties (e.g., 1.2.3 → 2.0.0)
[ ] Location: {proto-module}/gradle.properties
[ ] DO NOT proceed to PR creation without completing this step

Removal Planning

[ ] Document the planned removal version in proto comments (next MAJOR version)
[ ] Plan to delete old service, RPC methods, and superseded messages in the next MAJOR version bump
[ ] When the next MAJOR version is released: remove deprecated service, handler, and messages from proto files

Build Verification (MANDATORY)

[ ] Run ./gradlew clean build
[ ] Fix ALL compilation errors before proceeding
[ ] Loop: build → fix → build until SUCCESS

Validation

[ ] Verify errors appear in distributed tracing (not all marked as OK)
[ ] Run grpc-compliance-validate-repository --dir .
[ ] Run tests: ./gradlew test
[ ] ⚠️ Only create PR after build passes

Common Pitfalls

Creating Versioned Methods Instead of Versioned Services

Correct: Create TransferServiceV{N} with clean method names
Incorrect: Don't add transferV{N}() to an existing service

Placing New Version Definitions in the Same File as Previous Version

Correct: Create a separate *_v{N}.proto file for the new versioned service and its response messages
Incorrect: Adding the new versioned service/messages to the existing proto file
If multiple versions share one proto file, removing an old version requires careful, targeted edits; if each version lives in its own *_v{N}.proto file, removing an old version is as simple as deleting that file
The new version inherits the old file's package, preventing use of a project-specific package if the old file uses a generic one

Using Generic `protos.model` Package for NEW Versioned Services

Correct: Use project-specific package like com.bitso.iba.rate.v2
Incorrect: Using package protos.model; for NEW versioned services
Note: For EXISTING protos being moved, keep their original package

Not Checking for Existing Versioned Services

Required: Check if V2, V3, etc. already exist before choosing a version number
Incorrect: Blindly creating V2 when the service already has a V2 — this causes naming conflicts

Not Creating FailureHelper Utility Class

Required: Create FailureHelper in grpc module BEFORE implementing handlers
Incorrect: Inline error handling code in each service (duplicated boilerplate)
Location: {grpc-module}/src/main/java/com/bitso/{service}/grpc/util/FailureHelper.java

Forgetting to Compile Protos Before Implementing Handlers

Always run ./gradlew generateProto after proto changes
Then implement Java handlers using generated stubs

Creating PR with Broken Build

MANDATORY: Run ./gradlew clean build before creating PR
Fix ALL errors before proceeding - loop until build passes
Common issues: missing imports, wrong dependencies, proto not regenerated

Only Using Comments for Deprecation (Missing `option deprecated`)

Correct: Add both @replacedBy comments AND option deprecated = true to old service, RPCs, and superseded messages
Incorrect: Only adding javadoc @deprecated comments without the protobuf option
Comments are invisible to generated stubs - clients won't see compile-time warnings
The option deprecated = true generates @Deprecated in Java/Kotlin/Go, providing programmatic deprecation signals

Not Bumping MAJOR Version in Protobuf Module (CRITICAL ERROR)

gRPC migration is ALWAYS a breaking change - MAJOR version bump is mandatory
Adding new versioned services changes the error handling contract
Consumers must update their code to handle new error patterns
Using MINOR/PATCH violates semantic versioning and causes silent breakage
This mistake causes production incidents in downstream services

Consumer Migration Guide

// Before: Using original service with oneof Failure
TransferServiceGrpc.TransferServiceBlockingStub stub =
    TransferServiceGrpc.newBlockingStub(channel);

try {
  TransferResponse response = stub.transfer(request);
  if (response.hasFailure()) {
    // Handle failure
  }
} catch (Exception e) {
  // Handle error
}

// After: Using new versioned service with StatusRuntimeException
TransferServiceV2Grpc.TransferServiceV2BlockingStub stubV2 =
    TransferServiceV2Grpc.newBlockingStub(channel);

try {
  TransferResponseV2 response = stubV2.transfer(request);  // Same method name
  // Process success
} catch (StatusRuntimeException e) {
  Status.Code code = e.getStatus().getCode();
  if (code == Status.Code.FAILED_PRECONDITION) {
    // Extract business error details from google.rpc.Status
    com.google.rpc.Status status = StatusProto.fromThrowable(e);
    if (status != null && !status.getDetailsList().isEmpty()) {
      for (Any detail : status.getDetailsList()) {
        if (detail.is(TransferError.class)) {
          TransferError error = detail.unpack(TransferError.class);
          // Handle business error
        }
      }
    }
  }
}

References

references/ERROR_PATTERNS.md - Detailed error handling implementation patterns
RFC-33: gRPC Service Standards

Agent Skills: Error Handling Migration

Install this agent skill to your local

Skill Files