Agent Skills: Beta Testing Strategy

Beta Testing Strategy

End-to-end beta testing workflow for Apple platform apps — from TestFlight setup through feedback collection to launch readiness decision.

When This Skill Activates

Use this skill when the user:

• Wants to run a beta test or plan a beta program
• Needs to set up TestFlight for internal or external testing
• Asks how to collect user feedback during beta
• Wants to know if their app is ready to launch
• Needs help designing a beta feedback survey
• Asks "how many beta testers do I need?"
• Mentions "TestFlight", "beta testers", "soft launch", or "launch readiness"
• Wants to plan a structured beta rollout
• Needs a go/no-go framework for shipping

Process

Phase 1: TestFlight Program Strategy

Before recruiting testers, understand Apple's two-tier beta system.

Internal Testing

| Attribute | Details | |-----------|---------| | Max testers | 100 | | App Review required | No | | Build availability | Immediate after upload | | Tester requirement | Must be App Store Connect users | | Best for | Team, close collaborators, developer friends | | Expiration | 90 days from build upload |

Use internal testing for:

• Catching crashes and obvious bugs before anyone else sees the app
• Validating core flows work end-to-end
• Getting feedback from people who will be honest (friends, fellow devs)

External Testing

| Attribute | Details | |-----------|---------| | Max testers | 10,000 | | App Review required | Yes (Beta App Review — usually 24-48 hours) | | Build availability | After Beta App Review approval | | Tester requirement | Anyone with an email address and iOS/macOS device | | Best for | Real users, broader audience, validating product-market fit | | Expiration | 90 days from build upload |

Use external testing for:

• Validating with real users who have no context about your app
• Stress-testing with diverse devices, OS versions, and usage patterns
• Gathering signal on whether the value proposition resonates

Group Management

Create tester cohorts for targeted feedback:

| Cohort | Size | Purpose | What to Ask | |--------|------|---------|-------------| | Power Users | 10-20 | Deep feature testing, edge cases | Does this handle your advanced workflows? | | Casual Users | 20-50 | First-impression and onboarding quality | Was anything confusing in the first 5 minutes? | | Accessibility Testers | 5-10 | VoiceOver, Dynamic Type, color contrast | Can you complete core tasks with accessibility features? | | Domain Experts | 5-10 | Validate domain-specific correctness | Is the [domain] logic accurate and trustworthy? |

TestFlight group tips:

• Name groups clearly (e.g., "Wave 1 - Power Users", "Wave 2 - General")
• Use different "What to Test" descriptions per group
• Stagger group invitations — don't invite everyone at once

Phase 2: Beta Tester Recruitment

How Many Testers Per Stage

| Stage | Testers | Duration | Goal | |-------|---------|----------|------| | Internal alpha | 10-20 | 1-2 weeks | Crash-free, core flows work | | External wave 1 | 50-200 | 2 weeks | Validate UX, find confusion points | | External wave 2 | 200-1,000 | 2 weeks | Stress-test, validate at scale | | Open beta (optional) | 1,000+ | 1-2 weeks | Final validation, build buzz |

Rule of thumb: You need at least 50 external testers to get meaningful signal. Below 50, individual preferences dominate.

Where to Find Beta Testers

High-quality sources (engaged, will give feedback):

• Indie dev communities — Indie Dev Monday, iOS Dev Weekly Slack, Mastodon #iosdev
• Reddit communities — r/iOSBeta, r/apple, domain-specific subreddits (r/productivity, r/fitness, etc.)
• Twitter/X — Post "Looking for beta testers for [one-line pitch]" with a screenshot
• ProductHunt Upcoming — List your app, collect emails before launch
• Friends and family — Honest if you tell them honesty matters more than kindness

Medium-quality sources (volume, less feedback):

• BetaList — Submit for free listing
• Beta testing communities — BetaFamily, ErliBird
• Discord servers — App development and domain-specific servers

Low-quality sources (avoid for signal):

• Random giveaway sites (testers who just want free stuff)
• Paid beta testers (financially motivated, not genuine users)

Incentive Strategies

| Incentive | Best For | Notes | |-----------|----------|-------| | Early access | All testers | The default — being first is often enough | | Lifetime free / pro unlock | Power users | Strong motivator, limited cost to you | | Credit in app (About screen) | Engaged testers | Recognition matters to some users | | Direct access to developer | Power users | They feel heard, you get deep feedback | | Discount at launch | Wave 2+ testers | Good for larger cohorts |

What NOT to offer: Cash payment for testing. It attracts the wrong people and biases feedback.

Phase 3: Feedback Collection Methodology

Collect feedback through multiple channels — different methods catch different signals.

Channel 1: In-App Feedback Form

Build a simple feedback mechanism directly in the app. Three fields are enough:

1. What's broken? (bugs, crashes, errors)
2. What's confusing? (UX that doesn't make sense)
3. What's missing? (features you expected but didn't find)

Implementation tips:

• Add a "Send Feedback" button in Settings (always visible during beta)
• Include device info, OS version, and app version automatically
• Attach a screenshot option (but don't require it)
• Send to a dedicated email or use a simple form service
• Timestamp and tag by tester cohort

Channel 2: TestFlight Built-In Feedback

TestFlight's native feedback is surprisingly useful:

• Users take a screenshot, annotate it, and add text
• Feedback arrives in App Store Connect with device/OS metadata
• Crash reports are automatic
• No extra infrastructure needed

Tip: In your TestFlight "What to Test" field, be specific:

This week, please test:
1. Creating a new [item] from scratch
2. Editing an existing [item]
3. Sharing [item] with someone

Report anything confusing or broken via TestFlight feedback (screenshot + description).

Channel 3: Short Surveys (Max 5 Questions)

Send a survey at the end of each beta wave. Use AskUserQuestion to help design survey questions tailored to the app.

Template survey (adapt per app):

1. How would you rate the overall experience? (1-5 stars)
2. What was the most confusing part? (free text)
3. What feature would make you use this daily? (free text)
4. Would you pay for this app? (Yes / No / Maybe — if Yes, how much?)
5. How likely are you to recommend this to a friend? (0-10 NPS scale)

Survey rules:

• Maximum 5 questions — completion rate drops 20% per additional question
• Always include one open-ended question (the best insights come from free text)
• Send via email, not in-app (don't interrupt usage)
• Send 5-7 days after they start testing (enough time to form opinions)

Channel 4: 1-on-1 User Interviews

The highest-signal feedback channel. Do 5-8 interviews per beta wave.

Who to interview:

• 2-3 testers who used the app heavily (understand power user needs)
• 2-3 testers who tried it once and stopped (understand drop-off reasons)
• 1-2 testers from the accessibility cohort

Logistics:

• 20-30 minutes via video call or phone
• Record with permission (for your notes, not public)
• Take written notes even if recording

Phase 4: User Interview Guide

The 5 Key Questions

Ask these in order. Each builds on the previous:

1. "What were you trying to do when you opened the app?"

   • Reveals their mental model and expectations
   • Listen for: Does their goal match your intended use case?

2. "Walk me through what happened."

   • Have them narrate their experience step by step
   • Listen for: Where did they pause, backtrack, or get stuck?

3. "What did you expect to happen at [specific moment]?"

   • Reveals UX gaps between expectation and reality
   • Listen for: Mismatches between your design and their mental model

4. "What was the most confusing part?"

   • Direct question about pain points
   • Listen for: Recurring themes across multiple interviews

5. "If this app cost $X/month, what would make it worth paying for?"

   • Reveals perceived value and priority features
   • Listen for: What they mention first (that's the hero feature)

How to Listen (Interview Discipline)

Do:

• Nod and say "Tell me more about that"
• Ask "Why?" at least twice to get past surface answers
• Take notes on exact phrases they use (their words, not your interpretation)
• Let silence sit — people fill silence with valuable thoughts
• Ask "What else?" after they finish answering

Don't:

• Defend your design decisions ("Well, the reason it works that way is...")
• Explain how to use the feature ("Oh, you just need to swipe left")
• Lead the witness ("Don't you think the onboarding was clear?")
• Dismiss feedback ("That's an edge case" or "Nobody else reported that")
• Promise to fix things in the interview ("We'll definitely add that")

The golden rule: Your job is to understand their experience, not to educate them about your app. If they're confused, the app is confusing — full stop.

Phase 5: Interpreting Beta Feedback

Signal vs. Noise Framework

Not all feedback is equal. Use frequency to determine priority:

| Frequency | Classification | Action | |-----------|---------------|--------| | 1 person mentions it | Anecdote | Note it, don't act yet | | 3 people mention it | Pattern | Investigate, consider fixing | | 5+ people mention it | Must-fix | Fix before launch | | 10+ people mention it | Showstopper | Fix immediately, send new build |

Important: Weight feedback by tester quality. One thoughtful power user's detailed report is worth more than ten casual testers saying "looks good."

Categorization Framework

Assign every piece of feedback to a priority level:

| Priority | Category | Examples | Action | |----------|----------|----------|--------| | P0 — Critical | Crash / data loss | App crashes on launch, saved data disappears, sync destroys content | Fix immediately, push new build within 24 hours | | P1 — High | Broken flow | Cannot complete core task, flow dead-ends, save doesn't work | Fix before next beta wave | | P2 — Medium | UX confusion | Users don't find feature, misunderstand UI, take wrong path | Fix before launch | | P3 — Low | Nice-to-have | Feature requests, polish suggestions, "it would be cool if..." | Add to backlog, consider for v1.1 |

Distinguishing Problem Types

Two types of negative feedback require different solutions:

"I don't understand how to do X" = UX problem

• The feature exists but is hard to find or use
• Solution: Redesign the flow, add onboarding hints, improve labels
• Usually cheaper and faster to fix

"I don't need X" = Feature/product problem

• The feature exists but doesn't match user needs
• Solution: Rethink whether this feature belongs in v1, or pivot the approach
• More expensive to fix — may require cutting or rebuilding

How to tell the difference: Ask "If this feature were easier to use, would you use it?" If yes, it's UX. If no, it's a product problem.

Phase 6: Go/No-Go Launch Decision

The Decision Framework

After completing beta testing, evaluate across three categories:

Green Light (Ship It)

All of these must be true:

• [ ] Crash-free rate > 99.5% (measured over at least 1,000 sessions)
• [ ] Core flow completion rate > 80% (users can accomplish the primary task)
• [ ] NPS > 30 (more promoters than detractors)
• [ ] Zero P0 bugs open
• [ ] Zero P1 bugs open
• [ ] No data loss or privacy issues
• [ ] App Store Review Guidelines compliance verified
• [ ] Performance acceptable on oldest supported device

Yellow Light (Ship with Caution)

Acceptable to launch, but address soon:

• Minor UX confusion that doesn't block core flow
• Missing nice-to-have features that testers requested
• P2 bugs that affect < 10% of users
• NPS between 20-30 (decent but not enthusiastic)
• Polish issues (animations, visual alignment, copy)
• Crash-free rate 99.0-99.5%

Decision: Launch, but fix yellow items in v1.0.1 within 1-2 weeks.

Red Light (Do Not Ship)

If any of these are true, delay launch:

• Crash-free rate < 99% (app is unstable)
• Core flow completion rate < 60% (users can't do the main thing)
• Any P0 bugs open (crashes, data loss)
• Data loss or corruption bugs exist
• Privacy issues (data leaking, missing privacy labels, no consent flow)
• NPS < 0 (more detractors than promoters)
• App rejected in Beta App Review (signals App Store Review will also reject)
• Core flow broken on common device/OS combinations

Decision: Go back to development. Fix all red items. Run another beta wave. Re-evaluate.

Making the Call

Use this checklist format to present the decision:

# Go/No-Go Decision: [App Name] v[X.Y]

## Date: [Date]
## Beta Duration: [X weeks]
## Total Testers: [N]
## Sessions Analyzed: [N]

### Green Criteria
- [ ] Crash-free rate: [XX.X%] (target: >99.5%)
- [ ] Core flow completion: [XX%] (target: >80%)
- [ ] NPS score: [XX] (target: >30)
- [ ] P0 bugs: [0/N] open
- [ ] P1 bugs: [0/N] open
- [ ] Data loss issues: [None/Describe]
- [ ] Privacy compliance: [Pass/Fail]
- [ ] Oldest device performance: [Pass/Fail]

### Yellow Items (Ship but fix in v1.0.1)
- [Item 1]
- [Item 2]

### Red Items (Must fix before launch)
- [Item 1 — or "None"]

### Decision: [GO / NO-GO / CONDITIONAL GO]
### Reasoning: [1-2 sentences]
### Next Steps:
1. [Action item]
2. [Action item]
3. [Action item]

Phase 7: Beta Timeline

Recommended 7-Week Schedule

| Week | Phase | Activities | Deliverables | |------|-------|------------|-------------| | 1 | Setup | Configure TestFlight groups, write "What to Test" descriptions, prepare feedback form | TestFlight ready, feedback channels set up | | 2 | Internal Alpha | Invite 10-20 internal testers, fix crash-level bugs daily | Crash-free build, core flows validated | | 3 | External Wave 1 | Invite 50-200 external testers, monitor crash reports | First external feedback collected | | 4 | Wave 1 Analysis | Send survey, conduct 5-8 user interviews, categorize feedback | Feedback report, prioritized bug/UX list | | 5 | External Wave 2 | Fix P0/P1 issues, push new build, invite 200-1,000 testers | Improved build validated at scale | | 6 | Wave 2 Analysis | Send final survey, conduct 3-5 follow-up interviews | Final feedback report, NPS score | | 7 | Go/No-Go | Evaluate all data against decision framework, make launch call | Go/No-Go decision document |

Compressed schedule (4 weeks): Combine weeks 1-2, skip wave 2, use wave 1 data for go/no-go. Only recommended if the app is simple (< 5 screens) and developer has shipped before.

Extended schedule (10 weeks): Add a third external wave for large or complex apps (health apps, financial apps, apps with sync/collaboration). Extra time catches rare bugs and edge cases.

Output Format

Present the beta testing plan as:

# Beta Testing Plan: [App Name]

## TestFlight Configuration

### Internal Group
- **Testers**: [List or count]
- **Focus**: [What they're testing]
- **Duration**: [X days]

### External Group 1: [Name]
- **Size**: [N testers]
- **Cohort**: [Power users / casual / accessibility / domain]
- **What to Test**: [Specific tasks and flows]

### External Group 2: [Name]
- **Size**: [N testers]
- **Cohort**: [...]
- **What to Test**: [...]

## Recruitment Plan
- **Sources**: [Where to find testers]
- **Incentive**: [What to offer]
- **Outreach message**: [Draft]

## Feedback Channels
1. In-app feedback form: [Yes/No, what fields]
2. TestFlight feedback: [What to Test description]
3. Survey: [Questions]
4. User interviews: [How many, who to target]

## Timeline
| Week | Phase | Key Activities |
|------|-------|----------------|
| ... | ... | ... |

## Go/No-Go Criteria
- Crash-free rate target: >99.5%
- Core flow completion target: >80%
- NPS target: >30
- P0/P1 bugs: Must be zero

## Next Steps
1. [First action item]
2. [Second action item]
3. [Third action item]

Integration with Other Skills

This skill fits in the product development pipeline after implementation and before App Store submission:

1. product-agent discover    --> Validate the idea
2. prd-generator             --> Define features
3. architecture-spec         --> Technical design
4. implementation-guide      --> Build it
5. test-spec                 --> Automated tests
6. beta-testing (THIS SKILL) --> Validate with real users
7. release-review            --> Pre-submission audit
8. app-store                 --> App Store listing and submission

Inputs from other skills:

• test-spec provides automated test coverage (should be solid before beta)
• implementation-guide provides the feature list to test against
• prd-generator provides the user stories to validate

Outputs to other skills:

• Feedback data informs release-review checklist
• NPS and user quotes feed into app-store description and marketing
• Go/no-go decision determines if release-review proceeds

Common Mistakes to Avoid

Inviting too many testers too early

Bad:  Invite 500 external testers on day 1
Good: Start with 15 internal testers, fix crashes, THEN go external

Not giving testers direction

Bad:  "Please test the app and let me know what you think"
Good: "Please try creating a new project, adding 3 tasks, and marking one
      complete. Report anything confusing or broken."

Ignoring negative feedback

Bad:  "That tester just doesn't get it"
Good: "If 3 testers don't get it, my onboarding doesn't get it"

Running beta too short

Bad:  3 days of testing, then ship
Good: Minimum 2 weeks external testing with at least 50 testers

Not acting on feedback

Bad:  Collect feedback, file it away, ship unchanged
Good: Fix P0/P1 between waves, push new build, re-test

A beta test isn't a checkbox — it's your last chance to learn before the whole world sees your app. Treat tester feedback as a gift, even when it hurts.