NATS Server Knowledge Patch
Use this skill when configuring, upgrading, operating, or writing clients for NATS Server and JetStream. Check the project or deployment's actual server version before applying version-dependent advice, and prefer its configuration, code, and observed behavior when they differ.
Reference index
| Reference | Topics | | --- | --- | | Consumers and sourcing | Priority groups, pausing, promotion, durable sourcing, consumer reset | | Networking and accounts | Tracing, accounts, leafnodes, routes, gateways, TLS, MQTT, ACK subjects | | Operations and recovery | Strict requests, health, metadata, storage, overload, upgrades and downgrades | | Streams and publishing | TTL, delete markers, buffering, batches, counters, schedules, transforms |
Start with compatibility hazards
JetStream requests reject unknown fields
Strict request validation is enabled by default. Remove unknown JSON fields from clients instead of relying on the compatibility escape hatch. If a staged migration needs the earlier behavior temporarily:
jetstream {
strict: false
}
Core-to-JetStream ingestion is bounded
Core NATS publishes entering a stream can be dropped with
429 JSStreamTooManyRequests after either per-stream buffer limit is reached.
Publishers that need confirmation should use JetStream publishing and await
PubAcks. Tune only after measuring:
jetstream {
max_buffered_msgs: 50000
max_buffered_size: 256mib
}
The defaults are 10,000 messages and 128 MB.
Downgrades can rebuild or offline assets
Before moving back from servers with newer JetStream features, verify that the
destination release can safely recognize those assets. A first restart across
a stream-state format boundary may rescan message blocks, increasing CPU and
delaying health without losing data. WorkQueue and Interest sources also fall
back to less reliable ephemeral sourcing when downgraded from durable sourcing;
AckFlowControl consumers stay offline until a supporting server returns.
ACK permissions must allow both subject formats
Clients and permissions should accept both legacy and domain/account-aware ACK
and flow-control subjects. Catch-all $JS.ACK.> and $JS.FC.> rules already
work; narrow rules such as $JS.ACK.<stream>.> do not cover the v2 layout.
Always publish the reply subject received from the server unchanged.
Publishing quick reference
Choose the right batch mode
| Need | Stream option | Semantics |
| --- | --- | --- |
| All-or-nothing commit | AllowAtomicPublish | Stages a contiguous batch and acknowledges only its commit |
| High throughput with per-message checks | AllowBatchPublish | Flow-controlled fast publishing without atomic staging |
Atomic publishing requires API level 2, rejects asynchronous persistence, and
cannot be used on mirrors. Every atomic message carries one Nats-Batch-Id and
a contiguous Nats-Batch-Sequence; the last carries Nats-Batch-Commit: 1.
Fast and atomic batches can use a non-persisted end-of-batch commit message.
Enable per-message TTL deliberately
Set AllowMsgTTL on the stream before publishers send Nats-TTL:
StreamConfig{AllowMsgTTL: true}
Accepted values are integer seconds, Go durations such as 1h, and never.
Invalid or sub-second values reject the publish. never also bypasses stream
MaxAge, and AllowMsgTTL cannot later be disabled. Sources and mirrors store
the header but expire from it only when their own stream enables the feature.
Treat schedules as stored control messages
AllowMsgSchedules lets one stored message emit another message in the same
stream. Use Nats-Schedule with @at, a six-field UTC cron expression, an
alias, or @every; route with Nats-Schedule-Target or sample through
Nats-Schedule-Source. Nats-Schedule-TTL transfers to generated messages as
Nats-TTL, while the schedule record's own Nats-TTL limits that record.
Schedule subjects must be unique. The feature requires API level 2, cannot be disabled after enablement, and is rejected on sources and mirrors.
Counter streams accept increments, not ordinary messages
With AllowMsgCounter, every subject is an arbitrary-precision signed counter.
Publish Nats-Incr as a signed integer; the server stores and acknowledges the
new value. Counter mode is creation-only, requires Limits retention and API
level 2, and conflicts with mirrors, DiscardNew, message TTL, schedules, and
publishes without an increment.
Consumer and sourcing quick reference
Configure grouped pulls consistently
Grouped pull consumers require explicit acknowledgements and one
PriorityGroups entry. Pick a policy deliberately:
| Policy | Delivery behavior |
| --- | --- |
| overflow | Starts after the request's min_pending or min_ack_pending threshold is met |
| pinned_client | Selects one client and leaves others as standbys |
| prioritized | Gives the request work sooner, but work may flip-flop between clients |
For pinned clients, retain the Nats-Pin-Id response header and send it as
id on later pulls. On a 423 mismatch, clear it and retry without an ID.
Only PriorityTimeout is updatable; grouped mode and policy cannot be changed
in place.
Pause without declaring failure
Set PauseUntil at consumer creation or through the pause API. Delivery resumes
at the deadline, but heartbeats continue, so consumers should not interpret the
pause as a failed server or connection.
Durable sourcing moves the acknowledgement boundary
WorkQueue and Interest mirrors/sources use visible, replicated consumers named
JS_MIRROR_<suffix> or JS_SRC_<suffix>. Their AckFlowControl policy
acknowledges only after the receiving server persists data. It requires flow
control and heartbeats, acts like AckAll, forbids AckWait and BackOff, and
requires MaxDeliver: -1.
Use MaxAckPending to bound unacknowledged sourcing. For explicit lifecycle or
security control, pre-create the durable and reference its name and delivery
subject; then put start and filter settings on that consumer, not on the
StreamSource.
Reset delivery state with concurrency in mind
Publish an empty body or {"seq": N} to:
$JS.API.CONSUMER.RESET.<STREAM>.<CONSUMER>
An empty reset clears pending, redelivery, delivered, and consumer ack-floor
state while retaining the stream ack floor. A positive seq requests that the
next stream sequence be at least N, subject to delivery-policy bounds. Client
code must tolerate another actor resetting a non-ordered consumer and making
its delivery sequence non-monotonic.
Networking quick reference
Harden leafnode behavior per link
Use handshake_first: true inside the TLS block when TLS must precede the NATS
protocol handshake. Leafnode remotes can be added or removed on reload, and an
existing solicited remote can be suppressed with disabled: true.
The default leafnode dial timeout is one second. Increase dial_timeout globally
or on an individual remote for high-latency links:
leafnodes {
dial_timeout: 5s
}
Preserve user trace context
Set Nats-Trace-Dest to an inbox to receive distributed hop events. Add
Nats-Trace-Only: true to propagate tracing without delivering the message.
Servers preserve an existing traceparent; tracing logic should not assume it
will be rewritten.
Operations quick reference
Distinguish server and metadata health
js-server-only does not check the JetStream meta leader. Use js-meta-only
when meta-group health is the intended signal. Graceful SIGTERM exits with
status 0. Reject spaces in server, cluster, and gateway names before startup.
Detect unapplied configuration changes
Run the server with -t to hash a configuration file and compare that result
with varz.config_digest, the digest of the running configuration. A mismatch
indicates that the on-disk configuration has not been loaded.
Contain storage and consensus overload
A filestore write error freezes only the affected stream, logs write error,
and fails health checks. Other streams and core traffic continue, a replicated
stream may fail over, and the affected server must restart to recover.
Raft bounds memory and disk growth when proposals outpace commits. A lagging leader steps down for a healthier peer; if a majority is overloaded, capacity must recover before the cluster leaves its degraded state.
Applying this skill
- Determine the running server versions and whether a mixed-version upgrade or downgrade is in progress.
- Identify the affected surface: streams/publishing, consumers/sourcing, networking/accounts, or operations/recovery.
- Open the matching reference and check creation-only, restart-only, non-reloadable, and downgrade constraints before changing configuration.
- Update client parsers and permissions before enabling protocol-format flags.
- Exercise failure paths: PubAck rejection, leader changes, reset races, recovery, and configuration reloads.