Palantir Cost Tuning Skill

Palantir Cost Tuning

Overview

Optimize Foundry compute and API costs through incremental transforms, right-sized Spark profiles, efficient pagination, and usage monitoring.

Prerequisites

Active Foundry enrollment with build history
Access to Foundry resource usage metrics
Understanding of transform build patterns

Instructions

Step 1: Cost Drivers in Foundry

| Cost Category | Driver | Optimization | |---------------|--------|-------------| | Compute | Full rebuilds of large transforms | Use @incremental() | | Compute | Oversized Spark profiles | Right-size @configure profiles | | Storage | Redundant dataset snapshots | Configure retention policies | | API | High-frequency polling | Use webhooks instead | | API | Small page sizes | Use max page_size (500) |

Step 2: Convert Full Rebuilds to Incremental

from transforms.api import transform_df, Input, Output, incremental

# BEFORE: Full rebuild every run (expensive for large datasets)
@transform_df(Output("/out"), data=Input("/in"))
def expensive(data):
    return data.filter(data.status == "active")

# AFTER: Only processes new/changed rows
@incremental()
@transform_df(Output("/out"), data=Input("/in"))
def cheap(data):
    return data.filter(data.status == "active")

Step 3: Right-Size Spark Profiles

from transforms.api import configure

# DON'T: Default profile for everything
# DO: Match profile to actual data size

# Small data (< 1GB) — use lightweight transforms (no Spark)
from transforms.api import transform_polars
@transform_polars(Output("/out"), data=Input("/small_table"))
def small_job(data):
    return data.filter(data["status"] == "active")

# Medium data (1-50GB) — default profile is fine
@transform_df(Output("/out"), data=Input("/medium_table"))
def medium_job(data):
    return data.select("id", "name")

# Large data (50GB+) — explicit large profile
@configure(profile=["DRIVER_MEMORY_LARGE"])
@transform_df(Output("/out"), data=Input("/big_table"))
def large_job(data):
    return data.groupBy("region").count()

Step 4: Replace Polling with Webhooks

# EXPENSIVE: Polling every 30 seconds
import time
while True:
    result = client.ontologies.OntologyObject.list(
        ontology="co", object_type="Order", page_size=100,
    )
    process_new_orders(result.data)
    time.sleep(30)  # 2,880 API calls/day!

# CHEAP: Webhook-driven (0 polling API calls)
# Register webhook for ontology.object.created events
# See palantir-webhooks-events skill

Step 5: Monitor Usage

def log_api_usage(response):
    """Log rate limit headers to track usage patterns."""
    remaining = response.headers.get("X-RateLimit-Remaining", "?")
    limit = response.headers.get("X-RateLimit-Limit", "?")
    print(f"API usage: {remaining}/{limit} remaining")

Output

Incremental transforms reducing rebuild compute by 90%+
Right-sized Spark profiles matching actual data volumes
Webhook-driven architecture eliminating polling costs
Usage monitoring for ongoing optimization

Error Handling

| Optimization | Risk | Mitigation | |-------------|------|------------| | Incremental | Missed data on schema change | Schedule periodic full rebuild | | Polars (no Spark) | Data too large for memory | Fall back to Spark for > 1GB | | Aggressive caching | Stale data | Set TTL matching business requirements | | Webhook-only | Missed events | Periodic reconciliation job |

Resources

Next Steps

For reference architecture, see palantir-reference-architecture.

Agent Skills: Palantir Cost Tuning

Install this agent skill to your local

Skill Files