Back to tags
Tag

Agent Skills with tag: Inference

23 skills match this tag. Use tags to discover related Agent Skills and explore similar workflows.

data-processing-ray-data

Scalable data processing for ML workloads. Streaming execution across CPU/GPU, supports Parquet/CSV/JSON/images. Integrates with Ray Train, PyTorch, TensorFlow. Scales from single machine to 100s of nodes. Use for batch inference, data preprocessing, multi-modal data loading, or distributed ETL pipelines.

[DataProcessingRayData
Orchestra Research
Orchestra Research
33,1943,891

emerging-techniques-speculative-decoding

Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques. Use when optimizing inference speed (1.5-3.6× speedup), reducing latency for real-time applications, or deploying models with limited compute. Covers draft models, tree-based attention, Jacobi iteration, parallel token generation, and production deployment strategies.

[EmergingTechniquesSpeculativeDecoding
Orchestra Research
Orchestra Research
33,1943,891

inference-serving-llama-cpp

Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware. Use for edge deployment, M1/M2/M3 Macs, AMD/Intel GPUs, or when CUDA is unavailable. Supports GGUF quantization (1.5-8 bit) for reduced memory and 4-10× speedup vs PyTorch on CPU.

[InferenceServingLlama.cppCPU
Orchestra Research
Orchestra Research
33,1943,891

inference-serving-tensorrt-llm

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

[InferenceServingTensorRT-LLMNVIDIA
Orchestra Research
Orchestra Research
33,1943,891

inference-serving-vllm

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

[vLLMInferenceServingPagedAttention
Orchestra Research
Orchestra Research
33,1943,891

model-architecture-mamba

State-space model with O(n) complexity vs Transformers' O(n²). 5× faster inference, million-token sequences, no KV cache. Selective SSM with hardware-aware design. Mamba-1 (d_state=16) and Mamba-2 (d_state=128, multi-head). Models 130M-2.8B on HuggingFace.

[ModelArchitectureMambaState
Orchestra Research
Orchestra Research
33,1943,891

model-architecture-rwkv

RNN+Transformer hybrid with O(n) inference. Linear time, infinite context, no KV cache. Train like GPT (parallel), infer like RNN (sequential). Linux Foundation AI project. Production at Windows, Office, NeMo. RWKV-7 (March 2025). Models up to 14B parameters.

[RWKVModelArchitectureRNN
Orchestra Research
Orchestra Research
33,1943,891

optimization-awq

Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss. Use when deploying large models (7B-70B) on limited GPU memory, when you need faster inference than GPTQ with better accuracy preservation, or for instruction-tuned and multimodal models. MLSys 2024 Best Paper Award winner.

[OptimizationAWQQuantization4-Bit
Orchestra Research
Orchestra Research
33,1943,891

optimization-gguf

GGUF format and llama.cpp quantization for efficient CPU/GPU inference. Use when deploying models on consumer hardware, Apple Silicon, or when needing flexible quantization from 2-8 bit without GPU requirements.

[GGUFQuantizationllama.cppCPU
Orchestra Research
Orchestra Research
33,1943,891

optimization-gptq

Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4× memory reduction with <2% perplexity degradation, or for faster inference (3-4× speedup) vs FP16. Integrates with transformers and PEFT for QLoRA fine-tuning.

[OptimizationGPTQQuantization4-Bit
Orchestra Research
Orchestra Research
33,1943,891

optimization-hqq

Half-Quadratic Quantization for LLMs without calibration data. Use when quantizing models to 4/3/2-bit precision without needing calibration datasets, for fast quantization workflows, or when deploying with vLLM or HuggingFace Transformers.

[QuantizationHQQOptimizationMemory
Orchestra Research
Orchestra Research
33,1943,891

serving-llms-vllm

vLLM: high-throughput LLM serving, OpenAI API, quantization.

[vLLMInferenceServingPagedAttention
delorenj
delorenj
176

llama-cpp

llama.cpp local GGUF inference + HF Hub model discovery.

[llama.cppGGUFQuantizationHugging
delorenj
delorenj
176

lambda-labs-gpu-cloud

Reserved and on-demand GPU cloud instances for ML training and inference. Use when you need dedicated GPU instances with simple SSH access, persistent filesystems, or high-performance multi-node clusters for large-scale training.

[InfrastructureGPUCloudTraining
poorrican
poorrican
0

gguf-quantization

GGUF format and llama.cpp quantization for efficient CPU/GPU inference. Use when deploying models on consumer hardware, Apple Silicon, or when needing flexible quantization from 2-8 bit without GPU requirements.

[GGUFQuantizationllama.cppCPU
poorrican
poorrican
0

llama-cpp

llama.cpp local GGUF inference + HF Hub model discovery.

[llama.cppGGUFQuantizationHugging
poorrican
poorrican
0

serving-llms-vllm

vLLM: high-throughput LLM serving, OpenAI API, quantization.

[vLLMInferenceServingPagedAttention
poorrican
poorrican
0

tensorrt-llm

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

[InferenceServingTensorRT-LLMNVIDIA
poorrican
poorrican
0

lambda-labs-gpu-cloud

Reserved and on-demand GPU cloud instances for ML training and inference. Use when you need dedicated GPU instances with simple SSH access, persistent filesystems, or high-performance multi-node clusters for large-scale training.

[InfrastructureGPUCloudTraining
PoorRican
PoorRican
0

gguf-quantization

GGUF format and llama.cpp quantization for efficient CPU/GPU inference. Use when deploying models on consumer hardware, Apple Silicon, or when needing flexible quantization from 2-8 bit without GPU requirements.

[GGUFQuantizationllama.cppCPU
PoorRican
PoorRican
0

llama-cpp

llama.cpp local GGUF inference + HF Hub model discovery.

[llama.cppGGUFQuantizationHugging
PoorRican
PoorRican
0

serving-llms-vllm

vLLM: high-throughput LLM serving, OpenAI API, quantization.

[vLLMInferenceServingPagedAttention
PoorRican
PoorRican
0

tensorrt-llm

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

[InferenceServingTensorRT-LLMNVIDIA
PoorRican
PoorRican
0