Back to tags
Tag

Agent Skills with tag: Serving

8 skills match this tag. Use tags to discover related Agent Skills and explore similar workflows.

inference-serving-llama-cpp

Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware. Use for edge deployment, M1/M2/M3 Macs, AMD/Intel GPUs, or when CUDA is unavailable. Supports GGUF quantization (1.5-8 bit) for reduced memory and 4-10× speedup vs PyTorch on CPU.

[InferenceServingLlama.cppCPU
Orchestra Research
Orchestra Research
33,1943,891

inference-serving-tensorrt-llm

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

[InferenceServingTensorRT-LLMNVIDIA
Orchestra Research
Orchestra Research
33,1943,891

inference-serving-vllm

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

[vLLMInferenceServingPagedAttention
Orchestra Research
Orchestra Research
33,1943,891

serving-llms-vllm

vLLM: high-throughput LLM serving, OpenAI API, quantization.

[vLLMInferenceServingPagedAttention
delorenj
delorenj
176

serving-llms-vllm

vLLM: high-throughput LLM serving, OpenAI API, quantization.

[vLLMInferenceServingPagedAttention
poorrican
poorrican
0

tensorrt-llm

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

[InferenceServingTensorRT-LLMNVIDIA
poorrican
poorrican
0

serving-llms-vllm

vLLM: high-throughput LLM serving, OpenAI API, quantization.

[vLLMInferenceServingPagedAttention
PoorRican
PoorRican
0

tensorrt-llm

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

[InferenceServingTensorRT-LLMNVIDIA
PoorRican
PoorRican
0