File Upload Patterns
Priority: P0 (FOUNDATIONAL)
Secure file upload handling with validation and storage patterns.
- Magic Bytes: NEVER trust
content-typeheader or file extension.- Tool: Use
file-typeormmmagicto verify the actual buffer signature.
- Tool: Use
- Limits: Set strict
limits: { fileSize: 5000000 }(5MB) in Multer config to prevent DoS.
Streaming (Scalability)
- Memory Warning: Default Multer
MemoryStoragecrashes servers with large files. - Pattern: Use Streaming for any file > 10MB.
- Library:
multer-s3(direct upload to bucket) orbusboy(raw stream processing). - Architecture:
- Client requests Signed URL from API.
- Client uploads directly to S3/GCS (Bypassing API server completely).
- Pro Tip: This is the only way to scale file uploads infinitely.
- Library:
Processing
- Async: Don't process images/videos in the HTTP Request.
- Flow:
- Upload file.
- Push
FileUploadedEventto Queue (BullMQ). - Worker downloads, resizes/converts, and re-uploads.
Anti-Patterns
- No content-type trust: Always verify file magic bytes; MIME header can be spoofed.
- No MemoryStorage for large files: Use streaming or signed URL pattern for files > 10MB.
- No synchronous file processing: Offload image/video work to BullMQ workers via FileUploadedEvent.