File Upload Patterns
Priority: P0 (FOUNDATIONAL)
- Magic Bytes: NEVER trust
content-typeheader or file extension. - Tool: Use
file-typeormmmagicto verify actual buffer signature. - Limits: Set strict
limits: { fileSize: 5000000 }(5MB) in Multer config to prevent DoS.
Streaming (Scalability)
- Memory Warning: Default Multer
MemoryStoragecrashes servers with large files. - Pattern: Use Streaming for any file > 10MB.
- Library:
multer-s3(direct upload to bucket) orbusboy(raw stream processing). - Architecture:
- Client requests Signed URL from API.
- Client uploads directly to S3/GCS (Bypassing API server completely).
- Pro Tip: Only approach to scale file uploads infinitely.
Processing
- Async: Don't process images/videos in HTTP Request.
- Flow:
- Upload file.
- Push
FileUploadedEventto Queue (BullMQ). - Worker downloads, resizes/converts, and re-uploads.
Anti-Patterns
- No content-type trust: Always verify file magic bytes; MIME header can spoofed.
- No MemoryStorage for large files: Use streaming or signed URL pattern for files > 10MB.
- No synchronous file processing: Offload image/video work to BullMQ workers via FileUploadedEvent.