Creating a reusable file management service

Identifying files

Each file is given a unique key either explicitly by the caller or implicitly by the service.

Files are retrieved by passing a unique key.

Storage backends

The file service is backed by a storage backend. Storage backend instances are stateless and implement an interface to manage files using a specific file storage technology like S3 or the filesystem.

Each storage backend instance is scoped to a single conceptual "folder" wherein files are managed.

A file service can interface with multiple storage backends, each scoped to different folders. During file creation or retrieval through the file service, the specified folder controls which storage backend will be used internally.

File retrieval

There can be different requirements when retrieving files:

- streaming the file content

- reading the whole file content

- forwarding a signed URL to access the file content directly from the storage backend (e.g. S3 presigned URLs)

To accommodate those requirements, the file entities returned should provide functions to retrieve their content in different ways.

For client-side file retrieval, API endpoints can be used that load and return the content for a given file id. To separate file retrieval by folder and implement access control, each folder should be mapped to a different endpoint.