210 lines
9.0 KiB
Markdown
210 lines
9.0 KiB
Markdown
# User Storage System Report
|
|
|
|
## Overview
|
|
A persistent storage system based on "Drives"—scoped containers for each user. Files are tracked primarily in the database, while physical contents reside on disk using opaque identifiers. The backend prioritizes database lookups for metadata, falling back to direct disk access only when strictly necessary for serving content.
|
|
|
|
## Architecture Principles
|
|
|
|
### 1. Drives and Containers
|
|
Users organize storage into named "Drives". A Drive acts as a logical folder scoped to the owner.
|
|
```text
|
|
{data_directory}/drives/{user_id}/{drive_name}/{opaque_hash}
|
|
```
|
|
- `{drive_name}`: Human-readable scope (e.g., "My Projects").
|
|
- `{opaque_hash}`: Auto-generated alphanumeric string (UUID or SHA prefix).
|
|
- Disk directory structures mirror this hierarchy but use the hash as the final leaf node.
|
|
|
|
### 2. Database-First Metadata
|
|
Files are essentially database records that point to disk locations.
|
|
- **Filename**: Stored only in the database. Allows for easy renaming without touching physical files.
|
|
- **Path Resolution**: Access paths are constructed lazily via a resolve function, ensuring that even internal path formats can be changed without breaking external references.
|
|
- **Integrity**: Disk filenames (hashes) do not map deterministically to database IDs, preventing enumeration attacks on the filesystem level.
|
|
|
|
### 3. Minimal Backend Layers
|
|
Access flows through three strict stages:
|
|
1. **Auth**: Bearer token validation.
|
|
2. **DB Lookup**: Verify ownership (`user_id`) and Drive existence.
|
|
3. **Serve**: If requesting raw content, pass the resolved disk path to Quart's send functions.
|
|
|
|
This design keeps the router thin. The database handles all authorization logic before any file system calls are made.
|
|
|
|
## Data Model
|
|
|
|
Additions to `backend/src/database.py`:
|
|
|
|
```python
|
|
class Drive(models.Model):
|
|
id = fields.IntField(pk=True)
|
|
user = fields.ForeignKeyField("models.User", related_name="drives")
|
|
name = fields.CharField(max_length=255) # Readable drive name
|
|
created_at = fields.DatetimeField(auto_now_add=True)
|
|
|
|
class DriveFile(models.Model):
|
|
id = fields.IntField(pk=True)
|
|
drive = fields.ForeignKeyField("models.Drive", related_name="files")
|
|
|
|
filename = fields.CharField(max_length=500) # Readable name (DB only)
|
|
disk_hash = fields.CharField(max_length=64) # Internal disk identifier
|
|
mime_type = fields.CharField(max_length=255) # Content type
|
|
size_bytes = fields.BigIntField() # Physical size
|
|
|
|
accessed_at = fields.DatetimeField(null=True) # Last access timestamp
|
|
created_at = fields.DatetimeField(auto_now_add=True)
|
|
|
|
class Meta:
|
|
table = "drive_files"
|
|
```
|
|
|
|
**Design Notes:**
|
|
- Ownership is enforced via the foreign key chain (`User` -> `Drive` -> `File`).
|
|
- `disk_hash` allows us to rename files freely; the physical file remains at its generated path regardless of the database `filename`.
|
|
- Indexes on `drive_id` and `user_id` ensure fast listing queries.
|
|
|
|
## Backend Implementation
|
|
|
|
### Storage Service Interface
|
|
A dedicated module `backend/src/drive_service.py` handles operations.
|
|
|
|
```python
|
|
import os
|
|
from pathlib import Path
|
|
import database as db
|
|
|
|
DRIVES_ROOT = None
|
|
|
|
async def init_drives(data_directory):
|
|
global DRIVES_ROOT
|
|
DRIVES_ROOT = os.path.join(data_directory, "drives")
|
|
os.makedirs(DRIVES_ROOT, exist_ok=True)
|
|
|
|
def _generate_disk_hash():
|
|
return secrets.token_hex(8) # e.g., "a3f2b1..."
|
|
|
|
async def upload_file(user_id, drive_name, readable_filename, stream, mime_type=None):
|
|
# 1. Locate or create Drive
|
|
drive = await db.Drive.get_or_none(user_id=user_id, name=drive_name)
|
|
if not drive:
|
|
drive = await db.Drive.create(user_id=user_id, name=drive_name)
|
|
|
|
# 2. Prepare disk path
|
|
disk_hash = _generate_disk_hash()
|
|
drive_dir = os.path.join(DRIVES_ROOT, str(user_id), drive_name)
|
|
os.makedirs(drive_dir, exist_ok=True)
|
|
file_path = os.path.join(drive_dir, disk_hash)
|
|
|
|
# 3. Write stream and calculate size
|
|
size = 0
|
|
with open(file_path, "wb") as f:
|
|
while chunk := stream.read(8192):
|
|
f.write(chunk)
|
|
size += len(chunk)
|
|
|
|
# 4. Register in database
|
|
record = await db.DriveFile.create(
|
|
drive_id=drive.id,
|
|
filename=readable_filename,
|
|
disk_hash=disk_hash,
|
|
mime_type=mime_type or "application/octet-stream",
|
|
size_bytes=size,
|
|
)
|
|
return record.id
|
|
```
|
|
|
|
### Resolution and Serving Logic
|
|
When a request asks for a file, we verify permissions first, then resolve the physical location.
|
|
|
|
```python
|
|
async def get_drive_file_metadata(file_id):
|
|
"""Return JSON metadata for frontend display."""
|
|
file = await db.DriveFile.get_or_none(id=file_id)
|
|
if not file: raise Exception("File not found")
|
|
return {
|
|
"id": file.id,
|
|
"name": file.filename,
|
|
"size": file.size_bytes,
|
|
"mime": file.mime_type,
|
|
"created": file.created_at.isoformat(),
|
|
"accessed": file.accessed_at.isoformat() if file.accessed_at else None
|
|
}
|
|
|
|
async def resolve_drive_file_for_serving(file_id, current_user_id):
|
|
"""Resolve file to absolute disk path after permission check. Returns (path, mime)."""
|
|
# Eager load drive and verify ownership
|
|
file = await db.DriveFile.filter(id=file_id).select_related("drive__user").get_or_none()
|
|
if not file: raise FileNotFoundError("File not found")
|
|
if file.drive.user_id != current_user_id:
|
|
raise PermissionError("Access denied")
|
|
|
|
# Update access time asynchronously
|
|
from asyncio import create_task
|
|
create_task(db.DriveFile.filter(id=file_id).update(accessed_at=timezone.now()))
|
|
|
|
abs_path = os.path.join(DRIVES_ROOT, str(file.drive.user_id), file.drive.name, file.disk_hash)
|
|
if not os.path.exists(abs_path):
|
|
raise FileNotFoundError("File missing on disk")
|
|
|
|
return abs_path, file.mime_type
|
|
```
|
|
|
|
### App Integration
|
|
Attach these methods to the `App` class in `backend/src/app.py`:
|
|
- `app.upload_drive_file(...)`: Handles multipart uploads or websocket binary payloads.
|
|
- `app.resolve_file(...)`: Used by the server route to safely serve bytes.
|
|
|
|
## Server Actions & Protocol
|
|
|
|
We leverage the existing WebSocket action protocol for interactions and REST for byte streams.
|
|
|
|
| Action | Method | Payload | Description |
|
|
|--------|--------|---------|-------------|
|
|
| `list-drives` | WS | `{}` | Lists available drives for the logged-in user. |
|
|
| `list-files` | WS | `{driveName: "..."}` | Returns array of DB metadata objects for the specified drive. |
|
|
| `upload-file` | WS Binary | `{driveName, fileName, mime?}` | Sends JSON header followed by binary stream. Returns `fileId`. |
|
|
| `/files/<int:id>` | GET | N/A | Serves raw file download. Uses resolved path. Auth required. |
|
|
| `/files/<int:id>/meta` | GET | N/A | Lightweight metadata endpoint. Useful for checking existence before download. |
|
|
|
|
**WebSocket Upload Sequence Example:**
|
|
```javascript
|
|
// 1. Send metadata
|
|
ws.send(JSON.stringify({action: "upload-file", nonce: "n1", body: {driveName: "Docs", fileName: "report.pdf"}}));
|
|
|
|
// 2. Immediately follow with binary payload
|
|
ws.send(blob);
|
|
|
|
// 3. Server responds
|
|
{action: "file-uploaded", fileId: 42, nonce: "n1"}
|
|
```
|
|
|
|
## Frontend Considerations
|
|
|
|
Integration is minimal and relies on established Vue3 components.
|
|
|
|
1. **Drives View**:
|
|
- Fetches drives on mount. Clicking a drive triggers `list-files` with the drive name.
|
|
- Renders a simple table/list of `filename` and `size`.
|
|
|
|
2. **Upload Handling**:
|
|
- Standard drag-and-drop zone. On drop, initiates the WS upload sequence above.
|
|
- Updates local store immediately upon receiving the `fileId` response to show progress/success state instantly.
|
|
|
|
3. **Downloads**:
|
|
- Since files need resolution, we provide a dedicated REST endpoint `/files/<id>` secured by the standard `QuartAuth` bearer token.
|
|
- The frontend constructs an anchor tag `<a href="/files/{id}" download>Download</a>` rather than trying to stream binary over WS (to avoid memory bloat).
|
|
|
|
## Migration Steps
|
|
|
|
1. Append `Drive` and `DriveFile` models to `backend/src/database.py`.
|
|
2. Run `python -m tortoise.makemigrations --config tortoise_config --name add_drives` and apply migration.
|
|
3. Add `await app.init_drives(data_directory)` in `main()` initialization block.
|
|
4. Extend `server.websocket('/ws')` handlers to cover `upload-file` and `list-files`.
|
|
5. Add `/files/<int:id>` and `/files/<int:id>/meta` routes to `server.py`.
|
|
|
|
## Future Extensibility: CDN
|
|
To switch from local disk serving to a CDN:
|
|
- Modify `resolve_drive_file_for_serving` or the corresponding server route.
|
|
- Instead of returning the absolute path, use `disk_hash` to construct a signed CDN URL (e.g., `cdn.example.com/drives/{hash}`).
|
|
- Zero changes are required in the routing logic or frontend once the resolver abstracts the source location.
|
|
|
|
## Summary
|
|
This system separates logical file organization (Database) from physical storage (Disk). By using generated hashes on disk, we decouple user-facing names from storage implementation details. Permissions are checked against the DB owner relationship before any file IO occurs, providing a secure, low-latency access pattern ready for scaling.
|