hub/docs/drives-report.md

User Storage System Report

Overview

A persistent storage system based on "Drives"—scoped containers for each user. Files are tracked primarily in the database, while physical contents reside on disk using opaque identifiers. The backend prioritizes database lookups for metadata, falling back to direct disk access only when strictly necessary for serving content.

Architecture Principles

1. Drives and Containers

Users organize storage into named "Drives". A Drive acts as a logical folder scoped to the owner.

{data_directory}/drives/{user_id}/{drive_name}/{opaque_hash}

• {drive_name}: Human-readable scope (e.g., "My Projects").
• {opaque_hash}: Auto-generated alphanumeric string (UUID or SHA prefix).
• Disk directory structures mirror this hierarchy but use the hash as the final leaf node.

2. Database-First Metadata

Files are essentially database records that point to disk locations.

• Filename: Stored only in the database. Allows for easy renaming without touching physical files.
• Path Resolution: Access paths are constructed lazily via a resolve function, ensuring that even internal path formats can be changed without breaking external references.
• Integrity: Disk filenames (hashes) do not map deterministically to database IDs, preventing enumeration attacks on the filesystem level.

3. Minimal Backend Layers

Access flows through three strict stages:

1. Auth: Bearer token validation.
2. DB Lookup: Verify ownership (user_id) and Drive existence.
3. Serve: If requesting raw content, pass the resolved disk path to Quart's send functions.

This design keeps the router thin. The database handles all authorization logic before any file system calls are made.

Data Model

Additions to backend/src/database.py:

class Drive(models.Model):
    id = fields.IntField(pk=True)
    user = fields.ForeignKeyField("models.User", related_name="drives")
    name = fields.CharField(max_length=255)  # Readable drive name
    created_at = fields.DatetimeField(auto_now_add=True)

class DriveFile(models.Model):
    id = fields.IntField(pk=True)
    drive = fields.ForeignKeyField("models.Drive", related_name="files")
    
    filename = fields.CharField(max_length=500)       # Readable name (DB only)
    disk_hash = fields.CharField(max_length=64)       # Internal disk identifier
    mime_type = fields.CharField(max_length=255)      # Content type
    size_bytes = fields.BigIntField()                 # Physical size
    
    accessed_at = fields.DatetimeField(null=True)     # Last access timestamp
    created_at = fields.DatetimeField(auto_now_add=True)
    
    class Meta:
        table = "drive_files"

Design Notes:

• Ownership is enforced via the foreign key chain (User -> Drive -> File).
• disk_hash allows us to rename files freely; the physical file remains at its generated path regardless of the database filename.
• Indexes on drive_id and user_id ensure fast listing queries.

Backend Implementation

Storage Service Interface

A dedicated module backend/src/drive_service.py handles operations.

import os
from pathlib import Path
import database as db

DRIVES_ROOT = None

async def init_drives(data_directory):
    global DRIVES_ROOT
    DRIVES_ROOT = os.path.join(data_directory, "drives")
    os.makedirs(DRIVES_ROOT, exist_ok=True)

def _generate_disk_hash():
    return secrets.token_hex(8)  # e.g., "a3f2b1..."

async def upload_file(user_id, drive_name, readable_filename, stream, mime_type=None):
    # 1. Locate or create Drive
    drive = await db.Drive.get_or_none(user_id=user_id, name=drive_name)
    if not drive:
        drive = await db.Drive.create(user_id=user_id, name=drive_name)
    
    # 2. Prepare disk path
    disk_hash = _generate_disk_hash()
    drive_dir = os.path.join(DRIVES_ROOT, str(user_id), drive_name)
    os.makedirs(drive_dir, exist_ok=True)
    file_path = os.path.join(drive_dir, disk_hash)
    
    # 3. Write stream and calculate size
    size = 0
    with open(file_path, "wb") as f:
        while chunk := stream.read(8192):
            f.write(chunk)
            size += len(chunk)
            
    # 4. Register in database
    record = await db.DriveFile.create(
        drive_id=drive.id,
        filename=readable_filename,
        disk_hash=disk_hash,
        mime_type=mime_type or "application/octet-stream",
        size_bytes=size,
    )
    return record.id

Resolution and Serving Logic

When a request asks for a file, we verify permissions first, then resolve the physical location.

async def get_drive_file_metadata(file_id):
    """Return JSON metadata for frontend display."""
    file = await db.DriveFile.get_or_none(id=file_id)
    if not file: raise Exception("File not found")
    return {
        "id": file.id,
        "name": file.filename,
        "size": file.size_bytes,
        "mime": file.mime_type,
        "created": file.created_at.isoformat(),
        "accessed": file.accessed_at.isoformat() if file.accessed_at else None
    }

async def resolve_drive_file_for_serving(file_id, current_user_id):
    """Resolve file to absolute disk path after permission check. Returns (path, mime)."""
    # Eager load drive and verify ownership
    file = await db.DriveFile.filter(id=file_id).select_related("drive__user").get_or_none()
    if not file: raise FileNotFoundError("File not found")
    if file.drive.user_id != current_user_id:
        raise PermissionError("Access denied")
        
    # Update access time asynchronously
    from asyncio import create_task
    create_task(db.DriveFile.filter(id=file_id).update(accessed_at=timezone.now()))
    
    abs_path = os.path.join(DRIVES_ROOT, str(file.drive.user_id), file.drive.name, file.disk_hash)
    if not os.path.exists(abs_path):
        raise FileNotFoundError("File missing on disk")
        
    return abs_path, file.mime_type

App Integration

Attach these methods to the App class in backend/src/app.py:

• app.upload_drive_file(...): Handles multipart uploads or websocket binary payloads.
• app.resolve_file(...): Used by the server route to safely serve bytes.

Server Actions & Protocol

We leverage the existing WebSocket action protocol for interactions and REST for byte streams.

Action	Method	Payload	Description
list-drives	WS	{}	Lists available drives for the logged-in user.
list-files	WS	{driveName: "..."}	Returns array of DB metadata objects for the specified drive.
upload-file	WS Binary	{driveName, fileName, mime?}	Sends JSON header followed by binary stream. Returns fileId.
/files/<int:id>	GET	N/A	Serves raw file download. Uses resolved path. Auth required.
/files/<int:id>/meta	GET	N/A	Lightweight metadata endpoint. Useful for checking existence before download.

WebSocket Upload Sequence Example:

// 1. Send metadata
ws.send(JSON.stringify({action: "upload-file", nonce: "n1", body: {driveName: "Docs", fileName: "report.pdf"}}));

// 2. Immediately follow with binary payload
ws.send(blob);

// 3. Server responds
{action: "file-uploaded", fileId: 42, nonce: "n1"}

Frontend Considerations

Integration is minimal and relies on established Vue3 components.

1. Drives View:

   • Fetches drives on mount. Clicking a drive triggers list-files with the drive name.
   • Renders a simple table/list of filename and size.

2. Upload Handling:

   • Standard drag-and-drop zone. On drop, initiates the WS upload sequence above.
   • Updates local store immediately upon receiving the fileId response to show progress/success state instantly.

3. Downloads:

   • Since files need resolution, we provide a dedicated REST endpoint /files/<id> secured by the standard QuartAuth bearer token.
   • The frontend constructs an anchor tag <a href="/files/{id}" download>Download</a> rather than trying to stream binary over WS (to avoid memory bloat).

Migration Steps

1. Append Drive and DriveFile models to backend/src/database.py.
2. Run python -m tortoise.makemigrations --config tortoise_config --name add_drives and apply migration.
3. Add await app.init_drives(data_directory) in main() initialization block.
4. Extend server.websocket('/ws') handlers to cover upload-file and list-files.
5. Add /files/<int:id> and /files/<int:id>/meta routes to server.py.

Future Extensibility: CDN

To switch from local disk serving to a CDN:

• Modify resolve_drive_file_for_serving or the corresponding server route.
• Instead of returning the absolute path, use disk_hash to construct a signed CDN URL (e.g., cdn.example.com/drives/{hash}).
• Zero changes are required in the routing logic or frontend once the resolver abstracts the source location.

Summary

This system separates logical file organization (Database) from physical storage (Disk). By using generated hashes on disk, we decouple user-facing names from storage implementation details. Permissions are checked against the DB owner relationship before any file IO occurs, providing a secure, low-latency access pattern ready for scaling.