drives initial plan

docs/drives-phase-1.md

@@ -0,0 +1,69 @@
+# Phase 1: Data Model & Setup
+
+## Objective
+Establish the database schema and directory infrastructure that underpin the entire drive system. This is a prerequisite for all subsequent work.
+
+## Concepts
+
+### Drives and Containers
+Users organize storage into named "Drives". Each drive acts as a logical folder scoped to the owner.
+
+```text
+{data_directory}/drives/{user_id}/{drive_name}/{opaque_hash}
+```
+
+- `{drive_name}`: Human-readable scope (e.g., "My Projects").
+- `{opaque_hash}`: Auto-generated alphanumeric string (UUID or SHA prefix).
+
+### Disk Hashing Strategy
+Physical filenames are opaque hashes that do not map deterministically to database IDs, preventing enumeration attacks on the filesystem level. Renaming files updates only the database record; the physical file remains untouched.
+
+## Tasks
+
+### 1.1 Define Database Models
+
+Append the following models to `backend/src/database.py`:
+
+```python
+class Drive(models.Model):
+    id = fields.IntField(pk=True)
+    user = fields.ForeignKeyField("models.User", related_name="drives")
+    name = fields.CharField(max_length=255)
+    created_at = fields.DatetimeField(auto_now_add=True)
+
+class DriveFile(models.Model):
+    id = fields.IntField(pk=True)
+    drive = fields.ForeignKeyField("models.Drive", related_name="files")
+    
+    filename = fields.CharField(max_length=500)       # Readable name (DB only)
+    disk_hash = fields.CharField(max_length=64)       # Internal disk identifier
+    mime_type = fields.CharField(max_length=255)      # Content type
+    size_bytes = fields.BigIntField()                 # Physical size
+    
+    accessed_at = fields.DatetimeField(null=True)     # Last access timestamp
+    created_at = fields.DatetimeField(auto_now_add=True)
+    
+    class Meta:
+        table = "drive_files"
+```
+
+**Design Notes:**
+- Ownership is enforced via foreign key chain (`User` → `Drive` → `File`).
+- Indexes on `drive_id` and `user_id` ensure fast listing queries.
+
+### 1.2 Run Migrations
+
+Execute:
+```bash
+python -m tortoise.makemigrations --config tortoise_config --name add_drives
+python -m tortoise.migrate --config tortoise_config
+```
+
+### 1.3 Initialize Directory Structure
+
+In `backend/src/app.py`, call the drive initialization during startup:
+```python
+await app.init_drives(data_directory)
+```
+
+Ensure this happens after any DB connection is established so that `DRIVES_ROOT` can reference the data path correctly.

docs/drives-phase-2.md

@@ -0,0 +1,106 @@
+# Phase 2: Storage Service
+
+## Objective
+Implement the backend service layer that handles all disk operations: uploading files to hashed paths, resolving file metadata, and serving content. This module is `backend/src/drive_service.py`.
+
+## Concepts
+
+### Database-First Metadata
+Files are database records pointing to disk locations. Access paths are constructed lazily via a resolve function so internal path formats can change without breaking external references.
+
+### Minimal Backend Layers
+Access flows through three strict stages:
+1. **Auth**: Bearer token validation.
+2. **DB Lookup**: Verify ownership (`user_id`) and Drive existence.
+3. **Serve**: If requesting raw content, pass the resolved disk path to Quart's send functions.
+
+The router stays thin; the database handles all authorization before any file system calls.
+
+## Tasks
+
+### 2.1 Implement `drive_service.py`
+
+Create `backend/src/drive_service.py` with the following core functions:
+
+```python
+import os
+from pathlib import Path
+import secrets
+from datetime import timezone
+import database as db
+
+DRIVES_ROOT = None
+
+async def init_drives(data_directory):
+    """Initialize DRIVES_ROOT global from app startup."""
+    global DRIVES_ROOT
+    DRIVES_ROOT = os.path.join(data_directory, "drives")
+    os.makedirs(DRIVES_ROOT, exist_ok=True)
+
+def _generate_disk_hash():
+    """Generate a random hex string for filesystem-level filenames."""
+    return secrets.token_hex(8)
+
+async def upload_file(user_id, drive_name, readable_filename, stream, mime_type=None):
+    """Write an incoming stream to disk and register it in the DB. Returns file ID."""
+    # 1. Locate or create Drive
+    drive = await db.Drive.get_or_none(user_id=user_id, name=drive_name)
+    if not drive:
+        drive = await db.Drive.create(user_id=user_id, name=drive_name)
+    
+    # 2. Prepare disk path
+    disk_hash = _generate_disk_hash()
+    drive_dir = os.path.join(DRIVES_ROOT, str(user_id), drive_name)
+    os.makedirs(drive_dir, exist_ok=True)
+    file_path = os.path.join(drive_dir, disk_hash)
+    
+    # 3. Write stream and calculate size
+    size = 0
+    with open(file_path, "wb") as f:
+        while chunk := stream.read(8192):
+            f.write(chunk)
+            size += len(chunk)
+            
+    # 4. Register in database
+    record = await db.DriveFile.create(
+        drive_id=drive.id,
+        filename=readable_filename,
+        disk_hash=disk_hash,
+        mime_type=mime_type or "application/octet-stream",
+        size_bytes=size,
+    )
+    return record.id
+
+async def get_drive_file_metadata(file_id):
+    """Return JSON metadata for frontend display."""
+    file = await db.DriveFile.get_or_none(id=file_id)
+    if not file:
+        raise Exception("File not found")
+    return {
+        "id": file.id,
+        "name": file.filename,
+        "size": file.size_bytes,
+        "mime": file.mime_type,
+        "created": file.created_at.isoformat(),
+        "accessed": file.accessed_at.isoformat() if file.accessed_at else None
+    }
+
+async def resolve_drive_file_for_serving(file_id, current_user_id):
+    """Resolve file to absolute disk path after permission check. Returns (path, mime)."""
+    # Eager load drive and verify ownership
+    file = await db.DriveFile.filter(id=file_id).select_related("drive__user").get_or_none()
+    if not file:
+        raise FileNotFoundError("File not found")
+    if file.drive.user_id != current_user_id:
+        raise PermissionError("Access denied")
+        
+    # Update access time asynchronously
+    from asyncio import create_task
+    create_task(db.DriveFile.filter(id=file_id).update(accessed_at=timezone.now()))
+    
+    abs_path = os.path.join(DRIVES_ROOT, str(file.drive.user_id), file.drive.name, file.disk_hash)
+    if not os.path.exists(abs_path):
+        raise FileNotFoundError("File missing on disk")
+        
+    return abs_path, file.mime_type
+```

docs/drives-phase-3.md

@@ -0,0 +1,75 @@
+# Phase 3: Server Integration
+
+## Objective
+Attach drive operations to the Quart application's routing layer and WebSocket handlers. This phase implements the API endpoints (`/files/<int:id>` and `/files/<int:id>/meta`) and handles WebSocket binary uploads and list requests.
+
+## Concepts
+
+### Server Actions & Protocol
+We leverage the existing WebSocket action protocol for interactions and REST for byte streams:
+
+| Action | Method | Payload | Description |
+|--------|--------|---------|-------------|
+| `list-drives` | WS | `{}` | Lists available drives for the logged-in user. |
+| `list-files` | WS | `{driveName: "..."}` | Returns array of DB metadata objects for the specified drive. |
+| `upload-file` | WS Binary | `{driveName, fileName, mime?}` | Sends JSON header followed by binary stream. Returns `fileId`. |
+| `/files/<int:id>` | GET | N/A | Serves raw file download. Uses resolved path. Auth required. |
+| `/files/<int:id>/meta` | GET | N/A | Lightweight metadata endpoint for existence checks. |
+
+**WebSocket Upload Sequence Example:**
+```javascript
+// 1. Send metadata
+ws.send(JSON.stringify({action: "upload-file", nonce: "n1", body: {driveName: "Docs", fileName: "report.pdf"}}));
+
+// 2. Immediately follow with binary payload
+ws.send(blob);
+
+// 3. Server responds
+{action: "file-uploaded", fileId: 42, nonce: "n1"}
+```
+
+## Tasks
+
+### 3.1 Register Routes
+
+Extend `backend/src/app.py` with the following route handlers:
+
+```python
+from quart import Quart, request, send_file, make_response
+from . import drive_service
+
+app = Quart(__name__)
+
+@app.route("/files/<int:id>", methods=["GET"])
+async def serve_file(id):
+    try:
+        current_user_id = request.headers.get("Authorization").split(" ")[1] # or via auth middleware
+        file_path, mime_type = await drive_service.resolve_drive_file_for_serving(id, current_user_id)
+        return await send_file(file_path, mimetype=mime_type)
+    except FileNotFoundError as e:
+        return make_response("Not found", 404)
+    except PermissionError:
+        return make_response("Forbidden", 403)
+
+@app.route("/files/<int:id>/meta", methods=["GET"])
+async def get_file_meta(id):
+    try:
+        current_user_id = ... # same auth logic
+        # Verify ownership briefly before returning meta
+        file = await db.DriveFile.filter(id=id).select_related("drive__user").get_or_none()
+        if not file or file.drive.user_id != current_user_id:
+            raise Exception("Denied")
+        return drive_service.get_drive_file_metadata(id)
+    except Exception as e:
+        return make_response({"error": str(e)}, 404)
+```
+
+*Note: Replace explicit auth extraction with your framework's actual token validation mechanism.*
+
+### 3.2 Extend WebSocket Handler
+
+In `backend/src/app.py`, update the existing `server.websocket('/ws')` handler to cover new actions:
+
+- `list-drives`: Query `db.Drive.filter(user_id=user_id)`. Return a list of names and IDs.
+- `list-files`: Query `db.DriveFile.filter(drive__name=drive_name)` filtered by the authenticated user.
+- `upload-file`: Parse the first message for JSON headers (`driveName`, `fileName`, `mime`). Open the stream, call `drive_service.upload_file(...)`. Reply with `{action: "file-uploaded", fileId: ..., nonce: ...}`.

docs/drives-phase-4.md

@@ -0,0 +1,75 @@
+# Phase 4: Frontend Integration
+
+## Objective
+Build the Vue3 components that consume the drive API for listing, uploading, and downloading files. Integration is minimal and relies on established Vue3 patterns.
+
+## Concepts
+
+### Minimal Frontend Design
+- **Drives View**: Fetches drives on mount. Clicking a drive triggers `list-files` via WebSocket.
+- **Upload Handling**: Standard drag-and-drop zone. On drop, initiates WS upload sequence. Updates local store immediately upon receiving `fileId`.
+- **Downloads**: Uses dedicated REST endpoint `/files/<id>` secured by QuartAuth bearer token. Constructs `<a href="/files/{id}" download>Download</a>` rather than streaming binary over WebSocket.
+
+### Download Strategy
+Files require server-side resolution (ownership verification + path generation). Therefore:
+- The frontend never requests raw file bytes over WebSocket.
+- Instead, a direct anchor link to the secured REST endpoint handles downloads.
+- This avoids memory bloat on both client and server sides.
+
+## Tasks
+
+### 4.1 Drives View Component
+
+Create a component (`src/views/DrivesView.vue`) that:
+- Fetches available drives on mount using the `list-drives` WebSocket action.
+- Renders a list/table of drives with clickable names/IDs.
+- On click, stores selected drive name and emits an event to show the files list.
+
+### 4.2 Files List View
+
+Create or update the files view component to:
+- Call `list-files` with the selected `{driveName}` when the view mounts or drive changes.
+- Render a simple table displaying `filename` and `size` for each row.
+- Provide a "Download" button/link per row that navigates to `/files/{id}`.
+
+```vue
+<!-- Example file list item -->
+<a :href="`/files/${file.id}`" :download="file.name">
+  {{ file.name }} ({{ file.size }})
+</a>
+```
+
+### 4.3 Upload Zone
+
+Implement a drag-and-drop zone (`components/FileUploadZone.vue`):
+- Listen for `drop` events on supported file types.
+- On drop, initiate the WebSocket upload sequence:
+
+```javascript
+// 1. Send metadata header
+ws.send(JSON.stringify({
+  action: "upload-file",
+  nonce: crypto.randomUUID(),
+  body: {
+    driveName: currentDrive.value,
+    fileName: droppedFile.name,
+    mime: droppedFile.type
+  }
+}));
+
+// 2. Immediately follow with binary payload
+ws.send(droppedFile); // File object sent directly as binary
+
+// 3. Wait for server response
+// On receiving {action: "file-updated", fileId: ..., nonce: ...}:
+//   - Update local store to add the new file record
+//   - Optionally show a success toast
+```
+
+- Store uploaded files in local Vuex/Pinia store immediately after `fileId` receipt to reflect progress/success instantly without waiting for another poll cycle.
+
+### 4.4 Store Updates
+
+Extend your application store (Pinia/Vuex) to:
+- Hold state for `drives: []`, `currentDrive`, and `files: []`.
+- Handle incoming `file-uploaded` messages from WebSocket by inserting new records into the active drive's file array.

docs/drives-report.md

@@ -0,0 +1,209 @@
+# User Storage System Report
+
+## Overview
+A persistent storage system based on "Drives"—scoped containers for each user. Files are tracked primarily in the database, while physical contents reside on disk using opaque identifiers. The backend prioritizes database lookups for metadata, falling back to direct disk access only when strictly necessary for serving content.
+
+## Architecture Principles
+
+### 1. Drives and Containers
+Users organize storage into named "Drives". A Drive acts as a logical folder scoped to the owner.
+```text
+{data_directory}/drives/{user_id}/{drive_name}/{opaque_hash}
+```
+- `{drive_name}`: Human-readable scope (e.g., "My Projects").
+- `{opaque_hash}`: Auto-generated alphanumeric string (UUID or SHA prefix).
+- Disk directory structures mirror this hierarchy but use the hash as the final leaf node.
+
+### 2. Database-First Metadata
+Files are essentially database records that point to disk locations.
+- **Filename**: Stored only in the database. Allows for easy renaming without touching physical files.
+- **Path Resolution**: Access paths are constructed lazily via a resolve function, ensuring that even internal path formats can be changed without breaking external references.
+- **Integrity**: Disk filenames (hashes) do not map deterministically to database IDs, preventing enumeration attacks on the filesystem level.
+
+### 3. Minimal Backend Layers
+Access flows through three strict stages:
+1. **Auth**: Bearer token validation.
+2. **DB Lookup**: Verify ownership (`user_id`) and Drive existence.
+3. **Serve**: If requesting raw content, pass the resolved disk path to Quart's send functions.
+
+This design keeps the router thin. The database handles all authorization logic before any file system calls are made.
+
+## Data Model
+
+Additions to `backend/src/database.py`:
+
+```python
+class Drive(models.Model):
+    id = fields.IntField(pk=True)
+    user = fields.ForeignKeyField("models.User", related_name="drives")
+    name = fields.CharField(max_length=255)  # Readable drive name
+    created_at = fields.DatetimeField(auto_now_add=True)
+
+class DriveFile(models.Model):
+    id = fields.IntField(pk=True)
+    drive = fields.ForeignKeyField("models.Drive", related_name="files")
+    
+    filename = fields.CharField(max_length=500)       # Readable name (DB only)
+    disk_hash = fields.CharField(max_length=64)       # Internal disk identifier
+    mime_type = fields.CharField(max_length=255)      # Content type
+    size_bytes = fields.BigIntField()                 # Physical size
+    
+    accessed_at = fields.DatetimeField(null=True)     # Last access timestamp
+    created_at = fields.DatetimeField(auto_now_add=True)
+    
+    class Meta:
+        table = "drive_files"
+```
+
+**Design Notes:**
+- Ownership is enforced via the foreign key chain (`User` -> `Drive` -> `File`).
+- `disk_hash` allows us to rename files freely; the physical file remains at its generated path regardless of the database `filename`.
+- Indexes on `drive_id` and `user_id` ensure fast listing queries.
+
+## Backend Implementation
+
+### Storage Service Interface
+A dedicated module `backend/src/drive_service.py` handles operations.
+
+```python
+import os
+from pathlib import Path
+import database as db
+
+DRIVES_ROOT = None
+
+async def init_drives(data_directory):
+    global DRIVES_ROOT
+    DRIVES_ROOT = os.path.join(data_directory, "drives")
+    os.makedirs(DRIVES_ROOT, exist_ok=True)
+
+def _generate_disk_hash():
+    return secrets.token_hex(8)  # e.g., "a3f2b1..."
+
+async def upload_file(user_id, drive_name, readable_filename, stream, mime_type=None):
+    # 1. Locate or create Drive
+    drive = await db.Drive.get_or_none(user_id=user_id, name=drive_name)
+    if not drive:
+        drive = await db.Drive.create(user_id=user_id, name=drive_name)
+    
+    # 2. Prepare disk path
+    disk_hash = _generate_disk_hash()
+    drive_dir = os.path.join(DRIVES_ROOT, str(user_id), drive_name)
+    os.makedirs(drive_dir, exist_ok=True)
+    file_path = os.path.join(drive_dir, disk_hash)
+    
+    # 3. Write stream and calculate size
+    size = 0
+    with open(file_path, "wb") as f:
+        while chunk := stream.read(8192):
+            f.write(chunk)
+            size += len(chunk)
+            
+    # 4. Register in database
+    record = await db.DriveFile.create(
+        drive_id=drive.id,
+        filename=readable_filename,
+        disk_hash=disk_hash,
+        mime_type=mime_type or "application/octet-stream",
+        size_bytes=size,
+    )
+    return record.id
+```
+
+### Resolution and Serving Logic
+When a request asks for a file, we verify permissions first, then resolve the physical location.
+
+```python
+async def get_drive_file_metadata(file_id):
+    """Return JSON metadata for frontend display."""
+    file = await db.DriveFile.get_or_none(id=file_id)
+    if not file: raise Exception("File not found")
+    return {
+        "id": file.id,
+        "name": file.filename,
+        "size": file.size_bytes,
+        "mime": file.mime_type,
+        "created": file.created_at.isoformat(),
+        "accessed": file.accessed_at.isoformat() if file.accessed_at else None
+    }
+
+async def resolve_drive_file_for_serving(file_id, current_user_id):
+    """Resolve file to absolute disk path after permission check. Returns (path, mime)."""
+    # Eager load drive and verify ownership
+    file = await db.DriveFile.filter(id=file_id).select_related("drive__user").get_or_none()
+    if not file: raise FileNotFoundError("File not found")
+    if file.drive.user_id != current_user_id:
+        raise PermissionError("Access denied")
+        
+    # Update access time asynchronously
+    from asyncio import create_task
+    create_task(db.DriveFile.filter(id=file_id).update(accessed_at=timezone.now()))
+    
+    abs_path = os.path.join(DRIVES_ROOT, str(file.drive.user_id), file.drive.name, file.disk_hash)
+    if not os.path.exists(abs_path):
+        raise FileNotFoundError("File missing on disk")
+        
+    return abs_path, file.mime_type
+```
+
+### App Integration
+Attach these methods to the `App` class in `backend/src/app.py`:
+- `app.upload_drive_file(...)`: Handles multipart uploads or websocket binary payloads.
+- `app.resolve_file(...)`: Used by the server route to safely serve bytes.
+
+## Server Actions & Protocol
+
+We leverage the existing WebSocket action protocol for interactions and REST for byte streams.
+
+| Action | Method | Payload | Description |
+|--------|--------|---------|-------------|
+| `list-drives` | WS | `{}` | Lists available drives for the logged-in user. |
+| `list-files` | WS | `{driveName: "..."}` | Returns array of DB metadata objects for the specified drive. |
+| `upload-file` | WS Binary | `{driveName, fileName, mime?}` | Sends JSON header followed by binary stream. Returns `fileId`. |
+| `/files/<int:id>` | GET | N/A | Serves raw file download. Uses resolved path. Auth required. |
+| `/files/<int:id>/meta` | GET | N/A | Lightweight metadata endpoint. Useful for checking existence before download. |
+
+**WebSocket Upload Sequence Example:**
+```javascript
+// 1. Send metadata
+ws.send(JSON.stringify({action: "upload-file", nonce: "n1", body: {driveName: "Docs", fileName: "report.pdf"}}));
+
+// 2. Immediately follow with binary payload
+ws.send(blob);
+
+// 3. Server responds
+{action: "file-uploaded", fileId: 42, nonce: "n1"}
+```
+
+## Frontend Considerations
+
+Integration is minimal and relies on established Vue3 components.
+
+1. **Drives View**: 
+   - Fetches drives on mount. Clicking a drive triggers `list-files` with the drive name.
+   - Renders a simple table/list of `filename` and `size`.
+   
+2. **Upload Handling**:
+   - Standard drag-and-drop zone. On drop, initiates the WS upload sequence above.
+   - Updates local store immediately upon receiving the `fileId` response to show progress/success state instantly.
+
+3. **Downloads**:
+   - Since files need resolution, we provide a dedicated REST endpoint `/files/<id>` secured by the standard `QuartAuth` bearer token.
+   - The frontend constructs an anchor tag `<a href="/files/{id}" download>Download</a>` rather than trying to stream binary over WS (to avoid memory bloat).
+
+## Migration Steps
+
+1. Append `Drive` and `DriveFile` models to `backend/src/database.py`.
+2. Run `python -m tortoise.makemigrations --config tortoise_config --name add_drives` and apply migration.
+3. Add `await app.init_drives(data_directory)` in `main()` initialization block.
+4. Extend `server.websocket('/ws')` handlers to cover `upload-file` and `list-files`.
+5. Add `/files/<int:id>` and `/files/<int:id>/meta` routes to `server.py`.
+
+## Future Extensibility: CDN
+To switch from local disk serving to a CDN:
+- Modify `resolve_drive_file_for_serving` or the corresponding server route.
+- Instead of returning the absolute path, use `disk_hash` to construct a signed CDN URL (e.g., `cdn.example.com/drives/{hash}`).
+- Zero changes are required in the routing logic or frontend once the resolver abstracts the source location.
+
+## Summary
+This system separates logical file organization (Database) from physical storage (Disk). By using generated hashes on disk, we decouple user-facing names from storage implementation details. Permissions are checked against the DB owner relationship before any file IO occurs, providing a secure, low-latency access pattern ready for scaling.