Video upload and HLS database flow

Last updated: 10 September 2026

1. Result

A new video upload follows this path:

1. Laravel creates an upload authorization in MySQL.
2. The administrator's browser uploads the original file directly to evequality/incoming/....
3. Laravel verifies the S3 object and creates a durable video job plus an outbox message.
4. The outbox dispatcher sends the job identity to the video SQS queue.
5. The scale-to-zero conversion ASG starts one EC2 worker.
6. The worker uses FFmpeg/FFprobe to produce and verify HLS.
7. Laravel atomically publishes a media_versions row and updates the legacy video row.
8. The temporary original is deleted after the verified output has been active for the retention period.

The final HLS key is stored in media_versions.object_key. The active version ID is stored in media_assets.active_version_id. The full compatibility URL is also written to video.transfer_path.

This document describes the code currently implemented in this repository. Infrastructure resources exist in Terraform, but this does not mean they have already been deployed to production.

2. Main AWS and database flow

3. Database identity before upload

The administrator first creates or edits a legacy video record. AdminCatalogService ensures that it has two independent media assets:

The content asset is uniquely identified by:

media_assets.external_code holds the legacy video.videoCode. Uploading a replacement does not create another asset; it creates a newer source revision and later a newer media_versions row.

4. Stage-by-stage database changes

Stage A — authorize the browser upload

The UI calls POST /admin/api/media/uploads with the selected content asset, filename, byte size, MIME type and, for a single-part upload, a SHA-256 checksum.

Laravel validates administrator permission, asset state, allowed extension/MIME type and the maximum configured upload size. It then inserts:

The temporary source key has this shape:

Uploads at or above the default 100 MiB threshold use multipart S3 upload. Smaller files use one signed PUT. PHP never receives or stores the video bytes.

Stage B — upload to S3 and confirm

The browser uploads the file directly to the signed S3 location, then calls:

Laravel completes multipart upload when necessary and checks S3 metadata, actual byte count, content type, checksum and real file signature. On success one database transaction performs:

The unique job idempotency key is based on asset, source revision and profile. Repeating confirmation returns the existing job instead of creating duplicate processing work.

Stage C — durable SQS dispatch

The scheduled eve:dispatch-media-outbox command runs every minute.

The initial SQS payload includes job, asset, upload, source-revision, profile, processor and source-object details. Consumers use job_id and generation as the authoritative identity and reload current state from MySQL, so stale payload data cannot override the database.

If SQS publishing fails, the outbox returns to pending with exponential delay. After the outbox attempt limit, it becomes failed and the job becomes failed. A stale dispatch lock is recoverable.

Stage D — start and claim the heavy worker

The eve-staging-video-heavy queue has:

When at least one video message is visible, a CloudWatch alarm sets conversion ASG desired capacity to 1. The scheduled capacity reconciler also repairs missed starts. The ASG is configured as minimum 0, desired 0, maximum 1, with a candidate c8g.2xlarge ARM64 worker and 200 GiB encrypted gp3 temporary disk.

The EC2 command eve:process-media-queue receives one job at a time. Before processing, it enables ASG scale-in protection and creates:

During long FFmpeg work, the worker renews both the database lease and SQS message visibility. A scheduled recovery command expires dead leases and makes retryable work available again.

Stage E — FFmpeg conversion and S3 output

The worker downloads the confirmed incoming/ object into its private temporary directory. It requires free disk of at least three times the expected source size, with a 100 MB minimum.

Processing performs:

1. Normalize video to H.264/AAC MP4 locally with FFmpeg. This temporary legacy.mp4 is not uploaded.
2. Probe duration, dimensions and codecs with FFprobe.
3. Produce only renditions not larger than the source: configured 240p, 360p, 480p and 720p.
4. Create six-second HLS VOD segments, a playlist for every rendition and master.m3u8.
5. Upload the HLS inventory into the shared evequality bucket.

Example output:

The path builder prefers a safe existing legacy source directory. Otherwise it derives the board/class/subject/service/unit/chapter hierarchy. A later replacement version is isolated below versions/r{revision} when the asset already has versions.

The processor verifies that every internal playlist and segment reference is safe, relative, present and non-empty. It also validates duration, H.264, optional AAC, rendition count and duration tolerance.

Stage F — atomic publication

Only a still-valid processing lease may publish. Publication locks the job, attempt and asset and rejects a stale generation, superseded source revision, cancellation or blocked asset.

One MySQL transaction performs all authoritative changes:

The database stores a relative key such as:

The legacy compatibility column stores the full delivery URL:

Keeping the relative key in media_versions allows the CDN domain to change without rewriting every modern media row.

Important media_versions fields

5. Status timeline

6. Final table ownership

7. Failure, retry and cancellation

• A processor failure marks the attempt failed with error_code and safe error_context.
• Before the attempt limit, the job returns to dispatched with exponential delay and the SQS message is retried.
• At the job attempt limit, the job becomes failed; SQS can subsequently move the repeatedly failing message to its DLQ.
• A lost lease becomes lease_expired; recovery retries it only if budget remains.
• Candidate objects uploaded during a failed attempt are deleted best-effort before retry.
• A cancellation before processing changes the job to cancelled. During processing, cancel_requested_at causes the next heartbeat/publication guard to stop activation.
• An administrator can manually retry a failed/cancelled job. That raises generation, adds a new retry budget and creates a new outbox event. Messages from the old generation are ignored.
• Rollback is allowed only to an already verified, published version whose stored S3 inventory passes verification again. It changes active_version_id, refreshes video.transfer_path, and adds a publication_audits row.

8. Temporary source cleanup

The hourly source purger deletes the original only when all of these are true:

• media_uploads.status = confirmed;
• its key is below incoming/ in the configured shared bucket;
• the configured retention period has passed, default 24 hours;
• its exact asset and source revision is the current active media_versions row;
• that version is verified and published;
• the associated media_jobs row is published.

Cleanup sets source_delete_started_at, then source_deleted_at. Failure sets source_delete_error_code = source_delete_failed and allows a later retry. Final HLS objects are not deleted by this source purge.

Abandoned authorized uploads are checked every ten minutes. Their multipart operation is aborted, or their single object is deleted, and media_uploads.status becomes expired.

9. Existing offline-converted videos

The existing offline converter remains a separate path:

For this path, the importer:

• reads video.transfer_path only when hls_status = ready;
• converts the allowed CDN URL to a relative S3 key;
• validates the existing master.m3u8, rendition playlists and segments;
• creates or reuses the media_assets content row;
• inserts media_versions.object_key with processor = legacy_reference, format = hls, profile eve-2-legacy-hls-v1, and verification_status = verified;
• updates media_assets.active_version_id and records publication_audits.actor_type = legacy_import.

It does not create media_uploads, media_jobs, media_job_attempts or outbox_messages. It does not copy, move, rename or delete any S3 object. It also refuses to replace a modern active version or interfere with a busy modern job.

10. Video thumbnail relationship

The video and thumbnail are two separate uploads and two separate media assets. Uploading x.mp4 does not extract or publish a thumbnail automatically. The administrator selects the image/thumbnail asset and uploads an image through the image and thumbnail upload flow.

For a video thumbnail, the image Lambda writes thumbnail.webp and its variants beside the HLS output. Publication stores its primary key in the thumbnail asset's media_versions.object_key and writes the full CDN URL to video.videoThumbnail.