PDF upload and optimization database flow

Last updated: 10 September 2026

1. Result

A PDF is uploaded directly to evequality/incoming/..., recorded in MySQL, dispatched through the PDF SQS queue, optimized by a PDF Lambda using Ghostscript and qpdf, verified, and atomically published.

The final PDF key is stored in media_versions.object_key. media_assets.active_version_id points to the live version. Publication also writes the full CDN URL to the legacy pdfSolutions.pdflink column.

If the PDF is too large for the configured Lambda contract or the process times out, the same job is rerouted to the scale-to-zero EC2 heavy worker.

This document describes implemented code and Terraform, not a claim that these resources are already live.

2. Main AWS and database flow

3. Database identity

Each legacy pdfSolutions row has two independent assets:

For pdfSolutions.id = 70, the PDF content identity is:

media_assets.external_code contains pdfSolutions.pdf_code. A replacement PDF keeps the same asset but advances its source revision and creates a new version.

4. Stage-by-stage database changes

Stage A — authorize direct upload

The admin UI calls POST /admin/api/media/uploads. Laravel checks pdfs.update, requires .pdf plus application/pdf, validates the declared size and creates:

Temporary key example:

At the default 100 MiB multipart threshold, the browser uses signed per-part requests. All bytes travel directly from browser to S3.

Stage B — verify upload and create work

The confirmation endpoint completes multipart upload where applicable, reads S3 metadata and validates size, MIME, checksum and the real PDF signature.

Job creation is idempotent for asset, source revision and profile.

Stage C — outbox and PDF SQS

Every minute, the outbox dispatcher publishes pending work:

The PDF queue is encrypted, long-polls for 20 seconds, retains messages for four days, has a 360-second visibility timeout and redrives after three receives to a DLQ retained for 14 days.

The PDF Lambda event-source mapping has batch size one and partial-batch failure reporting.

Stage D — Lambda attempt and lease

The ARM64 PDF Lambda configuration is:

When the function claims work:

The local work directory must have at least three times the source size free, with a 100 MB floor.

Stage E — PDF optimization and validation

The processor performs these operations without rasterizing every page:

1. qpdf --check validates the source structure.
2. qpdf --show-encryption rejects encrypted PDFs.
3. pdfinfo records the source page count.
4. Ghostscript uses the /ebook profile, PDF 1.6, duplicate-image detection and font compression.
5. qpdf linearizes the result for faster web access.
6. If the optimized file is not smaller, the safe original is retained as the final document.pdf instead of publishing a larger file.
7. qpdf validates the output and pdfinfo proves that the page count did not change.

The result is uploaded as:

Later replacements can be isolated under versions/r{revision}. The verifier checks object existence, non-empty size, PDF signature and positive unchanged page count.

Stage F — publication transaction

The publisher first rejects stale leases, old generations, superseded source revisions, blocked assets and cancelled jobs. It then commits:

Example:

The new version table stores a relative key so delivery-domain changes do not require rewriting every version. The legacy field keeps the full URL for old consumers.

Important media_versions fields

5. PDF status timeline

The uploaded -> authorized recovery clears confirmation_started_at so the administrator can correct/retry confirmation before the authorization expires.

6. Heavy-worker fallback

If media_uploads.expected_bytes exceeds the default 100 MiB Lambda limit, or the Lambda processing command times out:

The CloudWatch video-queue alarm or one-minute reconciler starts the conversion ASG at desired capacity one. The EC2 worker claims a new attempt and runs the same PdfProcessor, but with the longer heavy-media timeout. Old-generation messages cannot activate output.

7. Table ownership summary

8. Failure, retry, rollback and cleanup

• Structural, encryption, page-count, command, disk or upload failures are stored on media_job_attempts.error_code; bounded context is stored in error_context.
• With retry budget remaining, media_jobs returns to dispatched with exponential delay and SQS retries the message.
• Exhausted jobs become failed; SQS moves repeatedly failed messages to the PDF DLQ after three receives.
• Candidate output from a failed attempt is deleted best-effort.
• The one-minute recovery command marks dead attempts lease_expired and retries only within budget.
• Manual retry increases generation and creates a fresh outbox row, preventing old messages from publishing.
• Rollback first verifies the stored PDF object and metadata, then changes active_version_id, updates pdflink and creates a publication audit.
• The hourly source purger deletes incoming/pdf/... only after the same revision is verified, published and active for the configured retention period, default 24 hours.
• Cleanup progress/failure is stored in media_uploads.source_delete_started_at, source_deleted_at and source_delete_error_code.

9. PDF thumbnail

The PDF processor does not generate a thumbnail. pdfSolutions.pdflogo belongs to a separate image asset (media_type = image, role = thumbnail). Its upload, Sharp variants, image queue and publication are described in the image and thumbnail upload flow.