Same-bucket HLS: MediaConvert without a second CloudFront

Back to Northlight Press. Covers already upload with a presigned PUT into a private bucket and read through CloudFront OAC. Next problem: long-form video. Editors drop a mezzanine file; readers need adaptive HLS, not a single giant MP4.

The first instinct on a video product is often: new bucket for bitrates, new CloudFront distribution, new hostname.

That instinct is right for some shops. It was the wrong default for Northlight’s UAT.

The constraint hiding in the API

Northlight’s backend already had one knobs for “where public media lives”:

AWS_S3_PUBLIC_DOMAIN=cdn-dev.northlight.press

Upload keys and (later) playlist URLs both go through toPublicMediaUrl(...). One domain. One mental model for Admin and the reader site.

A second CloudFront for HLS without a second domain (or a careful path-based multi-origin) breaks that contract. You either:

• teach every client a second host, or
• pretend two origins share one hostname without wiring the distribution for it

So the question was not “is a dedicated HLS bucket pure?” It was “what is the smallest architecture that still ships adaptive video under the domain we already promise?”

Decision

Reuse the existing CloudFront and put HLS objects in the same S3 bucket as uploads.

Piece	Choice
Source + HLS objects	same private bucket
CDN	existing cdn-dev distribution
HLS prefix	broadcasts/transcoded/{videoId}/{job}/
Public URLs	same AWS_S3_PUBLIC_DOMAIN

Prod can mirror the pattern on the prod bucket and prod CDN. Split buckets later if lifecycle rules or IAM isolation demand it. Mid-size newsrooms do not need that on day one.

The pipeline shape

1. Admin gets a presigned PUT (optionally via Transfer Acceleration for distant desks).
2. API stores sourceUrl, marks transcodingStatus: pending, submits MediaConvert with UserMetadata.videoId.
3. MediaConvert writes 360 / 480 / 720 HLS under the transcoded prefix.
4. Job COMPLETE or ERROR → EventBridge (same region) → SNS → Lambda.
5. Lambda POSTs { videoId, status, playlistFilePath, jobId } to the API webhook.
6. API sets ready + transcodedUrl using the CDN host.
7. Client playbackUrl: HLS when ready, else source.

Readers never talk to S3. Editors never hold long-lived AWS keys. MediaConvert talks s3:// to the bucket the way AWS expects (accelerate endpoints are for browser PUTs, not job IO).

Region is not optional

MediaConvert job state-change events show up on the event bus in the MediaConvert region.

If the bucket and jobs live in eu-north-1, then:

• EventBridge rule
• SNS topic
• Lambda

must also live in eu-north-1. CloudFront stays global. Mixing “SNS in us-east-1 because that is where we always put topics” is a silent empty subscription.

Same rule for the IAM role: create MediaConvert_Default_Role in this account with trust mediaconvert.amazonaws.com. Do not paste another project’s ARN and only swap the account id.

Why not copy the “many bitrate buckets” pattern?

Another product in the same orbit used separate bitrate buckets and dedicated CloudFront hostnames. That scales when:

• you already expose multiple media hosts, or
• lifecycle / billing / IAM need hard walls between source and renditions

Northlight’s UAT had neither. One public domain, one OAC origin, one Admin upload story. Adding a second CF for purity would have meant frontend and env churn for no reader-facing win.

Same-bucket HLS under a clear prefix is enough until ops data says otherwise.

Failure modes worth naming

• Wrong EventBridge event pattern (e.g. “all events” vs job state change) → overwrite the rule, then wonder why COMPLETE never fires.
• Lambda payload field drift (postId vs videoId) → webhook 400s while the job succeeded in AWS.
• Assuming Transfer Acceleration helps MediaConvert → it does not; keep accelerate on the signer for Admin uploads only.
• Webhook secret mismatch between Lambda env and API → silent rejects after a green MediaConvert console.

Takeaway

CDN and bucket topology should follow the URL contract your API already ships, not a generic video-platform diagram.

For Northlight: private S3 + existing CloudFront for reads, MediaConvert writing HLS into the same bucket, completion fan-out in the same region as the jobs. Split hosts when you have a reason. Until then, one domain keeps Admin, API, and players honest.