Attaching AVContentKey to app-created CMSampleBuffer fails with authorization error

Source#

Apple Developer Forums: https://developer.apple.com/forums/thread/713996

Relevant public API documentation:

Relevant WebKit change:

WebKit/WebKit#21689

Problem#

An application successfully obtains a FairPlay AVContentKey through AVContentKeySession, but attaching that key to an app-created CMSampleBuffer fails:

AVFoundationErrorDomain Code=-11836 "Cannot Open"
NSLocalizedFailureReason="This app is not authorized to play this file."
underlying NSOSStatusErrorDomain Code=-12161

The failure occurs at AVSampleBufferAttachContentKey (now deprecated in favor of CMReadySampleBuffer.attach(contentKey:)).

The key-loading path itself succeeds, so this should not initially be debugged as SPC/CKC generation failure.

Evidence level#

Public API documentation + public WebKit implementation evidence + community forum report.

Apple documents manual key attachment as a supported API operation for client-created sample buffers. Public WebKit source also shows that its FairPlay sample-buffer attachment path is entitlement-gated on Apple platforms. WebKit's 2023 adoption change states that macOS requires com.apple.private.coremedia.allow-fps-attachment; its entitlement-generation script contains this private entitlement for WebKit processes.

There is no public Apple documentation establishing that third-party App Store applications can request or receive this private entitlement. Therefore the entitlement requirement is strong implementation evidence, but entitlement availability to third-party apps must not be assumed.

Important distinction#

FairPlay key request succeeds
        ↓
AVContentKey exists
        ↓
key identity matches encrypted sample?
        ↓
caller authorized for manual FPS attachment?
        ↓
attach key to CMSampleBuffer
        ↓
enqueue for AVSampleBufferDisplayLayer / renderer

A successful CKC response proves that the application obtained a decryptor. It does not prove that the calling process is authorized to bind that decryptor to arbitrary app-created sample buffers.

Investigation#

1. Prove the failure boundary#

Record separately:

AVContentKeyRequest.status
AVContentKeySpecifier identifier
CKC processing success
AVContentKey callback received
sample key identifier / encryption metadata
attach(contentKey:) result
attach error domain/code

If the key request reaches a usable AVContentKey and only the attach call returns -11836/-12161, stop changing the KSM until evidence points back to key generation.

2. Verify key-to-sample suitability#

The current CoreMedia API states that the attached AVContentKey must have a contentKeySpecifier matching the sample's indications of suitability for the active content-key system.

Therefore test two independent hypotheses:

H1 — wrong key for sample
H2 — correct key, but process is not authorized to attach it

Do not collapse them into one generic DRM failure.

3. Use an A/B matrix#

A. Normal AVURLAsset / AVPlayer FairPlay playback
   same certificate + KSM + content key

B. Manual CMSampleBuffer path
   same key identity
   attach(contentKey:)

C. Clear, unencrypted CMSampleBuffer
   same renderer path, no key attachment

Interpretation:

A succeeds, B fails with authorization error
→ FairPlay service/key exchange is not the primary suspect

B fails only for mismatched KID
→ key/sample identity problem

B fails for a demonstrably matching key with -11836/-12161
→ authorization/entitlement boundary becomes primary

C fails
→ renderer/sample construction problem exists independently of FairPlay

4. Compare process entitlement context#

Do not attempt to add private Apple entitlements to a shipping app. Instead, inspect signed entitlements only to establish whether the failing third-party process and a known Apple/WebKit implementation operate under different authorization contexts.

Public WebKit source is useful as implementation evidence because its entitlement script explicitly adds:

com.apple.private.coremedia.allow-fps-attachment

for the relevant WebKit process configuration.

Platform behavior#

The important model is:

key acquisition capability
        ≠
manual decryptor attachment capability

AVContentKeySession manages content decryption keys. Manual sample-buffer attachment is a separate operation that can have additional authorization and key-suitability checks.

The public API surface alone should not be interpreted as proof that every application distribution context is entitled to every FairPlay attachment mode.

Solution / Guideline#

First prove that SPC → CKC → AVContentKey succeeds.
Verify that contentKeySpecifier matches the encrypted sample's key identity.
Treat -11836 with failure reason This app is not authorized to play this file at the attachment call as an authorization boundary, not as generic license-server failure.
Do not try to self-sign or inject a private Apple entitlement as a production workaround.
For a third-party product requirement that depends on manual FairPlay attachment, ask Apple through Feedback Assistant / DTS whether the intended architecture is supported for third-party distribution and which public API path should be used.
Prefer the current CMReadySampleBuffer.attach(contentKey:) API when targeting SDKs where it is available; AVSampleBufferAttachContentKey is deprecated.

Reusable engineering rule#

In DRM debugging, separate possession of a decryptor from authorization to use that decryptor in a particular media-processing path.

Use this state model:

key request
→ key response accepted
→ AVContentKey created
→ key/sample identity validated
→ operation authorization validated
→ key attached
→ sample decoded/rendered

The first failed transition is the root-cause boundary.

Verification#

A minimal repro should include:

single FairPlay KID
single encrypted sample sequence
one AVContentKeySession
logged contentKeySpecifier
logged sample encryption identifier
one attach(contentKey:) call
full NSError chain
signed entitlement dump for the test app
comparison run using ordinary AVPlayer FairPlay playback

If ordinary AVPlayer playback succeeds with the same content/key service while manual attachment consistently fails at the authorization check, the report has isolated the problem away from KSM/CKC generation.

References#

Apple Developer Forums — thread 713996
AVContentKey / AVContentKeyRecipient documentation
CMReadySampleBuffer.attach(contentKey:)
WebKit PR #21689, [Cocoa] Adopt AVSampleBufferAttachContentKey