Write-up
Embed CDN Caching Issue
What Happened

On July 12th, we updated our Embed feature. While the update worked correctly, a caching issue with how the release is distributed affected users' ability to access that version.

Impact
Technical Details
  • Our content delivery system was set to cache files for 24 hours, preventing immediate access to updated embed files.

  • Additionally, the Cache-Control header was not properly set, allowing a user’s browser to decide how long to cache files.

  • As a result, a user's browser may have cached an outdated `index.js` file, which attempted to fetch outdated resources from our CDN, failing to provide the correct embed functionality.

Actions Taken
  • We forced a cache refresh in CloudFormation to ensure all users could access the latest version.

  • To prevent this from happening again, we've:

    • Set specific Time-To-Live (TTL) values in our CloudFormation configuration.

    • Added Cache-Control headers for our embed files to manage caching effectively, setting the max age as 1-hour for files in /latest.

    • Implemented alerts to notify our team if CloudFormation 4xx error rates exceed a certain threshold.

    • Updated our CI/CD pipeline to automate cache invalidation and verify that the Cache-Control header is set on new files.

If you have any questions, please don't hesitate to contact our support team.

Thank you for your understanding and continued trust.