What happened

On July 19th, our backend queue kicked off a scheduled job to send internal roadmap update reports but sent duplicates of the same email.

Impact

• Some users that have subscribed to receive roadmap reports received duplicates of the same internal email

• Duplicate emails were only sent to internal management portal users

• No duplicate emails were sent to page subscribers

Technical details

• A rare edge case was encountered in the scheduled Sidekiq job

• Upon encountering this edge case, the backend application errored out and attempted to retry

• The job did not filter out successful sends and retries the entire original batch of users

Actions taken

• The problem was immediately reported to escalation engineer

• A fix was deployed to better handle this rare edge case and prevent retries on successful emails

• The fix includes more thorough logging to further investigate this edge case

If you have any questions, please don't hesitate to contact our support team.

Thank you for your understanding and continued trust.