We’ve had a few instances where our application will crash with an OOM, and we receive an email to what seems like the administrators of our Organization.
I was wondering if there are any other notification options available instead of e-mail, such as slack webhook or sentry notification so we can get alerted in a way which hits our more tactical alert systems a bit quicker.
Note: We probably could setup some sort of email rules to automate this, but before I go ahead and do that, I wanted to check if this was just a setting I was missing.
We don’t currently support custom notification options, though we’d like to get there someday. I do have one workaround you can use right now; there is a bit of work you’ll have to do on your own for it, though.
Since the OOM emails are generated from app logs (by checking if the message contains Out of memory: Killed process), it’s possible to stream app logs and wait for an OOM message. You can use fly-log-shipper to send logs to a log aggregator, or connect to our NATS endpoint (mentioned in the README of that repo) to stream logs to your own process.