Automate issue filing for production and staging problems

[icco]

I found #1151 interesting.

I don't know what kind of monitoring gittip uses, but it'd be nice to add alerts to gittip's monitoring that send messages to IRC and/or file a github issue when there are performance issues, or things that are wrong with the system that gittip needs someone to look at.

Another sort of thing we could alert about is #580, 500s, 404s, 403s, payment errors, fraud, long db queries, etc.

Thoughts? Depending on gittip's current monitoring, this is something I'd be willing to help out with.

[bruceadams]

@icco I don't think we have any automated monitoring. Does Twitter count? Clearly, automated monitoring would be a good thing.

[icco]

Interesting. Is there somewhere I can read up about gittip's deployment? I'd love to set up some monitoring and alerting.

[chadwhitacre]

We log exceptions to Sentry. I receive email notifications about those. We had them logged to IRC at one point but turned that off for fear of leaking sensitive information.

[chadwhitacre]

@icco We deploy a Python stack to Heroku. What questions can I answer about deployment?

[icco]

Hmm, I wonder if we could set up http://logstash.net/ easily with heroku. I'll see if I can rig up an example. Never tried with python and heroku, but I don't see why not.

[chadwhitacre]

Heroku has a whole "add-ons" ecosystem, and there are four logging add-ons available. I haven't been impressed with Loggly (#408, #433), nor Logentries or Papertrail (#626 (comment)). FlyData is new since last time I checked, so I've added the free plan to test it out. Logstash doesn't have an add-on, so we'd have to plumb it manually. You recommend Logstash, @icco?

[icco]

I've used it in the past. I like it because it's open source and has pretty expandable plugin system. I don't know if it's the best, and I don't know how well it will deal with heroku, but it's worth a shot. I've been really disappointed, as you apparently have as well, with the proprietary logging services heroku has to offer (which is actually part of the reason I moved all of my personal stuff off of heroku, that and a really slow free tier :p)

My theory is I can use a procfile to run logstash on heroku and backup to S3. But we'll see how that works. Gonna give it a try this weekend.

[chadwhitacre]

We have monitoring via Sentry. As we open that up to others, (#1443) let's reticket specific improvements.