Convert Rails logs to JSON

[bofstein]

Talk to Terry who has done this

[briri]

We will use Lograge to format the Rails logs. See #687 for details.

[briri]

Setup logstash-event and lograge gems and configured initializer.

The system now generates a new log/lograge-[env].log file.

The content of the new file that looks like this (each entry is a single line - expanded here for readability):

# Logfile created on 2025-01-29 19:02:50 +0000 by logger.rb/v1.6.1


{
  "method": "GET",
  "path": "/",
  "format": "html",
  "controller": "HomeController",
  "action": "index",
  "status": 200,
  "allocations": 88349,
  "duration": 747.98,
  "view": 602.2,
  "db": 141.32,
  "params": {},
  "ip":"172.20.0.1",
  "user_id": null,
  "@timestamp": "2025-01-29T22:17:51.457Z",
  "@version": "1",
  "message": "[200] GET / (HomeController#index)"
}

{
  "method": "POST",
  "path": "/users/sign_in",
  "format": "html",
  "controller": "Users::SessionsController",
  "action": "create",
  "status": 302,
  "allocations": 21407,
  "duration": 204.03,
  "view": 0.0,
  "db": 112.28,
  "location": "http://localhost:3000/plans",
  "unpermitted_params": [
    "authenticity_token",
    "user",
    "authenticity_token",
    "user",
    "user",
    "language_id",
    "language_id"
  ],
  "params": {
    "user": {
      "email": "[FILTERED]",
      "org_id": "15",
      "password": "[FILTERED]"
    }
  },
  "ip":"172.20.0.1",
  "user_id": 54373,
  "@timestamp": "2025-01-29T22:18:05.357Z",
  "@version": "1",
  "message": "[302] POST /users/sign_in (Users::SessionsController#create)"
}

{
  "method": "GET",
  "path": "/plans",
  "format": "html",
  "controller": "PlansController",
  "action": "index",
  "status": 200,
  "allocations": 348502,
  "duration": 614.84,
  "view": 349.04,
  "db": 155.43,
  "params": {},
  "ip":"172.20.0.1",
  "user_id": 54373,
  "@timestamp": "2025-01-29T22:18:05.994Z",
  "@version": "1",
  "message": "[200] GET /plans (PlansController#index)"
}

[briri]

This has been setup and deployed to stage and ready for when you are back @ashleygould

New log files are located in /dmp/apps/dmptool/shared/log/lograge_stage.log on the 2 stage hosts.

I followed the configuration Terry used for the Merritt admin tool. You can use whatever config you needed to inject/index his logs into OpenSearch. There is one exception though, I am including the params sent to the controller. The params property's contents are variable (numbers, strings, booleans, etc.) because they are specific to the request

Here is an example and entry (expanded to make it readable it is one line in the file):

{
  "method": "POST",
  "path": "/users/sign_in",
  "format": "html",
  "controller": "Users::SessionsController",
  "action": "create",
  "status": 302,
  "allocations": 21407,
  "duration": 204.03,
  "view": 0.0,
  "db": 112.28,
  "location": "http://localhost:3000/plans",
  "unpermitted_params": [
    "authenticity_token",
    "user",
    "authenticity_token",
    "user",
    "user",
    "language_id",
    "language_id"
  ],
  "params": {
    "user": {
      "email": "[FILTERED]",
      "org_id": "15",
      "password": "[FILTERED]"
    }
  },
  "ip":"172.20.0.1",
  "user_id": 54373,
  "@timestamp": "2025-01-29T22:18:05.357Z",
  "@version": "1",
  "message": "[302] POST /users/sign_in (Users::SessionsController#create)"
}