O+M 04-07-2023

[nickumia-reisys]

As part of day-to-day operation of Data.gov, there are many Operation and Maintenance (O&M) responsibilities. Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an O&M Triage role. One person on the team is assigned the Triage role which rotates each sprint. This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time.

Miscs

• Watch for user email requests
• Watch in #datagov-alerts and Vulnerable dependency notifications (daily email reports) for critical alerts.
• Monitor and improve Data.gov O&M Dashboard

Acceptance criteria

You are responsible for all O&M responsibilities this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

Daily Checklist

Check Deployments

• Catalog
• Inventory

Check Restarts

• Catalog
• Inventory

Check Snyk Scans

• Catalog
• Inventory

Check Catalog Auto Tasks

Note
You will need to update the chart values manually. Click the Action link in each issue and grab the values from monitor task output and check runtime.

• DB-Solr Sync
• Tracking Update

Check Harvesting Emails

• Catalog

Other

• New Relic Alerts Triaged
• Triage DMARC Report from Google

Weekly Checklist

• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Check Catalog Solr
• Catalog Dupe Check
• Invicti Scan

[hkdctol]

We will try one week shifts. Let's check in at daily sync on Friday 4/7.

[Jin-Sun-tts]

Friday 03/31

Check Deployments

• Catalog
• Inventory

Check Restarts

• Catalog
• Inventory

Check Snyk Scans

• Catalog
• Inventory

Check Catalog Auto Tasks

• DB-Solr Sync: 33507 re-indexed today

• Tracking Update: 43566 updated today

• Updated the graphic with number and time.
• Added new month for April,

Check Harvesting Emails

• Catalog: 55 harvest job error emails received at 10AMEST (will check again in the later afternoon)

Other

• New Relic Alerts Triaged: there are two on staging:

• Triage DMARC Report from Google:

Weekly Checklist

• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Check Catalog Solr: There were memory warning earlier, but all green now.

• Catalog Dupe Check
  

• Invicti Scan: no report received.

[Jin-Sun-tts]

Monday 04/03

https://github.com/GSA/data.gov/tree/om-status-updates

Check Catalog Auto Tasks

• DB-Solr Sync: 4698 packages need to be updated/added to Solr

• Tracking Update: 58192 package indexes to be rebuilt starting from 2023-03-31

Check Harvesting Emails

• Catalog: By 5pm EST, total received 10 harvesting job error notification

• most of them are missing required fields like :

  • Identifier: https://data.cdc.gov/api/views/728v-5uaa; Title: trails_lines; 3 Error(s) Found. ### ERROR Build Failing - Error importing PyZ3950 #1: 'programCode' is a required property; ### ERROR setup Route 53 ansible play  #2: 'keyword' is a required property; ### ERROR setup EC2/Autoscaling group/launch configuraiton #3: 'description':'' is too short.
  • Identifier: https://healthdata.gov/api/views/9q39-eeqb; Title: Lyme Innovation & Health Equity; 2 Error(s) Found. ### ERROR Build Failing - Error importing PyZ3950 #1: 'bureauCode' is a required property; ### ERROR setup Route 53 ansible play  #2: 'programCode' is a required property.
  • Identifier: https://datadiscovery.nlm.nih.gov/api/views/a392-5wn4; Title: NLM Catalog; 1 Error(s) Found. ### ERROR Build Failing - Error importing PyZ3950 #1: 'keyword' is a required property.

• some of them are like:

  • Element not found: Identification Citation Originator
  • Transformation to ISO failed
  • Element not found: Metadata Reference Information Contact Address Type
  • Element not found: Metadata Reference Information Contact Address State
  • Element not found: Identification Citation Originator
  • Transformation to ISO failed
  • Element not found: Metadata Reference Information Contact Address Type
  • Element not found: Metadata Reference Information Contact Address State
  • Element not found: Metadata Reference Information Contact Address Type

Other

• New Relic Alerts Triaged: no critical issue on production.

• Triage DMARC Report from Google:

[Jin-Sun-tts]

Tuesday 04/04

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• DB-Solr Sync: 4698 packages need to be updated/added to Solr

• Tracking Update: 66683 package indexes to be rebuilt starting from 2023-04-01 00:00:00

Check Harvesting Emails

• Catalog:
• no harvesting job reported yet at 10am EST
• 10 harvesting job reported yet at 5pm EST, most of them are related the missing required fields.

Other

• New Relic Alerts Triaged

2 or more locations failed on 'dashboard (production)' - resolved.

• Triage DMARC Report from Google:

[Jin-Sun-tts]

Wednesday 04/05

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• DB-Solr Sync: 4698 packages need to be updated/added to Solr

• Tracking Update: 62393 package indexes to be rebuilt starting from 2023-04-02 00:00:00

Check Harvesting Emails

• Catalog:
• no harvesting job reported yet at 10am EST
• 11 harvesting job errors reported by 5pm EST

Other

• New Relic Alerts Triaged

• Triage DMARC Report from Google:

[Jin-Sun-tts]

Thursday 04/06

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• DB-Solr Sync: 4697 packages need to be updated/added to Solr

• Tracking Update: 62150 package indexes to be rebuilt starting from 2023-04-03 00:00:00

Check Harvesting Emails

• Catalog:
• no harvesting job reported yet at 10am EST
• 9 harvesting job reported yet at 4pm EST, most of them are related the missing required fields.

Other

• New Relic Alerts Triaged

• Triage DMARC Report from Google:

[Jin-Sun-tts]

Friday 04/07

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• DB-Solr Sync: 10628 packages need to be updated/added to Solr

• Tracking Update: 55589 package indexes to be rebuilt starting from 2023-04-04 00:00:00

Check Harvesting Emails

• Catalog:
• 10amEST: 63 harvesting job errors reported since 4pm EST yesterday.

Other

• New Relic Alerts Triaged

• Triage DMARC Report from Google:

Weekly Checklist

• Audit Log (more info on AU-3 and AU-6 Log auditing)

• Check Catalog Solr:

• The leader and three follower are all green.

• Catalog Dupe Check

• Invicti Scan: no report received.