fix(seer grouping): Save grouphash metadata before adding seer results

[lobsterkatie]

Once we turned on the grouphash metadata backfill for existing grouphashes, a weird thing started sometimes happening with new grouphashes: when we went to update their metadata with Seer results, we got a Cannot update an instance that has not yet been created error. The breadcrumbs show us querying for the metadata record (the instance in question), and presumably we find it, since we then test whether the result is truthy and only proceed if it is. Indeed, in the stack locals of the error, we can see that there's a metadata object there.

One odd thing we noticed is that the metadata object retrieved has an id of None, as if it only exists in memory and hasn't yet been saved to the database. That shouldn't be possible, since it's created with objects.create, which should immediately save the record, but perhaps creation is buffered and we're running into a race condition? To test that hypothesis, this PR forces the object to be saved in cases where the id is None. It also bails before the problematic update call if that save doesn't fix the problem, so ingestion can proceed without errors. Finally, it adds logging so we can see both how often it happens and whether the fix works.

[codecov]

Codecov Report

Attention: Patch coverage is 25.00000% with 6 lines in your changes missing coverage. Please review.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/sentry/grouping/ingest/seer.py	25.00%	6 Missing ⚠️

❌ Your patch check has failed because the patch coverage (25.00%) is below the target coverage (50.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #85804       +/-   ##
===========================================
+ Coverage   46.67%   87.88%   +41.21%     
===========================================
  Files        9669     9683       +14     
  Lines      548150   548819      +669     
  Branches    21318    21318               
===========================================
+ Hits       255824   482325   +226501     
+ Misses     292019    66187   -225832     
  Partials      307      307               

[JoshFerge]

pasting slack comment for posterity:

so i'm thinking that the grouphash that we're passing into the function has a stale value of grouphashmetadata in cases where we're creating the grouphash for the same event. this is because when we create grouphash metadata, this doesn't automatically update the instantiated grouphash model that we have.
my thought is that we can call
grouphash.refresh_from_db() at the top of the create_or_update_grouphash_metadata_if_needed function, and that should fix this.
alternatively, i think we could also just say grouphash.metadata = metadata in

sentry/src/sentry/grouping/ingest/grouphash_metadata.py

Line 147 in e1b9e91

GroupHashMetadata.objects.create(**new_data)

[lobsterkatie]

pasting slack comment for posterity:

so i'm thinking that the grouphash that we're passing into the function has a stale value of grouphashmetadata in cases where we're creating the grouphash for the same event. this is because when we create grouphash metadata, this doesn't automatically update the instantiated grouphash model that we have. my thought is that we can call grouphash.refresh_from_db() at the top of the create_or_update_grouphash_metadata_if_needed function, and that should fix this. alternatively, i think we could also just say grouphash.metadata = metadata in

sentry/src/sentry/grouping/ingest/grouphash_metadata.py

Line 147 in e1b9e91

GroupHashMetadata.objects.create(**new_data)

Agreed through IRL convo that this isn't the issue (otherwise it'd never work, rather than working in 99%+ of cases). Will update here once we figure out what is.