Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Safty-data api involved #2760

Closed
wants to merge 25 commits into from
Closed

Conversation

ziv17
Copy link
Collaborator

@ziv17 ziv17 commented Jan 12, 2025

WIP, /involved api implementing Atalya's comment from 9-Jan on issue 2734.
Added load of safety_data involved and accident tables to the main.py process cbs command line.
No fine detail testing yet. Just to verify the query parameter and returned data.
You can view the API in swagger hub

@ziv17 ziv17 requested a review from atalyaalon January 12, 2025 05:29
@codecov-commenter
Copy link

codecov-commenter commented Jan 12, 2025

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 54.11765% with 156 lines in your changes missing coverage. Please review.

Project coverage is 54.06%. Comparing base (07a333b) to head (57dc88c).
Report is 17 commits behind head on dev.

Files with missing lines Patch % Lines
anyway/views/safety_data/involved_query_gb.py 35.64% 65 Missing ⚠️
anyway/views/safety_data/involved_query.py 51.69% 57 Missing ⚠️
anyway/flask_app.py 14.28% 24 Missing ⚠️
anyway/views/safety_data/sd_utils.py 80.39% 10 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##              dev    #2760      +/-   ##
==========================================
+ Coverage   52.57%   54.06%   +1.49%     
==========================================
  Files         122      125       +3     
  Lines       10232    10586     +354     
==========================================
+ Hits         5379     5723     +344     
- Misses       4853     4863      +10     
Flag Coverage Δ
unittests 54.06% <54.11%> (+1.49%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ziv17 ziv17 requested a review from citizen-dror January 12, 2025 17:30
Copy link
Collaborator

@atalyaalon atalyaalon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ziv17 all in all looks good. Well done!
I added a few comments.

In addition:

  1. I suggest using AccidentMarker instead of AccidentMarkerView. I assume that we won't have road_segment number for this table, but I suggest we'll handle this in a later PR, and for now I suggest keeping a placeholder for road_segment.
    Let me know if there are other challenges from using AccidentMarker instead of AccidentMarkerView.

  2. Do we keep data consistency for DB for these tables?
    (We had a similar discussion here)
    I think that in the case above with db.get_engine().begin() as conn: kept data consistency.
    I believe it can also be done with using db.session.commit() only after finishing deletions insertions.

  3. For better practice, I suggest you consider adding relationships for dimension tables. However I suggest adding it in a future pr, when we decide to handle the issue across our db, and not in this issue. I opened an issue for it here

@atalyaalon
Copy link
Collaborator

@ziv17 all in all looks good. Added a few comments, we can also discuss in tomorrow's meeting

@atalyaalon atalyaalon linked an issue Jan 21, 2025 that may be closed by this pull request
Load sd data chuncked.
@ziv17 ziv17 requested a review from atalyaalon February 2, 2025 20:43
@ziv17
Copy link
Collaborator Author

ziv17 commented Feb 2, 2025

Hi, I added /involved/groupby with most of the fields.

@ziv17 ziv17 changed the title Ssafty-data api involved Safty-data api involved Feb 4, 2025
@ziv17 ziv17 mentioned this pull request Feb 4, 2025
@atalyaalon
Copy link
Collaborator

atalyaalon commented Feb 5, 2025

@ziv17 I added relevant comments in this pr: #2768

I suggest that:

  1. Address the small issues I wrote in this pr
  2. Add an ETL pr in https://github.com/data-for-change/anyway-etl (without regular schedule) so we'll be able to load only the safety data tables in production, separately from CBS flow, for our own internal checks (of course loading should remain in regular CBS flow as you wrote it, we'll just add another ETL DAG to our Airflow).

After 1 and 2 we'll merge all prs.

@ziv17
Copy link
Collaborator Author

ziv17 commented Feb 9, 2025

This PR is superseded by #2768

@ziv17 ziv17 closed this Feb 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Safety Data new API
3 participants