Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use orjson to parse json faster #32153

Merged
merged 4 commits into from
Feb 25, 2020
Merged

Conversation

KapJI
Copy link
Member

@KapJI KapJI commented Feb 24, 2020

Proposed change

With #32128 and #32137 loading of logbook for the last 7 days improved from 22s to 11.3s.

I think it can be improved further. I noticed that json.loads takes quite a lot of time.
https://github.com/ijl/orjson is about 2 times faster than standard json.loads and saves another 1.25s or 11% of overall time.

Before:

Screen Shot 2020-02-24 at 10 19 46 PM

After:

Screen Shot 2020-02-24 at 10 20 29 PM

For encoding json.dumps is still used because orjson doesn't support custom serialiser in cls arg.

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Example entry for configuration.yaml:

# Example configuration.yaml

Additional information

  • This PR fixes or closes issue: fixes #
  • This PR is related to issue:
  • Link to documentation pull request:

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • The code has been formatted using Black (black --fast homeassistant tests)
  • Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • Untested files have been added to .coveragerc.

The integration reached or maintains the following Integration Quality Scale:

  • No score or internal
  • 🥈 Silver
  • 🥇 Gold
  • 🏆 Platinum

@springstan springstan changed the title [recorder] Use orjson to parse json faster Use orjson to parse json faster in recorder Feb 24, 2020
@codecov
Copy link

codecov bot commented Feb 24, 2020

Codecov Report

Merging #32153 into dev will increase coverage by 0.02%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##              dev   #32153      +/-   ##
==========================================
+ Coverage   94.73%   94.75%   +0.02%     
==========================================
  Files         767      768       +1     
  Lines       55505    55646     +141     
==========================================
+ Hits        52580    52727     +147     
+ Misses       2925     2919       -6
Impacted Files Coverage Δ
setup.py 88.02% <ø> (ø) ⬆️
homeassistant/components/recorder/models.py 93.93% <100%> (+0.06%) ⬆️
homeassistant/helpers/aiohttp_client.py 95.34% <100%> (+0.05%) ⬆️
homeassistant/util/json.py 85.71% <100%> (+0.2%) ⬆️
...istant/components/google_assistant/report_state.py 83.33% <0%> (-1.67%) ⬇️
homeassistant/components/alexa/state_report.py 83.63% <0%> (-0.63%) ⬇️
...meassistant/components/mqtt/vacuum/schema_state.py 91.13% <0%> (-0.59%) ⬇️
homeassistant/components/twitch/sensor.py 96.7% <0%> (-0.04%) ⬇️
homeassistant/components/rflink/light.py 100% <0%> (ø) ⬆️
homeassistant/components/darksky/sensor.py 96.81% <0%> (ø) ⬆️
... and 58 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9801810...abc2108. Read the comment docs.

@KapJI KapJI requested a review from a team as a code owner February 25, 2020 12:08
@KapJI KapJI changed the title Use orjson to parse json faster in recorder Use orjson to parse json faster Feb 25, 2020
@KapJI
Copy link
Member Author

KapJI commented Feb 25, 2020

@balloob I added it to http manifest because of failed CI job.
http component imports util.json and validation scripts fails even though this library is in main setup.py :(

@balloob
Copy link
Member

balloob commented Feb 25, 2020

It's a problem with CI that we should solve. We shouldn't hack around it.

@KapJI
Copy link
Member Author

KapJI commented Feb 25, 2020

Can you fix it then? :)

@balloob
Copy link
Member

balloob commented Feb 25, 2020

Trying but GitHub is not cooperating.

@balloob balloob merged commit 2365e2e into home-assistant:dev Feb 25, 2020
@balloob
Copy link
Member

balloob commented Feb 25, 2020

Realized this is using Rust, which means a big pain point for the community to install on RPi's in a venv. I will have to revert this PR.

balloob added a commit that referenced this pull request Feb 25, 2020
pvizeli pushed a commit that referenced this pull request Feb 25, 2020
@elupus
Copy link
Contributor

elupus commented Feb 26, 2020

I wonder if storing this in a sql native json data type is faster. Might just shift the work, but if it was smartly done it aught to be quicker.

@lock lock bot locked and limited conversation to collaborators Feb 28, 2020
@KapJI KapJI deleted the recorder-orjson branch April 27, 2021 18:31
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants