HLD for Memory_Statistics #1760

Arham-Nasir · 2024-07-23T08:38:56Z

This High-Level Design (HLD) document explains the Memory Statistics feature in SONiC. The aim is to improve memory monitoring and management to optimize network performance and reliability.

The Linked PR are as follows:

Repo	PR title	State
sonic-host-services	Support for Memory Statistics Host-Services
sonic-buildimage	Add YANG Model and Configuration Support for Memory Statistics
sonic-buildimage	Support for Memory Statisticsd Process and Configurations
sonic-utilities	Memory Statistics Config and Show Commands
sonic-swss-common	Add definition for MEMORY_STATISTICS table in schema

linux-foundation-easycla · 2024-07-23T08:39:00Z

The committers listed above are authorized under a signed CLA.

✅ login: Arham-Nasir (83fb3bc, 7749c55, c760669, ea8547c, 8bce9fd, 9f018b1, 4cf0b5d, 0dffed1)
✅ login: kanza-latif (ec2ce4c, 6c66870, 1c0ccc2, 62deb0f, 149fc81, c7e45be, c302011, 4b0d0b6)

zhangyanzhao · 2024-08-20T15:02:55Z

reviewer is expected and please leave your comments if you want to be a reviewer. Thanks.

zhangyanzhao · 2024-08-20T15:37:37Z

community review recording https://zoom.us/rec/share/DtubGviNDO1Rt8VRnHsqWpnF17tu62g_CSP8aTY-hjIZct50ChlNJVWsjBqLaYIl.eYe4gvza4Lsqjhs3

prgeor · 2024-08-20T21:32:28Z

doc/memory_statistics/memory_statistics_hld.md

+	  Metric            Current     High       Low        D1-D3      D3-D5      D5-D7      D7-D9      D9-D11     D11-D13    D13-D15    
+	                    Value       Value      Value      01Jun24    03Jun24    05Jun24    07Jun24    09Jun24    11Jun24    13Jun24    
+	--------------      --------    --------   --------   ---------  ---------  ---------  ---------  ---------  ---------  ---------
+	Total Memory        15.6G       15.6G      15.1G      15.1G      15.2G      15.3G      15.3G      15.4G      15.5G      15.5G      
+	Used Memory         2.3G        2.5G       2.0G       2.1G       2.2G       2.2G       2.3G       2.4G       2.3G       2.2G       
+	Free Memory         11.9G       12.4G      10.6G      11.0G      11.2G      11.4G      11.5G      11.7G      11.8G      11.9G      
+	Available Memory    13.0G       14.3G      12.4G      12.6G      12.7G      12.8G      12.9G      13.0G      13.1G      13.2G      
+    Cached Memory       1.2G        1.5G       1.0G       1.1G       1.2G       1.3G       1.4G       1.3G       1.4G       1.2G       
+    Buffer Memory       0.3G        0.4G       0.2G       0.2G       0.3G       0.3G       0.4G       0.3G       0.3G       0.4G       
+	Shared Memory       0.5G        0.6G       0.4G       0.5G       0.5G       0.5G       0.4G       0.5G       0.5G       0.5G       
+


@Arham-Nasirare these stats collected from the output of free linux command?

The data is not directly collected from the output of the free Linux command. Instead, they are gathered using the psutil Python library, which directly accesses system memory information via the same underlying mechanisms that the free command relies on. Specifically, the psutil.virtual_memory() function retrieves memory stats such as total, used, free, buffers, cached, and available memory.This approach allows for more direct access to memory data, avoiding the need to parse command output while providing flexibility for programmatic use.

qiluo-msft · 2024-08-20T21:43:39Z

doc/memory_statistics/memory_statistics_hld.md

+    Analysis Period:     From 2024-06-01 09:00:00 to 2024-06-15 09:00:00
+    Interval:	         2 days
+	--------------------------------------------------------------------------------
+	  Metric            Current     High       Low        D1-D3      D3-D5      D5-D7      D7-D9      D9-D11     D11-D13    D13-D15    


sonic-gnmi (telemetry) already has a comprehensive implementation https://github.com/sonic-net/sonic-gnmi/blob/64ed32b715e1673c3376701ee192d6292f5ec0ec/sonic_data_client/non_db_client.go#L235. Please check and ensure matrix are aligned.
Even better, suggest reuse:

delegate telemetry container as the memory collector

reduce this HLD scope to only memory-stats and storage, connecting to telemetry container by gnmi protocol

ensure this design could run inside and outside sonic switch

ensure this feature could be completely disabled inside a sonic switch and no performance burden if disabled

Thank you for the suggestion. We will review the sonic-gnmi implementation to align our metrics with the existing telemetry framework for system consistency. We will also explore reusing the telemetry container for memory collection and narrow the HLD scope to focus on memory statistics and storage.

Currently, we use the psutil.virtual_memory() function from the psutil library to gather memory data, which is similar to the getProcMeminfo() function in non_db_client.go, as both methods pull information from /proc/meminfo. We believe that psutil offers a more efficient and Pythonic approach compared to relying solely on the telemetry container for this purpose.

Can you elaborate more on how the design could run inside and outside the sonic switch.

if this design rely on telemetry container as the memory stats collector, it can run inside or outside sonic switch.

this thread is still applicable.

To provide some context on the current design, we gather memory statistics directly from SONiC using the psutil library. The collected data is stored as gzip files in the /var/log directory and is accessible via show commands through Unix sockets.

We appreciate your suggestion to leverage the telemetry container. As part of enhancements for this feature, we will certainly consider integrating with the telemetry container and aligning the design with your recommendations.

Thank you.

FengPan-Frank · 2024-08-21T07:59:49Z

doc/memory_statistics/images/enable_disable.svg

In enable/disable case, do you still use SIGHUP which means memstatsd will keep running? or just SIGTERM the memoryStats daemon.

hostcfgd will detect the enable/disable flag in ConfigDB. For disable, it sends SIGTERM to stop the daemon, where the daemon gracefully stops all processes, releases resources, and completely shuts down. For enable, it restarts the daemon using systemd, and SIGHUP is used for configuration reloads while the daemon is running.

Thanks, better to mention it explicitly, memorystats is not daemon but process controlled by hostcfgd, what if hostcfgd crashes, will it reload memorystats and recalculate retention?

There was a misunderstanding. Yes, we are introducing memorystatsd as a process and not a separate container. If the hostcfgd crashes, memorystats will restart and when the user enables memory statistics feature, it will use the default retention time unless the user explicitly mentions a custom retention period. Lets say, user enters 5 as retention period, then it will start the retention period from that time and will show the last 5 days data.

Add dynamic config reload and graceful shutdown for memorystatsd

Arham-Nasir · 2024-09-26T11:44:13Z

@qiluo-msft @xincunli-sonic @zbud-msft @FengPan-Frank We have linked the code PRs. Please help review the feature.

Signed-off-by: Arham-Nasir <[email protected]>

ridahanif96 · 2024-10-25T07:39:02Z

@qiluo-msft @xincunli-sonic @zbud-msft @FengPan-Frank We have linked the code PRs. Please help review the feature.
Target 202411, help with expedite review.

ridahanif96 · 2024-11-06T06:39:31Z

@qiluo-msft @xincunli-sonic @FengPan-Frank Please help review the feature.
Target 202411, help with expedite review.

ridahanif96 · 2024-11-15T12:49:44Z

@FengPan-Frank please help review and approve this feature. Code PRs are already in review. Target is 202411. We are running out of time for HLD review. Please help expedite review.

FengPan-Frank · 2024-11-20T12:58:57Z

@FengPan-Frank please help review and approve this feature. Code PRs are already in review. Target is 202411. We are running out of time for HLD review. Please help expedite review.

Pls get final signoff from Qi, I think code PR will need to be merged before HLD.

ridahanif96 · 2024-11-20T15:05:14Z

@FengPan-Frank please help review and approve this feature. Code PRs are already in review. Target is 202411. We are running out of time for HLD review. Please help expedite review.

Pls get final signoff from Qi, I think code PR will need to be merged before HLD.

@qiluo-msft help with this HLD review.

qiluo-msft · 2024-12-03T19:55:45Z

doc/memory_statistics/memory_statistics_hld.md

-| sampling_interval   | unit16      | Interval for memory data collection.       		     |
-| retention_period    | unit16      | Duration for which memory data is retained.		     |   
+| sampling_interval   | unit8      | Interval for memory data collection.       		     |
+| retention_period    | unit8      | Duration for which memory data is retained.		     |   


What are the units for interval and period? in seconds? IF yes, is uint8 too small?
typo: unit8->uint8

Thank you for your valuable feedback. To clarify:

The sampling_interval is provided by the user in minutes, with a configurable range from 3 to 15 minutes.

The retention_period is provided by the user in days, with a configurable range from 1 to 30 days.

Since the values for sampling_interval and retention_period fall within the range of uint8 (0 to 255), we believe uint8 is appropriate as the data type for representing the user input.

Additionally, the typo has been corrected to uint8 as suggested and the descriptions have been updated for further clarity

…ated descriptions and hld for clarity

zhangyanzhao · 2024-12-10T05:30:46Z

code PRs and HLD PR are not merged, move to backlog

xincunli-sonic

LGTM

HLD for Memory_Statistics

8bce9fd

kanza-latif and others added 9 commits August 19, 2024 13:25

Update memory_statistics_hld.md

ec2ce4c

UPdated Image

83fb3bc

Modifed diagram

ea8547c

updated image

7749c55

updated architecture diagram

149fc81

replaced the architecture diagram

c7e45be

Update memory_statistics_hld.md

62deb0f

Update memory_statistics_hld.md

4b0d0b6

Update memory_statistics_hld.md

1c0ccc2

zhangyanzhao assigned ridahanif96 and Arham-Nasir and unassigned ridahanif96 Aug 20, 2024

zhangyanzhao requested a review from qiluo-msft August 20, 2024 21:25

qiluo-msft requested review from zbud-msft, xincunli-sonic and FengPan-Frank August 20, 2024 21:26

prgeor reviewed Aug 20, 2024

View reviewed changes

qiluo-msft reviewed Aug 20, 2024

View reviewed changes

FengPan-Frank reviewed Aug 21, 2024

View reviewed changes

kanza-latif and others added 3 commits September 9, 2024 16:47

Update memory_statistics_hld.md

6c66870

updated the sequence diagrams and architecture diagram

c302011

Update memory_statistics_hld.md

c760669

Add dynamic config reload and graceful shutdown for memorystatsd

Arham-Nasir added 2 commits October 1, 2024 12:05

Updated the HLD

0dffed1

Signed-off-by: Arham-Nasir <[email protected]>

Delete doc/memory_statistics/images/enable_disable.svg

9f018b1

qiluo-msft reviewed Dec 3, 2024

View reviewed changes

Corrected data type to uint8 for MEMORY_STATISTICS parameters and upd…

4cf0b5d

…ated descriptions and hld for clarity

xincunli-sonic approved these changes Dec 10, 2024

View reviewed changes

ridahanif96 merged commit 07851ff into sonic-net:master Dec 14, 2024
1 check passed

ridahanif96 mentioned this pull request Dec 28, 2024

Memory Statistics Enhancement #1878

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HLD for Memory_Statistics #1760

HLD for Memory_Statistics #1760

Arham-Nasir commented Jul 23, 2024 •

edited by ridahanif96

Loading

linux-foundation-easycla bot commented Jul 23, 2024 •

edited

Loading

zhangyanzhao commented Aug 20, 2024

zhangyanzhao commented Aug 20, 2024

prgeor Aug 20, 2024

Arham-Nasir Aug 23, 2024

qiluo-msft Aug 20, 2024

Arham-Nasir Aug 23, 2024

qiluo-msft Nov 21, 2024

qiluo-msft Dec 3, 2024

Arham-Nasir Dec 4, 2024 •

edited

Loading

FengPan-Frank Aug 21, 2024

Arham-Nasir Aug 23, 2024

FengPan-Frank Aug 23, 2024

kanza-latif Aug 28, 2024

Arham-Nasir commented Sep 26, 2024 •

edited by ridahanif96

Loading

ridahanif96 commented Oct 25, 2024 •

edited

Loading

ridahanif96 commented Nov 6, 2024 •

edited

Loading

ridahanif96 commented Nov 15, 2024

FengPan-Frank commented Nov 20, 2024

ridahanif96 commented Nov 20, 2024

qiluo-msft Dec 3, 2024

Arham-Nasir Dec 4, 2024

zhangyanzhao commented Dec 10, 2024

xincunli-sonic left a comment

HLD for Memory_Statistics #1760

HLD for Memory_Statistics #1760

Conversation

Arham-Nasir commented Jul 23, 2024 • edited by ridahanif96 Loading

linux-foundation-easycla bot commented Jul 23, 2024 • edited Loading

zhangyanzhao commented Aug 20, 2024

zhangyanzhao commented Aug 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Arham-Nasir Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Arham-Nasir commented Sep 26, 2024 • edited by ridahanif96 Loading

ridahanif96 commented Oct 25, 2024 • edited Loading

ridahanif96 commented Nov 6, 2024 • edited Loading

ridahanif96 commented Nov 15, 2024

FengPan-Frank commented Nov 20, 2024

ridahanif96 commented Nov 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhangyanzhao commented Dec 10, 2024

xincunli-sonic left a comment

Choose a reason for hiding this comment

Arham-Nasir commented Jul 23, 2024 •

edited by ridahanif96

Loading

linux-foundation-easycla bot commented Jul 23, 2024 •

edited

Loading

Arham-Nasir Dec 4, 2024 •

edited

Loading

Arham-Nasir commented Sep 26, 2024 •

edited by ridahanif96

Loading

ridahanif96 commented Oct 25, 2024 •

edited

Loading

ridahanif96 commented Nov 6, 2024 •

edited

Loading