Heartbeat possible flaky test on monitors/active/http

[sayden]

@andrewvc I was just having issues with this. Forgive me if this is fixed already but I prefer to report and be wrong than to be right and do not report 😉

Flaky Test

• Test Name: heartbeat/monitors/active/http might be a flaky test
• Link: May be fixed already? I can't find neither assertions.go file or mapvaltest.go file which are the failing tests (look at the artifact link below)
• Branch: 6.7
• Artifact Link: https://beats-ci.elastic.co/job/elastic+beats+pull-request+multijob-linux/4920/beat=heartbeat,label=linux-immutable/console
• Notes: PR where the fail was consistently happening Cherry-pick #10592 to 6.x: Update and add missing descriptions to ETCD module #10611 (all retries of Jenkins were because of some failure in Heartbeat and some other Beat / libbeat)

Stack Trace

12:25:01 command [go test -race -tags integration -cover -coverprofile /tmp/gotestcover-146585084 github.com/elastic/beats/heartbeat/monitors/active/http]: exit status 1
12:25:01 --- FAIL: TestHTTPSx509Auth (1.42s)
12:25:01 	assertions.go:256: 
12:25:01 			Error Trace:	mapvaltest.go:40
12:25:01 			Error:      	mapval could not validate map
12:25:01 			Test:       	TestHTTPSx509Auth
12:25:01 			Messages:   	13 errors validating source: 
12:25:01 			            	(common.MapStr) (len=4) {"error":{"message":"Get https://127.0.0.1:38334: read tcp 127.0.0.1:59732-\u003e127.0.0.1:38334: i/o timeout","type":"io"},"http":{"url":"https://127.0.0.1:38334"},"monitor":{"duration":{"us":1015296},"id":"http@https://127.0.0.1:38334","ip":"127.0.0.1","scheme":"https","status":"down"},"tcp":{"port":38334,"rtt":{"connect":{"us":610}}}}
12:25:01 	assertions.go:256: 
12:25:01 			Error Trace:	mapvaltest.go:48
12:25:01 			Error:      	Received unexpected error:
12:25:01 			            	@path 'tls.rtt.handshake.us': expected this key to be present
12:25:01 			Test:       	TestHTTPSx509Auth
12:25:01 	assertions.go:256: 
12:25:01 			Error Trace:	mapvaltest.go:48
12:25:01 			Error:      	Received unexpected error:
12:25:01 			            	@path 'http.rtt.content.us': expected this key to be present
12:25:01 			Test:       	TestHTTPSx509Auth
12:25:01 	assertions.go:256: 
12:25:01 			Error Trace:	mapvaltest.go:48
12:25:01 			Error:      	Received unexpected error:
12:25:01 			            	@path 'error.type': unexpected field encountered during strict validation
12:25:01 			Test:       	TestHTTPSx509Auth
12:25:01 	assertions.go:256: 
12:25:01 			Error Trace:	mapvaltest.go:48
12:25:01 			Error:      	Received unexpected error:
12:25:01 			            	@path 'error.message': unexpected field encountered during strict validation
12:25:01 			Test:       	TestHTTPSx509Auth
12:25:01 	assertions.go:256: 
12:25:01 			Error Trace:	mapvaltest.go:48
12:25:01 			Error:      	Received unexpected error:
12:25:01 			            	@path 'monitor.status': objects not equal: actual(down) != expected(up)
12:25:01 			Test:       	TestHTTPSx509Auth
12:25:01 	assertions.go:256: 
12:25:01 			Error Trace:	mapvaltest.go:48
12:25:01 			Error:      	Received unexpected error:
12:25:01 			            	@path 'http.rtt.write_request.us': expected this key to be present
12:25:01 			Test:       	TestHTTPSx509Auth
12:25:01 	assertions.go:256: 
12:25:01 			Error Trace:	mapvaltest.go:48
12:25:01 			Error:      	Received unexpected error:
12:25:01 			            	@path 'tls.certificate_not_valid_after': expected this key to be present
12:25:01 			Test:       	TestHTTPSx509Auth
12:25:01 	assertions.go:256: 
12:25:01 			Error Trace:	mapvaltest.go:48
12:25:01 			Error:      	Received unexpected error:
12:25:01 			            	@path 'error': unexpected field encountered during strict validation
12:25:01 			Test:       	TestHTTPSx509Auth
12:25:01 	assertions.go:256: 
12:25:01 			Error Trace:	mapvaltest.go:48
12:25:01 			Error:      	Received unexpected error:
12:25:01 			            	@path 'tls.certificate_not_valid_before': expected this key to be present
12:25:01 			Test:       	TestHTTPSx509Auth
12:25:01 	assertions.go:256: 
12:25:01 			Error Trace:	mapvaltest.go:48
12:25:01 			Error:      	Received unexpected error:
12:25:01 			            	@path 'http.rtt.validate.us': expected this key to be present
12:25:01 			Test:       	TestHTTPSx509Auth
12:25:01 	assertions.go:256: 
12:25:01 			Error Trace:	mapvaltest.go:48
12:25:01 			Error:      	Received unexpected error:
12:25:01 			            	@path 'http.response.status_code': expected this key to be present
12:25:01 			Test:       	TestHTTPSx509Auth
12:25:01 	assertions.go:256: 
12:25:01 			Error Trace:	mapvaltest.go:48
12:25:01 			Error:      	Received unexpected error:
12:25:01 			            	@path 'http.rtt.response_header.us': expected this key to be present
12:25:01 			Test:       	TestHTTPSx509Auth
12:25:01 	assertions.go:256: 
12:25:01 			Error Trace:	mapvaltest.go:48
12:25:01 			Error:      	Received unexpected error:
12:25:01 			            	@path 'http.rtt.total.us': expected this key to be present
12:25:01 			Test:       	TestHTTPSx509Auth
12:25:01 FAIL
12:25:01 coverage: 62.4% of statements
12:25:01 FAIL	github.com/elastic/beats/heartbeat/monitors/active/http	4.878s

[andrewvc]

@sayden thanks for the ping!

I think it is indeed flakey (though very rare).

It's a tough one, because it definitely should work, and I can't repro it. I think the solution here is to have the test code wait 5 seconds or so for the HTTP endpoint to come up before running the proper test.

[andrewvc]

Fixed here: #10789 (I hope)

Thanks for writing this excellent report.

[sayden]

Feel free to close the issue when you think it's done