Integration tests fail in travis although they pass in dev. environment

[batikanu]

Issue details

Deploy from file integration tests pass in development environment but not in travis.

Environment

gulp integration-test or
gulp integration-test:prod
both pass in development environment.

Dashboard version: master branch
Kubernetes version: v1.2.0
Operating system: Ubuntu
Node.js version: v4.2.1
Go version: go1.5.2

Steps to reproduce

Run travis build of master branch
Run integration tests locally in your development environment.

Observed result

Tests fail in travis but not locally.

Expected result

In both environment the tests should pass

Comments

[f-higashi]

I can reporduce the same situation; tests succeed in local but tests fail in travis.
https://travis-ci.org/f-higashi/dashboard/builds/114188727

And I checked screencast in saucelabs.
http://saucelabs.com/jobs/80f635f03ff044e69017a5b8993fdcdf

It seems when 'deploy' button is hidden partially, clicking deploy button will fail.
It depends on amount of hidden area.

I found same issues in some web pages.

http://stackoverflow.com/questions/11676790/click-command-in-selenium-webdriver-does-not-work
(this page's solution is using sendkey("\r") instead of click(). I tried it but test failed.)
http://devadjust.exblog.jp/20073998/
(Japanese. this page's solution is showing button element. I don't try it. It is C#.
I didn't find how to execute same operation in Javascript.)

But when I set window size(1000x750) which is smaller than screen size(1024x768), tests succeeded fortunattly.
Maybe amount of hidden area has been changed.

https://saucelabs.com/jobs/d164baa478da4107a26e690514bf0d8c

It is not fundamental solution.
But we can test under the same conditions by this settings
So if tests fail in local, we add scroll operation.

--- a/src/test/integration/stories/deploy_not_existing_img_test.js
+++ b/src/test/integration/stories/deploy_not_existing_img_test.js
@@ -56,7 +56,7 @@ describe('Deploy not existing image story', () => {

   beforeAll(() => {
     // For empty cluster this should actually redirect to zerostate page
+    browser.driver.manage().window().setSize(1000, 750);
     browser.get('#/replicationcontrollers');

     zeroStatePage = new ZeroStatePageObject();

[f-higashi]

http://stackoverflow.com/questions/11676790/click-command-in-selenium-webdriver-does-not-work
(this page's solution is using sendkey("\r") instead of click(). I tried it but test failed.)

Sorry.
I failed "\r".
But I tried sendKeys("\n") just now and it succeeded.

[floreks]

Unfortunately problem still occurs. I've even configured selenium server + IE web driver on my notebook in order to run tests on IE locally but this doesn't help as they always pass. This has to be a problem with slow machine/freezing of browser on sauce labs machines. Watching the video it looks like browser just freezes and after it unfreezes tests are couple of steps ahead. It doesn't happen everytime. Sometimes tests pass but this is very random.

http://saucelabs.com/jobs/f6603a1edb7e4d259da0dcdfa22a7af3

Local tests:
https://cloud.githubusercontent.com/assets/2285385/13636119/6c424f20-e600-11e5-89d5-fd20324af470.gif

There are also issues reported regarding timeouts and random fails on IE:
https://github.com/angular/protractor/labels/browser%3A%20IE

[floreks]

From my point of view we have few options:

1. Try to run tests on better machines on sauce labs (if it is possible). Currently IE tests are very laggy.
2. Disable tests on IE for now and hope for some fixes/IEWebDriver update that will allow tests to pass.
3. Try to find solution in order to let tests pass. (this may take a long time and there is no guarantee that this is even possible)
4. Use our own server(windows VM) with selenium and IE driver configured for IE tests.

[floreks]

@bryk any thoughts on that topic?

[cheld]

@floreks Can we contact saucelab support? Even if we use the free service, they should be interested in feedback about the quality of their service. We should give it a try

[bryk]

Saucelabs is pretty stable recently. I say either close this bug or contact sauce that it is flaky.

What do you think?

[floreks]

It's stable because we've pretty much disabled all integration tests due to IE issues. :)

You can take a look at above saucelab screencast from IE tests before they were disabled. It looks like either browser freeze or cluster freeze when deploy was clicked.

[bryk]

Closing as stale.