I would like to see Cucumber as supported framework for e2e testing.

[asnov]

I would like to see Cucumber as supported framework for e2e testing.
Thank you, guys!

[samhatoum]

I would also like to see this.

I am one of the maintainers of Chimp which allows you to run Cucumber with end-to-end tests. I'd be happy to collaborate with you @ArtemGovorov to get this working

Cucumber.js has a command-line runner and it outputs a JSON report file. I'm guessing we'd need to add it as a runner somehow and transform the output to something that wallaby can parse.

Any pointers on how I can start to mess with it?

[ArtemGovorov]

Hi @samhatoum, Chimp looks awesome, thanks for sharing.

Regarding the integration, I'm not familiar much with how Chimp/Cucumber works, so I'll try to describe what would be required from wallaby side.

A new runner most certainly is required (and that part of the core is not yet available for external extending unfortunately).

Regarding the output, wallaby.js needs quite a few things:

• Apart from the runner, the JavaScript that Chimp/Cucumber executes has to instrumented by wallaby core (to collect the code coverage data).
• Wallaby also needs to hook into the testing framework (by adding a reporter), so that it knows when a test/suite is completed, able to filter what tests to skip/execute, etc.

By instrumenting files and adding the reporter, wallaby.js is able to get the information it needs. With our PhantomJs runner, we create a web socket connection using which wallaby.js sends the collected data to its core (server).

[samhatoum]

Thank you @ArtemGovorov

I'll try to improvise on each point to say how I can see it being done:

• The instrumentation can be done prior to a Cucumber.js execution
• When Cucumber.js is complete, it generates a json file, which we can transform to a wallaby-compatible report
• The next Cucumber.js run can be passed only the filtered features to run

Typically Cucumber.js runs in a node environment (it's also possible to run on the client, but not a typical practice).

It sounds like the reporter is where the work would be. Can I help in any way?

[ArtemGovorov]

The instrumentation can be done prior to a Cucumber.js execution

Yes.

When Cucumber.js is complete, it generates a json file, which we can transform to a wallaby-compatible report

This is the hard part. Wallaby needs quite a few data bits from the test execution host, ideally in real-time just as tests are executing.

Thanks, I may need some help with the reporter, I first should find some time and have a deeper look into Cucumber.js to better understand how it works, how to make it send what wallaby needs and generally what's the best approach of integrating with it.

[samhatoum]

This is the hard part. Wallaby needs quite a few data bits from the test execution host, ideally in real-time just as tests are executing.

Look no further than the StepResult event. This can give you realtime results as step are executed happen.

To use it with Wallaby.js, you'd have to create a file like step-result.js file, and put this in it:

module.exports = function() {
  this.StepResult(function(event){
    var stepResult = event.getPayloadItem('stepResult');
    var step = stepResult.getStep();
    //console.log(stepResult); // be sure to check this out
    //console.log(step);  // be sure to check this out

    console.log('\n - - - - - STEP RESULT - - - - - ');
    console.log('stepResult.isFailed - -', stepResult.isFailed());
    console.log('stepResult.isPending - -', stepResult.isPending());
    console.log('stepResult.isSkipped - -', stepResult.isSkipped());
    console.log('stepResult.isSuccessful - -', stepResult.isSuccessful());
    console.log('stepResult.isUndefined - -', stepResult.isUndefined());
    console.log('stepResult.getDuration - -', stepResult.getDuration());

    console.log('\n - - - - - STEP - - - - - ');
    console.log('step.setPreviousStep - -', step.setPreviousStep());
    console.log('step.isHidden - -', step.isHidden());
    console.log('step.isOutlineStep - -', step.isOutlineStep());
    console.log('step.getKeyword - -', step.getKeyword());
    console.log('step.getName - -', step.getName());
    console.log('step.hasUri - -', step.hasUri());
    console.log('step.getUri - -', step.getUri());
    console.log('step.getLine - -', step.getLine());
    console.log('step.getPreviousStep - -', step.getPreviousStep());
    console.log('step.hasPreviousStep - -', step.hasPreviousStep());
    console.log('step.getAttachment - -', step.getAttachment());
    console.log('step.getAttachmentContents - -', step.getAttachmentContents());
    console.log('step.getDocString - -', step.getDocString());
    console.log('step.getDataTable - -', step.getDataTable());
    console.log('step.hasAttachment - -', step.hasAttachment());
    console.log('step.hasDocString - -', step.hasDocString());
    console.log('step.hasDataTable - -', step.hasDataTable());
    console.log('step.ensureDataTableIsAttached - -', step.ensureDataTableIsAttached());
    console.log('step.isOutcomeStep - -', step.isOutcomeStep());
    console.log('step.isEventStep - -', step.isEventStep());
    console.log('step.hasOutcomeStepKeyword - -', step.hasOutcomeStepKeyword());
    console.log('step.hasEventStepKeyword - -', step.hasEventStepKeyword());
    console.log('step.isRepeatingOutcomeStep - -', step.isRepeatingOutcomeStep());
    console.log('step.isRepeatingEventStep - -', step.isRepeatingEventStep());
    console.log('step.hasRepeatStepKeyword - -', step.hasRepeatStepKeyword());
    console.log('step.isPrecededByOutcomeStep - -', step.isPrecededByOutcomeStep());
    console.log('step.isPrecededByEventStep - -', step.isPrecededByEventStep());
    console.log('step.getStepDefinition - -', step.getStepDefinition());
  });
}

And then when you run cucumberjs from Wallaby, just pass it an additional file using -r <path to>/step-result.js.

That should do it!

[ArtemGovorov]

Thanks Sam, will look into the StepResult.

[timReynolds]

Plus one comments are super annoying but I'd be interested in seeing Cucumber support for wallaby.

[eggyal]

I'd also like to see support for Cucumber, or else to understand how one can get Wallaby working with Yadda (which ostensibly overlays Gherkin-style BDD support atop one's test framework, including Jasmine and Mocha)—any thoughts?

[ArtemGovorov]

@eggyal if Yadda is based on Cucumber, then this issue is definitely a prerequisite before supporting it.

[eggyal]

@ArtemGovorov: No, it doesn't depend on Cucumber. AIUI, your existing test framework (Jasmine, Mocha, whatever) uses Yadda to parse Gherkin-like feature files in order to identify which test functions from your spec files should be run (and with what arguments).

The problem I've had is that it's only really practical with a single entry-point, and I think this would break Wallaby's caching?

[ArtemGovorov]

@eggyal Got it, thanks.

it's only really practical with a single-entry point, and I think this would break Wallaby's caching?

Not sure about caching, but can definitely see that a few of main wallaby features, such as running tests in parallel and incrementally executing tests, will not work if the framework expects a single-entry point for executing specs.

[eggyal]

Yes, one would invoke mocha ./yadda-entry-point.js or whatever.

So I guess Yadda's resolution of which spec functions to invoke would have to take place within Wallaby... is there a plugin API or somesuch that I can harness?

[ArtemGovorov]

@eggyal There's no (public) API at the moment to perform such integration.

[chrisns]

In terms of the yadda stuff I put this example together so at least the wallaby hackery/work can live somewhere.
https://github.com/chrisns/yadda/tree/wallaby/examples/mocha-wallaby
I was trying to see if I could fool wallaby into working, turns out its too clever :)
Tests run beautifully the first time, I was super excited, in fact I changed the feature file a couple of times and it was all working, then it for no discernible reason stopped and started showing various irrelevant errors.
Obviously coverage + inline beautiful wallaby goodness in the .feature file are probably a little far fetched for now at least, but it would be wicked to be able to execute the tests.

[celador]

@ArtemGovorov Any idea for when/if this might happen? or when "the core may be available for external extending". Wallaby is really nice, but Chimp.js is more full featured. I bought Wallaby, and will likely use it for writing js unit tests, but having Chimp.js/Cucumber support would be a killer feature. Wallaby will be a hard sell to my team without Chimp/Cucumber support. I work on a team with 40+ developers and it would be an easy sell to the team if it had Chimp support. We'd certainly purchase a license for each developer if we could have Chimp(Chrome, Gherkin, Webdriver.io API) + Wallaby(super fast testing and code coverage in the IDE) . Please, please, please work with @samhatoum to see if you can get this working.

[ArtemGovorov]

@celador I have done some estimates and looked into Cucumber/Chimp to better understand our options back in Feb, there are a few things that need to be implemented in wallaby.js core before implementing the support. Some of them are pretty big. Nevertheless, I'd like to add Chimp/Cucumber support at some point, but can't make any promises when it's going to happen.

[samhatoum]

So Chimp is a test runner, and Wallaby is a test runner. A test runner running a test runner is a bad idea!

Saying that, Chimp has a session manager that makes end-to-end a pleasurable experience by reusing open sessions. There may be some value in extracting the session manager into a separate module, which can then be imported into a Wallaby + Cucumber.js configuration - but I think I'll only be able to do that once we see the basic Wallaby + Cucumber combo.

@ArtemGovorov let me know if you need anything when you finally get around to doing this

[samhatoum]

Hey @ArtemGovorov. I know how busy life gets and how product backlogs work. Just wondering if you have any rough outlook on when you might start attacking Cucumber.js support. And as always, here to help if you need it as I've done some major digging into Cucumber.js.

Thanks!

[ArtemGovorov]

Hey @samhatoum, can't share any plans about the feature support yet. The team is busy at the moment rushing towards releasing this big feature. After releasing that one and clearing up a few smaller long wanted/waiting ones, we'll revisit the backlog, and of course will post some update here if anything changes.

[samhatoum]

@ArtemGovorov that looks incredible!! I love it and can't wait to try it.

[eggyal]

Wallaby App is indeed very cool, but the benefit it actually provides for us (over monitoring test results in-editor) doesn't come anywhere close to what we'd get from having Cucumber support.

I appreciate that there are other priorities, but it nevertheless saddens me that (15 months after it was first requested) there's still no indication this feature is even scheduled to be worked upon any time soon. Perhaps I've just become too used to how rapidly open-source tools iterate, or else how easy it is to add custom functionality where there are external APIs for plugins/modules!

I'll keep praying for this feature to appear sooooooooon!

[eggyal]

Maybe https://github.com/cucumber/microcuke (an official, minimal—only 500 SLOC, JS reference implementation) could be useful?

[tedvanderveen]

@ArtemGovorov can you please share status on this enhancement request?

[ArtemGovorov]

@tedvanderveen The status hasn't really changed, but I'll post an update here when it does.

[jfstephe]

For me, test coverage is not a great measure of quality (not that people are necessarily saying it is) but perhaps at best an indication of the quality of code. All the arguments about this aside, if you are running cucumber tests and non-cuke tests if you can only report the test coverage on part of the tests you are running that makes this less than useful for me. Was quite excited about this tool until I saw this was not supported.

ETA on this feature? 1 week? 3 months? 6 months? 12 months? Never? Can you give us a ball-park idea?

[ArtemGovorov]

@jfstephe

ETA on this feature? 1 week? 3 months? 6 months? 12 months? Never? Can you give us a ball-park idea?

Unfortunately I can't share ETA at the moment. It is highly unlikely that we add the support within the next 3 months.

[samhatoum]

@ArtemGovorov @charlierudolph is the maintainer of CucumberJS. I'm hoping this intro could accelerate this ticket :)

[eggyal]

FWIW, I see that #1144 was linked in the context of Cucumber being for E2E testing. That's not really correct, at all. Cucumber is merely an engine for running Gherkin files, which are themselves a way of specifying tests in natural language—those tests could be unit tests, integration tests, E2E tests or indeed anything at all.

Certainly it's true that the overhead of writing tests in natural language is typically reserved for cases where they are reviewed by a non-technical audience, e.g. acceptance tests (which are usually only E2E)—but there is nothing that requires that to be the case.

It may be that this issue was lumped together with E2E testing frameworks for brevity, but I thought it worth setting the record straight lest a misunderstanding or incorrect assumption have caused it to be given lower priority than it might otherwise have received.

[samhatoum]

I'm hawking this issue as Wallaby support for CucumberJS would make me jump for joy.

So I second the comment from @eggyal. At Xolv.io we don't use Cucumber for e2e testing at all, we use it to do drive development of the system. If wallaby is to run tests within the browser/node context, this is the same use case for Cucumber as it is for mocha/jasmine/tape. From wallaby's perspective, Cucumber should be treated just as another test framework.

I also want to reiterate that Chimp has nothing to do with this implementation.

More interestingly, a new development in CucumberJS 3.0 is that it now has an event-protocol for reporters and other plugins.

See here: https://docs.cucumber.io/event-protocol/

and here: https://github.com/cucumber/cucumber-js/blob/master/src/formatter/event_protocol_formatter.js

@ArtemGovorov I think this might give you the realtime-ness you're looking for in a decoupled fashion

[FlippieCoetser]

@ArtemGovorov any new feedback regarding this issue? WallabyJS is great having it support CucumberJS would be even better.

[eggyal]

@FlippieCoetser: Don't bother, it's been over three years with absolutely zero progress. Look elsewhere.

[samhatoum]

@ArtemGovorov or anyone else, could you please let us know what's happening with this feature request?

I'd be glad to help if it would get us this feature faster.

Thanks

[samhatoum]

@smcenlly perhaps you could help here?

[FlippieCoetser]

Having cucumber support in Wallabyjs would be awesome! a Clear Win-Win

[smcenlly]

We don't have any immediate plans to support Cucumber. Unfortunately the structure of our code doesn't allow us to expose an API at this point in time. We may be in a position to provide an API for additional test runners to report their output in around 6 months time when we complete one of our current larger projects. Please check back in with us then.

[josersleal]

Maybe you could place in yor own implementation of gherkin via https://www.npmjs.com/package/jasmine-cucumber or https://www.npmjs.com/package/mocha-gherkin or one of the many other existing implemetations on the frameworks allready supported y wallaby?
Have in mind that the important bit is Gherkin, cucumber is just one way to implement it...
At a personal oppinon pov, I believe that Gherkin is one of the most consensual items in QA, not supporting is is a major flaw in any QA offering.

[smcenlly]

While not the same as first-class support, you can still execute cucumber tests in wallaby using something like jest-cucumber. We put together a sample project that uses jest-cucumber. While reporting structure may be lacking, you can use wallaby for real-time feedback while you're coding.

[samhatoum]

So wallaby does indeed have a jest-cucumber support, but jest-cucumber is seriously lagging behind cucumber, not having newer gherkin keywords like Rules and not supporting Cucumber expressions, and not having World support. Basically it's a very cut down version of cucumber with a few twists.

I've used it in a DDD/BDD project and recently ripped it out and replaced it with cucumber as that's what was right for our team. It's probably good for lighter use-cases than our own.

That aside though, I would really love to see support for the official Cucumber from the Wallaby team. 7 years ago (wow!) I commented on where I saw the incision points would be in the framework to make this happen to enable parallaism, and the cucumber team is very collaborative and I'm sure would be willing to help out here.

Is there anything we can do from a community perspective to help push this along?

[Effect-Healthcare-Dev]

We also follow a BDD approach and do use Cucumber a lot. It would be awesome if Cucumber can work with Wallaby. Both are awesome; together, it will be a game-changer. Big please for cucumber support! I am also happy to support you in making it happen.

[ArtemGovorov]

Hey everyone,

There is significant investment required on our side to integrate and maintain Cucumber support, so we’ve been waiting to get more interest in Cucumber as a technology from our users. At this point in time, we don’t have enough interest to warrant adding the support and justify the investment, but please +1 and leave your comments if you're interested in the support. If you are a part of a large team of software developers, and are using Cucumber, and interested in purchasing a large number of Wallaby licenses, but the lack of Cucumber support is stopping you from doing so, please email us at [email protected].

Is there anything we can do from a community perspective to help push this along?
jest-cucumber is seriously lagging behind cucumber, not having newer gherkin keywords like Rules and not supporting Cucumber expressions, and not having World support. Basically it's a very cut down version of cucumber with a few twists.

While we don't have plans to add Cucumber support to Wallaby at the moment, we do have first-class Jest support and are definitely committed to support Jest as a runner for as long as possible. So in terms of the community help, I think the best way would be to contribute to jest-cucumber to keep it up-to-date with Cucumber. Not only will it make Wallaby work with it, it will also help the rest of the community to run Cucumber via Jest CLI too.

[jrutila]

I am running a combination of vitest and quickpickle succesfully with Wallaby.

[samhatoum]

very interesting @jrutila , do you have a reference repo by chance?

side note, would also love to see if this can work with Deno (issue here: denoland/deno#23882)

[jrutila]

very interesting @jrutila , do you have a reference repo by chance?

Sorry, it is my private repo, but just following the configuration instructions at quickpickle works as written. I don't recall doing any tricks, it just worked.