Loading project.yaml and Fixtures

[markhuot]

It is currently impossible to utilize the project.yaml and fixtures in a test scenario. I'm not sure if this is something you'd like to support because it's non-trivial to add, but I figured I'd open a PR with some suggestions for your consideration.

First, the motivation for this is to implement TDD (or any testing) on a front-end only build of a website within Craft. E.g., imagine a developer working entirely in the template layer and the Craft admin. They may create several sections, add fields to those sections, configure some globals, etc. Additionally they may have some "boilerplate" entries that define things like a representative blog post or the scaffolding of the homepage.

Currently, the above developer would have to save everything out as a project.yaml file so they could sync configuration between production and staging. But for testing they would then have to redefine the section and entry type configuration in a second place, in fixtures, because they load too late in the test setup.

The attached fix moves the project.yaml setup out of the _before flow and moves it deeper in to the loadFixutres flow directly. This ensures it loads before the fixtures but not before the app is initialized.

The major downside to this approach is that the current loadFixtures method is private as defined in the Yii2 Codeception module. That means the attached won't actually work. If this seems feasible, though, I'm happy to open a PR over there for consideration of the upstream update.

If this doesn't make any sense and there's a better way to do this I'm all ears, but I couldn't find anything in my reading of the docs or my perusal of the source.

[codecov]

Codecov Report

Merging #4711 into develop will decrease coverage by 5.23%.
The diff coverage is 0%.

@@              Coverage Diff              @@
##             develop    #4711      +/-   ##
=============================================
- Coverage      31.63%   26.39%   -5.24%     
- Complexity     12886    12887       +1     
=============================================
  Files            531      531              
  Lines          38612    38880     +268     
=============================================
- Hits           12214    10264    -1950     
- Misses         26398    28616    +2218

Impacted Files	Coverage Δ	Complexity Δ
src/test/Craft.php	32.14% <0%> (-1.37%)	68 <3> (+1)
src/records/AssetTransform.php	0% <0%> (-100%)	1% <0%> (ø)
src/utilities/AssetIndexes.php	18.18% <0%> (-81.82%)	5% <0%> (ø)
src/models/CategoryGroup.php	0% <0%> (-80.65%)	13% <0%> (ø)
src/models/UserGroup.php	0% <0%> (-76.48%)	6% <0%> (ø)
src/fields/Users.php	0% <0%> (-75%)	4% <0%> (ø)
src/fields/Entries.php	0% <0%> (-75%)	4% <0%> (ø)
src/utilities/SystemMessages.php	30.76% <0%> (-69.24%)	4% <0%> (ø)
src/records/Structure.php	0% <0%> (-66.67%)	3% <0%> (ø)
src/models/TagGroup.php	0% <0%> (-66.67%)	5% <0%> (ø)
... and 94 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3e7abe2...3a51222. Read the comment docs.

[aaronbushnell]

Hey @markhuot! You should be able to utilize both fixtures and project.yaml if I'm not mistaken? Here's how I have them setup in one of my tests:

tests/fixtures/data/entries.php

<?php

return [
    [
        "sectionId" => $this->sectionIds['example'],
        "typeId" => $this->typeIds['example']['example'],
        "title" => "My Entry 1"
    ],

    [
        "sectionId" => $this->sectionIds['example'],
        "typeId" => $this->typeIds['example']['example'],
        "title" => "My Entry 2"
    ],

    [
        "sectionId" => $this->sectionIds['example'],
        "typeId" => $this->typeIds['example']['example'],
        "title" => "My Entry 3"
    ]
];

tests/unit/ExampleTest.php

<?php

namespace myprojecttests;

use Codeception\Test\Unit;
use UnitTester;

use Craft;
use craft\elements\Entry;

use myprojecttests\fixtures\EntriesFixture;

class ExampleTest extends Unit
{
    // Public properties
    // =========================================================================

    /**
     * @var UnitTester
     */
    protected $tester;

    // Public methods
    // =========================================================================
    public function _fixtures(): array
    {
        return [
            'entries' => [
                'class' => EntriesFixture::class
            ]
        ];
    }

    // Tests
    // =========================================================================
    public function testTotalEntryFixtures()
    {
        // Confirm we have three entries
        $this->assertEquals(3, Entry::find()->section(['example'])->total());
    }
}

You'll notice I'm using

"sectionId" => $this->sectionIds['example'],
"typeId" => $this->typeIds['example']['example'],

Where example is the handle of a "Section" and the handle of the "Entry Type".

[markhuot]

@aaronbushnell, I'm not seeing the same thing. If you empty out your database and re-run the unit tests do you get a full install/setup for your tests? I can force it to work by updating codeception.yml to set dbSetup: {clean: false, setupCraft: false} and then the the tests don't affect my DB. But with those both set to true the fixtures and project.yaml seem to fight.

What's your codeception.yml look like?

[aaronbushnell]

If you empty out your database and re-run the unit tests do you get a full install/setup for your tests?

So I have my tests setup to purge my database and seed a new one (then it uses project.yaml to configure the schema my live site has)

What's your codeception.yml look like?

Here ya go!

actor: Tester
paths:
  tests: tests
  log: tests/_output
  data: tests/_data
  support: tests/_support
  envs: tests/_envs
settings:
  bootstrap: _bootstrap.php
params:
  - tests/.env
modules:
  config:
    \craft\test\Craft:
      configFile: 'tests/_craft/config/test.php'
      entryUrl: 'http://mysite.test/index.php'
      projectConfig: {file: 'config/project.yaml'}
      migrations: []
      plugins: []
      cleanup: true
      transaction: true
      dbSetup: {clean: true, setupCraft: true}

[markhuot]

Hrm, I wonder why my experience is so different… I’ll try it on a fresh clone later and see what happens. Maybe my version is out of date.

[gtettelaar]

@markhuot Just to check one thing. Do you have the useProjectConfigFile option enabled in tests/_craft/config/general.php

[angrybrad]

@markhuot any luck using a fresh clone? I'm not seeing the behavior you're describing, either.

[elivz]

In my case, adding a tests/_craft/config/general.php file and setting useProjectConfigFile fixed it. What was confusing is that without that, the project config actually was being loaded anyway, just after fixtures were run.

[gtettelaar]

In my case, adding a tests/_craft/config/general.php file and setting useProjectConfigFile fixed it. What was confusing is that without that, the project config actually was being loaded anyway, just after fixtures were run.

I'm assuming you mean the Project Config data (i.e. Sections, Sites, email settings e.t.c.) and not just that project.yml was being copied into tests/_craft/config?
@elivz

[elivz]

@gtettelaar Correct

[gtettelaar]

tl;dr
#4804 will resolve what was causing confusion when mixing Project Config with Testing.

@elivz Yea this was weird behaviour. Here's a basic explanation:

If you had useProjectConfigFile enabled when running tests Craft would execute the following lines of code when setting up the testing DB:

cms/src/migrations/Install.php

Lines 1019 to 1023 in 82e14b2

	if ($applyExistingProjectConfig) {
	// Save the existing system settings
	echo ' > applying existing project config ... ';
	$projectConfig->applyYamlChanges();
	echo "done\n";

This executed before any fixtures were applied. If you had useProjectConfigFile disabled Craft would not run this code so the data from project.yml would not be applied.

After this Codeception/Yii load your fixtures. In your case, this was happening without Project Config data. With for example @aaronbushnell the Project Config was loaded. All because of what useProjectConfigFile was set to.

Once that was done Codeception calls craft\test\Craft::_before(). Here because you specified a project.yml file in codeception.yml we call Craft::$app->getProjectConfig()->applyConfigChanges() with the Project Config data:

cms/src/test/Craft.php

Lines 170 to 178 in 82e14b2

	if ($projectConfig = $this->_getConfig('projectConfig')) {
	// Tests just beginning. . Reset the project config to its original state.
	TestSetup::setupProjectConfig($projectConfig['file']);
	\Craft::$app->getProjectConfig()->applyConfigChanges(
	Yaml::parse(file_get_contents($projectConfig['file']))
	);
	\Craft::$app->getProjectConfig()->saveModifiedConfigData();

Now regardless of if useProjectConfigFile was enabled if you supplied a project.yml file we would parse that and apply it's config to the Testing environment. The end result was the fact that your project config data would be present in your tests but not in your fixtures (because _before() is called after your fixtures are applied).

[elivz]

Yes, that describes what I have been seeing perfectly. I just wasn't able to track down the root cause. Everything is working great since I added the tests/_craft/config/general.php file, and this additional warning should make things much clearer in the future.

Thank you for all your hard work on this testing framework! It's really going to improve the whole plugin ecosystem in the long run.

[angrybrad]

Assuming #4804 resolves this, so going to go ahead and close, but feel free to re-open if you're still having issues, Mark.