✨ feat(fw,fill,cli): Default evm logs (#999)

* ✨ feat(fw,fill,cli): Default evm logs

* fix(fixtures): EVM dump logs with no common path

* feat(plugins/filler): Add flag to skip evm dump

---------

Co-authored-by: rahul <[email protected]>
Co-authored-by: Mario Vega <[email protected]>

.gitignore

@@ -68,4 +68,7 @@ assets
 *.html
 
 # Environment
-env.yaml
+env.yaml
+
+# Framework logs
+logs/

docs/CHANGELOG.md

@@ -61,6 +61,7 @@ Test fixtures for use by clients are available for each release on the [Github r
 - ✨ Add framework changes for EIP-7742, required for Prague devnet-5 ([#931](https://github.com/ethereum/execution-spec-tests/pull/931)).
 - ✨ Add the `eest make env` command that generates a default env file (`env.yaml`)([#996](https://github.com/ethereum/execution-spec-tests/pull/996)).
 - ✨ Generate Transaction Test type ([#933](https://github.com/ethereum/execution-spec-tests/pull/933)).
+- ✨ Add a default location for evm logs (`--evm-dump-dir`) when filling tests ([#999](https://github.com/ethereum/execution-spec-tests/pull/999)).
 
 ### 🔧 EVM Tools
 

docs/filling_tests/debugging_t8n_tools.md

@@ -2,13 +2,13 @@
 
 There are two flags that can help debugging `t8n` tools or the execution-spec-tests framework:
 
-1. `--evm-dump-dir`: Write debug information from `t8n` tool calls to the specified directory.
+1. `--evm-dump-dir` (Default: <repo>/logs/evm): Write debug information from `t8n` tool calls to the specified directory.
 2. `--traces`: Collect traces of the execution from the transition tool.
 3. `--verify-fixtures`: Run go-ethereum's `evm blocktest` command to verify the generated test fixtures.
 
 ## EVM Dump Directory
 
-The `--evm-dump-dir` flag tells the framework to write the inputs and outputs of every call made to the `t8n` command to the specified output directory. The aim is to help debugging or simply understand how a test is interacting with the EVM.
+The `--evm-dump-dir` flag tells the framework to write the inputs and outputs of every call made to the `t8n` command to the specified output directory. The aim is to help debugging or simply understand how a test is interacting with the EVM. The default location is `logs/evm` in the project root.
 
 Each test case receives its own sub-directory under the `--evm-dump-dir` that contains these files which can be easily accessed from the HTML test report generated by `fill` (located by default in the root of the `--output` directory).
 

docs/filling_tests/filling_tests_command_line.md

@@ -184,7 +184,7 @@ Arguments defining filler location and output:
 
 Arguments defining debug behavior:
   --evm-dump-dir EVM_DUMP_DIR, --t8n-dump-dir EVM_DUMP_DIR
-                        Path to dump the transition tool debug output.
+                        Path to dump the transition tool debug output.  (Default: <repo>/logs/evm)
 
 Specify the fork range to generate fixtures for:
   --forks               Display forks supported by the test framework and exit.

src/config/app.py

@@ -5,6 +5,8 @@
 - AppConfig: Holds configurations for the application framework.
 """
 
+from pathlib import Path
+
 from pydantic import BaseModel
 
 
@@ -15,3 +17,9 @@ class AppConfig(BaseModel):
 
     version: str = "3.0.0"
     """The version of the application framework."""
+
+    DEFAULT_LOGS_DIR: Path = Path(__file__).resolve().parent.parent.parent / "logs"
+    """The default directory where log files are stored."""
+
+    DEFAULT_EVM_LOGS_DIR: Path = DEFAULT_LOGS_DIR / "evm"
+    """The default directory where EVM log files are stored."""

src/ethereum_test_fixtures/collector.py

@@ -38,7 +38,9 @@ def get_module_relative_output_dir(test_module: Path, filler_path: Path) -> Path
     tests/shanghai/eip3855_push0/test_push0.py -> shanghai/eip3855_push0/test_push0
     """
     basename = test_module.with_suffix("").absolute()
-    basename_relative = basename.relative_to(filler_path.absolute())
+    basename_relative = basename.relative_to(
+        os.path.commonpath([filler_path.absolute(), basename])
+    )
     module_path = basename_relative.parent / basename_relative.stem
     return module_path
 

src/pytest_plugins/filler/filler.py

@@ -20,6 +20,7 @@
 from pytest_metadata.plugin import metadata_key  # type: ignore
 
 from cli.gen_index import generate_fixtures_index
+from config import AppConfig
 from ethereum_clis import TransitionTool
 from ethereum_test_base_types import Alloc, ReferenceSpec
 from ethereum_test_fixtures import BaseFixture, FixtureCollector, TestInfo
@@ -182,8 +183,19 @@ def pytest_addoption(parser: pytest.Parser):
         "--t8n-dump-dir",
         action="store",
         dest="base_dump_dir",
-        default="",
-        help="Path to dump the transition tool debug output.",
+        default=AppConfig().DEFAULT_EVM_LOGS_DIR,
+        help=(
+            "Path to dump the transition tool debug output. "
+            f"(Default: {AppConfig().DEFAULT_EVM_LOGS_DIR})"
+        ),
+    )
+    debug_group.addoption(
+        "--skip-evm-dump",
+        "--skip-t8n-dump",
+        action="store_true",
+        dest="skip_dump_dir",
+        default=False,
+        help=("Skip dumping the the transition tool debug output."),
     )
 
 
@@ -458,6 +470,8 @@ def base_dump_dir(request: pytest.FixtureRequest) -> Path | None:
     """
     The base directory to dump the evm debug output.
     """
+    if request.config.getoption("skip_dump_dir"):
+        return None
     base_dump_dir_str = request.config.getoption("base_dump_dir")
     if base_dump_dir_str:
         return Path(base_dump_dir_str)

whitelist.txt

@@ -78,6 +78,7 @@ codecopy
 codesize
 coinbase
 coincurve
+commonpath
 compilable
 config
 conftest