Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

♻️ refactor(gentest): Migrate transaction test to state_test #903

Merged
merged 3 commits into from
Nov 7, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions docs/writing_tests/types_of_tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,28 @@ There are currently two types of tests that can be produced by a test spec:

## State Tests

State tests span a single block and, ideally, a single transaction. For example:
### Purpose

Tests the effects of individual transactions (ideally a single one) that span a single block in a controlled environment.

### Use cases

- Test a single opcode behavior.
- Verify opcode gas costs.
- Test interactions between multiple smart contracts.
- Test creation of smart contracts.

!!! info

The fill function will automatically generate a `blockchain_test` fixture from `state_tests`, consisting of one block and one transaction.

## Blockchain Tests

Blockchain tests span multiple blocks which may or may not contain transactions and mainly focus on the block to block effects to the Ethereum state. For example:
### Purpose

Blockchain tests span multiple blocks which may or may not contain transactions and mainly focus on the block to block effects to the Ethereum state.

### Use cases

- Verify system-level operations such as coinbase balance updates or withdrawals.
- Verify fork transitions.
Expand All @@ -38,3 +50,11 @@ def test_blob_type_tx_pre_fork(
Reject blocks with blobs before blobs fork
"""
```

## Deciding on a test type

### Prefer `state_test` for single transactions

Whenever possible, use `state_test` to examine individual transactions. This method is more straightforward and less prone to external influences that can occur during block building.

This provides more targeted testing since it does not invoke the client's block-building machinery. This reduces the risk of encountering false positives, particularly in exception scenarios (e.g., see issue [#343: "Zero max_fee_per_blob_gas test is ineffective"](https://github.com/ethereum/execution-spec-tests/issues/343)).
2 changes: 1 addition & 1 deletion docs/writing_tests/writing_a_new_test.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Test Functions

Every test case is defined as a python function that defines a single `StateTest` or `BlockchainTest` by using one of the `state_test` or `blockchain_test` objects made available by the framework. Test cases, respectively test modules, must fulfill the following requirements:
Every test case is defined as a Python function that implements a single `StateTest` or `BlockchainTest` using the `state_test` or `blockchain_test` objects made available by the framework ([learn how to decide on a test type](./types_of_tests.md#deciding-on-a-test-type)). Test cases, and the respective test modules, must fulfill the following requirements:

| Requirement | When |
| -----------------------------------------------------------------------|---------------------------------------------|
Expand Down
4 changes: 2 additions & 2 deletions src/cli/gentest/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
from ethereum_test_base_types import Hash

from .source_code_generator import get_test_source
from .test_context_providers import BlockchainTestContextProvider
from .test_context_providers import StateTestProvider


@click.command()
Expand All @@ -27,7 +27,7 @@ def generate(transaction_hash: str, output_file: TextIO):
OUTPUT_FILE is the path to the output python script.
"""
provider = BlockchainTestContextProvider(transaction_hash=Hash(transaction_hash))
provider = StateTestProvider(transaction_hash=Hash(transaction_hash))

source = get_test_source(provider=provider, template_path="blockchain_test/transaction.py.j2")
output_file.write(source)
Expand Down
13 changes: 3 additions & 10 deletions src/cli/gentest/templates/blockchain_test/transaction.py.j2
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,7 @@ from typing import Dict

import pytest

from ethereum_test_tools import (
Account,
Block,
BlockchainTestFiller,
Environment,
Storage,
Transaction,
)
from ethereum_test_tools import Account, Environment, StateTestFiller, Storage, Transaction

REFERENCE_SPEC_GIT_PATH = "N/A"
REFERENCE_SPEC_VERSION = "N/A"
Expand All @@ -28,7 +21,7 @@ def env(): # noqa: D103
@pytest.mark.valid_from("Paris")
def test_transaction_{{ tx_hash }}( # noqa: SC200, E501
env: Environment,
blockchain_test: BlockchainTestFiller,
state_test: StateTestFiller,
):
"""
Gentest autogenerated test for tx.hash:
Expand All @@ -40,4 +33,4 @@ def test_transaction_{{ tx_hash }}( # noqa: SC200, E501

tx = {{ transaction | stringify }}

blockchain_test(genesis_environment=env, pre=pre, post=post, blocks=[Block(txs=[tx])])
state_test(env=env, pre=pre, post=post, tx=tx)
4 changes: 2 additions & 2 deletions src/cli/gentest/test_context_providers.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,9 @@ def get_context(self) -> Dict:
pass


class BlockchainTestContextProvider(Provider):
class StateTestProvider(Provider):
"""
Provides context required to generate a `blockchain_test` using pytest.
Provides context required to generate a `state_test` using pytest.
"""

transaction_hash: Hash
Expand Down