Yet Another Rigorous, Robust Refactor (YARRR)

README.md

@@ -1,20 +1,23 @@
 
-# Agential
+<h3 align="center">
+  <img
+    src="https://raw.githubusercontent.com/agential-ai/.github/main/profile/banner_dark.svg#gh-dark-mode-only"
+  />
+  <img
+    src="https://raw.githubusercontent.com/agential-ai/.github/main/profile/banner_light.svg#gh-light-mode-only"
+  />
+</h3>
 
-[![codecov](https://codecov.io/gh/agential-ai/agential/branch/main/graph/badge.svg)](https://codecov.io/gh/agential-ai/agential)
-
-
-## Features
 
+<h1 align="center">Language agent research made easy.<h5 align="center"><a href="https://www.youtube.com/watch?v=5syJjBQ_k6o">You're definitely not you when you're hungry for research.</a></h5></h1>
 
- Our primary goal is to provide easy-to-use and clean implementations of popular LLM-based agent methods: an encyclopedia! This library is one of our contributions for our research project empirically surveying and investigating the performance of these methods across a diverse set of reasoning/decision-making tasks. Learn more about this [here](https://equatorial-jobaria-9ad.notion.site/Project-Lifecycle-Management-70d65e9a76eb4c86b6aed007f717aa41?pvs=4)! 
 
-- Easy-to-Use Interface: Provides intuitive and user-friendly functions for rapid prototyping and development.
+<h3 align="center">
 
-- Clean Functions: Offers clean and well-structured functions, promoting readability and maintainability of code.
-
-- Modularized Implementations: Includes modularized implementations of popular LLM-based agents and agent-related methods, allowing users to leverage cutting-edge innovations from the literature.
+[![codecov](https://codecov.io/gh/agential-ai/agential/branch/main/graph/badge.svg)](https://codecov.io/gh/agential-ai/agential)
+</h3>
 
+**Agential provides clear implementations of popular LLM-based agents across a variety of reasoning/decision-making and language agent benchmarks, making it easy for researchers to evaluate and compare different agents.**
 
 ## Getting Started 
 
@@ -27,10 +30,13 @@ pip install agential
 Next, let's query the `ReActAgent`!
 
 ```python
+from agential.llm.llm import LLM
+from agential.cog.react.agent import ReActAgent
+
 question = 'Who was once considered the best kick boxer in the world, however he has been involved in a number of controversies relating to his "unsportsmanlike conducts" in the sport and crimes of violence outside of the ring?'
 
-llm = ChatOpenAI(openai_api_key="YOUR_API_KEY")
-agent = ReActAgent(llm=llm)
+llm = LLM("gpt-3.5-turbo")
+agent = ReActAgent(llm=llm, benchmark="hotpotqa")
 out = agent.generate(question=question)
 ```
 
@@ -39,45 +45,48 @@ out = agent.generate(question=question)
 
 ------------
 
-    ├── agential                       <- Source code for this project.
+    ├── agential                           <- Source code for this project.
     │   ├── cog   
-    │   │   ├── agent                  <- Model/agent-related modules.
-    │   │   │   
-    │   │   ├── eval                   <- Agent core modules.
-    │   │   │   
-    │   │   ├── functional                  
-    │   │   │
-    │   │   ├── modules           
-    │   │   │   ├── memory             <- Memory-related modules.
-    │   │   │   ├── plan               <- Planning-related modules.
-    │   │   │   ├── reflect            <- Reflecting-related modules.
-    │   │   │   └── score              <- Scoring-related modules.
-    │   │   │
-    │   │   ├── persona             
-    │   │   │
-    │   │   └── prompts             
+    │   │   ├── agent                      <- Model/agent-related modules.
+    │   │   │     ├── strategies           <- Strategies encapsulate agent logic for each benchmark/benchmark type.
+    │   │   │     │       ├── base.py  
+    │   │   │     │       ├── qa.py
+    │   │   │     │       ├── math.py
+    │   │   │     │       └── code.py
+    │   │   │     │
+    │   │   │     ├── agent.py             <- Agent class responsible for selecting the correct strategy, prompts/few-shots, and generating responses.
+    │   │   │     ├── functional.py        <- Functional methods for agent. The lowest level of abstraction.
+    │   │   │     ├── output.py            <- Output class responsible for formatting the response from the agents.
+    │   │   │     ├── prompts.py           <- Prompt templates.
+    │   │   │     └── <modules>.py         <- Any additional modules you may have for the strategies. Agnostic to benchmarks/benchmark-types.
     │   │
-    │   └── utils                      <- Utility methods.
+    │   ├── eval                           <- Evaluation-related modules.
+    │   │
+    │   ├── llm                            <- LLM class.
+    │   │
+    │   └── utils                          <- Utility methods.
     │       
-    ├── docs                           <- An mkdocs project.
+    ├── docs                               <- An mkdocs project.
     │
-    ├── models                         <- Trained and serialized models, model predictions,
-    │                                          or model summaries.
-    │       
-    ├── notebooks                      <- Jupyter notebooks. Naming convention is a number 
-    │                                    (for ordering), the creator's initials, and a short `-` delimited │ description, e.g. `1.0-jqp-initial-data-exploration`.
+    ├── notebooks                          <- Jupyter notebooks. Naming convention is a number 
+    │                                         (for ordering), the creator's initials, and a short `-` delimited │ description, e.g. `1.0-jqp-initial-data-exploration`.
     │  
+    ├── references                         <- Data dictionaries, manuals, and all other explanatory materials.
     │
-    ├── references                     <- Data dictionaries, manuals, and all other explanatory materials.
+    ├── reports                            <- Generated analysis as HTML, PDF, LaTeX, etc.
+    │   └── figures                        <- Generated graphics and figures to be used in reporting.
     │
-    ├── reports                        <- Generated analysis as HTML, PDF, LaTeX, etc.
-    │   └── figures                    <- Generated graphics and figures to be used in reporting.
-    │
-    └── tests                          <- Tests.
+    └── tests                              <- Tests.
 
 ---------
 
+## 🙏 Acknowledgement
 
-## Contributing
+## 😀 Contributing
 
 If you want to contribute, please check the [contributing.md](https://github.com/alckasoc/agential/blob/main/CONTRIBUTING.md) for guidelines!
+Please check out the [project document timeline](https://equatorial-jobaria-9ad.notion.site/Project-Lifecycle-Management-70d65e9a76eb4c86b6aed007f717aa41?pvs=4) on Notion and reach out to us if you have any questions!
+
+## 😶‍🌫️ Contact Us!
+
+If you have any questions or suggestions, please feel free to reach out to [email protected]!

agential/cog/base/agent.py

@@ -1,18 +1,87 @@
 """Base agent interface class."""
 
 from abc import ABC, abstractmethod
-from typing import Any
+from typing import Any, Dict
+
+from agential.cog.base.output import BaseOutput
+from agential.cog.base.strategies import BaseStrategy
+from agential.llm.llm import BaseLLM
 
 
 class BaseAgent(ABC):
-    """Base agent class providing a general interface for agent operations."""
+    """Base agent class providing a general interface for agent operations.
+
+    Parameters:
+        llm (BaseLLM): An instance of a language model used for generating initial answers
+            and critiques.
+        benchmark (str): The benchmark.
+        testing (bool, optional): Whether to run in testing mode. Defaults to False.
+    """
+
+    def __init__(
+        self,
+        llm: BaseLLM,
+        benchmark: str,
+        testing: bool = False,
+    ) -> None:
+        """Initialization."""
+        super().__init__()
+        self.llm = llm
+        self.benchmark = benchmark
+        self.testing = testing
 
     @abstractmethod
-    def reset(self, *args: Any, **kwargs: Any) -> Any:
-        """Resets the agent's state."""
-        raise NotImplementedError("Reset method not implemented.")
+    def get_fewshots(
+        self, benchmark: str, fewshot_type: str, **kwargs: Any
+    ) -> Dict[str, str]:
+        """Retrieve few-shot examples based on the benchmark.
+
+        Args:
+            benchmark (str): The benchmark name.
+            fewshot_type (str): The benchmark few-shot type.
+            **kwargs (Any): Additional arguments.
+
+        Returns:
+            Dict[str, str]: A dictionary of few-shot examples.
+        """
+        raise NotImplementedError
 
     @abstractmethod
-    def generate(self, *args: Any, **kwargs: Any) -> Any:
-        """Generate a response."""
-        raise NotImplementedError("Generate method not implemented.")
+    def get_prompts(self, benchmark: str, **kwargs: Any) -> Dict[str, str]:
+        """Retrieve the prompt instructions based on the benchmark.
+
+        Args:
+            benchmark (str): The benchmark name.
+            **kwargs (Any): Additional arguments.
+
+        Returns:
+            Dict[str, str]: A dictionary of prompt instructions.
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def get_strategy(self, benchmark: str, **kwargs: Any) -> BaseStrategy:
+        """Returns an instance of the appropriate strategy based on the provided benchmark.
+
+        Args:
+            benchmark (str): The benchmark name.
+            **kwargs (Dict[str, Any]): Additional keyword arguments to pass to
+                the strategy's constructor.
+
+        Returns:
+            BaseStrategy: An instance of the appropriate strategy.
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def generate(self, *args: Any, **kwargs: Any) -> BaseOutput:
+        """Generate a response.
+
+        Args:
+            *args (Any): Additional arguments.
+            **kwargs (Any): Additional keyword arguments.
+
+        Returns:
+                BaseOutput: The generated response.
+        """
+        raise NotImplementedError

agential/cog/base/output.py

@@ -0,0 +1,44 @@
+"""Base output class."""
+
+from typing import Any
+
+from pydantic import BaseModel, Field
+
+
+class BaseOutput(BaseModel):
+    """Base class for structured agent outputs.
+
+    Attributes:
+        answer (Any): The answer generated by the agent.
+        total_prompt_tokens (int): The total number of input tokens used.
+        total_completion_tokens (int): The total number of output tokens used.
+        total_tokens (int): The total number of tokens used.
+        total_cost (float): The total cost of the output.
+        total_prompt_cost (float): The total cost of the prompt tokens.
+        total_completion_cost (float): The total cost of the completion tokens.
+        total_prompt_time (float): The total time taken for the LLM API to generate the outputs in seconds.
+        total_time (float): The total time for the agent to finish generating in seconds.
+        additional_info (Any): A general attribute for additional information.
+    """
+
+    answer: Any = Field(..., description="The answer generated by the agent.")
+    total_prompt_tokens: int = Field(..., description="Total input tokens used.")
+    total_completion_tokens: int = Field(..., description="Total output tokens used.")
+    total_tokens: int = Field(..., description="Total tokens used.")
+    total_prompt_cost: float = Field(
+        ..., description="Total cost of the prompt tokens."
+    )
+    total_completion_cost: float = Field(
+        ..., description="Total cost of the completion tokens."
+    )
+    total_cost: float = Field(..., description="Total cost of the output.")
+    total_prompt_time: float = Field(
+        ...,
+        description="Total time taken for the LLM API to generate the outputs in seconds.",
+    )
+    total_time: float = Field(
+        ..., description="Total time for the agent to finish generating in seconds."
+    )
+    additional_info: Any = Field(
+        ..., description="Additional information related to the output."
+    )

agential/cog/base/strategies.py

@@ -1,33 +1,51 @@
 """Generic base strategy class."""
 
 from abc import ABC, abstractmethod
-from typing import Any, Dict
+from typing import Any
 
 from agential.llm.llm import BaseLLM
 
 
 class BaseStrategy(ABC):
-    """An abstract base class for defining strategies for generating responses with LLM-based agents."""
+    """An abstract base class for defining strategies for generating responses with LLM-based agents.
 
-    def __init__(self, llm: BaseLLM) -> None:
+    Parameters:
+        llm (BaseLLM): An instance of a language model used for generating responses.
+        testing (bool): Whether the generation is for testing purposes. Defaults to False.
+    """
+
+    def __init__(
+        self,
+        llm: BaseLLM,
+        testing: bool = False,
+    ) -> None:
         """Initialization."""
         self.llm = llm
+        self.testing = testing
 
     @abstractmethod
     def generate(
         self,
         *args: Any,
         **kwargs: Any,
     ) -> Any:
-        """Generates a response."""
-        pass
+        """Generates a response.
+
+        Args:
+            *args: Variable length argument list.
+            **kwargs: Arbitrary keyword arguments.
+
+        Returns:
+            Any: The generated response.
+        """
+        raise NotImplementedError
 
     @abstractmethod
     def reset(self, *args: Any, **kwargs: Any) -> None:
-        """Resets the strategy's internal state, if any."""
-        pass
+        """Resets the strategy's internal state, if any.
 
-    @abstractmethod
-    def create_output_dict(self, *args: Any, **kwargs: Any) -> Dict[str, Any]:
-        """Creates a dictionary containing the generated response."""
-        pass
+        Args:
+            *args: Variable length argument list.
+            **kwargs: Arbitrary keyword arguments.
+        """
+        raise NotImplementedError