Introducing TrickOps - An Extensible Sim Testing Framework (#1130)

* Introducing TrickOps - An Extensible Sim Testing Framework Features: * Multiple simultaneous sim builds, runs, file vs. file comparisons, arbitrary post-run analyses, valgrind of runs * Real-time progress bars for sim builds and runs * Exit code management lets users easily define success & failure * Failed comparisons can optionally generate koviz error reports See share/trick/trickops/README.md for details * Add GitHub Actions Workflow for TrickOps for Ubuntu:20.04 & CentOS latest * Adds python unit and doc tests to github actions for push / pull requests for Ubuntu:20.04 and CentOS 8:latest. MacOS still forthcoming. * Also updates documentation with TrickOps information * Remove duplicate / overwriting SIM_ definitions in ExampleWorkflow.py * Address Code Review / Discussion * Reduce set of sims tested in ExampleWorkflow.py to stable set * Add ExampleWorkflow.py to GitHub Actions * Clarify documentation and add image of TrickOps in action * Error/Ignore valgrind entries in YAML file if platform == darwin * Fix run.compare() logic error and add unit test to cover it Co-authored-by: Dan Jordan <[email protected]>
nasa · May 3, 2021 · 1d9ea79 · 1d9ea79
1 parent b4d7655
commit 1d9ea79
Show file tree

Hide file tree

Showing 22 changed files with 3,563 additions and 0 deletions.
diff --git a/.github/workflows/trickops.yml b/.github/workflows/trickops.yml
@@ -0,0 +1,79 @@
+name: TrickOps
+# This workflow is triggered on pushes to the repository.
+on: [push, pull_request]
+
+defaults:
+  run:
+    shell: bash
+
+jobs:
+  trickops-tests-ubuntu:
+    name: Unit Tests Ubuntu:20.04
+    runs-on: ubuntu-20.04
+    container: ubuntu:20.04
+    steps:
+      - uses: actions/checkout@master
+      - name: install dependencies
+        # Note that perl is for trick-gte which TrickOps runs and qt and everything after it is for koviz
+        run: |
+          export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get install -y git python3 python3-venv perl perl-modules-5.30 qtbase5-dev wget unzip g++ make flex bison
+      - name: create virtual environment
+        run: |
+          cd share/trick/trickops/
+          python3 -m venv .venv && source .venv/bin/activate && pip3 install -r requirements.txt
+      - name: get and build koviz
+        run: |
+          cd /tmp/ && wget -q https://github.com/nasa/koviz/archive/refs/heads/master.zip && unzip master.zip
+          cd /tmp/koviz-master/ &&  qmake && make
+      - name: run unit and doc tests
+        run: |
+          cd share/trick/trickops/tests/
+          source ../.venv/bin/activate
+          export PATH="/tmp/koviz-master/bin:${PATH}"
+          ./run_tests.py
+      - uses: actions/upload-artifact@master
+        if: ${{ always() }}
+        with:
+          name: doctests
+          path: |
+            share/trick/trickops/tests/*_doctest_log.txt
+            /tmp/log.*
+
+  trickops-tests-centos8:
+    name: Unit Tests CentOS:latest
+    runs-on: ubuntu-20.04
+    container: centos:latest
+    steps:
+      - uses: actions/checkout@master
+      - name: install dependencies
+        # Note that perl is for trick-gte which TrickOps runs and qt and everything after it is for koviz
+        run: |
+          dnf install -y git python3-devel which perl perl-Digest-MD5 qt5-qtbase-devel bison clang flex make gcc gcc-c++ wget
+      - name: create virtual environment
+        run: |
+          cd share/trick/trickops/
+          python3 -m venv .venv && source .venv/bin/activate && pip3 install -r requirements.txt
+      - name: get and build koviz
+        run: |
+          cd /tmp/ && wget -q https://github.com/nasa/koviz/archive/refs/heads/master.zip && unzip master.zip
+          cd /tmp/koviz-master/ &&  qmake-qt5 && make
+      - name: run unit and doc tests
+        run: |
+          cd share/trick/trickops/tests/
+          source ../.venv/bin/activate
+          export PATH="/tmp/koviz-master/bin:${PATH}"
+          ./run_tests.py
+      - uses: actions/upload-artifact@master
+        if: ${{ always() }}
+        with:
+          name: doctests
+          path: |
+            share/trick/trickops/tests/*_doctest_log.txt
+            /tmp/log.*
+
+# TODO: ExampleWorkflow.py is not included here because it needs a built Trick
+#  to function correctly and I don't want to duplicate the Trick build testing
+#  here to provide testing of what is essentially an example provided for
+#  documentation purposes.  If we could leverage artifacts from a previous
+#  stage and/or stable containers where Trick is already pre-built, we should
+#  consider adding ExampleWorfklow.py to testing in this file. -Jordan 4/2021
diff --git a/docs/documentation/Documentation-Home.md b/docs/documentation/Documentation-Home.md
@@ -66,5 +66,6 @@ The user guide contains information pertinent to Trick users. These pages will h
 
 01. [Miscellaneous Trick Tools](miscellaneous_trick_tools/Miscellaneous-Trick-Tools)  
     01. [Python Variable Server Client](miscellaneous_trick_tools/Python-Variable-Server-Client)
+    02. [TrickOps Sim Testing Framework](miscellaneous_trick_tools/TrickOps.md)
 
 01. [Software Requirements Specification](software_requirements_specification/SRS)
diff --git a/docs/documentation/miscellaneous_trick_tools/TrickOps.md b/docs/documentation/miscellaneous_trick_tools/TrickOps.md
diff --git a/docs/documentation/miscellaneous_trick_tools/trickops_example.png b/docs/documentation/miscellaneous_trick_tools/trickops_example.png
diff --git a/share/trick/trickops/ColorStr.py b/share/trick/trickops/ColorStr.py
@@ -0,0 +1,63 @@
+# Simple color string utility class
+class ColorStr:
+
+    def __init__( self ):
+        self.CODE={
+            'ENDC':0,  # RESET COLOR
+            'BOLD':1,
+            'UNDERLINE':4,
+            'BLINK':5,
+            'INVERT':7,
+            'CONCEALD':8,
+            'STRIKE':9,
+            'GREY30':90,
+            'GREY40':2,
+            'GREY65':37,
+            'GREY70':97,
+            'GREY20_BG':40,
+            'GREY33_BG':100,
+            'GREY80_BG':47,
+            'GREY93_BG':107,
+            'DARK_RED':31,
+            'RED':91,
+            'RED_BG':41,
+            'LIGHT_RED_BG':101,
+            'DARK_YELLOW':33,
+            'YELLOW':93,
+            'YELLOW_BG':43,
+            'LIGHT_YELLOW_BG':103,
+            'DARK_BLUE':34,
+            'BLUE':94,
+            'BLUE_BG':44,
+            'LIGHT_BLUE_BG':104,
+            'DARK_MAGENTA':35,
+            'PURPLE':95,
+            'MAGENTA_BG':45,
+            'LIGHT_PURPLE_BG':105,
+            'DARK_CYAN':36,
+            'AQUA':96,
+            'CYAN_BG':46,
+            'LIGHT_AUQA_BG':106,
+            'DARK_GREEN':32,
+            'GREEN':92,
+            'GREEN_BG':42,
+            'LIGHT_GREEN_BG':102,
+            'BLACK':30 \
+        }
+
+    def getCodes( self ):
+        return self.CODE
+
+    def termcode(self, num):
+        return '\033[%sm'%num
+
+    def colorstr(self, astr, color):
+        return self.termcode(self.CODE[color])+astr+self.termcode(self.CODE['ENDC'])
+
+    def showCodes( self ):
+        astr='yippy skippy'
+        codes = self.getCodes()
+        for key in sorted(codes.keys()):
+            print('%s: %s' % (key, self.colorstr( astr, key )))
+    def cprint( self, astr, color ):
+        print(self.colorstr( astr, color ))
diff --git a/share/trick/trickops/ExampleWorkflow.py b/share/trick/trickops/ExampleWorkflow.py
@@ -0,0 +1,143 @@
+#!/usr/bin/env python3
+import os, sys
+
+# Create a valid yml config file describing which sims to consider
+yml_content=(
+"""
+SIM_parse_s_define:
+    path: test/SIM_parse_s_define
+SIM_python_namespace:
+    path: test/SIM_python_namespace
+    runs:
+        RUN_test/unit_test.py:
+SIM_rti:
+    path: test/SIM_rti
+    runs:
+        RUN_test/unit_test.py:
+SIM_segments:
+    path: test/SIM_segments
+    runs:
+        RUN_test/input.py:
+SIM_stls:
+    path: test/SIM_stls
+    runs:
+        RUN_test/input.py:
+        RUN_test/unit_test.py:
+SIM_swig_template_scoping:
+    path: test/SIM_swig_template_scoping
+SIM_target_specific_variables:
+    path: test/SIM_target_specific_variables
+SIM_test_abstract:
+    path: test/SIM_test_abstract
+    runs:
+        RUN_test/input.py:
+SIM_test_dp:
+    path: test/SIM_test_dp
+    runs:
+        RUN_test/unit_test.py:
+        RUN_test/input.py:
+SIM_test_dr:
+    path: test/SIM_test_dr
+    runs:
+        RUN_test/unit_test.py:
+SIM_test_inherit:
+    path: test/SIM_test_inherit
+    runs:
+        RUN_test/input.py:
+SIM_test_io:
+    path: test/SIM_test_io
+    runs:
+        RUN_test/unit_test.py:
+SIM_test_ip:
+    path: test/SIM_test_ip
+    runs:
+        RUN_test/unit_test.py:
+SIM_test_sched:
+    path: test/SIM_test_sched
+    runs:
+        RUN_test/input.py:
+        RUN_test/unit_test.py:
+SIM_test_templates:
+    path: test/SIM_test_templates
+    runs:
+        RUN_test/unit_test.py:
+SIM_threads:
+    path: test/SIM_threads
+    runs:
+        RUN_test/sched.py:
+        RUN_test/amf.py:
+        RUN_test/async.py:
+        RUN_test/unit_test.py:
+SIM_threads_simple:
+    path: test/SIM_threads_simple
+    runs:
+        RUN_test/input.py:
+        RUN_test/sched.py:
+        RUN_test/async.py:
+SIM_trickcomm:
+    path: test/SIM_trickcomm
+    runs:
+        RUN_test/input.py:
+SIM_ball_L2:
+    path: trick_sims/Ball/SIM_ball_L2
+SIM_ball_L3:
+    path: trick_sims/Ball/SIM_ball_L3
+SIM_amoeba:
+    path: trick_sims/Cannon/SIM_amoeba
+SIM_cannon_aero:
+    path: trick_sims/Cannon/SIM_cannon_aero
+SIM_cannon_analytic:
+    path: trick_sims/Cannon/SIM_cannon_analytic
+SIM_cannon_eulercromer:
+    path: trick_sims/Cannon/SIM_cannon_eulercromer
+SIM_cannon_jet:
+    path: trick_sims/Cannon/SIM_cannon_jet
+SIM_cannon_numeric:
+    path: trick_sims/Cannon/SIM_cannon_numeric
+SIM_monte:
+    path: trick_sims/Cannon/SIM_monte
+SIM_Ball++_L1:
+    path: trick_sims/SIM_Ball++_L1/
+SIM_contact:
+    path: trick_sims/SIM_contact
+SIM_lander:
+    path: trick_sims/SIM_lander
+SIM_msd:
+    path: trick_sims/SIM_msd
+SIM_parachute:
+    path: trick_sims/SIM_parachute
+SIM_rocket:
+    path: trick_sims/SIM_rocket
+SIM_sat2d:
+    path: trick_sims/SIM_sat2d
+SIM_satellite:
+    path: trick_sims/SIM_satellite
+SIM_sun:
+    path: trick_sims/SIM_sun
+SIM_wheelbot:
+    path: trick_sims/SIM_wheelbot
+""")
+f = open("/tmp/config.yml", "w")
+f.write(yml_content)
+f.close()
+
+from TrickWorkflow import *
+class ExampleWorkflow(TrickWorkflow):
+    def __init__( self, quiet, trick_top_level='/tmp/trick'):
+        # Real projects already have trick somewhere, but for this test, just clone it
+        if not os.path.exists(trick_top_level):
+          os.system('cd %s && git clone https://github.com/nasa/trick' % (os.path.dirname(trick_top_level)))
+        # Base Class initialize, this creates internal management structures
+        TrickWorkflow.__init__(self, project_top_level=trick_top_level, log_dir='/tmp/',
+            trick_dir=trick_top_level, config_file="/tmp/config.yml", cpus=3, quiet=quiet)
+    def run( self):
+      build_jobs      = self.get_jobs(kind='build')
+      run_jobs        = self.get_jobs(kind='run')
+
+      builds_status = self.execute_jobs(build_jobs, max_concurrent=3, header='Executing all sim builds.')
+      runs_status   = self.execute_jobs(run_jobs,   max_concurrent=3, header='Executing all sim runs.')
+      self.report()           # Print Verbose report
+      self.status_summary()   # Print a Succinct summary
+      return (builds_status or runs_status or self.config_errors)
+if __name__ == "__main__":
+    sys.exit(ExampleWorkflow(quiet=(True if len(sys.argv) > 1 and 'quiet' in sys.argv[1] else False)).run())
diff --git a/share/trick/trickops/README.md b/share/trick/trickops/README.md
@@ -0,0 +1 @@
+../../../docs/documentation/miscellaneous_trick_tools/TrickOps.md
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		../../../docs/documentation/miscellaneous_trick_tools/TrickOps.md