Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add code compliance checks #214

Merged
merged 36 commits into from
Aug 19, 2023
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
ce620a6
Add tree-sitter
qu35t-code Aug 18, 2023
14f647f
Add needs to CI + Fix path
qu35t-code Aug 18, 2023
e3c3ede
Add debug messages for parser
qu35t-code Aug 18, 2023
33c767a
Fix regex
qu35t-code Aug 18, 2023
06a0789
Remove debug print
qu35t-code Aug 18, 2023
85fba63
Add missing colorecho function
qu35t-code Aug 18, 2023
53e2c2c
Add missing colorecho function
qu35t-code Aug 18, 2023
e530880
Add add-aliases-whitelist
qu35t-code Aug 18, 2023
42cad03
Add add-history,add-test-command and add-to-list whitelist
qu35t-code Aug 18, 2023
06992ca
Revert dockerfiles for pretty debug
qu35t-code Aug 18, 2023
badf2c6
Nesting CCC workflow
ShutdownRepo Aug 19, 2023
e32d29d
Adding TODOs in some packages
ShutdownRepo Aug 19, 2023
0d03b48
Restructuring conf and parser
ShutdownRepo Aug 19, 2023
68cb564
Naming code-check job
ShutdownRepo Aug 19, 2023
242b8f3
Fixed files and folders renaming
ShutdownRepo Aug 19, 2023
9a290b2
Fix missing add-aliases
ShutdownRepo Aug 19, 2023
02957e5
Add batcat alias to bat
ShutdownRepo Aug 19, 2023
e141bc2
Fix missing add-test-command
ShutdownRepo Aug 19, 2023
9dae44f
Fix missing add-to-list
ShutdownRepo Aug 19, 2023
e271e98
Fix messed up regex on package filename
ShutdownRepo Aug 19, 2023
81a5e58
Testing fix on add-history.conf
ShutdownRepo Aug 19, 2023
df21fc9
Testing matrix
ShutdownRepo Aug 19, 2023
ff21fe9
Testing manual install of Rust and Cargo
ShutdownRepo Aug 19, 2023
7356bb4
Finalizing CCC workflow v1
ShutdownRepo Aug 19, 2023
5a85949
Renaming CCC workflow calls
ShutdownRepo Aug 19, 2023
963a4d5
Removing varset calls
ShutdownRepo Aug 19, 2023
5b32733
Trying to replace varset vars with env vars
ShutdownRepo Aug 19, 2023
7740a47
Reverting varset calls
ShutdownRepo Aug 19, 2023
0c133a3
Remove tree-sitter and add better custom python parser
qu35t-code Aug 19, 2023
96a6b18
Comment code + Move and rename the file
qu35t-code Aug 19, 2023
5ec49fc
Rename the code-check file
qu35t-code Aug 19, 2023
af2a1ad
Typo filename in the wf
qu35t-code Aug 19, 2023
e341ebb
Fix add-history for functions
qu35t-code Aug 19, 2023
a667b81
Update code_compliance_check.py
ShutdownRepo Aug 19, 2023
426a0ed
Add tests commands to the whitelist for GUI tools
qu35t-code Aug 19, 2023
7979c02
Merge remote-tracking branch 'origin/ci/tree-sitter' into ci/tree-sitter
qu35t-code Aug 19, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 49 additions & 1 deletion .github/workflows/entrypoint_pull_request.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,57 @@ jobs:
echo "IMAGE_BASE_NAME=PR${PR_NUMBER}" >> $GITHUB_OUTPUT
echo "DOCKERFILE=${DOCKERFILE}" >> $GITHUB_OUTPUT

setup-code-check:
runs-on: ubuntu-latest

steps:
- name: Checkout
uses: actions/checkout@v3

- name: Install Rust and Cargo
uses: actions-rs/toolchain@v1
with:
profile: minimal
toolchain: stable
override: true

- name: Setup tree-sitter
run: |
cargo install tree-sitter-cli
mkdir $HOME/src
git -C $HOME/src/ clone https://github.com/tree-sitter/tree-sitter-bash.git
tree-sitter init-config

- name: Debug print tree-sitter parsing
run: |
tree-sitter query -c ./sources/assets/code-check/colorecho.conf "./sources/install/package*.sh"
tree-sitter query -c ./sources/assets/code-check/add-aliases.conf "./sources/install/package*.sh"
tree-sitter query -c ./sources/assets/code-check/add-history.conf "./sources/install/package*.sh"
tree-sitter query -c ./sources/assets/code-check/add-test-command.conf "./sources/install/package*.sh"
tree-sitter query -c ./sources/assets/code-check/add-to-list.conf "./sources/install/package*.sh"

- name: Check colorecho
run: tree-sitter query -c ./sources/assets/code-check/colorecho.conf "./sources/install/package*.sh"|python3 ./sources/assets/code-check/parser.py

- name: Check add-aliases
if: always()
run: tree-sitter query -c ./sources/assets/code-check/add-aliases.conf "./sources/install/package*.sh"|python3 ./sources/assets/code-check/parser.py

- name: Check add-history
if: always()
run: tree-sitter query -c ./sources/assets/code-check/add-history.conf "./sources/install/package*.sh"|python3 ./sources/assets/code-check/parser.py

- name: Check add-test-command
if: always()
run: tree-sitter query -c ./sources/assets/code-check/add-test-command.conf "./sources/install/package*.sh"|python3 ./sources/assets/code-check/parser.py

- name: Check add-to-list
if: always()
run: tree-sitter query -c ./sources/assets/code-check/add-to-list.conf "./sources/install/package*.sh"|python3 ./sources/assets/code-check/parser.py

build_belt:
name: Build belt
needs: varset
needs: [varset, setup-code-check]
strategy:
fail-fast: false
matrix:
Expand Down
66 changes: 33 additions & 33 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -19,39 +19,39 @@ WORKDIR /root/sources/install
# WARNING: package_most_used can't be used with other functions other than: package_base, post_install
# ./entrypoint.sh package_most_used

RUN echo "${TAG}-${VERSION}" > /opt/.exegol_version && \
chmod +x entrypoint.sh && \
./entrypoint.sh package_base && \
./entrypoint.sh package_desktop && \
./entrypoint.sh package_misc && \
./entrypoint.sh package_misc_configure && \
./entrypoint.sh package_c2 && \
./entrypoint.sh package_c2_configure && \
./entrypoint.sh package_wordlists && \
./entrypoint.sh package_wordlists_configure && \
./entrypoint.sh package_cracking && \
./entrypoint.sh package_cracking_configure && \
./entrypoint.sh package_osint && \
./entrypoint.sh package_osint_configure && \
./entrypoint.sh package_web && \
./entrypoint.sh package_web_configure && \
./entrypoint.sh package_ad && \
./entrypoint.sh package_ad_configure && \
./entrypoint.sh package_mobile && \
./entrypoint.sh package_iot && \
./entrypoint.sh package_rfid && \
./entrypoint.sh package_voip && \
./entrypoint.sh package_sdr && \
./entrypoint.sh package_network && \
./entrypoint.sh package_wifi && \
./entrypoint.sh package_forensic && \
./entrypoint.sh package_cloud && \
./entrypoint.sh package_steganography && \
./entrypoint.sh package_reverse && \
./entrypoint.sh package_crypto && \
./entrypoint.sh package_code_analysis && \
./entrypoint.sh post_install && \
rm -rf /root/sources /var/lib/apt/lists/*
RUN echo "${TAG}-${VERSION}" > /opt/.exegol_version
RUN chmod +x entrypoint.sh
RUN ./entrypoint.sh package_base
RUN ./entrypoint.sh package_desktop
RUN ./entrypoint.sh package_misc
RUN ./entrypoint.sh package_misc_configure
RUN ./entrypoint.sh package_c2
RUN ./entrypoint.sh package_c2_configure
RUN ./entrypoint.sh package_wordlists
RUN ./entrypoint.sh package_wordlists_configure
RUN ./entrypoint.sh package_cracking
RUN ./entrypoint.sh package_cracking_configure
RUN ./entrypoint.sh package_osint
RUN ./entrypoint.sh package_osint_configure
RUN ./entrypoint.sh package_web
RUN ./entrypoint.sh package_web_configure
RUN ./entrypoint.sh package_ad
RUN ./entrypoint.sh package_ad_configure
RUN ./entrypoint.sh package_mobile
RUN ./entrypoint.sh package_iot
RUN ./entrypoint.sh package_rfid
RUN ./entrypoint.sh package_voip
RUN ./entrypoint.sh package_sdr
RUN ./entrypoint.sh package_network
RUN ./entrypoint.sh package_wifi
RUN ./entrypoint.sh package_forensic
RUN ./entrypoint.sh package_cloud
RUN ./entrypoint.sh package_steganography
RUN ./entrypoint.sh package_reverse
RUN ./entrypoint.sh package_crypto
RUN ./entrypoint.sh package_code_analysis
RUN ./entrypoint.sh post_install
RUN rm -rf /root/sources /var/lib/apt/lists/*

WORKDIR /workspace

Expand Down
38 changes: 19 additions & 19 deletions ad.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -19,25 +19,25 @@ WORKDIR /root/sources/install
# WARNING: package_most_used can't be used with other functions other than: package_base, post_install
# ./entrypoint.sh package_most_used

RUN echo "${TAG}-${VERSION}" > /opt/.exegol_version && \
chmod +x entrypoint.sh && \
./entrypoint.sh package_base && \
./entrypoint.sh package_desktop && \
./entrypoint.sh package_misc && \
./entrypoint.sh package_misc_configure && \
./entrypoint.sh package_c2 && \
./entrypoint.sh package_c2_configure && \
./entrypoint.sh package_wordlists && \
./entrypoint.sh package_wordlists_configure && \
./entrypoint.sh package_cracking && \
./entrypoint.sh package_cracking_configure && \
./entrypoint.sh package_web && \
./entrypoint.sh package_web_configure && \
./entrypoint.sh package_ad && \
./entrypoint.sh package_ad_configure && \
./entrypoint.sh package_network && \
./entrypoint.sh post_install && \
rm -rf /root/sources /var/lib/apt/lists/*
RUN echo "${TAG}-${VERSION}" > /opt/.exegol_version
RUN chmod +x entrypoint.sh
RUN ./entrypoint.sh package_base
RUN ./entrypoint.sh package_desktop
RUN ./entrypoint.sh package_misc
RUN ./entrypoint.sh package_misc_configure
RUN ./entrypoint.sh package_c2
RUN ./entrypoint.sh package_c2_configure
RUN ./entrypoint.sh package_wordlists
RUN ./entrypoint.sh package_wordlists_configure
RUN ./entrypoint.sh package_cracking
RUN ./entrypoint.sh package_cracking_configure
RUN ./entrypoint.sh package_web
RUN ./entrypoint.sh package_web_configure
RUN ./entrypoint.sh package_ad
RUN ./entrypoint.sh package_ad_configure
RUN ./entrypoint.sh package_network
RUN ./entrypoint.sh post_install
RUN rm -rf /root/sources /var/lib/apt/lists/*

WORKDIR /workspace

Expand Down
10 changes: 5 additions & 5 deletions debug.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,11 @@ COPY sources /root/sources/

WORKDIR /root/sources/install

RUN echo "${TAG}-${VERSION}" > /opt/.exegol_version && \
chmod +x entrypoint.sh && \
./entrypoint.sh package_base_debug && \
./entrypoint.sh post_install && \
rm -rf /root/sources /var/lib/apt/lists/*
RUN echo "${TAG}-${VERSION}" > /opt/.exegol_version
RUN chmod +x entrypoint.sh
RUN ./entrypoint.sh package_base_debug
RUN ./entrypoint.sh post_install
RUN rm -rf /root/sources /var/lib/apt/lists/*

WORKDIR /workspace

Expand Down
20 changes: 10 additions & 10 deletions light.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -19,16 +19,16 @@ WORKDIR /root/sources/install
# WARNING: package_most_used can't be used with other functions other than: package_base, post_install
# ./entrypoint.sh package_most_used

RUN echo "${TAG}-${VERSION}" > /opt/.exegol_version && \
chmod +x entrypoint.sh && \
./entrypoint.sh package_base && \
./entrypoint.sh package_desktop && \
./entrypoint.sh package_most_used && \
./entrypoint.sh configure_john && \
./entrypoint.sh package_misc && \
./entrypoint.sh package_misc_configure && \
./entrypoint.sh post_install && \
rm -rf /root/sources /var/lib/apt/lists/*
RUN echo "${TAG}-${VERSION}" > /opt/.exegol_version
RUN chmod +x entrypoint.sh
RUN ./entrypoint.sh package_base
RUN ./entrypoint.sh package_desktop
RUN ./entrypoint.sh package_most_used
RUN ./entrypoint.sh configure_john
RUN ./entrypoint.sh package_misc
RUN ./entrypoint.sh package_misc_configure
RUN ./entrypoint.sh post_install
RUN rm -rf /root/sources /var/lib/apt/lists/*

WORKDIR /workspace

Expand Down
20 changes: 10 additions & 10 deletions osint.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -19,16 +19,16 @@ WORKDIR /root/sources/install
# WARNING: package_most_used can't be used with other functions other than: package_base, post_install
# ./entrypoint.sh package_most_used

RUN echo "${TAG}-${VERSION}" > /opt/.exegol_version && \
chmod +x entrypoint.sh && \
./entrypoint.sh package_base && \
./entrypoint.sh package_desktop && \
./entrypoint.sh package_misc && \
./entrypoint.sh package_misc_configure && \
./entrypoint.sh package_osint && \
./entrypoint.sh package_osint_configure && \
./entrypoint.sh post_install && \
rm -rf /root/sources /var/lib/apt/lists/*
RUN echo "${TAG}-${VERSION}" > /opt/.exegol_version
RUN chmod +x entrypoint.sh
RUN ./entrypoint.sh package_base
RUN ./entrypoint.sh package_desktop
RUN ./entrypoint.sh package_misc
RUN ./entrypoint.sh package_misc_configure
RUN ./entrypoint.sh package_osint
RUN ./entrypoint.sh package_osint_configure
RUN ./entrypoint.sh post_install
RUN rm -rf /root/sources /var/lib/apt/lists/*

WORKDIR /workspace

Expand Down
6 changes: 6 additions & 0 deletions sources/assets/code-check/add-aliases.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
(function_definition
name: (_) @name
body: (_) @body
(#not-match? @body "add-aliases")
(#not-match? @body "# CODE-CHECK-WHITELIST=.*add-aliases")
)
6 changes: 6 additions & 0 deletions sources/assets/code-check/add-history.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
(function_definition
name: (_) @name
body: (_) @body
(#not-match? @body "add-history")
(#not-match? @body "# CODE-CHECK-WHITELIST=.*add-history")
)
6 changes: 6 additions & 0 deletions sources/assets/code-check/add-test-command.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
(function_definition
name: (_) @name
body: (_) @body
(#not-match? @body "add-test-command")
(#not-match? @body "# CODE-CHECK-WHITELIST=.*add-test-command")
)
6 changes: 6 additions & 0 deletions sources/assets/code-check/add-to-list.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
(function_definition
name: (_) @name
body: (_) @body
(#not-match? @body "add-to-list")
(#not-match? @body "# CODE-CHECK-WHITELIST=.*add-to-list")
)
6 changes: 6 additions & 0 deletions sources/assets/code-check/colorecho.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
(function_definition
name: (_) @name
body: (_) @body
(#not-match? @body "colorecho")
(#not-match? @body "# CODE-CHECK-WHITELIST=.*colorecho")
)
58 changes: 58 additions & 0 deletions sources/assets/code-check/parser.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
#!/usr/bin/env python3

import sys
import re

blue = "\033[1;34m"
magenta = "\033[1;35m"
clear = "\033[0m"

def non_compliance(file_dict, input_string):
not_compliant = False
for element in file_dict["functions"]:
if not element.startswith("install_"):
continue
formatted_string = r"text: `({})`".format(element)
function_name_match = re.search(formatted_string, input_string)
if function_name_match:
content = r'text: `({})`\n.*?text: `(.*?)`'.format(element)
body_pattern = re.compile(content, re.DOTALL)
body_match = body_pattern.search(input_string)
if body_match:
print(f"{magenta}File : {file_dict['filename']}{clear}")
print(f"{blue}Function : {body_match.group(1)}{clear}")
print(body_match.group(2), end='\n\n')
not_compliant = True
return(not_compliant)

def get_functions_name(input_string):
lines = input_string.strip().split('\n')
file_pattern = re.compile(r'^(?P<path>.*/)?(?P<filename>\w+\.sh)')
name_pattern = re.compile(r'text: `(?P<name>[\w_]+)`')
i = 0
result = []
while i < len(lines):
line = lines[i]
if file_pattern.match(line):
current_file = {
"filename": line,
"functions": []
}
result.append(current_file)
i += 1
continue
if name_pattern.search(line):
current_file["functions"].append(name_pattern.search(line).group('name'))
i += 1
continue
i += 1
return result

if __name__ == "__main__":
input_string = sys.stdin.read()
files_and_functions = get_functions_name(input_string)
compliant = 0
for file_dict in files_and_functions:
if non_compliance(file_dict, input_string):
compliant = 1
exit(compliant)
Loading