Skip to main content

A generic SAST core built on top of semantic grep and regex

Project description

Generic SAST for Security Engineers. Powered by regex based pattern matcher and semantic aware semgrep.

Made with Love in India Tweet

PyPI version platform License python

Language grade: Python Requirements Status Build

Install

pip install libsast

Supports only Mac and Linux

Command Line Options

$ libsast
usage: libsast [-h] [-o OUTPUT] [-p PATTERN_FILE] [-s SGREP_PATTERN_FILE]
               [--sgrep-file-extensions SGREP_FILE_EXTENSIONS [SGREP_FILE_EXTENSIONS ...]]
               [--file-extensions FILE_EXTENSIONS [FILE_EXTENSIONS ...]]
               [--ignore-filenames IGNORE_FILENAMES [IGNORE_FILENAMES ...]]
               [--ignore-extensions IGNORE_EXTENSIONS [IGNORE_EXTENSIONS ...]]
               [--ignore-paths IGNORE_PATHS [IGNORE_PATHS ...]]
               [--show-progress] [-v]
               [path [path ...]]

positional arguments:
  path                  Path can be file(s) or directories

optional arguments:
  -h, --help            show this help message and exit
  -o OUTPUT, --output OUTPUT
                        Output filename to save JSON report.
  -p PATTERN_FILE, --pattern-file PATTERN_FILE
                        YAML pattern file, directory or url
  -s SGREP_PATTERN_FILE, --sgrep-pattern-file SGREP_PATTERN_FILE
                        sgrep rules directory
  --sgrep-file-extensions SGREP_FILE_EXTENSIONS [SGREP_FILE_EXTENSIONS ...]
                        File extensions that should be scanned with sgrep
  --file-extensions FILE_EXTENSIONS [FILE_EXTENSIONS ...]
                        File extensions that should be scanned with pattern
                        matcher
  --ignore-filenames IGNORE_FILENAMES [IGNORE_FILENAMES ...]
                        File name(s) to ignore
  --ignore-extensions IGNORE_EXTENSIONS [IGNORE_EXTENSIONS ...]
                        File extension(s) to ignore in lower case
  --ignore-paths IGNORE_PATHS [IGNORE_PATHS ...]
                        Path(s) to ignore
  --show-progress       Show scan progress
  -v, --version         Show libsast version

Example Usage

$ libsast -s tests/unit/assets/rules/semantic_grep/ -p tests/unit/assets/rules/pattern_matcher/ tests/unit/assets/files/
{
  "pattern_matcher": {
    "test_regex": {
      "files": [
        {
          "file_path": "tests/unit/assets/files/test_matcher.test",
          "match_lines": [
            28,
            28
          ],
          "match_position": [
            1143,
            1151
          ],
          "match_string": ".close()"
        }
      ],
      "metadata": {
        "description": "This is a rule to test regex",
        "id": "test_regex",
        "input_case": "exact",
        "pattern": "\\.close\\(\\)",
        "severity": "info",
        "type": "Regex"
      }
    },
    "test_regex_and": {
      "files": [
        {
          "file_path": "tests/unit/assets/files/test_matcher.test",
          "match_lines": [
            7,
            7
          ],
          "match_position": [
            194,
            256
          ],
          "match_string": ".loadUrl(\"file:///\" + Environment.getExternalStorageDirectory("
        },
        {
          "file_path": "tests/unit/assets/files/test_matcher.test",
          "match_lines": [
            3,
            3
          ],
          "match_position": [
            52,
            66
          ],
          "match_string": "webkit.WebView"
        }
      ],
      "metadata": {
        "description": "This is a rule to test regex_and",
        "id": "test_regex_and",
        "input_case": "exact",
        "pattern": [
          "\\.loadUrl\\(.*getExternalStorageDirectory\\(",
          "webkit\\.WebView"
        ],
        "severity": "error",
        "type": "RegexAnd"
      }
    },
    "test_regex_and_not": {
      "files": [
        {
          "file_path": "tests/unit/assets/files/test_matcher.test",
          "match_lines": [
            42,
            42
          ],
          "match_position": [
            1417,
            1426
          ],
          "match_string": "WKWebView"
        },
        {
          "file_path": "tests/unit/assets/files/test_matcher.test",
          "match_lines": [
            40,
            40
          ],
          "match_position": [
            1365,
            1374
          ],
          "match_string": "WKWebView"
        }
      ],
      "metadata": {
        "description": "This is a rule to test regex_and_not",
        "id": "test_regex_and_not",
        "input_case": "exact",
        "pattern": [
          "WKWebView",
          "\\.javaScriptEnabled=false"
        ],
        "severity": "warning",
        "type": "RegexAndNot"
      }
    },
    "test_regex_and_or": {
      "files": [
        {
          "file_path": "tests/unit/assets/files/test_matcher.test",
          "match_lines": [
            58,
            58
          ],
          "match_position": [
            1975,
            1990
          ],
          "match_string": "sendTextMessage"
        },
        {
          "file_path": "tests/unit/assets/files/test_matcher.test",
          "match_lines": [
            50,
            50
          ],
          "match_position": [
            1553,
            1573
          ],
          "match_string": "telephony.SmsManager"
        }
      ],
      "metadata": {
        "description": "This is a rule to test regex_and_or",
        "id": "test_regex_and_or",
        "input_case": "exact",
        "pattern": [
          "telephony.SmsManager",
          [
            "sendMultipartTextMessage",
            "sendTextMessage",
            "vnd.android-dir/mms-sms"
          ]
        ],
        "severity": "warning",
        "type": "RegexAndOr"
      }
    },
    "test_regex_multiline": {
      "files": [
        {
          "file_path": "tests/unit/assets/files/test_matcher.test",
          "match_lines": [
            52,
            52
          ],
          "match_position": [
            1588,
            1686
          ],
          "match_string": "public void onRequestPermissionsResult(int requestCode,String permissions[], int[] grantResults) {"
        },
        {
          "file_path": "tests/unit/assets/files/test_matcher.test",
          "match_lines": [
            10,
            11
          ],
          "match_position": [
            299,
            370
          ],
          "match_string": "public static ForgeAccount add(Context context, ForgeAccount account) {"
        }
      ],
      "metadata": {
        "description": "This is a rule to test regex",
        "id": "test_regex_multiline",
        "input_case": "exact",
        "pattern": "((?:public.+)+)",
        "severity": "info",
        "type": "Regex"
      }
    },
    "test_regex_or": {
      "files": [
        {
          "file_path": "tests/unit/assets/files/test_matcher.test",
          "match_lines": [
            26,
            26
          ],
          "match_position": [
            1042,
            1069
          ],
          "match_string": "Context.MODE_WORLD_READABLE"
        }
      ],
      "metadata": {
        "description": "This is a rule to test regex_or",
        "id": "test_regex_or",
        "input_case": "exact",
        "pattern": [
          "MODE_WORLD_READABLE|Context\\.MODE_WORLD_READABLE",
          "openFileOutput\\(\\s*\".+\"\\s*,\\s*1\\s*\\)"
        ],
        "severity": "error",
        "type": "RegexOr"
      }
    }
  },
  "semantic_grep": {
    "errors": [
      {
        "code": 3,
        "help": "If the code appears to be valid, this may be a semgrep bug.",
        "level": "warn",
        "long_msg": "Could not parse test_file.cpython-37-pytest-5.4.2.pyc as python",
        "short_msg": "parse error",
        "spans": [
          {
            "context_end": null,
            "context_start": null,
            "end": {
              "col": 5,
              "line": 1
            },
            "file": "test_file.cpython-37-pytest-5.4.2.pyc",
            "source_hash": "b59a27ae0ab7799f6304901e395a559d0e01027d90202d068ded5912f6c5795d",
            "start": {
              "col": 4,
              "line": 1
            }
          }
        ],
        "type": "SourceParseError"
      }
    ],
    "matches": {
      "boto-client-ip": {
        "files": [
          {
            "file_path": "tests/unit/assets/files/example_file.py",
            "match_lines": [
              4,
              4
            ],
            "match_position": [
              24,
              31
            ],
            "match_string": "8.8.8.8"
          }
        ],
        "metadata": {
          "cwe": "CWE Category",
          "description": "boto client using IP address",
          "owasp": "OWASP Category",
          "severity": "ERROR"
        }
      }
    }
  }
}

Python API

>>> from libsast import Scanner
>>> options = {'match_rules': '/Users/ajinabraham/Code/njsscan/njsscan/rules/pattern_matcher', 'sgrep_rules': '/Users/ajinabraham/Code/njsscan/njsscan/rules/semantic_grep', 'sgrep_extensions': {'', '.js'}, 'match_extensions': {'.hbs', '.sh', '.ejs', '.toml', '.mustache', '.tmpl', '.jade', '.json', '.ect', '.vue', '.yml', '.hdbs', '.tl', '.html', '.haml', '.dust', '.pug', '.tpl'}, 'ignore_filenames': {'bootstrap.min.js', '.DS_Store', 'bootstrap-tour.js', 'd3.min.js', 'tinymce.js', 'codemirror.js', 'tinymce.min.js', 'react-dom.production.min.js', 'react.js', 'jquery.min.js', 'react.production.min.js', 'codemirror-compressed.js', 'axios.min.js', 'angular.min.js', 'raphael-min.js', 'vue.min.js'}, 'ignore_extensions': {'.7z', '.exe', '.rar', '.zip', '.a', '.o', '.tz'}, 'ignore_paths': {'__MACOSX', 'jquery', 'fixtures', 'node_modules', 'bower_components', 'example', 'spec'}, 'show_progress': False}
>>> paths = ['../njsscan/tests/assets/dot_njsscan/']
>>> scanner = Scanner(options, paths)
>>> scanner.scan()
{'pattern_matcher': {'handlebar_mustache_template': {'files': [{'file_path': '../njsscan/tests/assets/dot_njsscan/ignore_ext.hbs', 'match_string': '{{{html}}}', 'match_position': (52, 62)}], 'metadata': {'id': 'handlebar_mustache_template', 'description': 'The Handlebar.js/Mustache.js template has an unescaped variable. Untrusted user input passed to this variable results in Cross Site Scripting (XSS).', 'type': 'Regex', 'pattern': '{{{.+}}}|{{[ ]*&[\\w]+.*}}', 'severity': 'ERROR', 'input_case': 'exact', 'cwe': "CWE-79: Improper Neutralization of Input During Web Page Generation ('Cross-site Scripting')", 'owasp': 'A1: Injection'}}}, 'semantic_grep': {'matches': {'node_aes_ecb': {'files': [{'file_path': '../njsscan/tests/assets/dot_njsscan/lorem_scan.js', 'match_position': (16, 86), 'match_lines': (14, 14), 'match_string': "let decipher = crypto.createDecipheriv('aes-128-ecb', Buffer.from(ENCRYPTION_KEY), iv);"}], 'metadata': {'owasp': 'A9: Using Components with Known Vulnerabilities', 'cwe': 'CWE-327: Use of a Broken or Risky Cryptographic Algorithm', 'description': 'AES with ECB mode is deterministic in nature and not suitable for encrypting large amount of repetitive data.', 'severity': 'ERROR'}}, 'node_deserialize': {'files': [{'file_path': '../njsscan/tests/assets/dot_njsscan/skip.js', 'match_position': (19, 44), 'match_lines': (8, 8), 'match_string': '        var obj = serialize.unserialize(str);'}], 'metadata': {'owasp': 'A8: Insecure Deserialization', 'cwe': 'CWE-502: Deserialization of Untrusted Data', 'description': "User controlled data in 'unserialize()' or 'deserialize()' function can result in Object Injection or Remote Code Injection.", 'severity': 'ERROR'}}, 'express_xss': {'files': [{'file_path': '../njsscan/tests/assets/dot_njsscan/skip.js', 'match_position': (9, 52), 'match_lines': (7, 10), 'match_string': '        var str = new Buffer(req.cookies.profile, \'base64\').toString();\n\n        var obj = serialize.unserialize(str);\n\n        if (obj.username) {\n\n            res.send("Hello " + escape(obj.username));'}], 'metadata': {'owasp': 'A1: Injection', 'cwe': "CWE-79: Improper Neutralization of Input During Web Page Generation ('Cross-site Scripting')", 'description': 'Untrusted User Input in Response will result in Reflected Cross Site Scripting Vulnerability.', 'severity': 'ERROR'}}, 'node_tls_reject': {'files': [{'file_path': '../njsscan/tests/assets/dot_njsscan/skip_dir/skip_me.js', 'match_position': (9, 58), 'match_lines': (9, 9), 'match_string': "        process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = '0';"}, {'file_path': '../njsscan/tests/assets/dot_njsscan/skip_dir/skip_me.js', 'match_position': (9, 55), 'match_lines': (18, 18), 'match_string': '        process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";'}], 'metadata': {'owasp': 'A6: Security Misconfiguration', 'cwe': 'CWE-295: Improper Certificate Validation', 'description': "Setting 'NODE_TLS_REJECT_UNAUTHORIZED' to 0 will allow node server to accept self signed certificates and is not an secure behaviour.", 'severity': 'ERROR'}}, 'node_curl_ssl_verify_disable': {'files': [{'file_path': '../njsscan/tests/assets/dot_njsscan/skip_dir/skip_me.js', 'match_position': (5, 35), 'match_lines': (45, 50), 'match_string': '    curl(url,\n\n        {\n\n            SSL_VERIFYPEER: 0\n\n        },\n\n        function (err) {\n\n            response.end(this.body);'}], 'metadata': {'owasp': 'A6: Security Misconfiguration', 'cwe': 'CWE-599: Missing Validation of OpenSSL Certificate', 'description': 'SSL Certificate verification for node-curl is disabled.', 'severity': 'ERROR'}}}, 'errors': []}}

Write you own Static Analysis tool

With libsast, you can write your own static analysis tools. libsast provides two matching engines:

  1. Pattern Matcher
  2. Semantic Grep

Pattern Matcher

Currently Pattern Matcher supports any language.

Use Regex 101 to write simple Python Regex rule patterns.

A sample rule looks like

- id: test_regex_or
  description: This is a rule to test regex_or
  input_case: exact
  pattern:
  - MODE_WORLD_READABLE|Context\.MODE_WORLD_READABLE
  - openFileOutput\(\s*".+"\s*,\s*1\s*\)
  severity: error
  type: RegexOr
  owasp: 'OWASP Category'

A rule consist of

  • id : A unique id for the rule
  • description: A description for the rule
  • input_case: It can be exact, upper or lower. Data will be converted to lower case/upper case/as it is before comparing with the regex.
  • pattern: List of patterns depends on type.
  • severity: It can be error, warning or info
  • type: Pattern Matcher supports Regex, RegexAnd, RegexOr, RegexAndOr, RegexAndNot
  • custom_field: Define your own custom fields that you can use as metadata
1. Regex - if regex1 in input
2. RegexAnd - if regex1 in input and regex2 in input
3. RegexOr - if regex1 in input or regex2 in input
4. RegexAndOr -  if regex1 in input and (regex2 in input or regex3 in input)
5. RegexAndNot - if regex1 in input and not regex2 in input

Example: Pattern Matcher Rule

Test your pattern matcher rules

$ libsast -p tests/unit/assets/rules/pattern_matcher/patterns.yaml tests/unit/assets/files/

Inbuilt Standard Mapping Support

Semantic Grep

Semantic Grep uses semgrep, a fast and syntax-aware semantic code pattern search for many languages: like grep but for code.

Currently it supports Python, Java, JavaScript, Go and C.

Use semgrep.live to write semantic grep rule patterns.

A sample rule for Python code looks like

rules:
  - id: boto-client-ip
    patterns:
      - pattern-inside: boto3.client(host="...")
      - pattern-regex: '\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}'
    message: "boto client using IP address"
    languages: [python]
    severity: ERROR
    metadata:
      owasp: 'OWASP Category'
      cwe: 'CWE Category'

See semgrep documentation here.

Example: Semantic Grep Rule

Test your semgrep rules

$ libsast -s tests/unit/assets/rules/semantic_grep/sgrep.yaml tests/unit/assets/files/

Realworld Implementations

  • njsscan SAST is built with libsast pattern matcher and semantic grep.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

libsast-1.0.8.tar.gz (30.7 kB view hashes)

Uploaded Source

Built Distribution

libsast-1.0.8-py3-none-any.whl (37.9 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page