zkygr3-artifactory-cleanup 1.1.0

Rules and cleanup policies for Artifactory

Artifactory cleanup

artifactory-cleanup is an extended and flexible cleanup tool for JFrog Artifactory.

The tool has simple YAML-defined cleanup configuration and can be extended with your own rules on Python. Everything must be as a code, even cleanup policies!

Tables of Contents

• Artifactory cleanup
• Tables of Contents
• Installation
• Usage
  • Notes
  • Commands
• Rules
  • Common
  • Delete
  • Keep
  • Docker
  • Filters
  • Create your own rule
• How to
  • How to connect self-signed certificates for docker?
  • How to clean up Conan repository?
  • How to keep latest N docker images?
• Release

Installation

As simple as one command!

# docker
docker pull devopshq/artifactory-cleanup
docker run --rm devopshq/artifactory-cleanup artifactory-cleanup --help

# python (later we call it 'cli')
python3 -mpip install artifactory-cleanup
artifactory-cleanup --help

Usage

Suppose you want to remove all artifacts older than N days from reponame repository. You should take the following steps:

1. Install artifactory-cleanup (see above)
2. Create a configuration file artifactory-cleanup.yaml. variables.

# artifactory-cleanup.yaml
artifactory-cleanup:
  server: https://repo.example.com/artifactory
  # $VAR is auto populated from environment variables
  user: $ARTIFACTORY_USERNAME
  password: $ARTIFACTORY_PASSWORD

  policies:
    - name: Remove all files from repo-name-here older than 7 days
      rules:
        - rule: Repo
          name: "reponame"
        - rule: DeleteOlderThan
          days: 7

3. Run the command TO SHOW (not remove) artifacts that will be deleted. By default artifactory-cleanup uses "dry mode".

# Set the credentials with delete permissions
export ARTIFACTORY_USERNAME=usernamehere
export ARTIFACTORY_PASSWORD=password

# docker
docker run --rm -v "$(pwd)":/app -e ARTIFACTORY_USERNAME -e ARTIFACTORY_PASSWORD devopshq/artifactory-cleanup artifactory-cleanup

# cli
artifactory-cleanup

4. Verify that right artifacts will be removed and add --destroy flag TO REMOVE artifacts:

# docker
docker run --rm -v "$(pwd)":/app -e ARTIFACTORY_USERNAME -e ARTIFACTORY_PASSWORD devopshq/artifactory-cleanup artifactory-cleanup --destroy

# cli
artifactory-cleanup --destroy

Looking for more examples? Check examples folder!

Notes

• Always specify version of artifactory-cleanup when using it in the production. 1.0.0 is just an example, find the latest version in pypi: https://pypi.org/project/artifactory-cleanup/

# docker
docker pull devopshq/artifactory-cleanup:1.0.0
docker run --rm devopshq/artifactory-cleanup:1.0.0 artifactory-cleanup --version

# python (later we call it 'cli')
python3 -mpip install artifactory-cleanup==1.0.0
artifactory-cleanup --help

• Use CI servers or cron-like utilities to run artifactory-cleanup every day (or every hour). TeamCity and GitHub have built-in support and show additional logs format
• Do not save credentials in the configuration file, use environment variables.
• Use --ignore-not-found flag to ignore errors when the repository is not found. It's useful when you have a configuration for multiple repositories and some of them are not found.
• Use --worker-count=<WORKER_NUM> to increase the number of workers. By default, it's 1. It's useful when you have a lot of artifacts and you want to speed up the process.

Commands

# Debug - "dry run" mode by default
# debug run - only print artifacts. it does not delete any artifacts
artifactory-cleanup

# Debug run only for policytestname.
artifactory-cleanup --policy-name policytestname

# REMOVE
# For remove artifacts use --destroy
artifactory-cleanup --destroy

# For remove artifacts use environment variable
export ARTIFACTORY_CLEANUP_DESTROY=True
artifactory-cleanup

# Specify config filename
artifactory-cleanup --config artifactory-cleanup.yaml

# Specify config filename using environment variable
export ARTIFACTORY_CLEANUP_CONFIG_FILE=artifactory-cleanup.yaml
artifactory-cleanup --config artifactory-cleanup.yaml

# Look in the future - shows what the tool WILL remove after 10 days
artifactory-cleanup --days-in-future=10

# Not satisfied with built-in rules? Write your own rules in python and connect them!
artifactory-cleanup --load-rules=myrule.py
docker run -v "$(pwd)":/app devopshq/artifactory-cleanup artifactory-cleanup --load-rules=myrule.py

# Save the table summary in a file
artifactory-cleanup --output=myfile.txt

# Save the summary in a Json file
artifactory-cleanup --output=myfile.txt --output-format=json

# Save the summary in a json file and append a list with removed artifacts
artifactory-cleanup --output=myfile.json --output-format json-with-artifact-list

Rules

Common

• Repo - Apply the rule to one repository. If no name is specified, it is taken from the rule name (in CleanupPolicy definition)

- rule: Repo
  name: reponame

# OR - if you have a single policy for the repo - you can name the policy as reponame
# Both configurations are equal
policies:
  - name: reponame
    rules:
      - rule: Repo

• RepoList - Apply the policy to list of repositories.

- rule: RepoList
  repos:
    - repo1
    - repo2
    - repo3

• RepoByMask - Apply rule to repositories matching by mask

- rule: RepoByMask
  mask: "*.banned"

• PropertyEq- Delete repository artifacts only with a specific property value (property_key is the name of the parameter, property_value is the value)

- rule: PropertyEq
  property_key: key-name
  property_value: 1

• PropertyNeq- Delete repository artifacts only if the value != specified. If there is no value, delete it anyway. Allows you to specify the deletion flag do_not_delete = 1

- rule: PropertyNeq
  property_key: key-name
  property_value: 1

Delete

• DeleteOlderThan - deletes artifacts that are older than N days

- rule: DeleteOlderThan
  days: 1

• DeleteWithoutDownloads - deletes artifacts that have never been downloaded (DownloadCount=0). Better to use with DeleteOlderThan rule

- rule: DeleteWithoutDownloads

• DeleteOlderThanNDaysWithoutDownloads - deletes artifacts that are older than N days and have not been downloaded

- rule: DeleteOlderThanNDaysWithoutDownloads
  days: 1

• DeleteNotUsedSince - delete artifacts that were downloaded, but for a long time. N days passed. Or not downloaded at all from the moment of creation and it's been N days

- rule: DeleteNotUsedSince
  days: 1

• DeleteEmptyFolders - Clean up empty folders in given repository list

- rule: DeleteEmptyFolders

• DeleteByRegexpName - delete artifacts whose name matches the specified regexp

- rule: DeleteByRegexpName
  regex_pattern: "\d"

Keep

• KeepLatestNFiles - Leaves the last (by creation time) files in the amount of N pieces. WITHOUT accounting subfolders

- rule: KeepLatestNFiles
  count: 1

• KeepLatestNFilesInFolder - Leaves the last (by creation time) files in the number of N pieces in each folder

- rule: KeepLatestNFilesInFolder
  count: 1

• KeepLatestVersionNFilesInFolder - Leaves the latest N (by version) files in each folder. The definition of the version is using regexp. By default it parses semver using the regex - ([\d]+\.[\d]+\.[\d]+)")

- rule: KeepLatestVersionNFilesInFolder
  count: 1
  custom_regexp: "[^\\d][\\._]((\\d+\\.)+\\d+)"

• KeepLatestNupkgNVersions - Leaves N nupkg (adds *.nupkg filter) in release feature builds

- rule: KeepLatestNupkgNVersions
  count: 1

Docker

• DeleteDockerImagesOlderThan - Delete docker images that are older than N days

- rule: DeleteDockerImagesOlderThan
  days: 1

• DeleteDockerImagesOlderThanNDaysWithoutDownloads - Deletes docker images that are older than N days and have not been downloaded

- rule: DeleteDockerImagesOlderThanNDaysWithoutDownloads
  days: 1

• DeleteDockerImagesNotUsed - Removes Docker image not downloaded since N days

- rule: DeleteDockerImagesNotUsed
  days: 1

• IncludeDockerImages - Apply to docker images with the specified names and tags

- rule: IncludeDockerImages
  masks: "*singlemask*"
- rule: IncludeDockerImages
  masks:
    - "*production*"
    - "*release*"

• ExcludeDockerImages - Exclude Docker images by name and tags.

- rule: ExcludeDockerImages
  masks:
    - "*production*"
    - "*release*"

• KeepLatestNVersionImagesByProperty(count=N, custom_regexp='some-regexp', number_of_digits_in_version=X) - Leaves N Docker images with the same major. (^\d+\.\d+\.\d+$) is the default regexp how to determine version which matches semver 1.1.1. If you need to add minor then set number_of_digits_in_version to 2 or if patch then set to 3 (by default we match major, which 1). Semver tags prefixed with v are supported by updating the regexp to include (an optional) v in the expression (e.g., (^v?\d+\.\d+\.\d+$)).

- rule: KeepLatestNVersionImagesByProperty
  count: 1
  custom_regexp: "[^\\d][\\._]((\\d+\\.)+\\d+)"

• DeleteDockerImageIfNotContainedInProperties(docker_repo='docker-local', properties_prefix='my-prop', image_prefix=None, full_docker_repo_name=None) - Remove Docker image, if it is not found in the properties of the artifact repository.

• DeleteDockerImageIfNotContainedInPropertiesValue(docker_repo='docker-local', properties_prefix='my-prop', image_prefix=None, full_docker_repo_name=None) - Remove Docker image, if it is not found in the properties of the artifact repository.

Filters

• IncludePath - Apply to artifacts by path / mask.

- rule: IncludePath
  masks: "*production*"
- rule: IncludePath
  masks:
   - "*production*"
   - "*develop*"

• IncludeFilename - Apply to artifacts by name/mask

- rule: IncludeFilename
  masks:
   - "*production*"
   - "*develop*"

• ExcludePath - Exclude artifacts by path/mask

- rule: ExcludePath
  masks:
   - "*production*"
   - "*develop*"

• ExcludeFilename - Exclude artifacts by name/mask

- rule: ExcludeFilename
  masks:
    - "*.tag.gz"
    - "*.zip"

Create your own rule

If you want to create your own rule, you can do it!

The basic flow how the tool calls Rules:

1. Rule.check(*args, **kwargs) - verify that the Rule configured right. Call other services to get more information.
2. Rule.aql_add_filter(filters) - add Artifactory Query Language expressions
3. Rule.aql_add_text(aql) - add text to the result aql query
4. artifactory-cleanup calls Artifactory with AQL and pass the result to the next step
5. Rule.filter(artifacts) - filter out artifacts. The method returns artifacts that will be removed!.
   • To keep artifacts use artifacts.keep(artifact) method

Create myrule.py file at the same folder as artifactory-cleanup.yaml:

# myrule.py
from typing import List

from artifactory_cleanup import register
from artifactory_cleanup.rules import Rule, ArtifactsList


class MySimpleRule(Rule):
    """
    This doc string is used as rule title

    For more methods look at Rule source code
    """

    def __init__(self, my_param: str, value: int):
        self.my_param = my_param
        self.value = value

    def aql_add_filter(self, filters: List) -> List:
        print(f"Today is {self.today}")
        print(self.my_param)
        print(self.value)
        return filters

    def filter(self, artifacts: ArtifactsList) -> ArtifactsList:
        """I'm here just to print the list"""
        print(self.my_param)
        print(self.value)
        # You can make requests to artifactory by using self.session:
        # url = f"/api/storage/{self.repo}"
        # r = self.session.get(url)
        # r.raise_for_status()
        return artifacts


# Register your rule in the system
register(MySimpleRule)

Use rule: MySimpleRule in configuration:

# artifactory-cleanup.yaml
- rule: MySimpleRule
  my_param: "Hello, world!"
  value: 42

Specify --load-rules to the command:

# docker
docker run -v "$(pwd)":/app devopshq/artifactory-cleanup artifactory-cleanup --load-rules=myrule.py

# cli
artifactory-cleanup --load-rules=myrule.py

How to

How to connect self-signed certificates for docker?

In case you have set up your Artifactory self-signed certificates, place all certificates of the chain of trust into the certificates folder and add additional argument to the command:

docker run -v "$(pwd)":/app -v "$(pwd)/certificates":/mnt/self-signed-certs/ devopshq/artifactory-cleanup artifactory-cleanup

How to clean up Conan repository?

We can handle conan's metadata by creating two policies:

1. First one removes files but keep all metadata.
2. Second one look at folders and if it contains only medata files - removes it (because there's no associated with metadata files)

The idea came from https://github.com/devopshq/artifactory-cleanup/issues/47

# artifactory-cleanup.yaml
artifactory-cleanup:
  server: https://repo.example.com/artifactory
  user: $ARTIFACTORY_USERNAME
  password: $ARTIFACTORY_PASSWORD

  policies:
    - name: Conan - delete files older than 60 days
      rules:
        - rule: Repo
          name: "conan-testing"
        - rule: DeleteNotUsedSince
          days: 60
        - rule: ExcludeFilename
          masks:
            - ".timestamp"
            - "index.json"
    - name: Conan - delete empty folders (to fix the index)
      rules:
        - rule: Repo
          name: "conan-testing"
        - rule: DeleteEmptyFolders
        - rule: ExcludeFilename
          masks:
            - ".timestamp"
            - "index.json"

How to keep latest N docker images?

We can combine docker rules with usual "files" rules!

The idea came from https://github.com/devopshq/artifactory-cleanup/issues/61

# artifactory-cleanup.yaml
artifactory-cleanup:
  server: https://repo.example.com/artifactory
  user: $ARTIFACTORY_USERNAME
  password: $ARTIFACTORY_PASSWORD

  policies:
    - name: Remove docker images, but keep last 3
      rules:
        # Select repo
        - rule: Repo
          name: docker-demo
        # Delete docker images older than 30 days
        - rule: DeleteDockerImagesOlderThan
          days: 30
        # Keep these tags for all images
        - rule: ExcludeDockerImages
          masks:
            - "*:latest"
            - "*:release*"
        # Exclude these docker tags
        - rule: ExcludePath
          masks: "*base-tools*"
        # Keep 3 docker tags for all images
        - rule: KeepLatestNFilesInFolder
          count: 3

Release

In order to provide a new release of artifactory-cleanup, there are two steps involved.

1. Bump the version in the setup.py
2. Bump the version in the init.py
3. Create a Git release tag (in format 1.0.1) by creating a release on GitHub