Skip to main content

An example cloudmesh command

Project description

Cloudmesh Catalog

Cloudmesh catalog can be used to store information about a service, software component, or project. The information included in it can be categorized so that a comparision is possible. The catalog is implemented as REST service so it can be integrated in other projects and searched programatically.

The catalog depends on the cloudmesh command shell which allows eay integration of new commands line environment. It projects a sample Interface for the catalog from the commandline

We also can create static web pages from the catalog while using the export feature and integrating the pages in for example hugo.

We are currently exploring hugo docsy as it provides an easy way to generate hirarchical web pages, but also leverages hugos tags and categories. Other export formats include markdown and bibtex.

Instalation for developers

  1. If you do not have yet create an ssh key and upload it to the github repository.

    ssh-keygen

    Upload the ~/.ssh/id_rsa.pub key to github

  2. Download cloudmesh with its source repositories

    Make sure you ave python 3.10.2

    On Mac or Linux do

    $ python3.10 -m venv ~/ENV3
    $ source ~/ENV3/bin/activate
    

    On Windows

    $ py --version # make sure its 3.10.2
    $ py -m venv ~/ENV3
    $ source ~/ENV3/bin/activate
    

    After that the instalation is the same on all operating systems.

    $ mkdir cm
    $ cd cm
    
    $ pip install cloudmesh-installer
    $ cloudmesh-installer -ssh install catalog
    $ cms help
    

    This will download all source code for the cloudmesh shell and compile from source.

  3. Now you are all ready to do programming and enhancing cloudmesh-catalog If you have any issues, contact laszewski@gmail.com

Manual page

A manual pasge shoudl be implemented in cloudmesh-catalog/catalog/command/catalog.py This manual page can be displayed with the following command:

$ cms help catalog help

To just see the usage type in

$ cms catalog

Managing the Service

TODO: The integration of data into the service is not yet completed.

TODO: service management on Windows is not yet completed.

On Linux and macOS we can already experiment with an early prototype that allows us starting, sopping, and getting the status of the service. This service has nnot yet been integrated with a database.

BUG

TODO: The adat is not yet integrated and we like to use cloudmesh/yamldb for it.

Adding catalog and registry data

TODO: To add catalog and registry data for new services, one must create new .yaml files in the appropriate folders: 'data/catalog/my_example.yaml' and 'data/registry/my_example.yaml'. Each file must follow yaml formatting similar to the following example.

Example file: Amazon Comprehend (Catalog), amazon_comprehend.yaml

---
id: amazon_comprehend:
name: Amazon Comprehend
title: Amazon Comprehend
author: Amazon
slug: amazon-comprehend
public: true
description: |
 Comprehend is Amazon's solution for cloud-based NLP.
 It is available with an AWS account. To use,
 it requires use of either the AWS Command Line
 Interface or an AWS SDK for Python, Java, or .NET.
 Notable features include functionality for giving
 batches of documents to be processed as well as
 submission of multiple jobs in a list. The DetectEntities
 function also allows use of a custom-trained
 model, but many other functions do not.
version: unknown
license: unknown
microservice: no
protocol: AWS API
owner: Amazon Web Services
modified: 9/29/2021
created: 11/29/2017
documentation: https://docs.aws.amazon.com/comprehend/index.html
source: unknown
specification: unknown
tags: ["nlp", "nlp service", "machine learning", "cloud service", "nlp api",
     "deep learning", "natural language processing", "artificial intelligence"]
categories: ["NLP"]
additional_metadata: unknown
endpoint: unknown
sla: https://aws.amazon.com/machine-learning/language/sla/
authors: The AWS team can be contacted through support ticket at https://aws.amazon.com/contact-us/
data: |
 User data is stored on Amazon servers under the associated AWS account and is protected under the AWS
 shared responsibility model as detailed here https://aws.amazon.com/compliance/shared-responsibility-model/

Using the Catalog and Registry classes

Written in catalog.py and registry.py are classes capable of reading and storing the data written in the .yaml files. Both use the same interface. Here is an example of the Catalog class in action:

# initialize the catalog using data found in the given directory
catalog = Catalog('data/catalog/')
# query the catalog for Amazon Comprehend data, save result to amazon_catalog_data
amazon_catalog_data = cat.query({'name': 'Amazon Comprehend'})
# add a new data file to the catalog
catalog.add('new_example/azure_language.yaml')
# save entire catalog to a pickle file
catalog.to_pickle('catalog.pkl')
# load from pickle file
catalog.from_pickle('catalog.pkl')
# print catalog data
print(catalog.data)

Using the yaml to markdown conversion script

The catalog command includes several prototype export formats that takes all files recursively in a directory or an explicit file and converts it to the specified output

This includes

cms catalog export bibtex --souce=SOURCE
cms catalog export hugo --souce=SOURCE
cms catalog export md --souce=SOURCE

The commands will create next to the yal file entreies for bibtex, hugo markdown, and markdown.

The templates are just suggestions and we may improve them based on our findings.

Checking entries

It is very important that any entry be checked for minimal yaml complience. Hence we implemented a command

cms catalog check --souce=SOURCE

which will check all file sin the specified directory. THIs check will ignore line legth limits if the line contains an http or https refernce. We also check the data format for YYYY-MM-DD.

We know that it may be problematic to distingush automatically between YYYY-MM-DD and YYYY-DD-MM. Hence we encourage you to be careful when adding entries.

Developer Video Tutorials

We are providing a number of developer video tutorials that help undesrtanding how we develop code and leverage the cloudmesh-cmd5 shell features:

Other videos are available at


Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cloudmesh-catalog-5.0.2.tar.gz (24.6 kB view details)

Uploaded Source

Built Distribution

cloudmesh_catalog-5.0.2-py2.py3-none-any.whl (22.7 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file cloudmesh-catalog-5.0.2.tar.gz.

File metadata

  • Download URL: cloudmesh-catalog-5.0.2.tar.gz
  • Upload date:
  • Size: 24.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.12.1

File hashes

Hashes for cloudmesh-catalog-5.0.2.tar.gz
Algorithm Hash digest
SHA256 1ea8891a5e00079959930d7833b429a0fe3b25720708840e1e299b2f5796ed04
MD5 2ea548fd06d3ee9c5fe289034d1ad553
BLAKE2b-256 2ec36a70a122cf9825e8b3e533b1072c5e24fcd336b3ceb625f96f65778a6128

See more details on using hashes here.

File details

Details for the file cloudmesh_catalog-5.0.2-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for cloudmesh_catalog-5.0.2-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 12ffe701a66ad056970f151a8115d099260150179445c35b18e429c4375dc2df
MD5 543bb685d4043b4fe2dbf81630717557
BLAKE2b-256 b7a0188e99b62ea80c83a9207f0dbf2c942fceb1bd6b5478104fd08d5e1da10f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page