ndexutil 0.14.0

Unsupported NDEx utilities

Warning: This repository is for development and features may change. Please use this at your own risk.

Dependencies

• ndex2

• networkx

• biothings_client

• requests

• requests-toolbelt

• pandas

• mygene

• jsonschema

• urllib3

• ijson

Optional Dependencies

Installed if pip install ndexutil[cytoscape] is run

• py4cytoscape

Compatibility

• Python 3.4+

Installation

git clone https://github.com/ndexbio/ndexutils
cd ndexutils
make dist
pip install dist/ndexutil*whl

OR via PyPI

pip install ndexutil

ndexmisctools.py

WARNING: Please consider this tool alpha and probably contains errors/bugs and could cause data corruption or loss. You have been warned

ndexmisctools.py lets caller perform some operations on NDEx via the command line. This tool follows format of ndexmisctools.py <COMMAND> <args>

For more information run ndexmisctools.py --help and ndexmisctools.py <COMMAND> --help

COMMANDS:

• addnodeattrib - adds an attribute to all nodes in a network

  Credentials to NDEx server must be stored in the configuration (default ~/.ndexutils.conf) and be formatted as follows:

  [addprofile]
  user=bob
  password=6ea8f0ab0b2e
  server=public.ndexbio.org

  The following command adds foo node attribute with value my new value to all nodes in the network 9025480b-6fbc-4efe-9cd8-b575ce49dfda

  ndexmisctools.py --profile addprofile addnodeattrib --uuid 9025480b-6fbc-4efe-9cd8-b575ce49dfda --name foo --value 'my new value'

  Warning

  This command changes the network in place, so it is a destructive call with NO undo

• copynetwork - copies NDEx network between accounts and even servers

  For copying the source and destination credentials must be stored in the configuration (default ~/.ndexutils.conf) and be formatted as follows:

  [mycopyprofile]
  source_user=bob
  source_password=6ea8f0ab0b2e
  source_server=public.ndexbio.org
  dest_user=smith
  dest_password=4efe9cd8
  dest_server=public.ndexbio.org

  The following command copies the network 9025480b-6fbc-4efe-9cd8-b575ce49dfda from source credentials defined in configuration to dest

  ndexmisctools.py --profile mycopyprofile copynetwork --uuid 9025480b-6fbc-4efe-9cd8-b575ce49dfda

• cytoscapelayout - updates layout for network in NDEx using Cytoscape

  NOTE: This tool requires installation of py4cytoscape

  and Cytoscape to be activately running (tested with Cytsocape 3.8+)

  This command requires four positional parameters. The first is the layout algorithm to use followed by (username, password, and server) which are credentials for NDEx server to update the layout on the network.

  Any of these credential fields set to ‘-’ will force this tool to obtain the information from (default ~/.ndexutils.conf) configuration file under the profile specified by the --profile field in this format:

  [<value of --profile>]
  user = <NDEx username>
  password = <NDEx password>
  server = <NDEx server ie public.ndexbio.org>

  Using credentials from myattrib profile, the following command adds grid layout to the network 9025480b-6fbc-4efe-9cd8-b575ce49dfda

  ndexmisctools.py --profile myattrib networkxlayout grid - - - --uuid 9025480b-6fbc-4efe-9cd8-b575ce49dfda

  To get a list of available layouts printed to standard out, replace the layout with listlayout. Just put fake values in for credentials and any text for the –uuid flag:

  ndexmisctools.py cytoscapelayout listlayout x y z --uuid ignored

• deletenetwork - deletes a network or all networks under a given networkset from NDEx

  Credentials to NDEx server must be stored in the configuration (default ~/.ndexutils.conf) and be formatted as follows:

  [deleteprofile]
  user=bob
  password=6ea8f0ab0b2e
  server=public.ndexbio.org

  The following command deletes the network with UUID 9025480b-6fbc-4efe-9cd8-b575ce49dfda from NDEx

  ndexmisctools.py --profile deleteprofile deletenetwork --uuid 9025480b-6fbc-4efe-9cd8-b575ce49dfda

• uploadnetwork - uploads network to NDEx

  This command uploads a network to NDEx

  Credentials must be stored in the configuration (default ~/.ndexutils.conf) and be formatted as follows:

  [myattrib]
  user=bob
  password=6ea8f0ab0b2e
  server=public.ndexbio.org

  The following command uploads foo.cx network to NDEx

  ndexmisctools.py --profile myattrib uploadnetwork foo.cx

• featurednetworkreport - generates report about featurend networks on NDEx

  This command requires one positional parameter server. For production use public.ndexbio.org

  If the positional parameter is set to ‘-’, it will force this tool to obtain the information from (default ~/.ndexutils.conf) configuration file under the profile specified by the --profile field in this format:

  [<value of --profile>]
  server = <NDEx server ie public.ndexbio.org>

  The command below queries the production server for the featured networks json file and then queries NDEx for all networks that are part of the featured networks.

  ndexmisctools.py featurednetworkreport public.ndexbio.org --output foo.csv

  NOTE: If a networkset contains more then 500 networks, only the first 500 are examined

• networkattribupdate - updates network attributes on network in NDEx

  WARNING: Currently name, version, and description CANNOT be updated with this command.

  Credentials must be stored in the configuration (default ~/.ndexutils.conf) and be formatted as follows:

  [myattrib]
  user=bob
  password=6ea8f0ab0b2e
  server=public.ndexbio.org

  The following command updates foo network attribute on the network 9025480b-6fbc-4efe-9cd8-b575ce49dfda

  ndexmisctools.py --profile myattrib networkattribupdate --uuid 9025480b-6fbc-4efe-9cd8-b575ce49dfda --name foo --type string --value 'my new value'

• networkxlayout - updates layout for network in NDEx using networkx library

  This command requires four positional parameters. The first is the layout algorithm to use followed by (username, password, and server) which are credentials for NDEx server to update the layout on the network.

  Any of these credential fields set to ‘-’ will force this tool to obtain the information from (default ~/.ndexutils.conf) configuration file under the profile specified by the --profile field in this format:

  [<value of --profile>]
  user = <NDEx username>
  password = <NDEx password>
  server = <NDEx server ie public.ndexbio.org>

  By default this tool only updates the cartesianLayout aspect. To update the full network use add the --updatefullnetwork flag.

  Using credentials from myattrib profile, the following command adds spring layout to the network 9025480b-6fbc-4efe-9cd8-b575ce49dfda

  ndexmisctools.py --profile myattrib networkxlayout spring - - - --uuid 9025480b-6fbc-4efe-9cd8-b575ce49dfda

• removenodeattrib - removes an attribute from all nodes in a network

  Credentials to NDEx server must be stored in the configuration (default ~/.ndexutils.conf) and be formatted as follows:

  [removeprofile]
  user=bob
  password=6ea8f0ab0b2e
  server=public.ndexbio.org

  The following command removes foo node attribute from all nodes in the network 9025480b-6fbc-4efe-9cd8-b575ce49dfda

  ndexmisctools.py --profile removeprofile removenodeattrib --uuid 9025480b-6fbc-4efe-9cd8-b575ce49dfda --name foo

  Warning

  This command changes the network in place, so it is a destructive call with NO undo

• systemproperty - updates showcase, visibility, and indexing for single network or all networks in networkset in NDEx

  NOTE: --showcase has no effect if network visibility is private

  Credentials must be stored in the configuration (default ~/.ndexutils.conf) and be formatted as follows:

  [myattrib]
  user=bob
  password=6ea8f0ab0b2e
  server=public.ndexbio.org

  The following command enables showcase and sets indexing to meta for network with id 9025480b-6fbc-4efe-9cd8-b575ce49dfda

  ndexmisctools.py --profile myattrib systemproperty --uuid 9025480b-6fbc-4efe-9cd8-b575ce49dfda --showcase --indexlevel meta

  The following command sets visibility to public for all networks in networkset with id e9580d43-ec14-4be8-9977-9de88e1d410a

  ndexmisctools.py --profile myattrib systemproperty --networksetid e9580d43-ec14-4be8-9977-9de88e1d410a --visibility public

• styleupdate - update style of network in NDEx

  Credentials must be stored in the configuration (default ~/.ndexutils.conf) and be formatted as follows:

  [styleprofile]
  user=bob
  password=6ea8f0ab0b2e
  server=public.ndexbio.org

  The following command updates style of network with UUID 9025480b-6fbc-4efe-9cd8-b575ce49dfda with style from network specified by this UUID e9580d43-ec14-4be8-9977-9de88e1d410a

  ndexmisctools.py --profile styleprofile styleupdate --uuid 9025480b-6fbc-4efe-9cd8-b575ce49dfda --styleuuid e9580d43-ec14-4be8-9977-9de88e1d410a

• tsvloader - Loads TSV files as networks into NDEx

  The tsvloader command loads an edge list file in tab separated format (hence TSV) and using a load plan, loads that data as a network into NDEx. This tool attempts to mimic behavior of the older tsv_uploader.py script located here: https://github.com/ndexbio/load-content This new version uses the more memory efficient StreamTSVLoader.

  This command requires five positional parameters. The first three (username, password, and server) are credentials for NDEx server to upload the network.

  Any of these first three credential fields set to ‘-’ will force this tool to obtain the information from (default ~/.ndexutils.conf) configuration file under the profile specified by the --profile field in this format:

  [<value of --profile>]
  user = <NDEx username>
  password = <NDEx password>
  server = <NDEx server ie public.ndexbio.org>

  The forth positional parameter tsv_file (see TSV Loader section below) should be set to edge list file in tab separated format and the fifth or last positional parameter load_plan should be set to the load plan. The load plan is a JSON formatted text file that maps the columns to nodes, edges, and attributes in the network.

  By default this tool does not generate much output to standard out/error. For more verbosity add three to five -v parameters to left of command name tsvloader as seen in examples below.

  NOTE: If using --layout to run a Cytoscape layout, py4cytoscape

  must be installed and Cytoscape to be activately running (tested with Cytsocape 3.8+)

  In example below the - - - tells the program to read the credentials from the configuration file

  ndexmisctools.py -vvvv tsvloader - - - datafile.tsv load.plan

  Here is an example where the name and description of the network is set and the -t specifies a template network used to get style and in this case since --copyattribs is set the network attributes (minus @context) are also copied to the new network

  ndexmisctools.py -vvv tsvloader bob xx public.ndexbio.org \
                   datafile.tsv loadplan.json --uppercaseheader  \
                   -t dafe07ca-0676-11ea-93e0-525400c25d22 \
                   --name mynetwork --description 'some text' --copyattribs

  In this example an alternate header is prepended via the --header flag.

  NOTE: The --header flag does NOT remove an existing header

  ndexmisctools.py -vv --profile foo tsvloader - - public.ndexbio.org \
                   datafile.tsv loadplan.json \
                   --header 'col1   col2    col3' \
                   -t some_cx_file.cx \
                   -u 48a26aa0-0677-11ea-93e0-525400c25d22

  If successful 0 is returned otherwise there was an error.

TSV Loader

This module contains the Tab Separated Variable Loader (TSV Loader, see ndexutil/streamtsvloader.py module) which generates an NDEx CX file from a tab separated text file of edge data and attributes.

To load data a load plan must be created. This plan tells the loader how to map the columns in the file to nodes, and edges. This load plan needs to validate against this load plan JSON schema

Example TSV file

SOURCE  TARGET  WEIGHT
ABCD    AAA1    0.555
GGGG    BBBB    0.305

SOURCE is the source node, TARGET is target node

A schema that could be:

{
"source_plan":
    {
        "node_name_column": "SOURCE"
    },
    "target_plan":
    {
        "node_name_column": "TARGET"
    },
    "edge_plan":
    {
        "default_predicate": "unknown",
        "property_columns": [
          {
            "column_name": "WEIGHT",
            "attribute_name": "weight",
            "data_type": "double"
          }
        ]
    }
}

Example below assumes the following:

• ./loadplan.json is the load plan in JSON format

• ./style.cx is a NDEx CX with a style.

import ndex2
from ndexutil.tsv.streamtsvloader import StreamTSVLoader

# using ndex2 client library read CX file as NiceCXNetwork object
style_network = ndex2.create_nice_cx_from_file('./style.cx')

loader = StreamTSVLoader('./loadplan.json', style_network)
with open('./input.tsv', 'r') as tsvfile:
    with open('./output.cx', 'w') as outfile:
        loader.write_cx_network(tsvfile, outfile)

Credits

History

0.14.0 (2022-09-03)

• tqdm and ijson are now required packages

• Added --indexlevel, --showcase, --visibility, and --destnetworkset to ndexmisctools.py copynetwork command to allow caller to set copied networks accessibility and to add copied network to a networkset. This command also stores the original NDEx network UUID in __copynewtork_orig_ndex_uuid network attribute

• Added new command to ndexmisctools.py named uploadnetwork that lets caller upload networks to NDEx

• Added following methods to ndexutil/ndex.py: get_network_summary(), get_nice_cx_from_server(), set_index_and_showcase(), save_network_to_ndex()

0.13.1 (2020-11-11)

• Fixed bug where --styleuuid was failing to update network in ndexmisctools.py styleupdate command

0.13.0 (2020-11-03)

• Added Cytoscape layouts to ndexmisctools.py tsvloader via --layout flag

• Passing - for layout to ndexmisctools.py cytoscapelayout will run force-directed-cl layout in Cytoscape

• ndexmisctools.py cytoscapelayout by default updates only the layout aspect on network in NDEx. This can be changed to perform a full network update with new --updatefullnetwork flag

0.12.0 (2020-10-30)

• Added --layout flag to ndexmisctools.py tsvloader command to enable layouts to be applied to network generated by loader

0.11.0 2020-10-26

• Added new command to ndexmisctools.py named networkxlayout that lets caller apply networkx layouts to network on NDEx

• Added new command to ndexmisctools.py named cytoscapelayout that lets caller apply Cytoscape layouts to network on NDEx

• Added new command to ndexmisctools.py named featurednetworkreport that generates a report about featured networks on NDEx

0.10.0 2020-02-19

• Added new command to ndexmisctools.py named addnodeattrib that lets caller add an attribute to all nodes in a network

• Added new command to ndexmisctools.py named removenodeattrib that lets caller remove an attribute from all nodes in a network

0.9.0.post1 2019-12-11

• Updated this history file adding a note about addition of styleupdate command to ndexmisctools.py in 0.9.0 release

0.9.0 2019-12-11

• Added –typeonly flag to networkattribupdate command in ndexmisctools.py that lets caller just update the data type for a given network attribute

• Added new command to ndexmisctools.py named deletenetwork that lets caller delete a network or delete all networks in a given networkset from NDEx

• Added new command to ndexmisctools.py named styleupdate that lets caller update the style of a network or all networks in a given set on NDEx

0.8.1 2019-11-15

• Fixed bug where ndexmisctools.py failed to run due to argparse bug in Python versions below 3.7

0.8.0 2019-11-15

• Fixed bug in StreamTSVLoader where 2 @context network attributes were added to network if @context was set in load plan and @context was passed in the network attributes parameter for write_cx_network()

• Added new command to ndexmisctools.py named tsvloader that lets caller load TSV files as networks into NDEx

0.7.0 2019-09-11

• Added new command to ndexmisctools.py named systemproperty that lets caller update showcase, visibility, and indexing for a single network or all networks in a networkset in NDEx

0.6.1 2019-07-12

• Fixed bug where ndexmisctools.py networkattributeupdate was creating duplicate network attributes for name, description, and value. This is due to server bug. To deal with this the code removes those entries when doing the network attribute update and forbids caller from trying to update those attributes

0.6.0 2019-07-10

• Added new commandline utility ndexmisctools.py which lets caller copy a network from one NDEx account to another and lets one update network attributes of a network in NDEx. WARNING: THIS IS AN UNTESTED ALPHA RELEASE AND MAY CONTAIN ERRORS

0.5.0 2019-06-06

• Added GeneSymbolSearcher class to loaderutils.py module

• Minor bug fix in NetworkIssueReport get_fullreport_as_string() where issue text is wrapped in str() in case its not a string

0.4.0 2019-05-23

• Added loaderutils.py module with two new classes, NetworkIssueReport and an abstract class NetworkUpdator

0.3.0 2019-05-14

• tsv2nicecx2.convert_pandas_to_nice_cx_with_load_plan now loads @context data into a @context network attribute instead of a separate aspect

0.2.0 2019-04-01

• StreamTSVLoader class added which enables loading of TSV data into in streaming format to handle loading of large networks