Neptune Fetcher
Project description
Neptune Fetcher
[!NOTE] This package is experimental.
Neptune Fetcher is a Python package designed to separate data retrieval capabilities from the regular neptune
package. This separation bypasses the need to initialize the heavy structures of the regular package, which makes data fetching more efficient and improves performance.
Installation
pip install neptune-fetcher
Example usage
Fetching data frame containing specified fields as columns
from neptune_fetcher import ReadOnlyProject
project = ReadOnlyProject("workspace/project")
# Fetch all runs with specific columns
runs_df = project.fetch_runs_df(
columns=["sys/name", "sys/modification_time"],
columns_regex="tree/.*", # added to columns specified with the "columns" parameter
)
Fetching data from specified runs
from neptune_fetcher import ReadOnlyProject
project = ReadOnlyProject("workspace/project")
for run in project.fetch_read_only_runs(with_ids=["RUN-1", "RUN-2"]):
run.prefetch(["parameters/optimizer", "parameters/init_lr"])
print(run["parameters/optimizer"].fetch())
print(run["parameters/init_lr"].fetch())
Listing run identifiers
from neptune_fetcher import ReadOnlyProject
project = ReadOnlyProject("workspace/project")
for run in project.list_runs():
print(run)
Fetching data from a single run
from neptune_fetcher import ReadOnlyProject, ReadOnlyRun
project = ReadOnlyProject("workspace/project")
run = ReadOnlyRun(project, with_id="TES-1")
run.prefetch(["parameters/optimizer", "parameters/init_lr"])
print(run["parameters/optimizer"].fetch())
print(run["parameters/init_lr"].fetch())
API reference
ReadOnlyProject
Representation of a Neptune project in a limited read-only mode.
Initialization
Initialize with the ReadOnlyProject class constructor.
Parameters:
Name | Type | Default | Description |
---|---|---|---|
project |
str , optional |
None |
Name of a project in the form workspace-name/project-name . If None , the value of the NEPTUNE_PROJECT environment variable is used. |
api_token |
str , optional |
None |
Your Neptune API token (or a service account's API token). If None , the value of the NEPTUNE_API_TOKEN environment variable is used. To keep your token secure, avoid placing it in source code. Instead, save it as an environment variable. |
proxies |
dict , optional |
None |
Argument passed to HTTP calls made via the Requests library. For details on proxies, see the Requests documentation. |
Example:
project = ReadOnlyProject("workspace/project", api_token="...")
list_runs()
Lists minimal information, like identifier and name, for every run in a project.
Example:
for run in project.list_runs():
print(run)
Returns:
Iterator
of dictionaries with Neptune run identifiers, custom identifiers and names.
fetch_runs()
Fetches a table containing Neptune IDs, custom run IDs and names of runs in the project.
Example:
df = project.fetch_runs()
Returns:
pandas.DataFrame
with three columns (sys/id
, sys/name
and sys/custom_run_id
) and rows corresponding to project runs.
fetch_runs_df()
Fetches the runs' metadata and returns them as a pandas DataFrame.
Parameters:
Name | Type | Default | Description |
---|---|---|---|
columns |
List[str] , optional |
None |
Names of columns to include in the table, as a list of field names. The Neptune ID ("sys/id" ) is included automatically. If None , all the columns of the experiments table are included. |
columns_regex |
str , optional |
None |
A regex pattern to filter columns by name. Use this parameter to include columns in addition to the ones specified by the columns parameter. |
names_regex |
str , optional |
None |
A regex pattern to filter the runs by name. When applied, it needs to limit the number of runs to 100 or fewer. |
custom_id_regex |
str , optional |
None |
A regex pattern to filter the runs by custom ID. When applied, it needs to limit the number of runs to 100 or fewer. |
with_ids |
List[str] , optional |
None |
List of multiple Neptune IDs. Example: ["NLU-1", "NLU-2"] . Matching any element of the list is sufficient to pass the criterion. |
custom_ids |
List[str] , optional |
None |
List of multiple Custom IDs. Example: ["nostalgic_shockley", "high_albattani"] . Matching any element of the list is sufficient to pass the criterion. |
states |
List[str] , optional |
None |
List of states. Possible values: "inactive" , "active" . "Active" means that at least one process is connected to the run. Matching any element of the list is sufficient to pass the criterion. |
owners |
List[str] , optional |
None |
List of multiple owners. Example: ["frederic", "josh"] . The owner is the user who created the run. Matching any element of the list is sufficient to pass the criterion. |
tags |
List[str] , optional |
None |
A list of tags. Example: "lightGBM" or ["pytorch", "cycleLR"] . Note: Only runs that have all specified tags will pass this criterion. |
trashed |
bool , optional |
False |
Whether to retrieve trashed runs. If True , only trashed runs are retrieved. If False , only non-trashed runs are retrieved. If None or left empty, all run objects are retrieved, including trashed ones. |
limit |
int , optional |
None |
Maximum number of runs to fetch. If None , all runs are fetched. |
sort_by |
str , optional |
sys/creation_time |
Name of the field to sort the results by. The field must represent a simple type (string, float, integer). |
ascending |
bool , optional |
False |
Whether to sort the entries in ascending order of the sorting column values. |
progress_bar |
bool , Type[ProgressBarCallback] , optional |
None |
Set to False to disable the download progress bar, or pass a type of ProgressBarCallback to use your own progress bar. If set to None or True , the default tqdm-based progress bar will be used. |
Example:
# Fetch all runs with specific columns
runs_df = project.fetch_runs_df(
columns=["sys/name", "sys/modification_time", "training/lr"]
)
# Fetch all runs with specific columns and extra columns that match a regex pattern
runs_df = project.fetch_runs_df(
columns=["sys/name", "sys/modification_time"],
columns_regex="tree/.*",
)
# Fetch runs by specific IDs
specific_runs_df = my_project.fetch_runs_df(
custom_ids=["nostalgic_shockley", "high_albattani"]
)
# Filter by name regex
specific_runs_df = my_project.fetch_runs_df(
names_regex="tree_3[2-4]+"
)
Returns:
pandas.DataFrame
: A pandas DataFrame containing metadata of the fetched runs.
fetch_read_only_runs()
List runs of the project in the form of ReadOnlyRun.
Parameters:
Name | Type | Default | Description |
---|---|---|---|
with_ids |
Optional[List[str]] |
None |
List of Neptune run IDs to fetch. |
custom_ids |
Optional[List[str]] |
None |
List of custom run IDs to fetch. |
Example:
for run in project.fetch_read_only_runs(custom_ids=["my-run-aa-1", "my-run-bb-2"]):
...
Returns: Iterator of ReadOnlyRun objects.
ReadOnlyRun
Representation of a Neptune run in a limited read-only mode.
Initialization
Can be created with the class constructor, or as a result of the fetch_read_only_runs()
method of the ReadOnlyProject class.
Parameters:
Name | Type | Default | Description |
---|---|---|---|
read_only_project |
ReadOnlyProject |
- | Source project from which run will be fetched. |
with_id |
Optional[str] |
None |
Neptune run ID to fetch. Example: RUN-1 . Exclusive with custom_id |
custom_id |
Optional[str] |
None |
Neptune Custom IDs to fetch. Example: my-run-aa-1 . Exclusive with with_id |
Example:
from neptune_fetcher import ReadOnlyProject, ReadOnlyRun
project = ReadOnlyProject("workspace/project", api_token="...")
run = ReadOnlyRun(project, with_id="TES-1")
.field_names
List of run field names.
Example:
for run in project.fetch_read_only_runs(with_ids=["TES-1", "TES-2"]):
print(list(run.field_names))
Returns: Iterator of run fields as strings.
Field lookup: run[field_name]
Used to access a specific field of a run. See Available types.
Returns: An internal object used to operate on a specific field.
Example:
custom_id = run["sys/custom_run_id"].fetch()
prefetch()
Pre-fetches a batch of fields to the internal cache.
Improves the performance of access to consecutive field values. Only simple field types are supported (int
, float
, str
).
Parameters:
Name | Type | Default | Description |
---|---|---|---|
paths |
List[str] |
- | List of paths to fetch to the cache. |
Example:
run.prefetch(["parameters/optimizer", "parameter/init_lr"])
# No more calls to the API
print(run["parameters/optimizer"].fetch())
print(run["parameter/init_lr"].fetch())
Available types
The following sections list the currently supported field types, along with their available data retrieval operations.
Integer
fetch()
Retrieves value either from the internal cache (see prefetch()
) or from the API.
Example:
batch_size = run["batch_size"].fetch()
Returns:
int
Float
fetch()
Retrieves value either from the internal cache (see prefetch()
) or from the API.
Example:
f1 = run["scores/f1"].fetch()
Returns:
float
String
fetch()
Retrieves value either from the internal cache (see prefetch()
) or from the API.
Example:
token = run["token"].fetch()
Returns:
str
Datetime
fetch()
Retrieves value either from the internal cache (see prefetch()
) or from the API.
Example:
created_at = run["sys/creation_time"].fetch()
Returns:
datetime.datetime
Object state
fetch()
Retrieves value either from the internal cache (see prefetch()
) or from the API.
Example:
state = run["sys/state"].fetch()
Returns:
str
Boolean
fetch()
Retrieves value either from the internal cache (see prefetch()
) or from the API.
Example:
status = run["sys/failed"].fetch()
Returns:
bool
Float series
fetch()
or fetch_last()
Retrieves last series value either from the internal cache (see prefetch()
) or from the API.
Example:
loss = run["loss"].fetch_last()
Returns:
Optional[float]
fetch_values()
Retrieves all series values from the API.
Parameters:
Name | Type | Default | Description |
---|---|---|---|
include_timestamp |
bool |
True | Whether the fetched data should include the timestamp field. |
Example:
values = run["loss"].fetch_values()
Returns:
pandas.DataFrame
License
This project is licensed under the Apache License Version 2.0. For more details, see Apache License Version 2.0.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for neptune_fetcher-0.4.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7733b147859f59407354dc588c22debd0bfe2a32920dbe61de2c30444133bd6c |
|
MD5 | ac7dd0c2ffd5a3bf1d3238d7217dee09 |
|
BLAKE2b-256 | 450db952d355ec33c78992e77444066afbafd578c975bee0b85da12306210e4c |