carbon-python-sdk 0.2.40

`carbon.auth.get_white_labeling`

Returns whether or not the organization is white labeled and which integrations are white labeled

:param current_user: the current user :param db: the database session :return: a WhiteLabelingResponse

🛠️ Usage

get_white_labeling_response = carbon.auth.get_white_labeling()

🔄 Return

WhiteLabelingResponse

🌐 Endpoint

/auth/v1/white_labeling get

`carbon.crm.get_account`

Get Account

🛠️ Usage

get_account_response = carbon.crm.get_account(
    id="id_example",
    data_source_id=1,
    include_remote_data=False,
    includes=["string_example"],
)

⚙️ Parameters

id: `str`

data_source_id: `int`

include_remote_data: `bool`

includes: List[`BaseIncludes`]

🔄 Return

Account

🌐 Endpoint

/integrations/data/crm/accounts/{id} get

`carbon.crm.get_accounts`

Get Accounts

🛠️ Usage

get_accounts_response = carbon.crm.get_accounts(
    data_source_id=1,
    include_remote_data=False,
    next_cursor="string_example",
    page_size=1,
    order_dir="asc",
    includes=[],
    filters={},
    order_by="created_at",
)

⚙️ Parameters

data_source_id: `int`

include_remote_data: `bool`

next_cursor: `Optional[str]`

page_size: `Optional[int]`

order_dir: `OrderDirV2Nullable`

includes: List[`BaseIncludes`]

order_by: `AccountsOrderByNullable`

⚙️ Request Body

AccountsRequest

🔄 Return

AccountResponse

🌐 Endpoint

/integrations/data/crm/accounts post

`carbon.crm.get_contact`

Get Contact

🛠️ Usage

get_contact_response = carbon.crm.get_contact(
    id="id_example",
    data_source_id=1,
    include_remote_data=False,
    includes=["string_example"],
)

⚙️ Parameters

id: `str`

data_source_id: `int`

include_remote_data: `bool`

includes: List[`BaseIncludes`]

🔄 Return

Contact

🌐 Endpoint

/integrations/data/crm/contacts/{id} get

`carbon.crm.get_contacts`

Get Contacts

🛠️ Usage

get_contacts_response = carbon.crm.get_contacts(
    data_source_id=1,
    include_remote_data=False,
    next_cursor="string_example",
    page_size=1,
    order_dir="asc",
    includes=[],
    filters={},
    order_by="created_at",
)

⚙️ Parameters

data_source_id: `int`

include_remote_data: `bool`

next_cursor: `Optional[str]`

page_size: `Optional[int]`

order_dir: `OrderDirV2Nullable`

includes: List[`BaseIncludes`]

order_by: `ContactsOrderByNullable`

⚙️ Request Body

ContactsRequest

🔄 Return

ContactsResponse

🌐 Endpoint

/integrations/data/crm/contacts post

`carbon.crm.get_lead`

Get Lead

🛠️ Usage

get_lead_response = carbon.crm.get_lead(
    id="id_example",
    data_source_id=1,
    include_remote_data=False,
    includes=["string_example"],
)

⚙️ Parameters

id: `str`

data_source_id: `int`

include_remote_data: `bool`

includes: List[`BaseIncludes`]

🔄 Return

Lead

🌐 Endpoint

/integrations/data/crm/leads/{id} get

`carbon.crm.get_leads`

Get Leads

🛠️ Usage

get_leads_response = carbon.crm.get_leads(
    data_source_id=1,
    include_remote_data=False,
    next_cursor="string_example",
    page_size=1,
    order_dir="asc",
    includes=[],
    filters={},
    order_by="created_at",
)

⚙️ Parameters

data_source_id: `int`

include_remote_data: `bool`

next_cursor: `Optional[str]`

page_size: `Optional[int]`

order_dir: `OrderDirV2Nullable`

includes: List[`BaseIncludes`]

order_by: `LeadsOrderByNullable`

⚙️ Request Body

LeadsRequest

🔄 Return

LeadsResponse

🌐 Endpoint

/integrations/data/crm/leads post

`carbon.crm.get_opportunities`

Get Opportunities

🛠️ Usage

get_opportunities_response = carbon.crm.get_opportunities(
    data_source_id=1,
    include_remote_data=False,
    next_cursor="string_example",
    page_size=1,
    order_dir="asc",
    includes=[],
    filters={
        "status": "WON",
    },
    order_by="created_at",
)

⚙️ Parameters

data_source_id: `int`

include_remote_data: `bool`

next_cursor: `Optional[str]`

page_size: `Optional[int]`

order_dir: `OrderDirV2Nullable`

includes: List[`BaseIncludes`]

order_by: `OpportunitiesOrderByNullable`

⚙️ Request Body

OpportunitiesRequest

🔄 Return

OpportunitiesResponse

🌐 Endpoint

/integrations/data/crm/opportunities post

`carbon.crm.get_opportunity`

Get Opportunity

🛠️ Usage

get_opportunity_response = carbon.crm.get_opportunity(
    id="id_example",
    data_source_id=1,
    include_remote_data=False,
    includes=["string_example"],
)

⚙️ Parameters

id: `str`

data_source_id: `int`

include_remote_data: `bool`

includes: List[`BaseIncludes`]

🔄 Return

Opportunity

🌐 Endpoint

/integrations/data/crm/opportunities/{id} get

OrganizationUserDataSourceQueryInput

`carbon.data_sources.query_user_data_sources`

User Data Sources

🛠️ Usage

query_user_data_sources_response = carbon.data_sources.query_user_data_sources(
    pagination={
        "limit": 10,
        "offset": 0,
    },
    order_by="created_at",
    order_dir="desc",
    filters={
        "source": "GOOGLE_CLOUD_STORAGE",
    },
)

⚙️ Parameters

order_by: `OrganizationUserDataSourceOrderByColumns`

order_dir: `OrderDir`

filters: `OrganizationUserDataSourceFilters`

⚙️ Request Body

🔄 Return

OrganizationUserDataSourceResponse

🌐 Endpoint

/user_data_sources post

`carbon.data_sources.revoke_access_token`

Revoke Access Token

🛠️ Usage

revoke_access_token_response = carbon.data_sources.revoke_access_token(
    data_source_id=1,
)

⚙️ Parameters

data_source_id: `int`

⚙️ Request Body

RevokeAccessTokenInput

🔄 Return

🌐 Endpoint

/revoke_access_token post

GetEmbeddingDocumentsBody

`carbon.embeddings.get_documents`

For pre-filtering documents, using tags_v2 is preferred to using tags (which is now deprecated). If both tags_v2 and tags are specified, tags is ignored. tags_v2 enables building complex filters through the use of "AND", "OR", and negation logic. Take the below input as an example:

{
    "OR": [
        {
            "key": "subject",
            "value": "holy-bible",
            "negate": false
        },
        {
            "key": "person-of-interest",
            "value": "jesus christ",
            "negate": false
        },
        {
            "key": "genre",
            "value": "religion",
            "negate": true
        }
        {
            "AND": [
                {
                    "key": "subject",
                    "value": "tao-te-ching",
                    "negate": false
                },
                {
                    "key": "author",
                    "value": "lao-tzu",
                    "negate": false
                }
            ]
        }
    ]
}

In this case, files will be filtered such that:

"subject" = "holy-bible" OR
"person-of-interest" = "jesus christ" OR
"genre" != "religion" OR
"subject" = "tao-te-ching" AND "author" = "lao-tzu"

Note that the top level of the query must be either an "OR" or "AND" array. Currently, nesting is limited to 3. For tag blocks (those with "key", "value", and "negate" keys), the following typing rules apply:

"key" isn't optional and must be a string
"value" isn't optional and can be any or list[any]
"negate" is optional and must be true or false. If present and true, then the filter block is negated in the resulting query. It is false by default.

When querying embeddings, you can optionally specify the media_type parameter in your request. By default (if not set), it is equal to "TEXT". This means that the query will be performed over files that have been parsed as text (for now, this covers all files except image files). If it is equal to "IMAGE", the query will be performed over image files (for now, .jpg and .png files). You can think of this field as an additional filter on top of any filters set in file_ids and

When hybrid_search is set to true, a combination of keyword search and semantic search are used to rank and select candidate embeddings during information retrieval. By default, these search methods are weighted equally during the ranking process. To adjust the weight (or "importance") of each search method, you can use the hybrid_search_tuning_parameters property. The description for the different tuning parameters are:

weight_a: weight to assign to semantic search
weight_b: weight to assign to keyword search

You must ensure that sum(weight_a, weight_b,..., weight_n) for all n weights is equal to 1. The equality has an error tolerance of 0.001 to account for possible floating point issues.

In order to use hybrid search for a customer across a set of documents, two flags need to be enabled:

Use the /modify_user_configuration endpoint to to enable sparse_vectors for the customer. The payload body for this request is below:

{
  "configuration_key_name": "sparse_vectors",
  "value": {
    "enabled": true
  }
}

Make sure hybrid search is enabled for the documents across which you want to perform the search. For the /uploadfile endpoint, this can be done by setting the following query parameter: generate_sparse_vectors=true

Carbon supports multiple models for use in generating embeddings for files. For images, we support Vertex AI's multimodal model; for text, we support OpenAI's text-embedding-ada-002 and Cohere's embed-multilingual-v3.0. The model can be specified via the embedding_model parameter (in the POST body for /embeddings, and a query parameter in /uploadfile). If no model is supplied, the text-embedding-ada-002 is used by default. When performing embedding queries, embeddings from files that used the specified model will be considered in the query. For example, if files A and B have embeddings generated with OPENAI, and files C and D have embeddings generated with COHERE_MULTILINGUAL_V3, then by default, queries will only consider files A and B. If COHERE_MULTILINGUAL_V3 is specified as the embedding_model in /embeddings, then only files C and D will be considered. Make sure that the set of all files you want considered for a query have embeddings generated via the same model. For now, do not set VERTEX_MULTIMODAL as an embedding_model. This model is used automatically by Carbon when it detects an image file.

🛠️ Usage

get_documents_response = carbon.embeddings.get_documents(
    query="a",
    k=1,
    tags={
        "key": "string_example",
    },
    query_vector=[3.14],
    file_ids=[1],
    parent_file_ids=[1],
    include_all_children=False,
    tags_v2={},
    include_tags=True,
    include_vectors=True,
    include_raw_file=True,
    hybrid_search=True,
    hybrid_search_tuning_parameters={
        "weight_a": 0.5,
        "weight_b": 0.5,
    },
    media_type="TEXT",
    embedding_model="OPENAI",
    include_file_level_metadata=False,
    high_accuracy=False,
    rerank={
        "model": "model_example",
    },
    file_types_at_source=["string_example"],
    exclude_cold_storage_files=False,
)

⚙️ Parameters

query: `str`

Query for which to get related chunks and embeddings.

k: `int`

Number of related chunks to return.

tags: `GetEmbeddingDocumentsBodyTags`

query_vector: `GetEmbeddingDocumentsBodyQueryVector`

file_ids: `GetEmbeddingDocumentsBodyFileIds`

parent_file_ids: `GetEmbeddingDocumentsBodyParentFileIds`

include_all_children: `bool`

Flag to control whether or not to include all children of filtered files in the embedding search.

tags_v2: `Optional[Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]]`

A set of tags to limit the search to. Use this instead of tags, which is deprecated.

include_tags: `Optional[bool]`

Flag to control whether or not to include tags for each chunk in the response.

include_vectors: `Optional[bool]`

Flag to control whether or not to include embedding vectors in the response.

include_raw_file: `Optional[bool]`

Flag to control whether or not to include a signed URL to the raw file containing each chunk in the response.

hybrid_search: `Optional[bool]`

Flag to control whether or not to perform hybrid search.

hybrid_search_tuning_parameters: `HybridSearchTuningParamsNullable`

media_type: `FileContentTypesNullable`

embedding_model: `EmbeddingGeneratorsNullable`

include_file_level_metadata: `Optional[bool]`

Flag to control whether or not to include file-level metadata in the response. This metadata will be included in the content_metadata field of each document along with chunk/embedding level metadata.

high_accuracy: `Optional[bool]`

Flag to control whether or not to perform a high accuracy embedding search. By default, this is set to false. If true, the search may return more accurate results, but may take longer to complete.

rerank: `RerankParamsNullable`

file_types_at_source: `GetEmbeddingDocumentsBodyFileTypesAtSource`

exclude_cold_storage_files: `bool`

Flag to control whether or not to exclude files that are not in hot storage. If set to False, then an error will be returned if any filtered files are in cold storage.

⚙️ Request Body

🔄 Return

DocumentResponseList

🌐 Endpoint

/embeddings post

EmbeddingsAndChunksQueryInput

`carbon.embeddings.get_embeddings_and_chunks`

Retrieve Embeddings And Content

🛠️ Usage

get_embeddings_and_chunks_response = carbon.embeddings.get_embeddings_and_chunks(
    filters={
        "user_file_id": 1,
        "embedding_model": "OPENAI",
    },
    pagination={
        "limit": 10,
        "offset": 0,
    },
    order_by="created_at",
    order_dir="desc",
    include_vectors=False,
)

⚙️ Parameters

filters: `EmbeddingsAndChunksFilters`

pagination: `Pagination`

order_by: `EmbeddingsAndChunksOrderByColumns`

order_dir: `OrderDir`

include_vectors: `bool`

⚙️ Request Body

🔄 Return

EmbeddingsAndChunksResponse

🌐 Endpoint

/text_chunks post

EmbeddingsAndChunksQueryInputV2

`carbon.embeddings.list`

Retrieve Embeddings And Content V2

🛠️ Usage

list_response = carbon.embeddings.list(
    filters={
        "include_all_children": False,
        "non_synced_only": False,
    },
    pagination={
        "limit": 10,
        "offset": 0,
    },
    order_by="created_at",
    order_dir="desc",
    include_vectors=False,
)

⚙️ Parameters

filters: `OrganizationUserFilesToSyncFilters`

pagination: `Pagination`

order_by: `OrganizationUserFilesToSyncOrderByTypes`

order_dir: `OrderDir`

include_vectors: `bool`

⚙️ Request Body

🔄 Return

EmbeddingsAndChunksResponse

🌐 Endpoint

/list_chunks_and_embeddings post

ChunksAndEmbeddingsUploadInput

`carbon.embeddings.upload_chunks_and_embeddings`

Upload Chunks And Embeddings

🛠️ Usage

upload_chunks_and_embeddings_response = carbon.embeddings.upload_chunks_and_embeddings(
    embedding_model="OPENAI",
    chunks_and_embeddings=[
        {
            "file_id": 1,
            "chunks_and_embeddings": [
                {
                    "chunk_number": 1,
                    "chunk": "chunk_example",
                }
            ],
        }
    ],
    overwrite_existing=False,
    chunks_only=False,
    custom_credentials={
        "key": {},
    },
)

⚙️ Parameters

embedding_model: `EmbeddingGenerators`

chunks_and_embeddings: List[`SingleChunksAndEmbeddingsUploadInput`]

overwrite_existing: `bool`

chunks_only: `bool`

custom_credentials: `ChunksAndEmbeddingsUploadInputCustomCredentials`

⚙️ Request Body

🔄 Return

🌐 Endpoint

/upload_chunks_and_embeddings post

OrganizationUserFileTagCreate

`carbon.files.create_user_file_tags`

A tag is a key-value pair that can be added to a file. This pair can then be used for searches (e.g. embedding searches) in order to narrow down the scope of the search. A file can have any number of tags. The following are reserved keys that cannot be used:

db_embedding_id
organization_id
user_id
organization_user_file_id

Carbon currently supports two data types for tag values - string and list<string>. Keys can only be string. If values other than string and list<string> are used, they're automatically converted to strings (e.g. 4 will become "4").

🛠️ Usage

create_user_file_tags_response = carbon.files.create_user_file_tags(
    tags={
        "key": "string_example",
    },
    organization_user_file_id=1,
)

⚙️ Parameters

tags: `OrganizationUserFileTagCreateTags`

organization_user_file_id: `int`

⚙️ Request Body

🔄 Return

🌐 Endpoint

/create_user_file_tags post

`carbon.files.delete`

Deprecated

Delete File Endpoint

🛠️ Usage

delete_response = carbon.files.delete(
    file_id=1,
)

⚙️ Parameters

file_id: `int`

🔄 Return

🌐 Endpoint

/deletefile/{file_id} delete

OrganizationUserFileTagsRemove

`carbon.files.delete_file_tags`

Delete File Tags

🛠️ Usage

delete_file_tags_response = carbon.files.delete_file_tags(
    tags=["string_example"],
    organization_user_file_id=1,
)

⚙️ Parameters

tags: `OrganizationUserFileTagsRemoveTags`

organization_user_file_id: `int`

⚙️ Request Body

🔄 Return

🌐 Endpoint

/delete_user_file_tags post

`carbon.files.delete_many`

Deprecated

Delete Files Endpoint

🛠️ Usage

delete_many_response = carbon.files.delete_many(
    file_ids=[1],
    sync_statuses=["string_example"],
    delete_non_synced_only=False,
    send_webhook=False,
    delete_child_files=False,
)

⚙️ Parameters

file_ids: `DeleteFilesQueryInputFileIds`

sync_statuses: List[`ExternalFileSyncStatuses`]

delete_non_synced_only: `bool`

send_webhook: `bool`

delete_child_files: `bool`

⚙️ Request Body

DeleteFilesQueryInput

🔄 Return

🌐 Endpoint

/delete_files post

DeleteFilesV2QueryInput

`carbon.files.delete_v2`

Delete Files V2 Endpoint

🛠️ Usage

delete_v2_response = carbon.files.delete_v2(
    filters={
        "include_all_children": False,
        "non_synced_only": False,
    },
    send_webhook=False,
    preserve_file_record=False,
)

⚙️ Parameters

filters: `OrganizationUserFilesToSyncFilters`

send_webhook: `bool`

preserve_file_record: `bool`

Whether or not to delete all data related to the file from the database, BUT to preserve the file metadata, allowing for resyncs. By default preserve_file_record is false, which means that all data related to the file as well as its metadata will be deleted. Note that even if preserve_file_record is true, raw files uploaded via the uploadfile endpoint still cannot be resynced.

⚙️ Request Body

🔄 Return

🌐 Endpoint

/delete_files_v2 post

`carbon.files.get_parsed_file`

Deprecated

This route is deprecated. Use /user_files_v2 instead.

🛠️ Usage

get_parsed_file_response = carbon.files.get_parsed_file(
    file_id=1,
)

⚙️ Parameters

file_id: `int`

🔄 Return

PresignedURLResponse

🌐 Endpoint

/parsed_file/{file_id} get

`carbon.files.get_raw_file`

Deprecated

This route is deprecated. Use /user_files_v2 instead.

🛠️ Usage

get_raw_file_response = carbon.files.get_raw_file(
    file_id=1,
)

⚙️ Parameters

file_id: `int`

🔄 Return

PresignedURLResponse

🌐 Endpoint

/raw_file/{file_id} get

ModifyColdStorageParametersQueryInput

`carbon.files.modify_cold_storage_parameters`

Modify Cold Storage Parameters

🛠️ Usage

modify_cold_storage_parameters_response = carbon.files.modify_cold_storage_parameters(
    filters={
        "include_all_children": False,
        "non_synced_only": False,
    },
    enable_cold_storage=True,
    hot_storage_time_to_live=1,
)

⚙️ Parameters

filters: `OrganizationUserFilesToSyncFilters`

enable_cold_storage: `Optional[bool]`

hot_storage_time_to_live: `Optional[int]`

⚙️ Request Body

🌐 Endpoint

/modify_cold_storage_parameters post

MoveToHotStorageQueryInput

`carbon.files.move_to_hot_storage`

Move To Hot Storage

🛠️ Usage

move_to_hot_storage_response = carbon.files.move_to_hot_storage(
    filters={
        "include_all_children": False,
        "non_synced_only": False,
    },
)

⚙️ Parameters

filters: `OrganizationUserFilesToSyncFilters`

⚙️ Request Body

🌐 Endpoint

/move_to_hot_storage post

OrganizationUserFilesToSyncQueryInput

`carbon.files.query_user_files`

{
    "OR": [
        {
            "key": "subject",
            "value": "holy-bible",
            "negate": false
        },
        {
            "key": "person-of-interest",
            "value": "jesus christ",
            "negate": false
        },
        {
            "key": "genre",
            "value": "religion",
            "negate": true
        }
        {
            "AND": [
                {
                    "key": "subject",
                    "value": "tao-te-ching",
                    "negate": false
                },
                {
                    "key": "author",
                    "value": "lao-tzu",
                    "negate": false
                }
            ]
        }
    ]
}

In this case, files will be filtered such that:

"subject" = "holy-bible" OR
"person-of-interest" = "jesus christ" OR
"genre" != "religion" OR
"subject" = "tao-te-ching" AND "author" = "lao-tzu"

"key" isn't optional and must be a string
"value" isn't optional and can be any or list[any]
"negate" is optional and must be true or false. If present and true, then the filter block is negated in the resulting query. It is false by default.

🛠️ Usage

query_user_files_response = carbon.files.query_user_files(
    pagination={
        "limit": 10,
        "offset": 0,
    },
    order_by="created_at",
    order_dir="desc",
    filters={
        "include_all_children": False,
        "non_synced_only": False,
    },
    include_raw_file=True,
    include_parsed_text_file=True,
    include_additional_files=True,
)

⚙️ Parameters

pagination: `Pagination`

order_by: `OrganizationUserFilesToSyncOrderByTypes`

order_dir: `OrderDir`

filters: `OrganizationUserFilesToSyncFilters`

include_raw_file: `Optional[bool]`

include_parsed_text_file: `Optional[bool]`

include_additional_files: `Optional[bool]`

⚙️ Request Body

🔄 Return

UserFilesV2

🌐 Endpoint

/user_files_v2 post

OrganizationUserFilesToSyncQueryInput

`carbon.files.query_user_files_deprecated`

Deprecated

This route is deprecated. Use /user_files_v2 instead.

🛠️ Usage

query_user_files_deprecated_response = carbon.files.query_user_files_deprecated(
    pagination={
        "limit": 10,
        "offset": 0,
    },
    order_by="created_at",
    order_dir="desc",
    filters={
        "include_all_children": False,
        "non_synced_only": False,
    },
    include_raw_file=True,
    include_parsed_text_file=True,
    include_additional_files=True,
)

⚙️ Parameters

pagination: `Pagination`

order_by: `OrganizationUserFilesToSyncOrderByTypes`

order_dir: `OrderDir`

filters: `OrganizationUserFilesToSyncFilters`

include_raw_file: `Optional[bool]`

include_parsed_text_file: `Optional[bool]`

include_additional_files: `Optional[bool]`

⚙️ Request Body

🔄 Return

FilesQueryUserFilesDeprecatedResponse

🌐 Endpoint

/user_files post

`carbon.files.resync`

Resync File

🛠️ Usage

resync_response = carbon.files.resync(
    file_id=1,
    chunk_size=1,
    chunk_overlap=1,
    force_embedding_generation=False,
    skip_file_processing=False,
)

⚙️ Parameters

file_id: `int`

chunk_size: `Optional[int]`

chunk_overlap: `Optional[int]`

force_embedding_generation: `bool`

skip_file_processing: `Optional[bool]`

⚙️ Request Body

ResyncFileQueryInput

🔄 Return

🌐 Endpoint

/resync_file post

BodyCreateUploadFileUploadfilePost

`carbon.files.upload`

This endpoint is used to directly upload local files to Carbon. The POST request should be a multipart form request. Note that the set_page_as_boundary query parameter is applicable only to PDFs for now. When this value is set, PDF chunks are at most one page long. Additional information can be retrieved for each chunk, however, namely the coordinates of the bounding box around the chunk (this can be used for things like text highlighting). Following is a description of all possible query parameters:

chunk_size: the chunk size (in tokens) applied when splitting the document
chunk_overlap: the chunk overlap (in tokens) applied when splitting the document
skip_embedding_generation: whether or not to skip the generation of chunks and embeddings
set_page_as_boundary: described above
embedding_model: the model used to generate embeddings for the document chunks
use_ocr: whether or not to use OCR as a preprocessing step prior to generating chunks. Valid for PDFs, JPEGs, and PNGs
generate_sparse_vectors: whether or not to generate sparse vectors for the file. Required for hybrid search.
prepend_filename_to_chunks: whether or not to prepend the filename to the chunk text

🛠️ Usage

upload_response = carbon.files.upload(
    file=open("/path/to/file", "rb"),
    chunk_size=1,
    chunk_overlap=1,
    skip_embedding_generation=False,
    set_page_as_boundary=False,
    embedding_model="string_example",
    use_ocr=False,
    generate_sparse_vectors=False,
    prepend_filename_to_chunks=False,
    max_items_per_chunk=1,
    parse_pdf_tables_with_ocr=False,
    detect_audio_language=False,
    transcription_service="assemblyai",
    include_speaker_labels=False,
    media_type="TEXT",
    split_rows=False,
    enable_cold_storage=False,
    hot_storage_time_to_live=1,
    generate_chunks_only=False,
    store_file_only=False,
)

⚙️ Parameters

file: `IO`

chunk_size: `Optional[int]`

Chunk size in tiktoken tokens to be used when processing file.

chunk_overlap: `Optional[int]`

Chunk overlap in tiktoken tokens to be used when processing file.

skip_embedding_generation: `bool`

Flag to control whether or not embeddings should be generated and stored when processing file.

set_page_as_boundary: `bool`

Flag to control whether or not to set the a page's worth of content as the maximum amount of content that can appear in a chunk. Only valid for PDFs. See description route description for more information.

embedding_model: Union[`TextEmbeddingGenerators`, `str`]

Embedding model that will be used to embed file chunks.

use_ocr: `bool`

Whether or not to use OCR when processing files. Valid for PDFs, JPEGs, and PNGs. Useful for documents with tables, images, and/or scanned text.

generate_sparse_vectors: `bool`

Whether or not to generate sparse vectors for the file. This is required for the file to be a candidate for hybrid search.

prepend_filename_to_chunks: `bool`

Whether or not to prepend the file's name to chunks.

max_items_per_chunk: `Optional[int]`

Number of objects per chunk. For csv, tsv, xlsx, and json files only.

parse_pdf_tables_with_ocr: `bool`

Whether to use rich table parsing when use_ocr is enabled.

detect_audio_language: `bool`

Whether to automatically detect the language of the uploaded audio file.

transcription_service: `TranscriptionServiceNullable`

The transcription service to use for audio files. If no service is specified, 'deepgram' will be used.

include_speaker_labels: `bool`

Detect multiple speakers and label segments of speech by speaker for audio files.

media_type: `FileContentTypesNullable`

The media type of the file. If not provided, it will be inferred from the file extension.

split_rows: `bool`

Whether to split tabular rows into chunks. Currently only valid for CSV, TSV, and XLSX files.

enable_cold_storage: `bool`

Enable cold storage for the file. If set to true, the file will be moved to cold storage after a certain period of inactivity. Default is false.

hot_storage_time_to_live: `Optional[int]`

Time in seconds after which the file will be moved to cold storage.

generate_chunks_only: `bool`

If this flag is enabled, the file will be chunked and stored with Carbon, but no embeddings will be generated. This overrides the skip_embedding_generation flag.

store_file_only: `bool`

If this flag is enabled, the file will be stored with Carbon, but no processing will be done.

⚙️ Request Body

🔄 Return

🌐 Endpoint

/uploadfile post

`carbon.files.upload_from_url`

Create Upload File From Url

🛠️ Usage

upload_from_url_response = carbon.files.upload_from_url(
    url="string_example",
    file_name="string_example",
    chunk_size=1,
    chunk_overlap=1,
    skip_embedding_generation=False,
    set_page_as_boundary=False,
    embedding_model="OPENAI",
    generate_sparse_vectors=False,
    use_textract=False,
    prepend_filename_to_chunks=False,
    max_items_per_chunk=1,
    parse_pdf_tables_with_ocr=False,
    detect_audio_language=False,
    transcription_service="assemblyai",
    include_speaker_labels=False,
    media_type="TEXT",
    split_rows=False,
    cold_storage_params={
        "enable_cold_storage": False,
    },
    generate_chunks_only=False,
    store_file_only=False,
)

⚙️ Parameters

url: `str`

file_name: `Optional[str]`

chunk_size: `Optional[int]`

chunk_overlap: `Optional[int]`

skip_embedding_generation: `bool`

set_page_as_boundary: `bool`

embedding_model: `EmbeddingGenerators`

generate_sparse_vectors: `bool`

use_textract: `bool`

prepend_filename_to_chunks: `bool`

max_items_per_chunk: `Optional[int]`

Number of objects per chunk. For csv, tsv, xlsx, and json files only.

parse_pdf_tables_with_ocr: `bool`

detect_audio_language: `bool`

transcription_service: `TranscriptionServiceNullable`

include_speaker_labels: `bool`

media_type: `FileContentTypesNullable`

split_rows: `bool`

cold_storage_params: `ColdStorageProps`

generate_chunks_only: `bool`

If this flag is enabled, the file will be chunked and stored with Carbon, but no embeddings will be generated. This overrides the skip_embedding_generation flag.

store_file_only: `bool`

If this flag is enabled, the file will be stored with Carbon, but no processing will be done.

⚙️ Request Body

UploadFileFromUrlInput

🔄 Return

🌐 Endpoint

/upload_file_from_url post

`carbon.files.upload_text`

🛠️ Usage

upload_text_response = carbon.files.upload_text(
    contents="aaaaa",
    name="string_example",
    chunk_size=1,
    chunk_overlap=1,
    skip_embedding_generation=False,
    overwrite_file_id=1,
    embedding_model="OPENAI",
    generate_sparse_vectors=False,
    cold_storage_params={
        "enable_cold_storage": False,
    },
    generate_chunks_only=False,
    store_file_only=False,
)

⚙️ Parameters

contents: `str`

name: `Optional[str]`

chunk_size: `Optional[int]`

chunk_overlap: `Optional[int]`

skip_embedding_generation: `bool`

overwrite_file_id: `Optional[int]`

embedding_model: `EmbeddingGeneratorsNullable`

generate_sparse_vectors: `Optional[bool]`

cold_storage_params: `ColdStorageProps`

generate_chunks_only: `bool`

If this flag is enabled, the file will be chunked and stored with Carbon, but no embeddings will be generated. This overrides the skip_embedding_generation flag.

store_file_only: `bool`

If this flag is enabled, the file will be stored with Carbon, but no processing will be done.

⚙️ Request Body

RawTextInput

🔄 Return

🌐 Endpoint

/upload_text post

`carbon.integrations.cancel`

Cancel Data Source Items Sync

🛠️ Usage

cancel_response = carbon.integrations.cancel(
    data_source_id=1,
)

⚙️ Parameters

data_source_id: `int`

⚙️ Request Body

SyncDirectoryRequest

🔄 Return

🌐 Endpoint

/integrations/items/sync/cancel post

ConnectDataSourceResponse

`carbon.integrations.connect_data_source`

Connect Data Source

🛠️ Usage

connect_data_source_response = carbon.integrations.connect_data_source(
    authentication={
        "source": "GOOGLE_DRIVE",
        "access_token": "access_token_example",
    },
    sync_options={
        "chunk_size": 1500,
        "chunk_overlap": 20,
        "skip_embedding_generation": False,
        "embedding_model": "OPENAI",
        "generate_sparse_vectors": False,
        "prepend_filename_to_chunks": False,
        "sync_files_on_connection": True,
        "set_page_as_boundary": False,
        "enable_file_picker": True,
        "sync_source_items": True,
        "incremental_sync": False,
    },
)

⚙️ Parameters

authentication: Union[`OAuthAuthentication`, `NotionAuthentication`, `OneDriveAuthentication`, `SharepointAuthentication`, `ConfluenceAuthentication`, `ZendeskAuthentication`, `ZoteroAuthentication`, `GitbookAuthetication`, `SalesforceAuthentication`, `FreskdeskAuthentication`, `S3Authentication`, `AzureBlobStorageAuthentication`, `GithubAuthentication`, `ServiceNowAuthentication`, `GuruAuthentication`, `GongAuthentication`]

sync_options: `SyncOptions`

⚙️ Request Body

ConnectDataSourceInput

🔄 Return

🌐 Endpoint

/integrations/connect post

FreshDeskConnectRequest

`carbon.integrations.connect_freshdesk`

Refer this article to obtain an API key https://support.freshdesk.com/en/support/solutions/articles/215517. Make sure that your API key has the permission to read solutions from your account and you are on a paid plan. Once you have an API key, you can make a request to this endpoint along with your freshdesk domain. This will trigger an automatic sync of the articles in your "solutions" tab. Additional parameters below can be used to associate data with the synced articles or modify the sync behavior.

🛠️ Usage

connect_freshdesk_response = carbon.integrations.connect_freshdesk(
    domain="string_example",
    api_key="string_example",
    tags={},
    chunk_size=1500,
    chunk_overlap=20,
    skip_embedding_generation=False,
    embedding_model="OPENAI",
    generate_sparse_vectors=False,
    prepend_filename_to_chunks=False,
    sync_files_on_connection=True,
    request_id="string_example",
    sync_source_items=True,
    file_sync_config={
        "auto_synced_source_types": ["ARTICLE"],
        "sync_attachments": False,
        "detect_audio_language": False,
        "transcription_service": "assemblyai",
        "include_speaker_labels": False,
        "split_rows": False,
        "generate_chunks_only": False,
        "store_file_only": False,
        "skip_file_processing": False,
    },
)

⚙️ Parameters

domain: `str`

api_key: `str`

tags: `Optional[Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]]`

chunk_size: `Optional[int]`

chunk_overlap: `Optional[int]`

skip_embedding_generation: `Optional[bool]`

embedding_model: `EmbeddingGeneratorsNullable`

generate_sparse_vectors: `Optional[bool]`

prepend_filename_to_chunks: `Optional[bool]`

sync_files_on_connection: `Optional[bool]`

request_id: `Optional[str]`

sync_source_items: `bool`

Enabling this flag will fetch all available content from the source to be listed via list items endpoint

file_sync_config: `FileSyncConfigNullable`

⚙️ Request Body

🔄 Return

🌐 Endpoint

/integrations/freshdesk post

`carbon.integrations.connect_gitbook`

You will need an access token to connect your Gitbook account. Note that the permissions will be defined by the user generating access token so make sure you have the permission to access spaces you will be syncing. Refer this article for more details https://developer.gitbook.com/gitbook-api/authentication. Additionally, you need to specify the name of organization you will be syncing data from.

🛠️ Usage

connect_gitbook_response = carbon.integrations.connect_gitbook(
    organization="string_example",
    access_token="string_example",
    tags={},
    chunk_size=1500,
    chunk_overlap=20,
    skip_embedding_generation=False,
    embedding_model="OPENAI",
    generate_sparse_vectors=False,
    prepend_filename_to_chunks=False,
    sync_files_on_connection=True,
    request_id="string_example",
    sync_source_items=True,
    file_sync_config={
        "auto_synced_source_types": ["ARTICLE"],
        "sync_attachments": False,
        "detect_audio_language": False,
        "transcription_service": "assemblyai",
        "include_speaker_labels": False,
        "split_rows": False,
        "generate_chunks_only": False,
        "store_file_only": False,
        "skip_file_processing": False,
    },
)

⚙️ Parameters

organization: `str`

access_token: `str`

tags: `Optional[Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]]`

chunk_size: `Optional[int]`

chunk_overlap: `Optional[int]`

skip_embedding_generation: `Optional[bool]`

embedding_model: `EmbeddingGenerators`

generate_sparse_vectors: `Optional[bool]`

prepend_filename_to_chunks: `Optional[bool]`

sync_files_on_connection: `Optional[bool]`

request_id: `Optional[str]`

sync_source_items: `bool`

Enabling this flag will fetch all available content from the source to be listed via list items endpoint

file_sync_config: `FileSyncConfigNullable`

⚙️ Request Body

GitbookConnectRequest

🔄 Return

🌐 Endpoint

/integrations/gitbook post

`carbon.integrations.connect_guru`

You will need an access token to connect your Guru account. To obtain an access token, follow the steps highlighted here https://help.getguru.com/docs/gurus-api#obtaining-a-user-token. The username should be your Guru username.

🛠️ Usage

connect_guru_response = carbon.integrations.connect_guru(
    username="string_example",
    access_token="string_example",
    tags={},
    chunk_size=1500,
    chunk_overlap=20,
    skip_embedding_generation=False,
    embedding_model="OPENAI",
    generate_sparse_vectors=False,
    prepend_filename_to_chunks=False,
    sync_files_on_connection=True,
    request_id="string_example",
    sync_source_items=True,
    file_sync_config={
        "auto_synced_source_types": ["ARTICLE"],
        "sync_attachments": False,
        "detect_audio_language": False,
        "transcription_service": "assemblyai",
        "include_speaker_labels": False,
        "split_rows": False,
        "generate_chunks_only": False,
        "store_file_only": False,
        "skip_file_processing": False,
    },
)

⚙️ Parameters

username: `str`

access_token: `str`

tags: `Optional[Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]]`

chunk_size: `Optional[int]`

chunk_overlap: `Optional[int]`

skip_embedding_generation: `Optional[bool]`

embedding_model: `EmbeddingGenerators`

generate_sparse_vectors: `Optional[bool]`

prepend_filename_to_chunks: `Optional[bool]`

sync_files_on_connection: `Optional[bool]`

request_id: `Optional[str]`

sync_source_items: `bool`

Enabling this flag will fetch all available content from the source to be listed via list items endpoint

file_sync_config: `FileSyncConfigNullable`

⚙️ Request Body

GuruConnectRequest

🔄 Return

🌐 Endpoint

/integrations/guru post

`carbon.integrations.create_aws_iam_user`

This endpoint can be used to connect S3 as well as Digital Ocean Spaces (S3 compatible)
For S3, create a new IAM user with permissions to:

List all buckets.
Read from the specific buckets and objects to sync with Carbon. Ensure any future buckets or objects carry the same permissions.

Once created, generate an access key for this user and share the credentials with us. We recommend testing this key beforehand. For Digital Ocean Spaces, generate the above credentials in your Applications and API page here https://cloud.digitalocean.com/account/api/spaces. Endpoint URL is required to connect Digital Ocean Spaces. It should look like <>.digitaloceanspaces.com

🛠️ Usage

create_aws_iam_user_response = carbon.integrations.create_aws_iam_user(
    access_key="string_example",
    access_key_secret="string_example",
    sync_source_items=True,
    endpoint_url="string_example",
)

⚙️ Parameters

access_key: `str`

access_key_secret: `str`

sync_source_items: `bool`

Enabling this flag will fetch all available content from the source to be listed via list items endpoint

endpoint_url: `Optional[str]`

You can specify a Digital Ocean endpoint URL to connect a Digital Ocean Space through this endpoint. The URL should be of format .digitaloceanspaces.com. It's not required for S3 buckets.

⚙️ Request Body

S3AuthRequest

🔄 Return

🌐 Endpoint

/integrations/s3 post

`carbon.integrations.get_oauth_url`

This endpoint can be used to generate the following URLs

An OAuth URL for OAuth based connectors
A file syncing URL which skips the OAuth flow if the user already has a valid access token and takes them to the success state.

🛠️ Usage

get_oauth_url_response = carbon.integrations.get_oauth_url(
    service="BOX",
    tags=None,
    scope="string_example",
    chunk_size=1500,
    chunk_overlap=20,
    skip_embedding_generation=False,
    embedding_model="OPENAI",
    zendesk_subdomain="string_example",
    microsoft_tenant="string_example",
    sharepoint_site_name="string_example",
    confluence_subdomain="string_example",
    generate_sparse_vectors=False,
    prepend_filename_to_chunks=False,
    max_items_per_chunk=1,
    salesforce_domain="string_example",
    sync_files_on_connection=True,
    set_page_as_boundary=False,
    data_source_id=1,
    connecting_new_account=False,
    request_id="string_example",
    use_ocr=False,
    parse_pdf_tables_with_ocr=False,
    enable_file_picker=True,
    sync_source_items=True,
    incremental_sync=False,
    file_sync_config={
        "auto_synced_source_types": ["ARTICLE"],
        "sync_attachments": False,
        "detect_audio_language": False,
        "transcription_service": "assemblyai",
        "include_speaker_labels": False,
        "split_rows": False,
        "generate_chunks_only": False,
        "store_file_only": False,
        "skip_file_processing": False,
    },
    automatically_open_file_picker=True,
    gong_account_email="string_example",
    servicenow_credentials={
        "instance_subdomain": "instance_subdomain_example",
        "client_id": "client_id_example",
        "client_secret": "client_secret_example",
        "redirect_uri": "redirect_uri_example",
    },
)

⚙️ Parameters

service: `OauthBasedConnectors`

tags: `Union[bool, date, datetime, dict, float, int, list, str, None]`

scope: `Optional[str]`

chunk_size: `Optional[int]`

chunk_overlap: `Optional[int]`

skip_embedding_generation: `Optional[bool]`

embedding_model: `EmbeddingGeneratorsNullable`

zendesk_subdomain: `Optional[str]`

microsoft_tenant: `Optional[str]`

sharepoint_site_name: `Optional[str]`

confluence_subdomain: `Optional[str]`

generate_sparse_vectors: `Optional[bool]`

prepend_filename_to_chunks: `Optional[bool]`

max_items_per_chunk: `Optional[int]`

Number of objects per chunk. For csv, tsv, xlsx, and json files only.

salesforce_domain: `Optional[str]`

sync_files_on_connection: `Optional[bool]`

Used to specify whether Carbon should attempt to sync all your files automatically when authorization is complete. This is only supported for a subset of connectors and will be ignored for the rest. Supported connectors: Intercom, Zendesk, Gitbook, Confluence, Salesforce, Freshdesk

set_page_as_boundary: `bool`

data_source_id: `Optional[int]`

Used to specify a data source to sync from if you have multiple connected. It can be skipped if you only have one data source of that type connected or are connecting a new account.

connecting_new_account: `Optional[bool]`

Used to connect a new data source. If not specified, we will attempt to create a sync URL for an existing data source based on type and ID.

request_id: `Optional[str]`

This request id will be added to all files that get synced using the generated OAuth URL

use_ocr: `Optional[bool]`

Enable OCR for files that support it. Supported formats: pdf, png, jpg

parse_pdf_tables_with_ocr: `Optional[bool]`

enable_file_picker: `bool`

Enable integration's file picker for sources that support it. Supported sources: BOX, DROPBOX, GOOGLE_DRIVE, ONEDRIVE, SHAREPOINT

sync_source_items: `bool`

Enabling this flag will fetch all available content from the source to be listed via list items endpoint

incremental_sync: `bool`

Only sync files if they have not already been synced or if the embedding properties have changed. This flag is currently supported by ONEDRIVE, GOOGLE_DRIVE, BOX, DROPBOX, INTERCOM, GMAIL, OUTLOOK, ZENDESK, CONFLUENCE, NOTION, SHAREPOINT, SERVICENOW. It will be ignored for other data sources.

file_sync_config: `FileSyncConfigNullable`

automatically_open_file_picker: `Optional[bool]`

Automatically open source file picker after the OAuth flow is complete. This flag is currently supported by BOX, DROPBOX, GOOGLE_DRIVE, ONEDRIVE, SHAREPOINT. It will be ignored for other data sources.

gong_account_email: `Optional[str]`

If you are connecting a Gong account, you need to input the email of the account you wish to connect. This email will be used to identify your carbon data source.

servicenow_credentials: `ServiceNowCredentialsNullable`

⚙️ Request Body

OAuthURLRequest

🔄 Return

OuthURLResponse

🌐 Endpoint

/integrations/oauth_url post

`carbon.integrations.list_confluence_pages`

Deprecated

This endpoint has been deprecated. Use /integrations/items/list instead.

To begin listing a user's Confluence pages, at least a data_source_id of a connected Confluence account must be specified. This base request returns a list of root pages for every space the user has access to in a Confluence instance. To traverse further down the user's page directory, additional requests to this endpoint can be made with the same data_source_id and with parent_id set to the id of page from a previous request. For convenience, the has_children property in each directory item in the response list will flag which pages will return non-empty lists of pages when set as the parent_id.

🛠️ Usage

list_confluence_pages_response = carbon.integrations.list_confluence_pages(
    data_source_id=1,
    parent_id="string_example",
)

⚙️ Parameters

data_source_id: `int`

parent_id: `Optional[str]`

⚙️ Request Body

ListRequest

🔄 Return

ListResponse

🌐 Endpoint

/integrations/confluence/list post

`carbon.integrations.list_conversations`

List all of your public and private channels, DMs, and Group DMs. The ID from response can be used as a filter to sync messages to Carbon
types: Comma separated list of types. Available types are im (DMs), mpim (group DMs), public_channel, and private_channel. Defaults to public_channel.
cursor: Used for pagination. If next_cursor is returned in response, you need to pass it as the cursor in the next request
data_source_id: Data source needs to be specified if you have linked multiple slack accounts
exclude_archived: Should archived conversations be excluded, defaults to true

🛠️ Usage

list_conversations_response = carbon.integrations.list_conversations(
    types="public_channel",
    cursor="string_example",
    data_source_id=1,
    exclude_archived=True,
)

⚙️ Parameters

types: `str`

cursor: `Optional[str]`

data_source_id: `Optional[int]`

exclude_archived: `bool`

🌐 Endpoint

/integrations/slack/conversations get

ListDataSourceItemsRequest

`carbon.integrations.list_data_source_items`

List Data Source Items

🛠️ Usage

list_data_source_items_response = carbon.integrations.list_data_source_items(
    data_source_id=1,
    parent_id="string_example",
    filters={},
    pagination={
        "limit": 10,
        "offset": 0,
    },
    order_by="name",
    order_dir="asc",
)

⚙️ Parameters

data_source_id: `int`

parent_id: `Optional[str]`

filters: `ListItemsFiltersNullable`

pagination: `Pagination`

order_by: `ExternalSourceItemsOrderBy`

order_dir: `OrderDirV2`

⚙️ Request Body

🔄 Return

ListDataSourceItemsResponse

🌐 Endpoint

/integrations/items/list post

`carbon.integrations.list_folders`

After connecting your Outlook account, you can use this endpoint to list all of your folders on outlook. This includes both system folders like "inbox" and user created folders.

🛠️ Usage

list_folders_response = carbon.integrations.list_folders(
    data_source_id=1,
)

⚙️ Parameters

data_source_id: `Optional[int]`

🌐 Endpoint

/integrations/outlook/user_folders get

`carbon.integrations.list_gitbook_spaces`

After connecting your Gitbook account, you can use this endpoint to list all of your spaces under current organization.

🛠️ Usage

list_gitbook_spaces_response = carbon.integrations.list_gitbook_spaces(
    data_source_id=1,
)

⚙️ Parameters

data_source_id: `int`

🌐 Endpoint

/integrations/gitbook/spaces get

`carbon.integrations.list_labels`

After connecting your Gmail account, you can use this endpoint to list all of your labels. User created labels will have the type "user" and Gmail's default labels will have the type "system"

🛠️ Usage

list_labels_response = carbon.integrations.list_labels(
    data_source_id=1,
)

⚙️ Parameters

data_source_id: `Optional[int]`

🌐 Endpoint

/integrations/gmail/user_labels get

`carbon.integrations.list_outlook_categories`

After connecting your Outlook account, you can use this endpoint to list all of your categories on outlook. We currently support listing up to 250 categories.

🛠️ Usage

list_outlook_categories_response = carbon.integrations.list_outlook_categories(
    data_source_id=1,
)

⚙️ Parameters

data_source_id: `Optional[int]`

🌐 Endpoint

/integrations/outlook/user_categories get

`carbon.integrations.list_repos`

Once you have connected your GitHub account, you can use this endpoint to list the repositories your account has access to. You can use a data source ID or username to fetch from a specific account.

🛠️ Usage

list_repos_response = carbon.integrations.list_repos(
    per_page=30,
    page=1,
    data_source_id=1,
)

⚙️ Parameters

per_page: `int`

page: `int`

data_source_id: `Optional[int]`

🌐 Endpoint

/integrations/github/repos get

`carbon.integrations.sync_azure_blob_files`

After optionally loading the items via /integrations/items/sync and integrations/items/list, use the container name and file name as the ID in this endpoint to sync them into Carbon. Additional parameters below can associate data with the selected items or modify the sync behavior

🛠️ Usage

sync_azure_blob_files_response = carbon.integrations.sync_azure_blob_files(
    ids=[{}],
    tags={},
    chunk_size=1500,
    chunk_overlap=20,
    skip_embedding_generation=False,
    embedding_model="OPENAI",
    generate_sparse_vectors=False,
    prepend_filename_to_chunks=False,
    max_items_per_chunk=1,
    set_page_as_boundary=False,
    data_source_id=1,
    request_id="string_example",
    use_ocr=False,
    parse_pdf_tables_with_ocr=False,
    file_sync_config={
        "auto_synced_source_types": ["ARTICLE"],
        "sync_attachments": False,
        "detect_audio_language": False,
        "transcription_service": "assemblyai",
        "include_speaker_labels": False,
        "split_rows": False,
        "generate_chunks_only": False,
        "store_file_only": False,
        "skip_file_processing": False,
    },
)

⚙️ Parameters

ids: List[`AzureBlobGetFileInput`]

tags: `Optional[Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]]`

chunk_size: `Optional[int]`

chunk_overlap: `Optional[int]`

skip_embedding_generation: `Optional[bool]`

embedding_model: `EmbeddingGenerators`

generate_sparse_vectors: `Optional[bool]`

prepend_filename_to_chunks: `Optional[bool]`

max_items_per_chunk: `Optional[int]`

Number of objects per chunk. For csv, tsv, xlsx, and json files only.

set_page_as_boundary: `bool`

data_source_id: `Optional[int]`

request_id: `Optional[str]`

use_ocr: `Optional[bool]`

parse_pdf_tables_with_ocr: `Optional[bool]`

file_sync_config: `FileSyncConfigNullable`

⚙️ Request Body

AzureBlobFileSyncInput

🔄 Return

🌐 Endpoint

/integrations/azure_blob_storage/files post

`carbon.integrations.sync_azure_blob_storage`

This endpoint can be used to connect Azure Blob Storage.

For Azure Blob Storage, follow these steps:

Create a new Azure Storage account and grant the following permissions:
- List containers.
- Read from specific containers and blobs to sync with Carbon. Ensure any future containers or blobs carry the same permissions.
Generate a shared access signature (SAS) token or an access key for the storage account.

Once created, provide us with the following details to generate the connection URL:

Storage Account KeyName.
Storage Account Name.

🛠️ Usage

sync_azure_blob_storage_response = carbon.integrations.sync_azure_blob_storage(
    account_name="string_example",
    account_key="string_example",
    sync_source_items=True,
)

⚙️ Parameters

account_name: `str`

account_key: `str`

sync_source_items: `bool`

⚙️ Request Body

AzureBlobAuthRequest

🔄 Return

🌐 Endpoint

/integrations/azure_blob_storage post

`carbon.integrations.sync_confluence`

Deprecated

This endpoint has been deprecated. Use /integrations/files/sync instead.

After listing pages in a user's Confluence account, the set of selected page ids and the connected account's data_source_id can be passed into this endpoint to sync them into Carbon. Additional parameters listed below can be used to associate data to the selected pages or alter the behavior of the sync.

🛠️ Usage

sync_confluence_response = carbon.integrations.sync_confluence(
    data_source_id=1,
    ids=["string_example"],
    tags={},
    chunk_size=1500,
    chunk_overlap=20,
    skip_embedding_generation=False,
    embedding_model="OPENAI",
    generate_sparse_vectors=False,
    prepend_filename_to_chunks=False,
    max_items_per_chunk=1,
    set_page_as_boundary=False,
    request_id="string_example",
    use_ocr=False,
    parse_pdf_tables_with_ocr=False,
    incremental_sync=False,
    file_sync_config={
        "auto_synced_source_types": ["ARTICLE"],
        "sync_attachments": False,
        "detect_audio_language": False,
        "transcription_service": "assemblyai",
        "include_speaker_labels": False,
        "split_rows": False,
        "generate_chunks_only": False,
        "store_file_only": False,
        "skip_file_processing": False,
    },
)

⚙️ Parameters

data_source_id: `int`

ids: Union[`List[str]`, List[`SyncFilesIds`]]

tags: `Optional[Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]]`

chunk_size: `Optional[int]`

chunk_overlap: `Optional[int]`

skip_embedding_generation: `Optional[bool]`

embedding_model: `EmbeddingGeneratorsNullable`

generate_sparse_vectors: `Optional[bool]`

prepend_filename_to_chunks: `Optional[bool]`

max_items_per_chunk: `Optional[int]`

Number of objects per chunk. For csv, tsv, xlsx, and json files only.

set_page_as_boundary: `bool`

request_id: `Optional[str]`

use_ocr: `Optional[bool]`

parse_pdf_tables_with_ocr: `Optional[bool]`

incremental_sync: `bool`

file_sync_config: `FileSyncConfigNullable`

⚙️ Request Body

SyncFilesRequest

🔄 Return

🌐 Endpoint

/integrations/confluence/sync post

`carbon.integrations.sync_data_source_items`

Sync Data Source Items

🛠️ Usage

sync_data_source_items_response = carbon.integrations.sync_data_source_items(
    data_source_id=1,
)

⚙️ Parameters

data_source_id: `int`

⚙️ Request Body

SyncDirectoryRequest

🔄 Return

🌐 Endpoint

/integrations/items/sync post

`carbon.integrations.sync_files`

After listing files and folders via /integrations/items/sync and integrations/items/list, use the selected items' external ids as the ids in this endpoint to sync them into Carbon. Sharepoint items take an additional parameter root_id, which identifies the drive the file or folder is in and is stored in root_external_id. That additional paramter is optional and excluding it will tell the sync to assume the item is stored in the default Documents drive.

🛠️ Usage

sync_files_response = carbon.integrations.sync_files(
    data_source_id=1,
    ids=["string_example"],
    tags={},
    chunk_size=1500,
    chunk_overlap=20,
    skip_embedding_generation=False,
    embedding_model="OPENAI",
    generate_sparse_vectors=False,
    prepend_filename_to_chunks=False,
    max_items_per_chunk=1,
    set_page_as_boundary=False,
    request_id="string_example",
    use_ocr=False,
    parse_pdf_tables_with_ocr=False,
    incremental_sync=False,
    file_sync_config={
        "auto_synced_source_types": ["ARTICLE"],
        "sync_attachments": False,
        "detect_audio_language": False,
        "transcription_service": "assemblyai",
        "include_speaker_labels": False,
        "split_rows": False,
        "generate_chunks_only": False,
        "store_file_only": False,
        "skip_file_processing": False,
    },
)

⚙️ Parameters

data_source_id: `int`

ids: Union[`List[str]`, List[`SyncFilesIds`]]

tags: `Optional[Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]]`

chunk_size: `Optional[int]`

chunk_overlap: `Optional[int]`

skip_embedding_generation: `Optional[bool]`

embedding_model: `EmbeddingGeneratorsNullable`

generate_sparse_vectors: `Optional[bool]`

prepend_filename_to_chunks: `Optional[bool]`

max_items_per_chunk: `Optional[int]`

Number of objects per chunk. For csv, tsv, xlsx, and json files only.

set_page_as_boundary: `bool`

request_id: `Optional[str]`

use_ocr: `Optional[bool]`

parse_pdf_tables_with_ocr: `Optional[bool]`

incremental_sync: `bool`

file_sync_config: `FileSyncConfigNullable`

⚙️ Request Body

SyncFilesRequest

🔄 Return

🌐 Endpoint

/integrations/files/sync post

`carbon.integrations.sync_git_hub`

Refer this article to obtain an access token https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens. Make sure that your access token has the permission to read content from your desired repos. Note that if your access token expires you will need to manually update it through this endpoint.

🛠️ Usage

sync_git_hub_response = carbon.integrations.sync_git_hub(
    username="string_example",
    access_token="string_example",
    sync_source_items=False,
)

⚙️ Parameters

username: `str`

access_token: `str`

sync_source_items: `bool`

Enabling this flag will fetch all available content from the source to be listed via list items endpoint

⚙️ Request Body

GithubConnectRequest

🔄 Return

🌐 Endpoint

/integrations/github post

`carbon.integrations.sync_gitbook`

You can sync upto 20 Gitbook spaces at a time using this endpoint. Additional parameters below can be used to associate data with the synced pages or modify the sync behavior.

🛠️ Usage

sync_gitbook_response = carbon.integrations.sync_gitbook(
    space_ids=["string_example"],
    data_source_id=1,
    tags={},
    chunk_size=1500,
    chunk_overlap=20,
    skip_embedding_generation=False,
    embedding_model="OPENAI",
    generate_sparse_vectors=False,
    prepend_filename_to_chunks=False,
    request_id="string_example",
    file_sync_config={
        "auto_synced_source_types": ["ARTICLE"],
        "sync_attachments": False,
        "detect_audio_language": False,
        "transcription_service": "assemblyai",
        "include_speaker_labels": False,
        "split_rows": False,
        "generate_chunks_only": False,
        "store_file_only": False,
        "skip_file_processing": False,
    },
)

⚙️ Parameters

space_ids: `GitbookSyncRequestSpaceIds`

data_source_id: `int`

tags: `Optional[Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]]`

chunk_size: `Optional[int]`

chunk_overlap: `Optional[int]`

skip_embedding_generation: `Optional[bool]`

embedding_model: `EmbeddingGenerators`

generate_sparse_vectors: `Optional[bool]`

prepend_filename_to_chunks: `Optional[bool]`

request_id: `Optional[str]`

file_sync_config: `FileSyncConfigNullable`

⚙️ Request Body

GitbookSyncRequest

🌐 Endpoint

/integrations/gitbook/sync post

`carbon.integrations.sync_gmail`

Once you have successfully connected your gmail account, you can choose which emails to sync with us using the filters parameter. Filters is a JSON object with key value pairs. It also supports AND and OR operations. For now, we support a limited set of keys listed below.

label: Inbuilt Gmail labels, for example "Important" or a custom label you created.
after or before: A date in YYYY/mm/dd format (example 2023/12/31). Gets emails after/before a certain date. You can also use them in combination to get emails from a certain period.
is: Can have the following values - starred, important, snoozed, and unread
from: Email address of the sender
to: Email address of the recipient
in: Can have the following values - sent (sync emails sent by the user)
has: Can have the following values - attachment (sync emails that have attachments)

Using keys or values outside of the specified values can lead to unexpected behaviour.

An example of a basic query with filters can be

{
    "filters": {
            "key": "label",
            "value": "Test"
        }
}

Which will list all emails that have the label "Test".

You can use AND and OR operation in the following way:

{
    "filters": {
        "AND": [
            {
                "key": "after",
                "value": "2024/01/07"
            },
            {
                "OR": [
                    {
                        "key": "label",
                        "value": "Personal"
                    },
                    {
                        "key": "is",
                        "value": "starred"
                    }
                ]
            }
        ]
    }
}

This will return emails after 7th of Jan that are either starred or have the label "Personal". Note that this is the highest level of nesting we support, i.e. you can't add more AND/OR filters within the OR filter in the above example.

🛠️ Usage

sync_gmail_response = carbon.integrations.sync_gmail(
    filters={},
    tags={},
    chunk_size=1500,
    chunk_overlap=20,
    skip_embedding_generation=False,
    embedding_model="OPENAI",
    generate_sparse_vectors=False,
    prepend_filename_to_chunks=False,
    data_source_id=1,
    request_id="string_example",
    sync_attachments=False,
    file_sync_config={
        "auto_synced_source_types": ["ARTICLE"],
        "sync_attachments": False,
        "detect_audio_language": False,
        "transcription_service": "assemblyai",
        "include_speaker_labels": False,
        "split_rows": False,
        "generate_chunks_only": False,
        "store_file_only": False,
        "skip_file_processing": False,
    },
    incremental_sync=False,
)

⚙️ Parameters

filters: `Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]`

tags: `Optional[Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]]`

chunk_size: `Optional[int]`

chunk_overlap: `Optional[int]`

skip_embedding_generation: `Optional[bool]`

embedding_model: `EmbeddingGenerators`

generate_sparse_vectors: `Optional[bool]`

prepend_filename_to_chunks: `Optional[bool]`

data_source_id: `Optional[int]`

request_id: `Optional[str]`

sync_attachments: `Optional[bool]`

file_sync_config: `FileSyncConfigNullable`

incremental_sync: `bool`

⚙️ Request Body

GmailSyncInput

🔄 Return

🌐 Endpoint

/integrations/gmail/sync post

`carbon.integrations.sync_outlook`

Once you have successfully connected your Outlook account, you can choose which emails to sync with us using the filters and folder parameter. "folder" should be the folder you want to sync from Outlook. By default we get messages from your inbox folder.
Filters is a JSON object with key value pairs. It also supports AND and OR operations. For now, we support a limited set of keys listed below.

category: Custom categories that you created in Outlook.
after or before: A date in YYYY/mm/dd format (example 2023/12/31). Gets emails after/before a certain date. You can also use them in combination to get emails from a certain period.
is: Can have the following values: flagged
from: Email address of the sender

An example of a basic query with filters can be

{
    "filters": {
            "key": "category",
            "value": "Test"
        }
}

Which will list all emails that have the category "Test".

Specifying a custom folder in the same query

{
    "folder": "Folder Name",
    "filters": {
            "key": "category",
            "value": "Test"
        }
}

You can use AND and OR operation in the following way:

{
    "filters": {
        "AND": [
            {
                "key": "after",
                "value": "2024/01/07"
            },
            {
                "OR": [
                    {
                        "key": "category",
                        "value": "Personal"
                    },
                    {
                        "key": "category",
                        "value": "Test"
                    },
                ]
            }
        ]
    }
}

This will return emails after 7th of Jan that have either Personal or Test as category. Note that this is the highest level of nesting we support, i.e. you can't add more AND/OR filters within the OR filter in the above example.

🛠️ Usage

sync_outlook_response = carbon.integrations.sync_outlook(
    filters={},
    tags={},
    folder="Inbox",
    chunk_size=1500,
    chunk_overlap=20,
    skip_embedding_generation=False,
    embedding_model="OPENAI",
    generate_sparse_vectors=False,
    prepend_filename_to_chunks=False,
    data_source_id=1,
    request_id="string_example",
    sync_attachments=False,
    file_sync_config={
        "auto_synced_source_types": ["ARTICLE"],
        "sync_attachments": False,
        "detect_audio_language": False,
        "transcription_service": "assemblyai",
        "include_speaker_labels": False,
        "split_rows": False,
        "generate_chunks_only": False,
        "store_file_only": False,
        "skip_file_processing": False,
    },
    incremental_sync=False,
)

⚙️ Parameters

filters: `Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]`

tags: `Optional[Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]]`

folder: `Optional[str]`

chunk_size: `Optional[int]`

chunk_overlap: `Optional[int]`

skip_embedding_generation: `Optional[bool]`

embedding_model: `EmbeddingGenerators`

generate_sparse_vectors: `Optional[bool]`

prepend_filename_to_chunks: `Optional[bool]`

data_source_id: `Optional[int]`

request_id: `Optional[str]`

sync_attachments: `Optional[bool]`

file_sync_config: `FileSyncConfigNullable`

incremental_sync: `bool`

⚙️ Request Body

OutlookSyncInput

🔄 Return

🌐 Endpoint

/integrations/outlook/sync post

GithubFetchReposRequest

`carbon.integrations.sync_repos`

You can retreive repos your token has access to using /integrations/github/repos and sync their content. You can also pass full name of any public repository (username/repo-name). This will store the repo content with carbon which can be accessed through /integrations/items/list endpoint. Maximum of 25 repositories are accepted per request.

🛠️ Usage

sync_repos_response = carbon.integrations.sync_repos(
    repos=["string_example"],
    data_source_id=1,
)

⚙️ Parameters

repos: `GithubFetchReposRequestRepos`

data_source_id: `Optional[int]`

⚙️ Request Body

🌐 Endpoint

/integrations/github/sync_repos post

`carbon.integrations.sync_rss_feed`

Rss Feed

🛠️ Usage

sync_rss_feed_response = carbon.integrations.sync_rss_feed(
    url="string_example",
    tags={},
    chunk_size=1500,
    chunk_overlap=20,
    skip_embedding_generation=False,
    embedding_model="OPENAI",
    generate_sparse_vectors=False,
    prepend_filename_to_chunks=False,
    request_id="string_example",
)

⚙️ Parameters

url: `str`

tags: `Optional[Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]]`

chunk_size: `Optional[int]`

chunk_overlap: `Optional[int]`

skip_embedding_generation: `Optional[bool]`

embedding_model: `EmbeddingGenerators`

generate_sparse_vectors: `Optional[bool]`

prepend_filename_to_chunks: `Optional[bool]`

request_id: `Optional[str]`

⚙️ Request Body

RSSFeedInput

🔄 Return

🌐 Endpoint

/integrations/rss_feed post

`carbon.integrations.sync_s3_files`

After optionally loading the items via /integrations/items/sync and integrations/items/list, use the bucket name and object key as the ID in this endpoint to sync them into Carbon. Additional parameters below can associate data with the selected items or modify the sync behavior

🛠️ Usage

sync_s3_files_response = carbon.integrations.sync_s3_files(
    ids=[{}],
    tags={},
    chunk_size=1500,
    chunk_overlap=20,
    skip_embedding_generation=False,
    embedding_model="OPENAI",
    generate_sparse_vectors=False,
    prepend_filename_to_chunks=False,
    max_items_per_chunk=1,
    set_page_as_boundary=False,
    data_source_id=1,
    request_id="string_example",
    use_ocr=False,
    parse_pdf_tables_with_ocr=False,
    file_sync_config={
        "auto_synced_source_types": ["ARTICLE"],
        "sync_attachments": False,
        "detect_audio_language": False,
        "transcription_service": "assemblyai",
        "include_speaker_labels": False,
        "split_rows": False,
        "generate_chunks_only": False,
        "store_file_only": False,
        "skip_file_processing": False,
    },
)

⚙️ Parameters

ids: List[`S3GetFileInput`]

tags: `Optional[Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]]`

chunk_size: `Optional[int]`

chunk_overlap: `Optional[int]`

skip_embedding_generation: `Optional[bool]`

embedding_model: `EmbeddingGenerators`

generate_sparse_vectors: `Optional[bool]`

prepend_filename_to_chunks: `Optional[bool]`

max_items_per_chunk: `Optional[int]`

Number of objects per chunk. For csv, tsv, xlsx, and json files only.

set_page_as_boundary: `bool`

data_source_id: `Optional[int]`

request_id: `Optional[str]`

use_ocr: `Optional[bool]`

parse_pdf_tables_with_ocr: `Optional[bool]`

file_sync_config: `FileSyncConfigNullable`

⚙️ Request Body

S3FileSyncInput

🔄 Return

🌐 Endpoint

/integrations/s3/files post

`carbon.integrations.sync_slack`

You can list all conversations using the endpoint /integrations/slack/conversations. The ID of conversation will be used as an input for this endpoint with timestamps as optional filters.

🛠️ Usage

sync_slack_response = carbon.integrations.sync_slack(
    filters={
        "conversation_id": "conversation_id_example",
    },
    tags={},
    chunk_size=1500,
    chunk_overlap=20,
    skip_embedding_generation=False,
    embedding_model="OPENAI",
    generate_sparse_vectors=False,
    prepend_filename_to_chunks=False,
    data_source_id=1,
    request_id="string_example",
)

⚙️ Parameters

filters: `SlackFilters`

tags: `Optional[Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]]`

chunk_size: `Optional[int]`

chunk_overlap: `Optional[int]`

skip_embedding_generation: `Optional[bool]`

embedding_model: `EmbeddingGenerators`

generate_sparse_vectors: `Optional[bool]`

prepend_filename_to_chunks: `Optional[bool]`

data_source_id: `Optional[int]`

request_id: `Optional[str]`

⚙️ Request Body

SlackSyncRequest

🌐 Endpoint

/integrations/slack/sync post

`carbon.organizations.get`

Get Organization

🛠️ Usage

get_response = carbon.organizations.get()

🔄 Return

OrganizationResponse

🌐 Endpoint

/organization get

UpdateOrganizationInput

`carbon.organizations.update`

Update Organization

🛠️ Usage

update_response = carbon.organizations.update(
    global_user_config={},
    data_source_configs={
        "key": {
            "allowed_file_formats": [],
        },
    },
)

⚙️ Parameters

global_user_config: `UserConfigurationNullable`

data_source_configs: `UpdateOrganizationInputDataSourceConfigs`

⚙️ Request Body

🔄 Return

🌐 Endpoint

/organization/update post

`carbon.organizations.update_stats`

Use this endpoint to reaggregate the statistics for an organization, for example aggregate_file_size. The reaggregation process is asyncronous so a webhook will be sent with the event type being FILE_STATISTICS_AGGREGATED to notify when the process is complee. After this aggregation is complete, the updated statistics can be retrieved using the /organization endpoint. The response of /organization willalso contain a timestamp of the last time the statistics were reaggregated.

🛠️ Usage

update_stats_response = carbon.organizations.update_stats()

🔄 Return

🌐 Endpoint

/organization/statistics post

`carbon.users.delete`

Delete Users

🛠️ Usage

delete_response = carbon.users.delete(
    customer_ids=["string_example"],
)

⚙️ Parameters

customer_ids: `DeleteUsersInputCustomerIds`

⚙️ Request Body

DeleteUsersInput

🔄 Return

🌐 Endpoint

/delete_users post

`carbon.users.get`

User Endpoint

🛠️ Usage

get_response = carbon.users.get(
    customer_id="string_example",
)

⚙️ Parameters

customer_id: `str`

⚙️ Request Body

UserRequestContent

🔄 Return

UserResponse

🌐 Endpoint

/user post

`carbon.users.list`

List users within an organization

🛠️ Usage

list_response = carbon.users.list(
    pagination={
        "limit": 10,
        "offset": 0,
    },
    filters={},
    order_by="created_at",
    order_dir="asc",
    include_count=False,
)

⚙️ Parameters

pagination: `Pagination`

filters: `ListUsersFilters`

order_by: `ListUsersOrderByTypes`

order_dir: `OrderDirV2`

include_count: `bool`

⚙️ Request Body

ListUsersRequest

🔄 Return

UserListResponse

🌐 Endpoint

/list_users post

ModifyUserConfigurationInput

`carbon.users.toggle_user_features`

Deprecated

Toggle User Features

🛠️ Usage

toggle_user_features_response = carbon.users.toggle_user_features(
    configuration_key_name="string_example",
    value={},
)

⚙️ Parameters

configuration_key_name: `str`

value: `Dict[str, Union[bool, date, datetime, dict, float, int, list, str, None]]`

⚙️ Request Body

🔄 Return

🌐 Endpoint

/modify_user_configuration post

`carbon.users.update_users`

Update Users

🛠️ Usage

update_users_response = carbon.users.update_users(
    customer_ids=["string_example"],
    auto_sync_enabled_sources=["string_example"],
    max_files=-1,
    max_files_per_upload=-1,
)

⚙️ Parameters

customer_ids: `UpdateUsersInputCustomerIds`

auto_sync_enabled_sources: Union[List[`DataSourceType`], `str`]

List of data source types to enable auto sync for. Empty array will remove all sources and the string \"ALL\" will enable it for all data sources

max_files: `Optional[int]`

Custom file upload limit for the user over all user's files across all uploads. If set, then the user will not be allowed to upload more files than this limit. If not set, or if set to -1, then the user will have no limit.

max_files_per_upload: `Optional[int]`

Custom file upload limit for the user across a single upload. If set, then the user will not be allowed to upload more files than this limit in a single upload. If not set, or if set to -1, then the user will have no limit.

⚙️ Request Body

UpdateUsersInput

🔄 Return

🌐 Endpoint

/update_users post

`carbon.utilities.fetch_urls`

Deprecated

Extracts all URLs from a webpage.

Args: url (str): URL of the webpage

Returns: FetchURLsResponse: A response object with a list of URLs extracted from the webpage and the webpage content.

🛠️ Usage

fetch_urls_response = carbon.utilities.fetch_urls(
    url="url_example",
)

⚙️ Parameters

url: `str`

🔄 Return

FetchURLsResponse

🌐 Endpoint

/fetch_urls get

`carbon.utilities.fetch_webpage`

Fetch Urls V2

🛠️ Usage

fetch_webpage_response = carbon.utilities.fetch_webpage(
    url="string_example",
)

⚙️ Parameters

url: `str`

⚙️ Request Body

FetchURLsRequest

🌐 Endpoint

/fetch_webpage post

YoutubeTranscriptResponse

`carbon.utilities.fetch_youtube_transcripts`

Fetches english transcripts from YouTube videos.

Args: id (str): The ID of the YouTube video. raw (bool): Whether to return the raw transcript or not. Defaults to False.

Returns: dict: A dictionary with the transcript of the YouTube video.

🛠️ Usage

fetch_youtube_transcripts_response = carbon.utilities.fetch_youtube_transcripts(
    id="id_example",
    raw=False,
)

⚙️ Parameters

id: `str`

raw: `bool`

🔄 Return

🌐 Endpoint

/fetch_youtube_transcript get

`carbon.utilities.process_sitemap`

Retrieves all URLs from a sitemap, which can subsequently be utilized with our web_scrape endpoint.

🛠️ Usage

process_sitemap_response = carbon.utilities.process_sitemap(
    url="url_example",
)

⚙️ Parameters

url: `str`

🌐 Endpoint

/process_sitemap get

`carbon.utilities.scrape_sitemap`

Extracts all URLs from a sitemap and performs a web scrape on each of them.

Args: sitemap_url (str): URL of the sitemap

Returns: dict: A response object with the status of the scraping job message.-->

🛠️ Usage

scrape_sitemap_response = carbon.utilities.scrape_sitemap(
    url="string_example",
    tags={
        "key": "string_example",
    },
    max_pages_to_scrape=1,
    chunk_size=1500,
    chunk_overlap=20,
    skip_embedding_generation=False,
    enable_auto_sync=False,
    generate_sparse_vectors=False,
    prepend_filename_to_chunks=False,
    html_tags_to_skip=[],
    css_classes_to_skip=[],
    css_selectors_to_skip=[],
    embedding_model="OPENAI",
    url_paths_to_include=[],
    url_paths_to_exclude=[],
    urls_to_scrape=[],
    download_css_and_media=False,
    generate_chunks_only=False,
    store_file_only=False,
)

⚙️ Parameters

url: `str`

tags: `SitemapScrapeRequestTags`

max_pages_to_scrape: `Optional[int]`

chunk_size: `Optional[int]`

chunk_overlap: `Optional[int]`

skip_embedding_generation: `Optional[bool]`

enable_auto_sync: `Optional[bool]`

generate_sparse_vectors: `Optional[bool]`

prepend_filename_to_chunks: `Optional[bool]`

html_tags_to_skip: `SitemapScrapeRequestHtmlTagsToSkip`

css_classes_to_skip: `SitemapScrapeRequestCssClassesToSkip`

css_selectors_to_skip: `SitemapScrapeRequestCssSelectorsToSkip`

embedding_model: `EmbeddingGenerators`

url_paths_to_include: `SitemapScrapeRequestUrlPathsToInclude`

url_paths_to_exclude: `SitemapScrapeRequestUrlPathsToExclude`

urls_to_scrape: `SitemapScrapeRequestUrlsToScrape`

download_css_and_media: `Optional[bool]`

Whether the scraper should download css and media from the page (images, fonts, etc). Scrapes might take longer to finish with this flag enabled, but the success rate is improved.

generate_chunks_only: `bool`

If this flag is enabled, the file will be chunked and stored with Carbon, but no embeddings will be generated. This overrides the skip_embedding_generation flag.

store_file_only: `bool`

If this flag is enabled, the file will be stored with Carbon, but no processing will be done.

⚙️ Request Body

SitemapScrapeRequest

🌐 Endpoint

/scrape_sitemap post

UtilitiesScrapeWebRequest

`carbon.utilities.scrape_web`

Conduct a web scrape on a given webpage URL. Our web scraper is fully compatible with JavaScript and supports recursion depth, enabling you to efficiently extract all content from the target website.

🛠️ Usage

scrape_web_response = carbon.utilities.scrape_web(
    body=[
        {
            "url": "url_example",
            "recursion_depth": 3,
            "max_pages_to_scrape": 100,
            "chunk_size": 1500,
            "chunk_overlap": 20,
            "skip_embedding_generation": False,
            "enable_auto_sync": False,
            "generate_sparse_vectors": False,
            "prepend_filename_to_chunks": False,
            "html_tags_to_skip": [],
            "css_classes_to_skip": [],
            "css_selectors_to_skip": [],
            "embedding_model": "OPENAI",
            "url_paths_to_include": [],
            "download_css_and_media": False,
            "generate_chunks_only": False,
            "store_file_only": False,
        }
    ],
)

⚙️ Request Body

🌐 Endpoint

/web_scrape post

`carbon.utilities.search_urls`

Perform a web search and obtain a list of relevant URLs.

As an illustration, when you perform a search for “content related to MRNA,” you will receive a list of links such as the following:

- https://tomrenz.substack.com/p/mrna-and-why-it-matters

- https://www.statnews.com/2020/11/10/the-story-of-mrna-how-a-once-dismissed-idea-became-a-leading-technology-in-the-covid-vaccine-race/

- https://www.statnews.com/2022/11/16/covid-19-vaccines-were-a-success-but-mrna-still-has-a-delivery-problem/

- https://joomi.substack.com/p/were-still-being-misled-about-how

Subsequently, you can submit these links to the web_scrape endpoint in order to retrieve the content of the respective web pages.

Args: query (str): Query to search for

Returns: FetchURLsResponse: A response object with a list of URLs for a given search query.

🛠️ Usage

search_urls_response = carbon.utilities.search_urls(
    query="query_example",
)

⚙️ Parameters

query: `str`

🔄 Return

FetchURLsResponse

🌐 Endpoint

/search_urls get

`carbon.utilities.user_webpages`

User Web Pages

🛠️ Usage

user_webpages_response = carbon.utilities.user_webpages(
    filters={},
    pagination={
        "limit": 10,
        "offset": 0,
    },
    order_by="created_at",
    order_dir="asc",
)

⚙️ Parameters

filters: `UserWebPagesFilters`

pagination: `Pagination`

order_by: `UserWebPageOrderByTypes`

order_dir: `OrderDirV2`

⚙️ Request Body

UserWebPagesRequest

🌐 Endpoint

/user_webpages post

`carbon.webhooks.add_url`

Add Webhook Url

🛠️ Usage

add_url_response = carbon.webhooks.add_url(
    url="string_example",
)

⚙️ Parameters

url: `str`

⚙️ Request Body

AddWebhookProps

🔄 Return

Webhook

🌐 Endpoint

/add_webhook post

`carbon.webhooks.delete_url`

Delete Webhook Url

🛠️ Usage

delete_url_response = carbon.webhooks.delete_url(
    webhook_id=1,
)

⚙️ Parameters

webhook_id: `int`

🔄 Return

🌐 Endpoint

/delete_webhook/{webhook_id} delete

`carbon.webhooks.urls`

Webhook Urls

🛠️ Usage

urls_response = carbon.webhooks.urls(
    pagination={
        "limit": 10,
        "offset": 0,
    },
    order_by="created_at",
    order_dir="desc",
    filters={
        "ids": [],
    },
)

⚙️ Parameters

pagination: `Pagination`

order_by: `WebhookOrderByColumns`

order_dir: `OrderDir`

filters: `WebhookFilters`

⚙️ Request Body

WebhookQueryInput

🔄 Return

WebhookQueryResponse

🌐 Endpoint

/webhooks post