API Reference

Create

class lqs.client.core.create.Create(app)

Create a digestion.

Parameters:

log_id (str) – The ID of the log to which the digestion should be added.
name (str, optional) – The name of the digestion. Defaults to None.
context (dict, optional) – The context to use for the digestion. Defaults to None.
note (str, optional) – A note about the digestion. Defaults to None.
locked (bool, optional) – Whether the digestion is locked. Defaults to False.
workflow_id (str, optional) – The ID of the workflow to use for the digestion. Defaults to None.
workflow_context (dict, optional) – The context to use for the workflow. Defaults to None.
state (ProcessState, optional) – The state of the digestion. Defaults to ProcessState.ready.

Returns:

A data response with the created digestion.

Return type:

DigestionDataResponse

digestion_part(digestion_id: UUID, sequence: int, locked: bool | None = False, workflow_id: UUID | None = None, workflow_context: dict | None = None, state: ProcessState = ProcessState.ready, index: List[DigestionPartIndex] | None = None, lock_token: str | None = None)

Create a digestion part.

Parameters:

digestion_id (str) – The ID of the digestion to which the digestion part should be added.
sequence (int) – The sequence of the digestion part.
locked (bool, optional) – Whether the digestion part is locked. Defaults to False.
workflow_id (str, optional) – The ID of the workflow to use for the digestion part. Defaults to None.
workflow_context (dict, optional) – The context to use for the workflow. Defaults to None.
state (ProcessState, optional) – The state of the digestion part. Defaults to ProcessState.ready.
index (int, optional) – The index of the digestion part. Defaults to None.

Returns:

A data response with the created digestion part.

Return type:

DigestionPartDataResponse

Create a digestion topic.

Parameters:

digestion_id (str) – The ID of the digestion to which the digestion topic should be added.
topic_id (str) – The ID of the topic to be digested.
start_time (int, optional) – The start time of the digestion topic. Defaults to None.
end_time (int, optional) – The end time of the digestion topic. Defaults to None.
frequency (float, optional) – The frequency of the digestion topic. Defaults to None.
query_data_filter (dict, optional) – The data filter of the digestion topic. Defaults to None.
context_filter (dict, optional) – The context filter of the digestion topic. Defaults to None.

Returns:

A data response with the created digestion topic.

Return type:

DigestionTopicDataResponse

group(name: str, note: str | None = None, context: dict | None = None, locked: bool | None = False, default_workflow_id: UUID | None = None)

Create a group.

Parameters:

name (str) – The name of the group.
note (str, optional) – A note about the group. Defaults to None.
context (dict, optional) – The context to use for the group. Defaults to None.
locked (bool, optional) – Whether the group is locked. Defaults to False.
default_workflow_id (str, optional) – The ID of the default workflow for the group. Defaults to None.

Returns:

A data response with the created group.

Return type:

GroupDataResponse

Create a hook.

Parameters:

workflow_id (str) – The ID of the workflow to which the hook should be added.
trigger_process (str) – The process to trigger.
trigger_state (str) – The state to trigger.
name (str, optional) – The name of the hook. Defaults to None.
note (str, optional) – A note about the hook. Defaults to None.
context (dict, optional) – The context to use for the hook. Defaults to None.
managed (bool, optional) – Whether the hook is managed. Defaults to False.
disabled (bool, optional) – Whether the hook is disabled. Defaults to False.
uri (str, optional) – The URI of the hook. Defaults to None.
secret (str, optional) – The secret of the hook. Defaults to None.

Returns:

A data response with the created hook.

Return type:

HookDataResponse

Create an ingestion.

Parameters:

log_id (str) – The ID of the log to which the ingestion should be added.
name (str, optional) – The name of the ingestion. Defaults to None.
note (str, optional) – A note about the ingestion. Defaults to None.
context (dict, optional) – The context to use for the ingestion. Defaults to None.
object_store_id (str, optional) – The ID of the object store to use for the ingestion. Defaults to None.
object_key (str, optional) – The key of the object to use for the ingestion. Defaults to None.
locked (bool, optional) – Whether the ingestion is locked. Defaults to False.
workflow_id (str, optional) – The ID of the workflow to use for the ingestion. Defaults to None.
workflow_context (dict, optional) – The context to use for the workflow. Defaults to None.
state (ProcessState, optional) – The state of the ingestion. Defaults to ProcessState.ready.

Returns:

A data response with the created ingestion.

Return type:

IngestionDataResponse

ingestion_part(ingestion_id: UUID, sequence: int, source: str | None = None, locked: bool | None = False, workflow_id: UUID | None = None, workflow_context: dict | None = None, state: ProcessState = ProcessState.ready, index: List[IngestionPartIndex] | None = None, lock_token: str | None = None)

Create an ingestion part.

Parameters:

ingestion_id (str) – The ID of the ingestion to which the ingestion part should be added.
sequence (int) – The sequence of the ingestion part.
source (str, optional) – The source of the ingestion part. Defaults to None.
locked (bool, optional) – Whether the ingestion part is locked. Defaults to False.
workflow_id (str, optional) – The ID of the workflow to use for the ingestion part. Defaults to None.
workflow_context (dict, optional) – The context to use for the workflow. Defaults to None.
state (ProcessState, optional) – The state of the ingestion part. Defaults to ProcessState.queued.
index (int, optional) – The index of the ingestion part. Defaults to None.

Returns:

A data response with the created ingestion part.

Return type:

IngestionPartDataResponse

label(value: str, note: str | None = None)

Create a label.

Parameters:

value (str) – The value of the label.
note (str, optional) – A note about the label. Defaults to None.

Returns:

A data response with the created label.

Return type:

LabelDataResponse

Create a log.

Parameters:

group_id (str) – The ID of the group to which the log should be added.
name (str) – The name of the log.
note (str, optional) – A note about the log. Defaults to None.
context (dict, optional) – The context to use for the log. Defaults to None.
locked (bool, optional) – Whether the log is locked. Defaults to False.
default_workflow_id (str, optional) – The ID of the default workflow for the log. Defaults to None.

Returns:

A data response with the created log.

Return type:

LogDataResponse

log_object(key: str, log_id: UUID, content_type: str | None = None, lock_token: str | None = None)

Create a log object.

Parameters:

key (str) – The key of the log object.
log_id (str) – The ID of the log to which the log object should be added.
content_type (str, optional) – The content type of the log object. Defaults to None.

Returns:

A data response with the created log object.

Return type:

LogObjectDataResponse

log_object_part(object_key: str, size: int, log_id: UUID, part_number: int | None = None)

Create a log object part.

Parameters:

object_key (str) – The key of the log object to which the log object part should be added.
size (int) – The size of the log object part.
log_id (str) – The ID of the log to which the log object part should be added.
part_number (int, optional) – The part number of the log object part. Defaults to None.

Returns:

A data response with the created log object part.

Return type:

LogObjectPartDataResponse

object(key: str, object_store_id: UUID, content_type: str | None = None)

Create an object.

Parameters:

key (str) – The key of the object.
object_store_id (str) – The ID of the object store to which the object should be added.
content_type (str, optional) – The content type of the object. Defaults to None.

Returns:

A data response with the created object.

Return type:

ObjectDataResponse

object_part(object_key: str, size: int, object_store_id: UUID, part_number: int | None = None)

Create an object part.

Parameters:

object_key (str) – The key of the object to which the object part should be added.
size (int) – The size of the object part.
object_store_id (str) – The ID of the object store to which the object part should be added.
part_number (int, optional) – The part number of the object part. Defaults to None.

Returns:

A data response with the created object part.

Return type:

ObjectPartDataResponse

Create an object store.

Parameters:

bucket_name (str) – The name of the bucket.
access_key_id (str, optional) – The access key ID of the object store. Defaults to None.
secret_access_key (str, optional) – The secret access key of the object store. Defaults to None.
region_name (str, optional) – The region name of the object store. Defaults to None.
endpoint_url (str, optional) – The endpoint URL of the object store. Defaults to None.
note (str, optional) – A note about the object store. Defaults to None.
context (dict, optional) – The context to use for the object store. Defaults to None.
disabled (bool, optional) – Whether the object store is disabled. Defaults to False.

Returns:

A data response with the created object store.

Return type:

ObjectStoreDataResponse

Create a query.

Parameters:

log_id (str) – The ID of the log to which the query should be added.
name (str, optional) – The name of the query. Defaults to None.
note (str, optional) – A note about the query. Defaults to None.
context (dict, optional) – The context to use for the query. Defaults to None.
statement (str, optional) – The statement of the query. Defaults to None.
parameters (dict, optional) – The parameters of the query. Defaults to None.

Returns:

A data response with the created query.

Return type:

QueryDataResponse

Create a record.

Parameters:

timestamp (int) – The timestamp of the record.
topic_id (str) – The ID of the topic to which the record should be added.
note (str, optional) – A note about the record. Defaults to None.
context (dict, optional) – The context to use for the record. Defaults to None.
locked (bool, optional) – Whether the record is locked. Defaults to False.
query_data (dict, optional) – A JSON representation of the record’s message data which is queryable. Defaults to None.
auxiliary_data (dict, optional) – A JSON representation of the record’s message data which is not queryable. Defaults to None.
data_offset (int, optional) – The data offset of the record. Defaults to None.
data_length (int, optional) – The data length of the record. Defaults to None.
chunk_compression (str, optional) – The chunk compression of the record. Defaults to None.
chunk_offset (int, optional) – The chunk offset of the record. Defaults to None.
chunk_length (int, optional) – The chunk length of the record. Defaults to None.
source (str, optional) – The source of the record. Defaults to None.

Returns:

A data response with the created record.

Return type:

RecordDataResponse

Create a tag.

Parameters:

label_id (str) – The ID of the label to which the tag should be added.
log_id (str) – The ID of the log to which the tag should be added.
topic_id (str, optional) – The ID of the topic to which the tag should be added. Defaults to None.
note (str, optional) – A note about the tag. Defaults to None.
context (dict, optional) – The context to use for the tag. Defaults to None.
start_time (int, optional) – The start time of the tag. Defaults to None.
end_time (int, optional) – The end time of the tag. Defaults to None.

Returns:

A data response with the created tag.

Return type:

TagDataResponse

Create a topic.

Parameters:

log_id (str) – The ID of the log to which the topic should be added.
name (str) – The name of the topic.
note (str, optional) – A note about the topic. Defaults to None.
context (dict, optional) – The context to use for the topic. Defaults to None.
associated_topic_id (str, optional) – The ID of the associated topic. Defaults to None.
locked (bool, optional) – Whether the topic is locked. Defaults to False.
strict (bool, optional) – Whether the topic is strict. Defaults to False.
type_name (str, optional) – The type name of the topic. Defaults to None.
type_encoding (str, optional) – The type encoding of the topic. Defaults to None.
type_data (str, optional) – The type data of the topic. Defaults to None.
type_schema (str, optional) – The type schema of the topic. Defaults to None.

Returns:

A data response with the created topic.

Return type:

TopicDataResponse

Create a workflow.

Parameters:

name (str) – The name of the workflow.
note (str, optional) – A note about the workflow. Defaults to None.
context (dict, optional) – The context to use for the workflow. Defaults to None.
default (bool, optional) – Whether the workflow is default. Defaults to False.
disabled (bool, optional) – Whether the workflow is disabled. Defaults to False.
managed (bool, optional) – Whether the workflow is managed. Defaults to False.
context_schema (dict, optional) – The context schema of the workflow. Defaults to None.

Returns:

A data response with the created workflow.

Return type:

WorkflowDataResponse

List

class lqs.client.core.list.List(app)

digestion(id: UUID | None = None, group_id: UUID | None = None, log_id: UUID | None = None, workflow_id: UUID | None = None, workflow_id_null: bool | None = None, state: str | None = None, name: str | None = None, name_like: str | None = None, progress_null: bool | None = None, progress_gte: float | None = None, progress_lte: float | None = None, error_filter: str | None = None, note_like: str | None = None, context_filter: str | None = None, workflow_context_filter: str | None = None, include_count: bool | None = True, offset: int | None = 0, limit: int | None = 100, order: str | None = 'created_at', sort: str | None = 'ASC', created_by: UUID | None = None, updated_by: UUID | None = None, deleted_by: UUID | None = None, updated_by_null: bool | None = None, deleted_by_null: bool | None = None, updated_at_null: bool | None = None, deleted_at_null: bool | None = None, created_at_lte: datetime | None = None, updated_at_lte: datetime | None = None, deleted_at_lte: datetime | None = None, created_at_gte: datetime | None = None, updated_at_gte: datetime | None = None, deleted_at_gte: datetime | None = None): Lists digestions.

digestion_part(id: UUID | None = None, group_id: UUID | None = None, log_id: UUID | None = None, sequence: int | None = None, digestion_id: UUID | None = None, workflow_id: UUID | None = None, workflow_id_null: bool | None = None, state: str | None = None, include_count: bool | None = True, offset: int | None = 0, limit: int | None = 100, order: str | None = 'created_at', sort: str | None = 'ASC', created_by: UUID | None = None, updated_by: UUID | None = None, deleted_by: UUID | None = None, updated_by_null: bool | None = None, deleted_by_null: bool | None = None, updated_at_null: bool | None = None, deleted_at_null: bool | None = None, created_at_lte: datetime | None = None, updated_at_lte: datetime | None = None, deleted_at_lte: datetime | None = None, created_at_gte: datetime | None = None, updated_at_gte: datetime | None = None, deleted_at_gte: datetime | None = None): Lists digestion parts.

digestion_topic(id: UUID | None = None, digestion_id: UUID | None = None, group_id: UUID | None = None, log_id: UUID | None = None, topic_id: UUID | None = None, start_time_null: bool | None = None, start_time_gte: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, start_time_lte: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, end_time_null: bool | None = None, end_time_gte: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, end_time_lte: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, include_count: bool | None = True, offset: int | None = 0, limit: int | None = 100, order: str | None = 'created_at', sort: str | None = 'ASC', created_by: UUID | None = None, updated_by: UUID | None = None, deleted_by: UUID | None = None, updated_by_null: bool | None = None, deleted_by_null: bool | None = None, updated_at_null: bool | None = None, deleted_at_null: bool | None = None, created_at_lte: datetime | None = None, updated_at_lte: datetime | None = None, deleted_at_lte: datetime | None = None, created_at_gte: datetime | None = None, updated_at_gte: datetime | None = None, deleted_at_gte: datetime | None = None): Lists digestion topics.

group(id: UUID | None = None, name: str | None = None, name_like: str | None = None, default_workflow_id: UUID | None = None, default_workflow_id_null: bool | None = None, include_count: bool | None = True, offset: int | None = 0, limit: int | None = 100, order: str | None = 'created_at', sort: str | None = 'ASC', created_by: UUID | None = None, updated_by: UUID | None = None, deleted_by: UUID | None = None, updated_by_null: bool | None = None, deleted_by_null: bool | None = None, updated_at_null: bool | None = None, deleted_at_null: bool | None = None, created_at_lte: datetime | None = None, updated_at_lte: datetime | None = None, deleted_at_lte: datetime | None = None, created_at_gte: datetime | None = None, updated_at_gte: datetime | None = None, deleted_at_gte: datetime | None = None): Lists groups.

hook(id: UUID | None = None, workflow_id: UUID | None = None, trigger_process: str | None = None, trigger_state: str | None = None, name: str | None = None, name_like: str | None = None, note_like: str | None = None, managed: bool | None = None, disabled: bool | None = None, uri: str | None = None, uri_like: str | None = None, include_count: bool | None = True, offset: int | None = 0, limit: int | None = 100, order: str | None = 'created_at', sort: str | None = 'ASC', created_by: UUID | None = None, updated_by: UUID | None = None, deleted_by: UUID | None = None, updated_by_null: bool | None = None, deleted_by_null: bool | None = None, updated_at_null: bool | None = None, deleted_at_null: bool | None = None, created_at_lte: datetime | None = None, updated_at_lte: datetime | None = None, deleted_at_lte: datetime | None = None, created_at_gte: datetime | None = None, updated_at_gte: datetime | None = None, deleted_at_gte: datetime | None = None): Lists hooks.

ingestion(id: UUID | None = None, group_id: UUID | None = None, log_id: UUID | None = None, object_store_id: UUID | None = None, name: str | None = None, name_like: str | None = None, object_key: str | None = None, object_key_like: str | None = None, workflow_id: UUID | None = None, workflow_id_null: bool | None = None, workflow_context_filter: str | None = None, state: str | None = None, progress_null: bool | None = None, progress_gte: float | None = None, progress_lte: float | None = None, error_filter: str | None = None, note_like: str | None = None, context_filter: str | None = None, include_count: bool | None = True, offset: int | None = 0, limit: int | None = 100, order: str | None = 'created_at', sort: str | None = 'ASC', created_by: UUID | None = None, updated_by: UUID | None = None, deleted_by: UUID | None = None, updated_by_null: bool | None = None, deleted_by_null: bool | None = None, updated_at_null: bool | None = None, deleted_at_null: bool | None = None, created_at_lte: datetime | None = None, updated_at_lte: datetime | None = None, deleted_at_lte: datetime | None = None, created_at_gte: datetime | None = None, updated_at_gte: datetime | None = None, deleted_at_gte: datetime | None = None): Lists ingestions.

ingestion_part(id: UUID | None = None, group_id: UUID | None = None, log_id: UUID | None = None, ingestion_id: UUID | None = None, sequence: int | None = None, source: str | None = None, workflow_id: UUID | None = None, workflow_id_null: bool | None = None, state: str | None = None, include_count: bool | None = True, offset: int | None = 0, limit: int | None = 100, order: str | None = 'created_at', sort: str | None = 'ASC', created_by: UUID | None = None, updated_by: UUID | None = None, deleted_by: UUID | None = None, updated_by_null: bool | None = None, deleted_by_null: bool | None = None, updated_at_null: bool | None = None, deleted_at_null: bool | None = None, created_at_lte: datetime | None = None, updated_at_lte: datetime | None = None, deleted_at_lte: datetime | None = None, created_at_gte: datetime | None = None, updated_at_gte: datetime | None = None, deleted_at_gte: datetime | None = None): Lists ingestion parts.

label(id: UUID | None = None, value: str | None = None, value_like: str | None = None, note_like: str | None = None, include_count: bool | None = True, offset: int | None = 0, limit: int | None = 100, order: str | None = 'created_at', sort: str | None = 'ASC', created_by: UUID | None = None, updated_by: UUID | None = None, deleted_by: UUID | None = None, updated_by_null: bool | None = None, deleted_by_null: bool | None = None, updated_at_null: bool | None = None, deleted_at_null: bool | None = None, created_at_lte: datetime | None = None, updated_at_lte: datetime | None = None, deleted_at_lte: datetime | None = None, created_at_gte: datetime | None = None, updated_at_gte: datetime | None = None, deleted_at_gte: datetime | None = None): Lists labels.

log(id: UUID | None = None, group_id: UUID | None = None, default_workflow_id: UUID | None = None, default_workflow_id_null: bool | None = None, name: str | None = None, name_like: str | None = None, locked: bool | None = None, note_like: str | None = None, context_filter: str | None = None, start_time_null: bool | None = None, start_time_gte: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, start_time_lte: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, end_time_null: bool | None = None, end_time_gte: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, end_time_lte: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, record_size_gte: int | None = None, record_size_lte: int | None = None, record_count_gte: int | None = None, record_count_lte: int | None = None, object_size_gte: int | None = None, object_size_lte: int | None = None, object_count_gte: int | None = None, object_count_lte: int | None = None, tag_label_ids_includes: UUID | List[UUID] | None = None, include_count: bool | None = True, offset: int | None = 0, limit: int | None = 100, order: str | None = 'created_at', sort: str | None = 'ASC', created_by: UUID | None = None, updated_by: UUID | None = None, deleted_by: UUID | None = None, updated_by_null: bool | None = None, deleted_by_null: bool | None = None, updated_at_null: bool | None = None, deleted_at_null: bool | None = None, created_at_lte: datetime | None = None, updated_at_lte: datetime | None = None, deleted_at_lte: datetime | None = None, created_at_gte: datetime | None = None, updated_at_gte: datetime | None = None, deleted_at_gte: datetime | None = None): Lists logs.

log_object(log_id: UUID, processing: bool | None = False, max_keys: int | None = None, prefix: str | None = None, start_after: str | None = None, delimiter: str | None = None, continuation_token: str | None = None): Lists log objects.

log_object_part(object_key: str, log_id: UUID, max_parts: int | None = None, part_number_marker: int | None = None): Lists log object parts.

object(object_store_id: UUID, processing: bool | None = False, max_keys: int | None = None, prefix: str | None = None, start_after: str | None = None, delimiter: str | None = None, continuation_token: str | None = None): Lists objects.

object_part(object_key: str, object_store_id: UUID, max_parts: int | None = None, part_number_marker: int | None = None): Lists object parts.

object_store(id: UUID | None = None, bucket_name: str | None = None, bucket_name_like: str | None = None, access_key_id: str | None = None, region_name: str | None = None, endpoint_url: str | None = None, note: str | None = None, note_like: str | None = None, disabled: bool | None = None, include_count: bool | None = True, offset: int | None = 0, limit: int | None = 100, order: str | None = 'created_at', sort: str | None = 'ASC', created_by: UUID | None = None, updated_by: UUID | None = None, deleted_by: UUID | None = None, updated_by_null: bool | None = None, deleted_by_null: bool | None = None, updated_at_null: bool | None = None, deleted_at_null: bool | None = None, created_at_lte: datetime | None = None, updated_at_lte: datetime | None = None, deleted_at_lte: datetime | None = None, created_at_gte: datetime | None = None, updated_at_gte: datetime | None = None, deleted_at_gte: datetime | None = None): Lists object stores.

query(id: UUID | None = None, log_id: UUID | None = None, name: str | None = None, name_like: str | None = None, note_like: str | None = None, statement: str | None = None, statement_like: str | None = None, workflow_id: UUID | None = None, workflow_id_null: bool | None = None, workflow_context_filter: str | None = None, context_filter: str | None = None, include_count: bool | None = True, offset: int | None = 0, limit: int | None = 100, order: str | None = 'created_at', sort: str | None = 'ASC', created_by: UUID | None = None, updated_by: UUID | None = None, deleted_by: UUID | None = None, updated_by_null: bool | None = None, deleted_by_null: bool | None = None, updated_at_null: bool | None = None, deleted_at_null: bool | None = None, created_at_lte: datetime | None = None, updated_at_lte: datetime | None = None, deleted_at_lte: datetime | None = None, created_at_gte: datetime | None = None, updated_at_gte: datetime | None = None, deleted_at_gte: datetime | None = None): Lists queries.

record(topic_id: UUID, timestamp: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, log_id: UUID | None = None, ingestion_id: UUID | None = None, workflow_id: UUID | None = None, workflow_id_null: bool | None = None, error_filter: str | None = None, note_like: str | None = None, source: str | None = None, query_data_filter: str | None = None, context_filter: str | None = None, altered: bool | None = None, frequency: float | None = None, timestamp_gt: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, timestamp_lt: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, timestamp_gte: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, timestamp_lte: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, data_length_gte: int | None = None, data_length_lte: int | None = None, data_offset_gte: int | None = None, data_offset_lte: int | None = None, chunk_compression: str | None = None, chunk_offset_gte: int | None = None, chunk_offset_lte: int | None = None, chunk_length_gte: int | None = None, chunk_length_lte: int | None = None, include_auxiliary_data: bool | None = False, include_raw_data: bool | None = False, include_count: bool | None = True, offset: int | None = 0, limit: int | None = 100, order: str | None = 'timestamp', sort: str | None = 'ASC', created_by: UUID | None = None, updated_by: UUID | None = None, deleted_by: UUID | None = None, updated_by_null: bool | None = None, deleted_by_null: bool | None = None, updated_at_null: bool | None = None, deleted_at_null: bool | None = None, created_at_lte: datetime | None = None, updated_at_lte: datetime | None = None, deleted_at_lte: datetime | None = None, created_at_gte: datetime | None = None, updated_at_gte: datetime | None = None, deleted_at_gte: datetime | None = None): Lists records.

tag(id: UUID | None = None, log_id: UUID | None = None, label_id: UUID | None = None, topic_id: UUID | None = None, note: str | None = None, note_like: str | None = None, context_filter: str | None = None, start_time_null: bool | None = None, start_time_gte: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, start_time_lte: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, end_time_null: bool | None = None, end_time_gte: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, end_time_lte: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, include_count: bool | None = True, offset: int | None = 0, limit: int | None = 100, order: str | None = 'created_at', sort: str | None = 'ASC', created_by: UUID | None = None, updated_by: UUID | None = None, deleted_by: UUID | None = None, updated_by_null: bool | None = None, deleted_by_null: bool | None = None, updated_at_null: bool | None = None, deleted_at_null: bool | None = None, created_at_lte: datetime | None = None, updated_at_lte: datetime | None = None, deleted_at_lte: datetime | None = None, created_at_gte: datetime | None = None, updated_at_gte: datetime | None = None, deleted_at_gte: datetime | None = None): Lists tags.

topic(id: UUID | None = None, log_id: UUID | None = None, group_id: UUID | None = None, name: str | None = None, name_like: str | None = None, associated_topic_id: UUID | None = None, latched: bool | None = None, strict: bool | None = None, locked: bool | None = None, context_filter: str | None = None, start_time_null: bool | None = None, start_time_gte: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, start_time_lte: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, end_time_null: bool | None = None, end_time_gte: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, end_time_lte: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)] | None = None, record_size_gte: int | None = None, record_size_lte: int | None = None, record_count_gte: int | None = None, record_count_lte: int | None = None, type_name: str | None = None, type_name_like: str | None = None, type_encoding: TypeEncoding | None = None, type_data: str | None = None, type_data_like: str | None = None, type_schema_filter: str | None = None, include_count: bool | None = True, offset: int | None = 0, limit: int | None = 100, order: str | None = 'created_at', sort: str | None = 'ASC', created_by: UUID | None = None, updated_by: UUID | None = None, deleted_by: UUID | None = None, updated_by_null: bool | None = None, deleted_by_null: bool | None = None, updated_at_null: bool | None = None, deleted_at_null: bool | None = None, created_at_lte: datetime | None = None, updated_at_lte: datetime | None = None, deleted_at_lte: datetime | None = None, created_at_gte: datetime | None = None, updated_at_gte: datetime | None = None, deleted_at_gte: datetime | None = None): Lists topics.

workflow(id: UUID | None = None, name: str | None = None, name_like: str | None = None, default: bool | None = None, disabled: bool | None = None, managed: bool | None = None, context_schema_filter: str | None = None, include_count: bool | None = True, offset: int | None = 0, limit: int | None = 100, order: str | None = 'created_at', sort: str | None = 'ASC', created_by: UUID | None = None, updated_by: UUID | None = None, deleted_by: UUID | None = None, updated_by_null: bool | None = None, deleted_by_null: bool | None = None, updated_at_null: bool | None = None, deleted_at_null: bool | None = None, created_at_lte: datetime | None = None, updated_at_lte: datetime | None = None, deleted_at_lte: datetime | None = None, created_at_gte: datetime | None = None, updated_at_gte: datetime | None = None, deleted_at_gte: datetime | None = None): Lists workflows.

Fetch

class lqs.client.core.fetch.Fetch(app)

digestion(digestion_id: UUID)

Fetches a digestion by ID.

Args:: digestion_id: The ID of the digestion to fetch.
Returns:: A data response for the digestion.

digestion_part(digestion_part_id: UUID, digestion_id: UUID | None = None)

Fetches a digestion part by ID.

Args:: digestion_part_id: The ID of the digestion part to fetch. digestion_id: The ID of the digestion to which the digestion part belongs.
Returns:: A data response for the digestion part.

digestion_topic(digestion_topic_id: UUID, digestion_id: UUID | None = None)

Fetches a digestion topic by ID.

Args:: digestion_topic_id: The ID of the digestion topic to fetch. digestion_id: The ID of the digestion to which the digestion topic belongs.
Returns:: A data response for the digestion topic.

group(group_id: UUID)

Fetches a group by ID.

Args:: group_id: The ID of the group to fetch.
Returns:: A data response for the group.

hook(hook_id: UUID, workflow_id: UUID | None = None)

Fetches a hook by ID.

Args:: hook_id: The ID of the hook to fetch. workflow_id: The ID of the workflow to which the hook belongs.
Returns:: A data response for the hook.

ingestion(ingestion_id: UUID)

Fetches an ingestion by ID.

Args:: ingestion_id: The ID of the ingestion to fetch.
Returns:: A data response for the ingestion.

ingestion_part(ingestion_part_id: UUID, ingestion_id: UUID | None = None)

Fetches an ingestion part by ID.

Args:: ingestion_part_id: The ID of the ingestion part to fetch. ingestion_id: The ID of the ingestion to which the ingestion part belongs.
Returns:: A data response for the ingestion part.

label(label_id: UUID)

Fetches a label by ID.

Args:: label_id: The ID of the label to fetch.
Returns:: A data response for the label.

log(log_id: UUID)

Fetches a log by ID.

Args:: log_id: The ID of the log to fetch.
Returns:: A data response for the log.

log_object(object_key: str, log_id: UUID, redirect: bool | None = False, offset: int | None = None, length: int | None = None) → ObjectDataResponse | bytes

Fetches an object by key.

Args:: object_key: The key of the object to fetch. log_id: The ID of the log to which the object belongs. redirect: Whether to redirect to the object store or return the object directly. Defaults to False. offset: The offset from which to read the object. length: The length of the object to read.
Returns:: A data response for the object or the object itself as bytes if redirect is True.

log_object_part(object_key: str, part_number: int, log_id: UUID)

Fetches an object part by key and part number.

Args:: object_key: The key of the object to fetch. part_number: The part number of the object to fetch. log_id: The ID of the log to which the object belongs.
Returns:: A data response for the object part.

object(object_key: str, object_store_id: UUID, redirect: bool | None = False, offset: int | None = None, length: int | None = None) → ObjectDataResponse | bytes

Fetches an object by key.

Args:: object_key: The key of the object to fetch. object_store_id: The ID of the object store to which the object belongs. redirect: Whether to redirect to the object store or return the object directly. Defaults to False. offset: The offset from which to read the object. length: The length of the object to read.
Returns:: A data response for the object or the object itself as bytes if redirect is True.

object_part(object_key: str, part_number: int, object_store_id: UUID)

Fetches an object part by key and part number.

Args:: object_key: The key of the object to fetch. part_number: The part number of the object to fetch. object_store_id: The ID of the object store to which the object belongs.
Returns:: A data response for the object part.

object_store(object_store_id: UUID)

Fetches an object store by ID.

Args:: object_store_id: The ID of the object store to fetch.
Returns:: A data response for the object store.

query(query_id: UUID, log_id: UUID | None = None)

Fetches a query by ID.

Args:: query_id: The ID of the query to fetch. log_id: The ID of the log to which the query belongs.
Returns:: A data response for the query.

record(timestamp: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)], topic_id: UUID, include_auxiliary_data: bool = False, include_raw_data: bool = False, auxiliary_context: str | None = None)

Fetches a record by timestamp and topic ID.

Args:: timestamp: The timestamp of the record to fetch. topic_id: The ID of the topic to which the record belongs. include_auxiliary_data: Whether to include auxiliary data in the record. Defaults to False. include_raw_data: Whether to include raw data in the record. Defaults to False. auxiliary_context: The context for the auxiliary data generation to include in the record. Defaults to None.
Returns:: A data response for the record.

tag(tag_id: UUID, log_id: UUID | None = None)

Fetches a tag by ID.

Args:: tag_id: The ID of the tag to fetch. log_id: The ID of the log to which the tag belongs.
Returns:: A data response for the tag.

topic(topic_id: UUID)

Fetches a topic by ID.

Args:: topic_id: The ID of the topic to fetch.
Returns:: A data response for the topic.

workflow(workflow_id: UUID)

Fetches a workflow by ID.

Args:: workflow_id: The ID of the workflow to fetch.
Returns:: A data response for the workflow.

Update

class lqs.client.core.update.Update(app)

digestion(digestion_id: UUID, data: DigestionUpdateRequest, lock_token: str | None = None)

Update a digestion.

Args:: digestion_id: The id of the digestion to update. data: The data to update the digestion with. lock_token: TODO: update this description
Returns:: A data response containing the updated digestion.

digestion_part(digestion_part_id: UUID, data: DigestionPartUpdateRequest, digestion_id: UUID | None = None, lock_token: str | None = None)

Update a digestion part.

Args:: digestion_part_id: The id of the digestion part to update. data: The data to update the digestion part with. digestion_id (optional): The id of the digestion to which the digestion part belongs.
Returns:: A data response containing the updated digestion part.

digestion_topic(digestion_topic_id: UUID, data: DigestionTopicUpdateRequest, digestion_id: UUID | None = None, lock_token: str | None = None)

Update a digestion topic.

Args:: digestion_topic_id: The id of the digestion topic to update. data: The data to update the digestion topic with. digestion_id (optional): The id of the digestion to which the digestion topic belongs.
Returns:: A data response containing the updated digestion topic.

group(group_id: UUID, data: GroupUpdateRequest, lock_token: str | None = None)

Update a group.

Args:: group_id: The id of the group to update. data: The data to update the group with.
Returns:: A data response containing the updated group.

hook(hook_id: UUID, data: HookUpdateRequest, workflow_id: UUID | None = None)

Update a hook.

Args:: hook_id: The id of the hook to update. data: The data to update the hook with. workflow_id (optional): The id of the workflow to which the hook belongs.
Returns:: A data response containing the updated hook.

ingestion(ingestion_id: UUID, data: IngestionUpdateRequest, lock_token: str | None = None)

Update an ingestion.

Args:: ingestion_id: The id of the ingestion to update. data: The data to update the ingestion with.
Returns:: A data response containing the updated ingestion.

ingestion_part(ingestion_part_id: UUID, data: IngestionPartUpdateRequest, ingestion_id: UUID | None = None, lock_token: str | None = None)

Update an ingestion part.

Args:: ingestion_part_id: The id of the ingestion part to update. data: The data to update the ingestion part with. ingestion_id (optional): The id of the ingestion to which the ingestion part belongs.
Returns:: A data response containing the updated ingestion part.

label(label_id: UUID, data: LabelUpdateRequest)

Update a label.

Args:: label_id: The id of the label to update. data: The data to update the label with.
Returns:: A data response containing the updated label.

log(log_id: UUID, data: LogUpdateRequest, lock_token: str | None = None)

Update a log.

Args:: log_id: The id of the log to update. data: The data to update the log with.
Returns:: A data response containing the updated log.

log_object(log_id: UUID, object_key: str, data: ObjectUpdateRequest, lock_token: str | None = None)

Update a log object.

Args:: log_id: The id of the log to which the object belongs. object_key: The key of the object to update. data: The data to update the object with.
Returns:: A data response containing the updated object.

object(object_store_id: UUID, object_key: str, data: ObjectUpdateRequest)

Update an object.

Args:: object_store_id: The id of the object store to which the object belongs. object_key: The key of the object to update. data: The data to update the object with.
Returns:: A data response containing the updated object.

object_store(object_store_id: UUID, data: ObjectStoreUpdateRequest)

Update an object store.

Args:: object_store_id: The id of the object store to update. data: The data to update the object store with.
Returns:: A data response containing the updated object store.

query(query_id: UUID, data: QueryUpdateRequest, log_id: UUID | None = None, lock_token: str | None = None)

Update a query.

Args:: query_id: The id of the query to update. data: The data to update the query with. log_id (optional): The id of the log to which the query belongs.
Returns:: A data response containing the updated query.

record(timestamp: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)], topic_id: UUID, data: RecordUpdateRequest, lock_token: str | None = None)

Update a record.

Args:: timestamp: The timestamp of the record to update. topic_id: The id of the topic to which the record belongs. data: The data to update the record with.
Returns:: A data response containing the updated record.

tag(tag_id: UUID, data: TagUpdateRequest, log_id: UUID | None = None, lock_token: str | None = None)

Update a tag.

Args:: tag_id: The id of the tag to update. data: The data to update the tag with. log_id (optional): The id of the log to which the tag belongs.
Returns:: A data response containing the updated tag.

topic(topic_id: UUID, data: TopicUpdateRequest, lock_token: str | None = None)

Update a topic.

Args:: topic_id: The id of the topic to update. data: The data to update the topic with.
Returns:: A data response containing the updated topic.

workflow(workflow_id: UUID, data: WorkflowUpdateRequest)

Update a workflow.

Args:: workflow_id: The id of the workflow to update. data: The data to update the workflow with.
Returns:: A data response containing the updated workflow.

Delete

class lqs.client.core.delete.Delete(app)

digestion(digestion_id: UUID, lock_token: str | None = None)

Delete a digestion.

Args:: digestion_id (UUID): ID of the digestion to delete.
Returns:: None

digestion_part(digestion_part_id: UUID, digestion_id: UUID | None = None, lock_token: str | None = None)

Delete a digestion part.

Args:: digestion_part_id (UUID): ID of the digestion part to delete. digestion_id (UUID, optional): ID of the digestion to which the digestion part belongs.
Returns:: None

digestion_topic(digestion_topic_id: UUID, digestion_id: UUID | None = None, lock_token: str | None = None)

Delete a digestion topic.

Args:: digestion_topic_id (UUID): ID of the digestion topic to delete. digestion_id (UUID, optional): ID of the digestion to which the digestion topic belongs.
Returns:: None

group(group_id: UUID, lock_token: str | None = None)

Delete a group.

Args:: group_id (UUID): ID of the group to delete.
Returns:: None

hook(hook_id: UUID, workflow_id: UUID | None = None)

Delete a hook.

Args:: hook_id (UUID): ID of the hook to delete. workflow_id (UUID, optional): ID of the workflow to which the hook belongs.
Returns:: None

ingestion(ingestion_id: UUID, lock_token: str | None = None)

Delete an ingestion.

Args:: ingestion_id (UUID): ID of the ingestion to delete.
Returns:: None

ingestion_part(ingestion_part_id: UUID, ingestion_id: UUID | None, lock_token: str | None = None)

Delete an ingestion part.

Args:: ingestion_part_id (UUID): ID of the ingestion part to delete. ingestion_id (UUID, optional): ID of the ingestion to which the ingestion part belongs.
Returns:: None

label(label_id: UUID)

Delete a label.

Args:: label_id (UUID): ID of the label to delete.
Returns:: None

log(log_id: UUID, lock_token: str | None = None)

Delete a log.

Args:: log_id (UUID): ID of the log to delete.
Returns:: None

log_object(log_id: UUID, object_key: str, lock_token: str | None = None)

Delete a log object.

Args:: log_id (UUID): ID of the log to which the log object belongs. object_key (str): Key of the log object to delete.
Returns:: None

object(object_store_id: UUID, object_key: str)

Delete an object.

Args:: object_store_id (UUID): ID of the object store to which the object belongs. object_key (str): Key of the object to delete.
Returns:: None

object_store(object_store_id: UUID)

Delete an object store.

Args:: object_store_id (UUID): ID of the object store to delete.
Returns:: None

query(query_id: UUID, log_id: UUID | None = None, lock_token: str | None = None)

Delete a query.

Args:: query_id (UUID): ID of the query to delete. log_id (UUID, optional): ID of the log to which the query belongs.
Returns:: None

record(timestamp: big_int_wrap, return_type=PydanticUndefined, when_used=json), Path(PydanticUndefined)], topic_id: UUID, lock_token: str | None = None)

Delete a record.

Args:: timestamp (int): Timestamp of the record to delete. topic_id (UUID): ID of the topic to which the record belongs.
Returns:: None

tag(tag_id: UUID, log_id: UUID | None = None, lock_token: str | None = None)

Delete a tag.

Args:: tag_id (UUID): ID of the tag to delete. log_id (UUID, optional): ID of the log to which the tag belongs.
Returns:: None

topic(topic_id: UUID, lock_token: str | None = None)

Delete a topic.

Args:: topic_id (UUID): ID of the topic to delete.
Returns:: None

workflow(workflow_id: UUID)

Delete a workflow.

Args:: workflow_id (UUID): ID of the workflow to delete.
Returns:: None

Utils

class lqs.client.utils.Utils(app: RESTClient)

calculate_etag(file_path: str, part_size: int = 104857600) → str

Calculate the ETag of a file.

This function calculates the ETag of a file assuming it is uploaded as a multipart upload.

Parameters:

file_path (str) – The path to the file to calculate the ETag for.
part_size (int, optional) – The size of each chunk to read from the file. Defaults to 100 * 1024 * 1024.

Returns:

The calculated ETag.

Return type:

str

download(object_key: str, file_path: str | None = None, object_store_id: str | None = None, log_id: str | None = None, continue_download: bool = False, skip_if_exists: bool = False, overwrite: bool = False, part_size: int = 104857600, max_workers: int | None = 10, start_offset: int = 0, end_offset: int | None = None, max_attempts: int = 3, backoff_factor: float = 5.0, connect_timeout: int = 60, read_timeout: int = 600) → int

Download an object’s data directly to a file.

Parameters:

object_key (str) – The key of the object.
file_path (str, optional) – The path to the file to download to. Defaults to None.
object_store_id (str, optional) – The object store id to use. Defaults to None.
log_id (str, optional) – The log id to use. Defaults to None.
continue_download (bool, optional) – Whether to continue an existing download. Defaults to False.
skip_if_exists (bool, optional) – Whether to skip the download if the file already exists. Defaults to False.
overwrite (bool, optional) – Whether to overwrite the file if it already exists. Defaults to False.
part_size (int, optional) – The size of each part to download. Defaults to 100 * 1024 * 1024.
max_workers (int, optional) – The maximum number of workers to use for parallel downloads. Defaults to 10.
start_offset (int, optional) – The offset to start downloading from. Defaults to 0.
end_offset (int, optional) – The offset to stop downloading at. Defaults to None.
max_attempts (int, optional) – The maximum number of attempts to make. Defaults to 3.
backoff_factor (float, optional) – The backoff factor to use. Defaults to 5.0.
connect_timeout (int, optional) – The connect timeout to use. Defaults to 60.
read_timeout (int, optional) – The read timeout to use. Defaults to 600.

Raises:

FileExistsError – If the file already exists and skip_if_exists is False.
Exception – If the file already exists and continue_download is False.

Returns:

The number of bytes downloaded.

Return type:

int

fetch_by_name_or_create(resource_type: str | None = None, resource_id: str | None = None, resource_name: str | None = None, list_params: dict = {}, create_if_missing: bool = True, create_params: dict = {}, create_func: Callable | None = None, list_func: Callable | None = None, fetch_func: Callable | None = None) → Any

Fetch or create a resource by name.

This function fetches a resource by id if provided, or by name if provided, or creates the resource if it doesn’t exist and create_if_missing is True.

If no resource id or name is provided, the function returns None.

Parameters:

resource_type (str) – The type of the resource.
resource_id (str, optional) – The id of the resource. Defaults to None.
resource_name (str, optional) – The name of the resource. Defaults to None.
list_params (dict, optional) – Additional parameters to use when listing the resource. Defaults to {}.
create_if_missing (bool, optional) – Whether to create the resource if it doesn’t exist. Defaults to True.
create_params (dict, optional) – Additional parameters to use when creating the resource. Defaults to {}.
create_func (Callable, optional) – The function to use when creating the resource. Defaults to None.
list_func (Callable, optional) – The function to use when listing the resource. Defaults to None.
fetch_func (Callable, optional) – The function to use when fetching the resource. Defaults to None.

Raises:

NotFoundException – If no resource is found and create_if_missing is False.

Returns:

The fetched or created resource, or None if no resource is found or created.

Return type:

Any

fetch_object_data_part(offset: int, length: int, object_key: str, object_store_id: str | None = None, log_id: str | None = None, max_attempts: int = 3, backoff_factor: float = 5.0, connect_timeout: int = 60, read_timeout: int = 600, data_container: bytearray | None = None) → tuple[bytearray, int]

Fetch a part of an object’s data.

Parameters:

offset (int) – The offset to start downloading from.
length (int) – The length of the data to download.
object_key (str) – The key of the object.
object_store_id (str, optional) – The object store id to use. Defaults to None.
log_id (str, optional) – The log id to use. Defaults to None.
max_attempts (int, optional) – The maximum number of attempts to make. Defaults to 3.
backoff_factor (float, optional) – The backoff factor to use. Defaults to 5.0.
connect_timeout (int, optional) – The connect timeout to use. Defaults to 60.
read_timeout (int, optional) – The read timeout to use. Defaults to 600.
data_container (bytearray, optional) – A bytearray to use as a container for the fetched data. Defaults to None.

Returns:

The downloaded data and the offset of the data.

Return type:

tuple[bytearray, int]

find_local_logs(root_dir: str, log_file_regexes: List[str] = ['/(?P<log_name>[^/]*?)\\.bag$', '/(?P<log_name>[^/]*?)\\.log$', '/(?P<log_name>[^/]*?)\\.mcap$', '/(?P<log_name>[^/]*?)\\.jsonl$', '/(?P<log_name>[^/]*)\\.log/log0$', '/(?P<log_name>[^/]*)/log0$', '/(?P<log_name>[^/]*)/manifests\\/[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-4[0-9a-fA-F]{3}-[89aAbB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}$'], max_depth: int | None = None) → List[dict]

Generate a list of log parameters from a directory of log files.

This function searches the root directory for log files using the given regular expressions. The log parameters include the log_path, log_directory, log_name, and group_name. The log_name and group_name are extracted from the corresponding named capture groups of the regular expression. If the log_name captured group is followed by a slash, the log directory is also extracted, otherwise it is set to None.

Parameters:

root_dir (str) – The directory to search for log files.
log_file_regexes (List[str], optional) – A list of regular expressions to match log files. Defaults to DEFAULT_LOG_FILE_REGEXES.
max_depth (int, optional) – The maximum depth to search for log files. Defaults to None.

Returns:

A list of log parameters.

Return type:

List[dict]

find_object_store_logs(object_store_id: UUID | None = None, bucket_name: str | None = None, root_prefix: str = '', skip_prefixes: List[str] = [], skip_duplicates: bool = True, log_file_regexes: List[str] = ['/(?P<log_name>[^/]*?)\\.bag$', '/(?P<log_name>[^/]*?)\\.log$', '/(?P<log_name>[^/]*?)\\.mcap$', '/(?P<log_name>[^/]*?)\\.jsonl$', '/(?P<log_name>[^/]*)\\.log/log0$', '/(?P<log_name>[^/]*)/log0$', '/(?P<log_name>[^/]*)/manifests\\/[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-4[0-9a-fA-F]{3}-[89aAbB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}$'], max_objects_to_list: int | None = 100000, fail_if_more_than_max_objects_to_list: bool = False) → List[dict]

Find logs in an object store.

This function finds log objects in a given object store bucket. It uses a list of regular expressions to match log file names. One of object_store_id or bucket_name must be provided, but not both. Searching starts from the root_prefix and continues recursively. If skip_duplicates is set to True, only unique logs based on the found log name, group name, the object’s Etag, and the object’s size, and the first unique object encountered is what is used. Specific prefixes can be skipped using skip_prefixes. We only list up to max_objects_to_list objects, and if fail_if_more_than_max_objects_to_list is set to True, we raise an exception if we find more objects than max_objects_to_list.

Parameters:

object_store_id (UUID, optional) – The id of the object store to find logs in (mutually exclusive with bucket_name).
bucket_name (str, optional) – The name of the object store bucket to find logs in (mutually exclusive with object_store_id).
root_prefix (str, optional) – The prefix to start searching from. Defaults to “”.
skip_prefixes (List[str], optional) – A list of prefixes to skip. Defaults to [].
skip_duplicates (bool, optional) – Whether to skip duplicate logs. Defaults to True.
log_file_regexes (List[str], optional) – A list of regular expressions to match log file names. Defaults to DEFAULT_LOG_FILE_REGEXES.
max_objects_to_list (int, optional) – The maximum number of objects to list. Defaults to 100_000.
fail_if_more_than_max_objects_to_list (bool, optional) – Whether to raise an exception if more objects are found than max_objects_to_list. Defaults to False.

Returns:

A list of dictionaries containing log parameters.

Return type:

List[dict]

get_object_meta(object_key: str, object_store_id: str | None = None, log_id: str | None = None, offset: int = 0, length: int | None = None) → Object

Get the metadata for either a Log Object or an Object Store Object.

Parameters:

object_key (str) – The key of the object.
object_store_id (str, optional) – The object store id to use. Defaults to None.
log_id (str, optional) – The log id to use. Defaults to None.
offset (int, optional) – The offset to use. Defaults to 0.
length (int, optional) – The length to use. Defaults to None.

Raises:

ValueError – If both object_store_id and log_id are provided.
ValueError – If neither object_store_id nor log_id are provided.

Returns:

The object metadata.

Return type:

Object

get_object_store(object_store_id: UUID | None = None, bucket_name: str | None = None) → ObjectStore

Get an object store.

This function gets an object store by ID or bucket name. If both object_store_id and bucket_name are provided, an exception is raised.

Parameters:

object_store_id (UUID, optional) – The id of the object store. Defaults to None.
bucket_name (str, optional) – The name of the object store bucket. Defaults to None.

Raises:

Exception – If neither object_store_id nor bucket_name is provided.

Returns:

The object store.

Return type:

ObjectStore

get_record_image(record_data: dict, max_size: int | None = None, format: str = 'WEBP', format_params: dict = {}, renormalize: bool = True, reset_position: bool = True, return_bytes: bool = False) → Image | BytesIO | None

A convenience method which takes deserialized record data from a standard image topic and returns the image as a PIL Image or BytesIO object.

Parameters:

record_data (dict) – The record data.
max_size (int, optional) – The maximum width or height to downscale to. Defaults to None, which means no downscaling.
format (str, optional) – The output format to use. Defaults to “WEBP”.
format_params (dict, optional) – The format parameters to use. Defaults to {}.
renormalize (bool, optional) – Whether to renormalize the image, which is necessary for visualization in some cases. Defaults to True.
reset_position (bool, optional) – Whether to reset the position offset position of the BytesIO object. Defaults to True.
return_bytes (bool, optional) – Whether to return the image as a BytesIO object. Defaults to False.

Returns:

The image, either as a PIL Image or BytesIO object, or None if the record data does not contain an image.

Return type:

Union[ImagePIL.Image, io.BytesIO, None]

iter_object_data_parts(object_key: str, object_store_id: str | None = None, log_id: str | None = None, part_size: int = 104857600, max_workers: int | None = 10, start_offset: int = 0, end_offset: int | None = None, max_attempts: int = 3, backoff_factor: float = 5.0, connect_timeout: int = 60, read_timeout: int = 600, return_as_completed: bool = False) → Iterable[tuple[bytearray, int]]

Yield parts of an object’s data.

Parameters:

object_key (str) – The key of the object.
object_store_id (str, optional) – The object store id to use. Defaults to None.
log_id (str, optional) – The log id to use. Defaults to None.
part_size (int, optional) – The size of each part to download. Defaults to 100 * 1024 * 1024.
max_workers (int, optional) – The maximum number of workers to use for parallel downloads. Defaults to 10.
start_offset (int, optional) – The offset to start downloading from. Defaults to 0.
end_offset (int, optional) – The offset to stop downloading at. Defaults to None.
max_attempts (int, optional) – The maximum number of attempts to make. Defaults to 3.
backoff_factor (float, optional) – The backoff factor to use. Defaults to 5.0.
connect_timeout (int, optional) – The connect timeout to use. Defaults to 60.
read_timeout (int, optional) – The read timeout to use. Defaults to 600.
return_as_completed (bool, optional) – Whether to return the results as they are completed. Defaults to False.

Yields:

The downloaded data and the offset of the data.

Return type:

Iterable[tuple[bytearray, int]]

iter_record_data(records: Iterable[Record], deserialize_results: bool = False, transcoder: Transcode | None = None, density_threshold: float = 0.9, max_contiguous_size: int = 100000000, max_contiguous_records: int = 1000, max_workers: int | None = 2, stop_event: Event | None = None) → Iterator[tuple[Record, bytes | dict]]

Given a set of records, yield the record and its data.

Parameters:

records (Iterable[Record]) – The records to use.
deserialize_results (bool, optional) – Whether to deserialize the results. Defaults to False.
transcoder (Transcode, optional) – The transcoder to use. Defaults to None.
density_threshold (float, optional) – The density threshold to use. Defaults to 0.9.
max_contiguous_size (int, optional) – The maximum contiguous size to use. Defaults to 100 * 1000 * 1000.
max_contiguous_records (int, optional) – The maximum contiguous records to use. Defaults to 1000.
max_workers (int | None, optional) – The maximum number of workers to use. Defaults to 2.
stop_event (Event, optional) – An event to signal stopping the iteration. Defaults to None.

Yields:

The record and the record data.

Return type:

tuple[Record, dict | bytes]

Iterate over records for a topic.

Parameters:

topic (Topic | UUID | str) – The topic to use.
start_time (int, optional) – The start time to use. Defaults to None.
end_time (int, optional) – The end time to use. Defaults to None.
per_request_limit (int, optional) – The limit to use for each request. Defaults to 1000.
frequency (float, optional) – The frequency to use for each request. Defaults to None.
include_auxiliary_data (bool, optional) – Whether to include auxiliary data. Defaults to False.

Yields:

The record.

Return type:

Record

Iterate over records for multiple topics.

Parameters:

topics (List[Topic | UUID | str]) – The topics to use.
start_time (int, optional) – The start time to use. Defaults to None.
end_time (int, optional) – The end time to use. Defaults to None.
per_request_limit (int, optional) – The limit to use for each request. Defaults to 1000.
frequency (float, optional) – The frequency to use for each request. Defaults to None.
include_auxiliary_data (bool, optional) – Whether to include auxiliary data. Defaults to False.

Yields:

The record.

Return type:

Record

list_all_log_object_parts(log_id: str, object_key: str) → list[ObjectPart]

List all parts of a log object.

Parameters:

log_id (str) – The log id to list the object parts of.
object_key (str) – The key of the object to list the parts of.

Returns:

The list of object parts.

Return type:

list[ObjectPart]

sync_log_objects(root_dir: str, log_file_regexes: List[str] = ['/(?P<log_name>[^/]*?)\\.bag$', '/(?P<log_name>[^/]*?)\\.log$', '/(?P<log_name>[^/]*?)\\.mcap$', '/(?P<log_name>[^/]*?)\\.jsonl$', '/(?P<log_name>[^/]*)\\.log/log0$', '/(?P<log_name>[^/]*)/log0$', '/(?P<log_name>[^/]*)/manifests\\/[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-4[0-9a-fA-F]{3}-[89aAbB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}$'], path_filter_regexes: List[str] | None = None, max_depth: int | None = None, group_id: str | None = None, group_name: str | None = None, create_group_if_missing: bool = True, group_note: str | None = None, group_context: dict | None = None, log_id: str | None = None, log_name: str | None = None, create_log_if_missing: bool = True, log_note: str | None = None, log_context: dict | None = None, key_prefix: str = None, part_size: int | None = None, max_workers: int = 32, skip_if_exists: bool = False, continue_upload: bool = True, skip_if_complete: bool = True, overwrite: bool = False, verify_integrity: bool = True, manage_memory: bool = True, fail_if_empty_file: bool = True, skip_if_empty_file: bool = False, warn_if_empty_file: bool = True, create_ingestions: bool = True, skip_existing_ingestions: bool = True, ingestion_state: ProcessState = ProcessState.queued, ingestion_note: str | None = None, ingestion_context: dict | None = None, ingestion_workflow_id: str | None = None, ingestion_workflow_context: dict | None = None, retry_count: int | None = None, verbosity: str | None = 'info') → int

Sync log files from a local directory to LogQS.

This function searches the root directory for log files using the given regular expressions. It then uploads the log files to LogQS and, optionally, creates ingestions for them.

The name used for the LogQS log resource of a found log file is extracted from the first capture group of the regular expression. If the captured group is followed by a slash, the log directory is also extracted.

If the log with the extracted name does not exist in the group, it is created. If the group does not exist, it is created if create_group_if_missing is True.

If log_id or log_name is provided, the logs are uploaded to the specified log instead of the log given by the name captured when finding the log files. If log_id is not provided, the log_name is used to find or create the log to upload the logs to. Similarly, if the group_id is not provided, group_name is used to find or create the group.

If create_ingestions is True, ingestions are created for the logs. If skip_existing_ingestions is True, ingestions are only created for logs that don’t already have corresponding ingestions.

Parameters:

root_dir (str) – The directory to search for log files.
log_file_regexes (List[str], optional) – A list of regular expressions to match log files. Defaults to DEFAULT_LOG_FILE_REGEXES.
path_filter_regexes (List[str], optional) – A list of regular expressions to match file paths. Defaults to None, meaning all files will be uploaded.
max_depth (int, optional) – The maximum depth to search for log files. Defaults to None.
group_id (str, optional) – The group id to upload the logs to. Defaults to None.
group_name (str, optional) – The group name to upload the logs to. Defaults to None.
create_group_if_missing (bool, optional) – Whether to create the group if it doesn’t exist. Defaults to True.
group_note (str, optional) – A note to use when creating the group. Defaults to None.
group_context (dict, optional) – A context to use when creating the group. Defaults to None.
log_id (str, optional) – The log id to upload the logs to. Defaults to None.
log_name (str, optional) – The log name to upload the logs to. Defaults to None.
create_log_if_missing (bool, optional) – Whether to create the log if it doesn’t exist. Defaults to True.
log_note (str, optional) – A note to use when creating the log. Defaults to None.
log_context (dict, optional) – A context to use when creating the log. Defaults to None.
key_prefix (str, optional) – A prefix to add to the object keys. Defaults to None.
part_size (int, optional) – The size of each part to upload. Defaults to None, which means the part size will be calculated based on the file size.
max_workers (int, optional) – The maximum number of workers to use for parallel uploads. Defaults to 32.
skip_if_exists (bool, optional) – Whether to skip the upload if the object already exists. Defaults to False.
continue_upload (bool, optional) – Whether to continue an existing upload. Defaults to True.
skip_if_complete (bool, optional) – Whether to skip the continued upload if the object is already complete. Defaults to True.
overwrite (bool, optional) – Whether to overwrite the object if it already exists. Defaults to False.
verify_integrity (bool, optional) – Whether to verify the integrity of the uploaded objects. Defaults to True.
manage_memory (bool, optional) – Whether to try to manage memory usage during the upload. Defaults to True.
fail_if_empty_file (bool, optional) – Whether to raise an exception if a file is empty. Defaults to True.
skip_if_empty_file (bool, optional) – Whether to skip the upload if a file is empty. Defaults to False.
warn_if_empty_file (bool, optional) – Whether to log a warning if a file is empty. Defaults to True.
create_ingestions (bool, optional) – Whether to create ingestions for the logs. Defaults to True.
skip_existing_ingestions (bool, optional) – Whether to skip creating ingestions for logs that already have ingestions. Defaults to True.
ingestion_state (str, optional) – The state to set for the ingestions. Defaults to “queued”.
ingestion_note (str, optional) – A note to add to the ingestions. Defaults to None.
ingestion_context (dict, optional) – A context to add to the ingestions. Defaults to None.
ingestion_workflow_id (str, optional) – The workflow id to use when creating the ingestions. Defaults to None.
ingestion_workflow_context (dict, optional) – The workflow context to use when creating the ingestions. Defaults to None.
retry_count (int, optional) – The number of times to retry creating the ingestions. Defaults to None, which falls back to the client’s configured retry count.
verbosity (str, optional) – The verbosity level to use. One of “debug”, “info”, or “error”. Defaults to “info”.

Returns:

The number of uploaded log files.

Return type:

int

sync_object_store_logs(object_store_id: UUID | None = None, bucket_name: str | None = None, root_prefix: str = '', skip_prefixes: List[str] = [], skip_duplicates: bool = True, log_file_regexes: List[str] = ['/(?P<log_name>[^/]*?)\\.bag$', '/(?P<log_name>[^/]*?)\\.log$', '/(?P<log_name>[^/]*?)\\.mcap$', '/(?P<log_name>[^/]*?)\\.jsonl$', '/(?P<log_name>[^/]*)\\.log/log0$', '/(?P<log_name>[^/]*)/log0$', '/(?P<log_name>[^/]*)/manifests\\/[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-4[0-9a-fA-F]{3}-[89aAbB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}$'], max_objects_to_list: int | None = 100000, fail_if_more_than_max_objects_to_list: bool = False, group_id: str | None = None, group_name: str | None = None, create_group_if_missing: bool = True, group_note: str | None = None, group_context: dict | None = None, ignore_group_for_matching: bool = True, log_id: str | None = None, log_name: str | None = None, create_log_if_missing: bool = True, log_note: str | None = None, log_context: dict | None = None, create_ingestions: bool = True, skip_existing_ingestions: bool = True, ingestion_state: ProcessState = ProcessState.queued, ingestion_note: str | None = None, ingestion_context: dict | None = None, ingestion_workflow_id: str | None = None, ingestion_workflow_context: dict | None = None, retry_count: int | None = None, verbosity: str | None = 'info', dry_run: bool = False) → int

Ingest logs from an object store.

This function creates logs and ingestions for log objects found in a given object store. It uses a list of regular expressions to match log file names. One of object_store_id or bucket_name must be provided, but not both. Searching starts from the root_prefix and continues recursively. If skip_duplicates is set to True, only unique logs based on the found log name, group name, the object’s Etag, and the object’s size, and the first unique object encountered is what is used. Specific prefixes can be skipped using skip_prefixes. We only list up to max_objects_to_list objects, and if fail_if_more_than_max_objects_to_list is set to True, we raise an exception if we find more objects than max_objects_to_list. If group_id or group_name is provided, we use the provided group, otherwise we create a group based on the found log parameters. If log_id or log_name is provided, we use the provided log, otherwise we create a log based on the found log parameters.

Parameters:

object_store_id (UUID, optional) – The id of the object store to find logs in (mutually exclusive with bucket_name).
bucket_name (str, optional) – The name of the object store bucket to find logs in (mutually exclusive with object_store_id).
root_prefix (str, optional) – The prefix to start searching from. Defaults to “”.
skip_prefixes (List[str], optional) – A list of prefixes to skip. Defaults to [].
skip_duplicates (bool, optional) – Whether to skip duplicate logs. Defaults to True.
log_file_regexes (List[str], optional) – A list of regular expressions to match log file names. Defaults to DEFAULT_LOG_FILE_REGEXES.
max_objects_to_list (int, optional) – The maximum number of objects to list. Defaults to 100_000.
fail_if_more_than_max_objects_to_list (bool, optional) – Whether to raise an exception if more objects are found than max_objects_to_list. Defaults to False.
group_id (str, optional) – The id of the group to create logs in. Defaults to None.
group_name (str, optional) – The name of the group to create logs in. Defaults to None.
create_group_if_missing (bool, optional) – Whether to create the group if it doesn’t exist. Defaults to True.
group_note (str, optional) – The note to use when creating the group. Defaults to None.
group_context (dict, optional) – The context to use when creating the group. Defaults to None.
ignore_group_for_matching (bool, optional) – Whether to ignore the group when matching logs. Suitable if logs have been moved to different groups since last sync. Defaults to False.
log_id (str, optional) – The id of the log to create ingestions for. Defaults to None.
log_name (str, optional) – The name of the log to create ingestions for. Defaults to None.
create_log_if_missing (bool, optional) – Whether to create the log if it doesn’t exist. Defaults to True.
log_note (str, optional) – The note to use when creating the log. Defaults to None.
log_context (dict, optional) – The context to use when creating the log. Defaults to None.
create_ingestions (bool, optional) – Whether to create ingestions for the found logs. Defaults to True.
skip_existing_ingestions (bool, optional) – Whether to skip existing ingestions. Defaults to True.
ingestion_state (ProcessState, optional) – The state to use when creating ingestions. Defaults to ProcessState.queued.
ingestion_note (str, optional) – The note to use when creating ingestions. Defaults to None.
ingestion_context (dict, optional) – The context to use when creating ingestions. Defaults to None.
ingestion_workflow_id (str, optional) – The workflow id to use when creating ingestions. Defaults to None.
ingestion_workflow_context (dict, optional) – The workflow context to use when creating ingestions. Defaults to None.
retry_count (int, optional) – The number of times to retry the operation. Defaults to None.
verbosity (str, optional) – The verbosity level to use for logging. Defaults to “info”.
dry_run (bool, optional) – Whether to perform a dry run. Defaults to False.

Returns:

The number of ingestions created.

Return type:

int

upload_log_object(log_id: str, file_path: str, object_key: str | None = None, key_replacement: tuple[str, str] = None, key_prefix: str = None, part_size: int | None = None, max_workers: int = 32, skip_if_exists: bool = False, continue_upload: bool = False, skip_if_complete: bool = True, overwrite: bool = False, show_progress: bool = True, verify_integrity: bool = True, manage_memory: bool = True) → tuple[Object, list[ObjectPart]]

Upload a file to a log.

The file is uploaded as a log object, meaning it is associated with a single log given by log_id. The file is split into parts of size part_size, which are uploaded in parallel using a maximum of max_workers workers. If no part_size is provided, the part size will be determined based on the file size to try to optimize the upload. If manage_memory is set to True, the upload will try to avoid using too much memory by limiting the number of parts that are uploaded in parallel. Note that larger values for part_size and max_workers will generally result in faster uploads, but may also result in higher memory usage.

If skip_if_exists is set to True, the upload will be skipped if the object already exists. If continue_upload is set to True, any existing parts of the object will be skipped and the upload will continue from where it left off. If continue_upload is set to True and skip_if_complete is set to True, the upload will be skipped if the object is already complete. If overwrite is set to True, any existing object with the same key will be deleted before the upload.

Parameters:

log_id (str) – The log id to upload the object to.
file_path (str) – The path to the file to upload.
object_key (str, optional) – The key to use for the object. Defaults to None.
key_replacement (tuple[str, str], optional) – A tuple of strings to replace in the object key. Defaults to None.
key_prefix (str, optional) – A prefix to add to the object key. Defaults to None.
part_size (int, optional) – The size of each part to upload in bytes. Defaults to None, which means the part size will be calculated based on the file size.
max_workers (int, optional) – The maximum number of workers to use for parallel uploads. Defaults to 32.
skip_if_exists (bool, optional) – Whether to skip the upload if the object already exists. Defaults to False.
continue_upload (bool, optional) – Whether to continue an existing upload. Defaults to True.
skip_if_complete (bool, optional) – Whether to skip the continued upload if the object is already complete. Defaults to True.
overwrite (bool, optional) – Whether to overwrite the object if it already exists. Defaults to False.
show_progress (bool, optional) – Whether to show a progress bar for the upload. Defaults to True.
verify_integrity (bool, optional) – Whether to verify the integrity of the uploaded object. Defaults to True.
manage_memory (bool, optional) – Whether to try to manage memory usage during the upload. Defaults to True.

Raises:

ConflictException – If existing resources conflict with the upload.

Returns:

The uploaded object and its parts.

Return type:

tuple[Object, list[ObjectPart]]

upload_log_object_from_memory(log_id: str, file_like: BytesIO | StringIO, object_key: str, key_replacement: tuple[str, str] = None, key_prefix: str = None, part_size: int = 104857600, max_workers: int | None = 8, skip_if_exists: bool = False, continue_upload: bool = True, skip_if_complete: bool = True, overwrite: bool = False) → tuple[Object, list[ObjectPart]]

Upload a file-like object to a log.

The file is uploaded as a log object, meaning it is associated with a single log given by log_id. The file is split into parts of size part_size, which are uploaded in parallel using a maximum of max_workers workers. Note that larger values for part_size and max_workers will generally result in faster uploads, but may also result in higher memory usage.

If skip_if_exists is set to True, the upload will be skipped if the object already exists. If continue_upload is set to True, any existing parts of the object will be skipped and the upload will continue from where it left off. If continue_upload is set to True and skip_if_complete is set to True, the upload will be skipped if the object is already complete. If overwrite is set to True, any existing object with the same key will be deleted before the upload.

Parameters:

log_id (str) – The log id to upload the object to.
file_like (io.BytesIO | io.StringIO) – The file-like object to upload.
object_key (str) – The key to use for the object.
key_replacement (tuple[str, str], optional) – A tuple of strings to replace in the object key. Defaults to None.
key_prefix (str, optional) – A prefix to add to the object key. Defaults to None.
part_size (int, optional) – The size of each part to upload. Defaults to 100 * 1024 * 1024.
max_workers (int, optional) – The maximum number of workers to use for parallel uploads. Defaults to 8.
skip_if_exists (bool, optional) – Whether to skip the upload if the object already exists. Defaults to False.
continue_upload (bool, optional) – Whether to continue an existing upload. Defaults to True.
skip_if_complete (bool, optional) – Whether to skip the continued upload if the object is already complete. Defaults to True.
overwrite (bool, optional) – Whether to overwrite the object if it already exists. Defaults to False.

Raises:

ConflictException – If existing resources conflict with the upload.

Returns:

The uploaded object and its parts.

Return type:

tuple[Object, list[ObjectPart]]

upload_log_object_part(log_id: str, object_key: str, part_number: int, file_path: str, offset: int, size: int, verify_integrity: bool = True, max_attempts: int = 5, backoff_factor: float = 5.0, connect_timeout: int = 60, read_timeout: int = 600)

Upload a part of a file to a log object.

Parameters:

log_id (str) – The log id to upload the object part to.
object_key (str) – The key of the object to upload the part to.
part_number (int) – The part number of the object part.
file_path (str) – The path to the file to upload.
offset (int) – The offset in the file to start reading from.
size (int) – The size of the part to upload.
verify_integrity (bool, optional) – Whether to verify the integrity of the uploaded part. Defaults to True.
max_attempts (int, optional) – The maximum number of attempts to upload the part. Defaults to 5.
backoff_factor (float, optional) – The backoff factor for retrying the upload. Defaults to 5.0.
connect_timeout (int, optional) – The connection timeout for the upload. Defaults to 60.
read_timeout (int, optional) – The read timeout for the upload. Defaults to 600.

Raises:

Exception – If the upload fails.

Returns:

The uploaded object part.

Return type:

ObjectPart

upload_log_object_part_from_memory(log_id: str, object_key: str, part_number: int, part_data: bytes | str, size: int, max_attempts: int = 3, backoff_factor: float = 5.0, connect_timeout: int = 60, read_timeout: int = 600)

Upload a part of a file to a log object.

Parameters:

log_id (str) – The log id to upload the object part to.
object_key (str) – The key of the object to upload the part to.
part_number (int) – The part number of the object part.
part_data (bytes | str) – The data to upload.
size (int) – The size of the part to upload.
max_attempts (int, optional) – The maximum number of attempts to upload the part. Defaults to 3.
backoff_factor (float, optional) – The backoff factor for retrying the upload. Defaults to 5.0.
connect_timeout (int, optional) – The connection timeout for the upload. Defaults to 60.
read_timeout (int, optional) – The read timeout for the upload. Defaults to 600.

Raises:

Exception – If the upload fails.

Returns:

The uploaded object part.

Return type:

ObjectPart

upload_log_objects(log_id: str, file_dir: str, path_filter_regexes: List[str] | None = None, key_replacement: tuple[str, str] = None, key_prefix: str = None, part_size: int | None = None, max_workers: int = 32, skip_if_exists: bool = False, continue_upload: bool = True, skip_if_complete: bool = True, overwrite: bool = False, verify_integrity: bool = True, fail_if_empty_dir: bool = True, fail_if_empty_file: bool = True, skip_if_empty_file: bool = False, warn_if_empty_file: bool = True, manage_memory: bool = True) → List[tuple[Object, list[ObjectPart]]]

Upload a directory of files to a log.

The files are uploaded as log objects, meaning they are associated with a single log given by log_id. The files are split into parts of size part_size, which are uploaded in parallel using a maximum of max_workers workers. If no part_size is provided, the part size will be determined based on the file size to try to optimize the upload. Note that larger values for part_size and max_workers will generally result in faster uploads, but may also result in higher memory usage and connection issues.

If skip_if_exists is set to True, the upload will be skipped if the object already exists. If continue_upload is set to True, any existing parts of the object will be skipped and the upload will continue from where it left off. If continue_upload is set to True and skip_if_complete is set to True, the upload will be skipped if the object is already complete. If overwrite is set to True, any existing object with the same key will be deleted before the upload.

Parameters:

log_id (str) – The log id to upload the objects to.
file_dir (str) – The path to the directory to upload.
path_filter_regexes (List[str], optional) – A list of regular expressions to match file paths. Defaults to None, which means all files will be uploaded.
key_replacement (tuple[str, str], optional) – A tuple of strings to replace in the object keys. Defaults to None.
key_prefix (str, optional) – A prefix to add to the object keys. Defaults to None.
part_size (int, optional) – The size of each part to upload. Defaults to None, which means the part size will be calculated based on the file size.
max_workers (int, optional) – The maximum number of workers to use for parallel uploads. Defaults to 32.
skip_if_exists (bool, optional) – Whether to skip the upload if the object already exists. Defaults to False.
continue_upload (bool, optional) – Whether to continue an existing upload. Defaults to True.
skip_if_complete (bool, optional) – Whether to skip the continued upload if the object is already complete. Defaults to True.
overwrite (bool, optional) – Whether to overwrite the object if it already exists. Defaults to False.
verify_integrity (bool, optional) – Whether to verify the integrity of the uploaded objects. Defaults to True.
fail_if_empty_dir (bool, optional) – Whether to raise an exception if no files are found in the directory. Defaults to True.
fail_if_empty_file (bool, optional) – Whether to raise an exception if a file is empty. Defaults to True.
skip_if_empty_file (bool, optional) – Whether to skip the upload if a file is empty. Defaults to False.
warn_if_empty_file (bool, optional) – Whether to log a warning if a file is empty. Defaults to True.
manage_memory (bool, optional) – Whether to try to manage memory usage during the upload. Defaults to True.

Raises:

Exception – If no files are found in the directory and fail_if_empty_dir is True.

Returns:

A list of tuples of uploaded objects and their parts.

Return type:

list[tuple[Object, list[ObjectPart]]]

verify_integrity(file_path: str, etag: str, part_size: int = 104857600) → bool

Verify the integrity of a file given an ETag.

This function verifies the integrity of a file assuming it is uploaded as a multipart upload.

Parameters:

file_path (str) – The path to the file to verify the integrity of.
etag (str) – The ETag of the file.
part_size (int, optional) – The size of each chunk to read from the file. Defaults to 100 * 1024 * 1024.

Returns:

Whether the file is intact.

Return type:

bool

verify_object_integrity(file_path: str, object: Object, part_size: int = 104857600) → bool

Verify the integrity of a file given an object.

This function verifies the integrity of a file assuming it is uploaded as a multipart upload.

Parameters:

file_path (str) – The path to the file to verify the integrity of.
object (Object) – The object to verify the integrity of.
part_size (int, optional) – The size of each chunk to read from the file. Defaults to 100 * 1024 * 1024.

Returns:

Whether the file is intact.

Return type:

bool