Object Stores

In LogQS, Object Stores represent integrations with external object store services (namely, S3-compatible services) which can be used to store and retrieve log data. All of the log data in LogQS is stored in an object store, whether it be a LogQS managed object store or a user-configured object store.

Creating an Object Store

To connect an AWS S3 bucket to LogQS, the user will need to first create the bucket in AWS and then create an IAM user with access to that bucket. The user will need to provide the access key ID and secret access key for that IAM user when creating the object store in LogQS. Note that this process will be similar for other S3-compatible object stores (such as Cloudflare R2 or MinIO), but the user may need to provide a custom endpoint URL when creating the object store in LogQS. For more information about AWS S3, refer to the AWS S3 User Guide.

When creating an S3 bucket, ensure that the bucket is created in the same region as your LogQS instance for optimal performance. Similarly, note that using a bucket in a different region may incur additional data transfer costs. LogQS will read data multiple times from the bucket when ingesting logs, performing digestions, and creating record artifacts (such as images), so it is important to consider these factors when selecting a bucket location.

When a user creates an object store, they submit access key credentials. The secret_access_key is encrypted in the database and can’t be fetched or changed after creation.

The user must create IAM credentials providing access to the underlying bucket in AWS. At a minimum, LogQS needs to be able to read objects from the bucket. The following policy can be used to create a policy which allow LogQS to list and read objects from the bucket. The user should replace <BUCKET NAME> with the name of their bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "LogQSReadOnlyAccess",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucket",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::<BUCKET NAME>/*",
                "arn:aws:s3:::<BUCKET NAME>"
            ]
        }
    ]
}

LogQS also has the ability to write objects to an object store, assuming the necessary permissions are in place. The following policy can be used to create a policy which allows LogQS to write objects to the bucket. The user should replace <BUCKET NAME> with the name of their bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "LogQSWriteOnlyAccess",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts",
                "s3:AbortMultipartUpload"
            ],
            "Resource": [
                "arn:aws:s3:::<BUCKET NAME>/*",
                "arn:aws:s3:::<BUCKET NAME>"
            ]
        }
    ]
}

Combining these two policies, the user can create a policy which allows LogQS to read and write objects to the bucket.

An object store has a read_only field which is a boolean indicating whether the object store is read-only. If true, the object store can only be used for reading objects and not writing objects. LogQS enforces this, so even if the policy allows writing, setting this field to true will prevent writing to the object store through the LogQS API. Note, however, that if the user has access to the underlying object store (e.g., AWS S3 bucket) through other means (such as through LogQS provided session credentials), they may still be able to write to the object store directly, bypassing LogQS. In order to prevent this, the user should ensure that the IAM policy associated with the access key only allows read access to the bucket.

Object stores also have a key_prefix field which is used to programmtically “scope” which objects can be accessed by the LogQS. This is useful for partioning data within the object store. The key_prefix is a string which is prepended to the object key when accessing objects in the object store. For example, if the key_prefix is logs/, and the object key is my_log.bag, then the full object key will be logs/my_log.bag. If a user supplies an object_key which already contains the prefix, then the prefix will not be prepended again. Similarly, any operations performed on objects in the object store will automatically have the prefix prepended to the object key. For example, if the key_prefix is logs/ and a user lists all objects in the object store, then the user will only be able to see objects with keys that start with logs/. If the user tries to list objects with keys that do not start with logs/, they will not be able to see those objects. Similar to the read_only field, this is enforced by LogQS when accessing the object store through the LogQS API, but if the user has direct access to the underlying object store, they may be able to bypass this restriction. Again, to prevent this, the user should ensure that the IAM policy associated with the access key only allows access to the desired prefix within the bucket.

Object stores are configured with the following fields:

name: A string representing the name of the object store. This field is used for reference and must be unique within the DataStore.
bucket_name: The name of the bucket in the object store where the log data is stored.
region_name: The region where the object store is located (e.g., us-east-1).
access_key_id: The access key ID used to authenticate with the object store.
secret_access_key: The secret access key used to authenticate with the object store.
endpoint_url: The optional URL of the object store’s API endpoint (if not populated, the default AWS S3 endpoint will be used).
key_prefix: An optional string representing a prefix enforced on all object keys in the object store. This is useful for namespacing objects within a bucket.
default: A boolean indicating whether this object store is the default object store for the DataStore. There can only be one default object store per DataStore.
managed: A boolean indicating whether this object store is managed by LogQS. If true, the object store is managed by LogQS and cannot be modified or deleted by the user.
disabled: A boolean indicating whether this object store is disabled. If true, the object store is disabled and cannot be used for ingestions or digestions.
read_only: A boolean indicating whether this object store is read-only. If true, objects in the object store can only be read, not written to.
note: A free-form text field that can be used to store any additional information about the object store.
context: A JSON object that can be used to store any additional information about the object store.

When creating an object store, the name, bucket_name, region_name, access_key_id, and secret_access_key fields are required. The other fields are optional.

Using an Object Store

Various resources in LogQS depend on Object Stores, most noteably Ingestions. When creating an ingestion, the user must specify which Object Store to use for that ingestion along with a key for an object in that object store to ingest. LogQS will then use the access credentials provided when creating the object store to read the object from the object store.

To specify an Object Store for these cases, you must first list or fetch the desired Object Store using the LogQS client. For example, to list all Object Stores in the DataStore:

object_stores = lqs.list.object_store().data

for obj_store in object_stores:
    print(f"Object Store: {obj_store.name} (ID: {obj_store.id})")

To fetch a specific Object Store by its ID:

object_store_id = "your-object-store-id"
object_store = lqs.fetch.object_store(object_store_id=object_store_id).data

print(f"Fetched Object Store: {object_store.name} (ID: {object_store.id})")

To fetch an Object Store by its name:

object_store_name = "your-object-store-name"
object_store = lqs.list.object_store(name=object_store_name).data[0]

print(f"Fetched Object Store: {object_store.name} (ID: {object_store.id})")

Note that you can list Object Stores by bucket name as well, but there can be multiple Object Stores with the same bucket name (i.e., multiple Object Stores configured to use the same underlying bucket with different access permissions, etc.), so it’s recommended to use either the Object Store ID or name for fetching a specific Object Store.

Once you have the desired Object Store, you can use list objects in that Object Store using the LogQS client. For example, to list all objects in the Object Store with a specific prefix:

prefix = "logs/"
objects = lqs.list.object(
    object_store_id=object_store.id,
    prefix=prefix
).data

for obj in objects:
    print(f"Log Object: {obj.key} (Size: {obj.size} bytes)")

Objects in an Object Store can be downloaded using a presigned URL supplied by LogQS. These presigned URLs are only generated for objects when fetching (not listing) objects. To fetch a specific object and get its presigned URL:

object_key = "logs/my_log.bag"
log_object = lqs.fetch.object(
    object_store_id=object_store.id,
    object_key=object_key
).data

print(f"Presigned URL for '{log_object.key}': {log_object.presigned_url}")

This presigned URL can then be used to download the object directly from the object store using standard HTTP methods. For example, using the requests library in Python:

import requests

response = requests.get(log_object.presigned_url)

if response.status_code == 200:
    with open("downloaded_log.bag", "wb") as f:
        f.write(response.content)
    print("Log file downloaded successfully.")
else:
    print(f"Failed to download log file. Status code: {response.status_code}")

The LogQS client offers a utility method for downloading objects easily and effeciently:

lqs.utils.download(
    object_store_id=object_store.id,
    object_key=object_key
)

For more information, refer to the Utils.download() section of the API reference.

Session Credentials

In some cases, such as for uploading many/large objects, it may be beneficial to avoid the overhead involved with utilizing the LogQS API for each object operation. To facilitate this, LogQS can provide temporary session credentials for an Object Store which can be used to access the underlying object store directly (bypassing the LogQS API). These session credentials are temporary and expire after a set amount of time. These credentials can be used with standard S3-compatible clients (such as boto3 in Python) or the AWS CLI.

You can fetch temporary session credentials for an Object Store using the LogQS client:

session_credentials = lqs.fetch.object_store_session_credentials(
    object_store_id=object_store.id
).data

print(f"Access Key ID: {session_credentials.access_key_id}")
print(f"Secret Access Key: {session_credentials.secret_access_key}")
print(f"Session Token: {session_credentials.session_token}")
print(f"Expiration: {session_credentials.expiration}")

These session credentials can then be used with standard S3-compatible clients. For example, using boto3 in Python:

import boto3

s3_client = boto3.client(
    "s3",
    aws_access_key_id=session_credentials.access_key_id,
    aws_secret_access_key=session_credentials.secret_access_key,
    aws_session_token=session_credentials.session_token,
    region_name=object_store.region_name,
    endpoint_url=object_store.endpoint_url  # Optional, only if using a custom endpoint
)

# List objects in the bucket
response = s3_client.list_objects_v2(Bucket=object_store.bucket_name)

for obj in response.get("Contents", []):
    print(f"Object Key: {obj['Key']} (Size: {obj['Size']} bytes)")

# Upload an object to the bucket
s3_client.put_object(
    Bucket=object_store.bucket_name,
    Key="logs/new_log.bag",
    Body=b"Log file content"
)

# Download an object from the bucket
response = s3_client.get_object(
    Bucket=object_store.bucket_name,
    Key="logs/my_log.bag"
)
log_data = response["Body"].read()
with open("downloaded_log.bag", "wb") as f:
    f.write(log_data)