Storing Artifact Values in Amazon S3

Note

If you encounter issues you cannot resolve, simply ask in our Slack community’s #support channel. We are always happy and ready to help you!

To use S3 as LineaPy’s serialized value location, you can run the following command in your notebook to change your storage backend:

lineapy.options.set('artifact_storage_dir', 's3://your-bucket/your-artifact-folder')

You should configure your AWS account just like you would for AWS CLI or boto3, and LineaPy will use the default AWS credentials to access the S3 bucket.

If you want to use other profiles available in your AWS configuration, you can set the profile name with:

lineapy.options.set('storage_options', {'profile': 'ANOTHER_AWS_PROFILE'})

which is equivalent to setting your environment variable AWS_PROFILE to the profile name.

If you really need to set your AWS credentials directly in the running session (strongly discouraged as it may result in accidentally saving these credentials in plain text), you can set them with:

lineapy.options.set('storage_options', {'key': 'AWS KEY', 'secret': 'AWS SECRET'})

which is equivalent to setting environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

To learn more about which S3 configuration items that you can set in storage_options, you can see the parameters of s3fs.S3FileSystem since fsspec is passing storage_options items to s3fs.S3FileSystem to access S3 under the hood.