Skip to main content
Quick navigation

Large message/batch handling

Introduction

Amazon S3 or Amazon Simple Storage Service is a service offered by Amazon Web Services that provides object storage through a web service interface.

Large message/batch handling interceptor will save the actual message produced to gateway into Amazon Simple Storage Service.

It helps to protect data or optimize storage in actual kafka.

Configuration

keytypedefaultdescription
topicString.*Topics that match this regex will have the interceptor applied
s3ConfigS3Amazon S3 Configuration
minimumSizeInBytesintOnly upload to s3 for batch/message record has size greater than or equal to this minimumSizeInBytes

S3

keytypedescription
accessKeyStringS3 access key
secretKeyStringS3 secret key
bucketNameStringS3 bucket name
uriStringS3 uri
regionStringS3 Region
localDiskDirectoryStringLocal temp storage, used when we download file from S3 while fetching messages

Examples

Large batches

Each batch above the minimumSizeInBytes threshold will be saved in one file on s3.

{
"name": "myLargeBatchHandlingPlugin",
"pluginClass": "io.conduktor.gateway.interceptor.LargeBatchHandlingPlugin",
"priority": 100,
"config": {
"topic": "topic.*",
"minimumSizeInBytes": 1024,
"s3Config": {
"accessKey": "myAccessKey",
"secretKey": "mySecretKey",
"bucketName": "myBucketName",
"uri": "http://myexampleuri",
"region": "us-east-1",
"localDiskDirectory": "myStorage/"
}
}
}

Large messages

Each individual message above the minimumSizeInBytes threshold will be saved in one file on s3.

{
"name": "myLargeMessageHandlingPlugin",
"pluginClass": "io.conduktor.gateway.interceptor.LargeMessageHandlingPlugin",
"priority": 100,
"config": {
"topic": "topic.*",
"minimumSizeInBytes": 1024,
"s3Config": {
"accessKey": "myAccessKey",
"secretKey": "mySecretKey",
"bucketName": "myBucketName",
"uri": "http://myexampleuri",
"region": "us-east-1",
"localDiskDirectory": "myStorage/"
}
}
}