Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How can I get a log file in jsonl form? #594

Open
gurudykim opened this issue Feb 5, 2024 · 0 comments
Open

How can I get a log file in jsonl form? #594

gurudykim opened this issue Feb 5, 2024 · 0 comments
Labels
question Further information is requested

Comments

@gurudykim
Copy link

I have a log file where data in jsonl form is accumulated.
{}
{}
{} It builds up underneath in form.
The first time you connect the connector, read the file
After that, it is not reflected when added.
I will post the properties I used at the bottom.

I couldn't find it, so I ended up posting it here
I'm sorry if it's basic.

name = conn_name
connector.class = io.streamthoughts.kafka.connect.filepulse.source.FilePulseSourceConnector
tasks.max = 1
value.converter = io.confluent.connect.json.JsonSchemaConverter
fs.listing.class = io.streamthoughts.kafka.connect.filepulse.fs.LocalFSDirectoryListing
fs.listing.filters = io.streamthoughts.kafka.connect.filepulse.fs.filter.RegexFileListFilter
tasks.reader.class = io.streamthoughts.kafka.connect.filepulse.fs.reader.LocalRowFileInputReader
topic = topicname

tasks.file.status.storage.class = io.streamthoughts.kafka.connect.filepulse.state.KafkaFileObjectStateBackingStore

filters = JSONFilter
filters.JSONFilter.source=message


fs.listing.interval.ms = 3000
fs.cleanup.policy.class = io.streamthoughts.kafka.connect.filepulse.fs.clean.LogCleanupPolicy

tasks.file.status.storage.topic = topicname.status
fs.scan.directory.path = /home/elasticsearch_user/confluent/util/filepulse/guru/
fs.scan.interval.ms = 3000

offset.attributes.string = name

fs.scan.filters = io.streamthoughts.kafka.connect.filepulse.scanner.local.filter.RegexFileListFilter

filters.JSONFilter.type = io.streamthoughts.kafka.connect.filepulse.filter.JSONFilter
filters.JSONFilter.merge = true
filters.JSONFilter.target=payload

tasks.file.status.storage.producer.bootstrap.servers = http://localhost:9092
tasks.file.status.storage.topic.replication.factor = 1

fs.listing.directory.path = /home/elasticsearch_user/confluent/util/filepulse/guru/

tasks.file.status.storage.bootstrap.servers = http://localhost:9092
tasks.file.status.storage.topic.partitions = 1

tasks.file.status.storage.consumer.bootstrap.servers = http://localhost:9092
file.filter.regex.pattern = sample.json

key.converter.schemas.enable = false
value.converter.schema.registry.url = http://localhost:8081


@gurudykim gurudykim added the question Further information is requested label Feb 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant