timesketch_importer monitoring files and folders

[cvandeplas]

Is your feature request related to a problem? Please describe.
A feature we're missing is the ability to continuously monitor for new files and folders, and updated files and ingest them into Timesketch, comparable to what Splunk universal forwarder does.

Describe the solution you'd like
The ability to monitor a folder, and all subfolders for new files, or changed files and ingest the data into timesketch.
The sketch_name and timeline_name may need to be dynamic, for example based on the folder path of the files.

Describe alternatives you've considered

• timesketch-importer.sh is not an option as
  • it only imports new files, when they are fully there (file close),
  • it doesn't support monitoring changes in files (new entries added).
  • It also relies on filesystem features (inotify) which may not be available with NFS or SMB
  • and requires files to be available on the timesketch instance itself

A draft PR will be raised in a moment to request input on a proposed implementation.

[jkppr]

We've discussed this in the Timesketch team and we agree that continuous monitoring is a valuable use-case. However, we feel it is out of scope for the core timesketch-importer CLI tool.

The timesketch-importer is designed as a synchronous, one-off utility. Transforming it into a daemon that manages state, handles file-system polling, and tracks file offsets would significantly increase the maintenance complexity and change the nature of the tool.

But, you don't need to fork or copy the importer's code to get what you want. We have designed the timesketch_api_client and timesketch_import_client specifically so you can build custom tools like this easily.

All the authentication, credential encryption, and configuration logic you mentioned lives in the library's config module, not the script. You can initialize a client in your own monitoring script with a single call:
ts_client = timesketch_api_client.config.get_client()

We recommend building this as a standalone project (e.g., timesketch-forwarder script) that imports our libraries. This gives you the freedom to choose your monitoring engine (Watchdog, Polling, etc.) and naming logic without being restricted by the CLI's structure.

If you like you can contribute it as a script into https://github.com/google/timesketch/tree/master/contrib or if you decide to host it as your own repository, let us know and we can mention it in our docs on timesketch.org so others can find it as well.

[cvandeplas]

I'll pick this up, will not be done immediately, but is indeed something that would be good to work on. Thanks a lot for the input that was provided.