Can I subscribe to FTP events when files are added or changed in a folder? #1661

robinrodricks · 2024-10-05T07:15:58Z

Question:

I would like to ask wether it's possible to subscribe on ftp command events via FluentFTP, because I would like to se wether it's possible to make a solution where I don't have to pull a ftp every 5 seconds.

When files are being added to a particular folder I would be notified.

That way I don't have to pull and check ftp for new files every 5 seconds (not super great for ftp) - I will only pull, when I get notified.

Proposed solution:

This is a very common usecase especially on the cloud, where FTP uploads are used to trigger cloud workflows.

I suppose we could consider creating a new class for this. Something like FtpFolderMonitor. This class would then allow you to monitor a specific remote folder on the FTP server. It would trigger events when files were added/removed. Internally it would poll the folder and check for new files (this is the only way technically possibly via the FTP protocol).

Config:

Config - the FTP client config (FTP client will be internally created automatically and kept alive based on this config)
FolderPath - which folder to monitor
PollInterval - how often to check the folder - default : 1 sec
WaitTillFileFullyUploaded - flag to enable detection of partially uploaded files (this works by monitoring the filesize and only firing an event when the filesize is stable)

Events:

FilesChanged - if the date/filesize of any files changed, this event is fired with the list of changed files
FilesAdded - when new files are added, this event is fired with the list of added files
FilesDeleted - when files are removed, this event is fired with the list of deleted files
ChangeDetected - a generic event fired when anything changes

In event handlers, file lists are provided as List<string> or List<FtpListItem>.

The text was updated successfully, but these errors were encountered:

FanDjango · 2024-10-05T11:29:43Z

Maybe I'm too stoopid, but

are we talking about monitoring for local folder changes triggering an event

or

are we talking about monitoring (either with a permanent connection or regular repeated connections) remote folder changes triggering an event

???

markat1 · 2024-10-05T12:39:43Z

I like @robinrodricks idea of having a wrapper class FtpFolderMonitor that would allow monitoring a specific folder!

Would it be possible to pub/sub on FtpFolderMonitor?

robinrodricks · 2024-10-07T12:58:14Z

are we talking about monitoring (either with a permanent connection or regular repeated connections) remote folder changes triggering an event

@FanDjango monitoring of a remote folder.

Would it be possible to pub/sub on FtpFolderMonitor?

how would pub/sub be implemented?

FanDjango · 2024-10-07T16:57:07Z

monitoring of a remote folder

So you would propose to do a MLST, LIST or NLST regularly? And compare the contents?

PollInterval - how often to check the folder - default : 1 sec

You will be kicked from the server. Either because you don't ever transfer a file and time out (some servers enforce this), or if you try to circumvent this by QUIT and reconnect, you will be noticed and banned.

In any case you are severely misusing the FTP server, you might get away with it if you reduce the timing to, say every 5 minutes.

Unless of course it is your own server, then you are free to do this.

Otherwise, this is a job for NFS, RSYNC or other stuff higher up the tree.

how would pub/sub be implemented?

I don't even know what that is. Enlighten me, please, someone.

robinrodricks · 2024-10-08T06:26:59Z

So you would propose to do a MLST, LIST or NLST regularly? And compare the contents?

Yes.

You will be kicked from the server.

Then we can keep the default to 10 seconds or 60 secs.

Unless of course it is your own server

Yes this would be the most common use case. Or even if its a web host, they should allow polling every 1 min.

I don't even know that that is. Enlighten me, please, someone.

Its normally used on the cloud. I'm not sure why the user is asking us to implement it at the library level.

https://aws.amazon.com/what-is/pub-sub-messaging/

markat1 · 2024-10-09T04:37:54Z

Reason I ask for pub/sub is some sort of control of broadcasting certain events to specific subscribers. It was just the architecture that came to mind.

Anyway, I guess pub/sub doesn't have to be part of this solution - it could be integrated seperately, which problably makes more sense.

robinrodricks · 2024-10-09T06:24:10Z

Reason I ask for pub/sub is some sort of control of broadcasting certain events to specific subscribers

You can surely handle that as part of your user code.

robinrodricks · 2024-10-09T11:39:42Z

@markat1 I have committed the first version of these classes. Can you download the FluentFTP project, build from source and try using it?

New classes:

AsyncFtpFolderMonitor - async version
FtpFolderMonitor - sync version

Usage:

var client = new FtpClient(...);
var monitor = new FtpFolderMonitor(client, "/remote/folder");

// optional config
monitor.WaitTillFileFullyUploaded = false;
monitor.Recursive = false;
monitor.PollInterval = 60;

// add events
monitor.FilesAdded += your_event;
monitor.FilesDeleted += your_event;
monitor.FilesChanged += your_event;
monitor.Start();

Since I'm very busy I am hoping you can fix minor bugs and submit a PR with a working version. I don't have time to test it on my end right now.

I have used System.Threading.Timer. I am not sure if it works in all use cases. Please change this to whatever timer is best suited to such projects.

File change detection is done by filesize. I suppose an alternate way would be using the date modified.

markat1 · 2024-10-10T07:55:11Z

I testet following in a console program - I have replaced connection information in the examples shown here.

both Sync and async version - doesn't respond back on filesAdded, FilesDeleted or FilesdDeleted unfortunately

using FluentFTP;
using FluentFTP.Monitors;

var client = new FtpClient("Server", "User", "Password");

client.Connect();

Console.WriteLine("Running FtpFolderMonitor");

var monitor = new FtpFolderMonitor(client, "OutboxDirectory");

monitor.WaitTillFileFullyUploaded = false;
monitor.Recursive = false;
monitor.PollInterval = 5;

monitor.FilesAdded += Monitor_FilesAdded;
monitor.FilesChanged += Monitor_FilesChanged;
monitor.FilesDeleted += Monitor_FilesDeleted;

void Monitor_FilesAdded(object? sender, List<string> e) {
	Console.WriteLine("File added");
}

void Monitor_FilesChanged(object? sender, List<string> e) {
	Console.WriteLine("File changed");
}
void Monitor_FilesDeleted(object? sender, List<string> e) {
	Console.WriteLine("File removed");
}

Console.ReadLine();

using FluentFTP;
using FluentFTP.Monitors;


var asyncclient = new AsyncFtpClient("Server", "User", "Password");

client.Connect();

Console.WriteLine("Running FtpFolderMonitor");

var monitor = new AsyncFtpFolderMonitor(asyncclient,"OutboxDirectory");

monitor.WaitTillFileFullyUploaded = false;
monitor.Recursive = false;
monitor.PollInterval = 5;

monitor.FilesAdded += Monitor_FilesAdded;
monitor.FilesChanged += Monitor_FilesChanged;
monitor.FilesDeleted += Monitor_FilesDeleted;

void Monitor_FilesAdded(object? sender, List<string> e) {
	Console.WriteLine("File added");
}

void Monitor_FilesChanged(object? sender, List<string> e) {
	Console.WriteLine("File changed");
}
void Monitor_FilesDeleted(object? sender, List<string> e) {
	Console.WriteLine("File removed");
}

Console.ReadLine();

robinrodricks · 2024-10-10T11:05:28Z

You have missed to start the monitor.

monitor.Start();

Just test one version (sync is fine).

The class is easy to understand. You can add breakpoints into the PollFolder method and see what's going on.

Since I'm very busy I am hoping you can fix minor bugs and submit a PR with a working version. I don't have time to test it on my end right now.

markat1 · 2024-10-16T16:21:39Z

I played around with it for a couple of hours - does work fine - maybe a small change would be to put restart time in the try catch finally - just to make sure it's always running

/// <summary>
/// Polls the FTP folder for changes
/// </summary>
private async void PollFolder(object state) {
	try {

		// exit if not connected
		if (!_ftpClient.IsConnected) {
			return;
		}

		// stop the timer
		StopTimer();

		// Step 1: Get the current listing
		var currentListing = await GetCurrentListing();

		// Step 2: Handle unstable files if WaitTillFileFullyUploaded is true
		if (WaitTillFileFullyUploaded) {
			currentListing = HandleUnstableFiles(currentListing);
		}

		// Step 3: Compare current listing to last listing
		var filesAdded = new List<string>();
		var filesChanged = new List<string>();
		var filesDeleted = new List<string>();

		foreach (var file in currentListing) {
			if (!_lastListing.TryGetValue(file.Key, out long lastSize)) {
				filesAdded.Add(file.Key);
			}
			else if (lastSize != file.Value) {
				filesChanged.Add(file.Key);
			}
		}

		filesDeleted = _lastListing.Keys.Except(currentListing.Keys).ToList();

		// Trigger events
		if (filesAdded.Count > 0) FilesAdded?.Invoke(this, filesAdded);
		if (filesChanged.Count > 0) FilesChanged?.Invoke(this, filesChanged);
		if (filesDeleted.Count > 0) FilesDeleted?.Invoke(this, filesDeleted);

		if (filesAdded.Count > 0 || filesChanged.Count > 0 || filesDeleted.Count > 0) {
			ChangeDetected?.Invoke(this, EventArgs.Empty);
		}

		// Step 4: Update last listing
		_lastListing = currentListing;
	}
	catch (Exception ex) {
		// Log the exception or handle it as needed
		Console.WriteLine($"Error polling FTP folder: {ex.Message}");
	}
	finally {
		// restart the timer
		StartTimer(PollFolder);
	}
	                  
	
}

FanDjango · 2024-10-17T17:48:42Z

Why don't you create a PR from this, then it can be an official change? I'll merge it then.

markat1 · 2024-10-21T16:29:45Z

hmm don't think I'm allowed to make a pull request - anyway it's just a small change :)

FanDjango · 2024-10-21T17:46:40Z

Never mind, I'll take care of it...

See #1661

FanDjango · 2024-10-22T18:09:55Z

I merged it.

robinrodricks added enhancement discussion labels Oct 5, 2024

robinrodricks mentioned this issue Oct 5, 2024

FluentFTP github page could use a discussion tab? #1660

Open

robinrodricks changed the title ~~Can I subscribe to FTP events when files are changed in a folder?~~ Can I subscribe to FTP events when files are added or changed in a folder? Oct 5, 2024

robinrodricks added major work in progress labels Oct 9, 2024

FanDjango added a commit to FanDjango/FluentFTP that referenced this issue Oct 22, 2024

See robinrodricks#1661

688e4a2

FanDjango added a commit that referenced this issue Oct 22, 2024

Merge pull request #1668 from FanDjango/master

8994ced

See #1661

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can I subscribe to FTP events when files are added or changed in a folder? #1661

Can I subscribe to FTP events when files are added or changed in a folder? #1661

robinrodricks commented Oct 5, 2024 •

edited

Loading

FanDjango commented Oct 5, 2024

markat1 commented Oct 5, 2024 •

edited

Loading

robinrodricks commented Oct 7, 2024

FanDjango commented Oct 7, 2024 •

edited

Loading

robinrodricks commented Oct 8, 2024

markat1 commented Oct 9, 2024 •

edited

Loading

robinrodricks commented Oct 9, 2024

robinrodricks commented Oct 9, 2024 •

edited

Loading

markat1 commented Oct 10, 2024 •

edited

Loading

robinrodricks commented Oct 10, 2024 •

edited

Loading

markat1 commented Oct 16, 2024

FanDjango commented Oct 17, 2024

markat1 commented Oct 21, 2024 •

edited

Loading

FanDjango commented Oct 21, 2024

FanDjango commented Oct 22, 2024

Can I subscribe to FTP events when files are added or changed in a folder? #1661

Can I subscribe to FTP events when files are added or changed in a folder? #1661

Comments

robinrodricks commented Oct 5, 2024 • edited Loading

Question:

Proposed solution:

FanDjango commented Oct 5, 2024

markat1 commented Oct 5, 2024 • edited Loading

robinrodricks commented Oct 7, 2024

FanDjango commented Oct 7, 2024 • edited Loading

robinrodricks commented Oct 8, 2024

markat1 commented Oct 9, 2024 • edited Loading

robinrodricks commented Oct 9, 2024

robinrodricks commented Oct 9, 2024 • edited Loading

New classes:

Usage:

markat1 commented Oct 10, 2024 • edited Loading

robinrodricks commented Oct 10, 2024 • edited Loading

markat1 commented Oct 16, 2024

FanDjango commented Oct 17, 2024

markat1 commented Oct 21, 2024 • edited Loading

FanDjango commented Oct 21, 2024

FanDjango commented Oct 22, 2024

robinrodricks commented Oct 5, 2024 •

edited

Loading

markat1 commented Oct 5, 2024 •

edited

Loading

FanDjango commented Oct 7, 2024 •

edited

Loading

markat1 commented Oct 9, 2024 •

edited

Loading

robinrodricks commented Oct 9, 2024 •

edited

Loading

markat1 commented Oct 10, 2024 •

edited

Loading

robinrodricks commented Oct 10, 2024 •

edited

Loading

markat1 commented Oct 21, 2024 •

edited

Loading