-
Notifications
You must be signed in to change notification settings - Fork 108
Networking: File Transfers
An important goal for ELKS has been to become fully networked - which means a reliable TCP/IP network stack and drivers to support the physical networks. SLIP and serial port connections has been available for several years while the ktcp user space implementation of TCP/IP has been constantly improved. As of December 2021 Ethernet was fully supported via the PC ISA bus and the ubiquitous NE1K and NE2K ethernet interfaces. A year later (December 2022) more interface types are supported, refer til the main wiki for details.
ELKS supports the telnet-protocol (inbound and outbound) for interactive access, the ftp-protocol (inbound and outbound) for file transfers and http via the ELKS httpd server. Recent improvements in networking support includes serial port reliability & speed and not the least a ftp/ftpd pair rewritten specifically for ELKS.
Serial
Transferring limited size ascii files can be done with reasonable reliability via serial lines and terminal emulation. Using cat > filename on ELKS and then pasting text into the terminal is reasonably efficient. If this fails marginally (losing a few bytes here and there), turn off echo in order to reduce the interrupt load on the system:
stty raw; cat > filename; stty -raw
or use the sercat command:
sercat > filename
Terminate such transfers with ^D (EOF).
To check whether the transfer was error free, use the sum command at both ends, after ensuring that the byte count is exactly the same (ls -l). Note that it takes only an extra newline or CR at the end of the file to make the sum command useless.
Even binary files may be transferred this way by first converting them to ascii. Use the hexdump command on Linux:
hexdump -e '"%06.6_ax:" 8/1 " %02x" " " 8/1 " %02x" " " ' -e '16/1 "%_p" "\n"' "$@"
to create the ascii file, then transfer it and use hd -r on ELKS to convert it back.
A far more efficient (and traditional!) way is using uuencode and uudecode. Instead of turning 16 bytes into 69 which hd does, uuencode converts 2 bytes into 3 ascii characters. The commands are part of the uucp package. The (simplified) syntax is
uuencode newname < inputfile > outputfile
where newname is the filename to be used when the file is decoded and recreated. Even the filemode (protection codes) is stored in the encoded file and will be restored during decoding – if possible. To decode, simply give the command
uudecode < inputfile
and the original file (and filename + modes) will be recreated. There are many options to override the defaults, check the man-pages for details.
Transferring text or uuencoded binary files from ELKS via serial is usually easy (and reliable) since most systems will have no problems keeping up with the speed of the ELKS system.
Mechanisms to make serial transfers binary and fully reliable - Kermit, X/Y/Z-modem etc. - are forthcoming.
As mentioned above, ELKS now supports several different ISA-based Ethernet interfaces and a solid TCP/IP implementation in the ktcp user space process. Application level telnet, ftp and http protocols are supported. The quality and diversity of this support is improving constantly. Until recently, the urlget program was the key to move files into ELKS. Now, the availability of ftp (client and server) opens ELKS to the world so to speak. The ftp user application and the ftpd server deamon implement most basic ftp commands, enabling the use of visual file managers like FileZilla with ELKS.
Verify that the ethernet interface was actually configured correctly during boot by looking at the console boot messages where you should see something like one of these lines (depending on your configuration):
eth: ne0 at 0x380, irq 9, (ne1k) MAC 00:1f:11:02:60:2d, flags 0x80
eth: wd0 at 0x320, irq 11, ram 0xcc00, (wd8013) MAC 00:00:C0:BC:8F:4B, flags 0x80
eth: 3c0 at 0x300, irq 15, (3c509) MAC 00:a0:24:67:7f:b1 (HWconf: 330/9), flags 0x80
Use the ps command to check that the network is up and running (look for the ktcp-process and the telnetd and ftpd daemons):
# ps
PID GRP TTY USER STAT CSEG DSEG HEAP FREE SIZE COMMAND
1 0 root S 3240 33ef 3072 2012 12864 /bin/init 4
22 22 1 root S 3564 36ff 0 1966 13136 /bin/getty /dev/tty1
23 23 S0 root S 3cca 71d3 1164 8860 66688 -/bin/sh
24 24 S1 root S 3cca 76e0 1164 8788 66688 -/bin/sh
11 11 root S 513b 63b2 3072 32690 75696 ktcp -b -p wd0 10.0.2.15 10.0.2.2 255.255.255.0
15 15 root S 5591 4c2e 0 1996 15376 telnetd
18 18 root S 5c26 5719 0 11406 32992 ftpd
27 23 S0 root R 4e67 5f9f 1024 1188 13568 ps
#
If it's not, run the net start command, and check for errors in the output:
# net start
Starting networking on wd0
ktcp -b -p wd0 10.0.2.15 10.0.2.2 255.255.255.0
ktcp: ip 10.0.2.15, gateway 10.0.2.2, netmask 255.255.255.0
ktcp: /dev/wd0 mac 00.00.c0.bc.8f.4b mtu 1500
Starting daemons 'telnetd' 'ftpd -d'
#
net start and the corresponding net stop commands are automatically run at startup if net=xxx is included in the ELKS config-file /bootopts, ´xxx´being one of ne0, wd0, 3c0. The same configfile includes other options that define the network configuration. For more information, check out the chapter on Boot options in the main wiki. Here is a sample working /bootopts file:
## boot options max size 511 bytes
console=ttyS0,38400 debug net=ne0 3 # serial console, multiuser, networking
#TZ=MDT7 # timezone
#QEMU=1 net=ne0 # ftpd on QEMU
HOSTNAME=elks1 # set network IP from /etc/hosts
GATEWAY=10.0.2.2
ne0=9,0x380,,0x80
#wd0=11,0x320,0xCC00,0x80
#3c0=15,0x300,,0x80
#comirq=,,7,10
#init=/bin/init 3 n # multiuser serial no /etc/rc.d/rc.sys
#init=/bin/sh # singleuser shell
#console=ttyS0,19200 # serial console
#root=hda1 ro # root hd partition 1, read-only
Which daemons to start automatically is configured in the /etc/net.cfg file. The examples in the file are reasonably self-explanatory.
The interactive ftp client works like most ftp-clients for Linux, Unix and other OSes. Please refer to Linux ftp-man pages for general info. The following paragraphs discuss the exceptions to the 'standard'.
Command line options
The ftp-client recognises the following command line options:
-u - username
-p - password
-P - Start in PASV (passive) mode. This is the default.
-A - Start in PORT (active) mode
-i - Disable multifile prompting.
-q - Qemu mode, normally set automatically via the environment-variable QEMU, see below.
-v - Verbose mode, equivalent to debug = 1, which is now the default.
-d - Debug on. The default debug level is 1, repeat -d to increase the level top max 4.
Thus a typical command line for ftp may be
# ftp -u user 10.0.2.2 5050
or simply
# ftp
The former will connect to the given host or address and prompt for username and password, like this:
# ftp -u myusername 10.0.2.2 21
Connected to 10.0.2.2.
220 raspi2b FTP server (Version 6.4/OpenBSD/Linux-ftpd-0.17) ready.
Name (10.0.2.2:myusername):
Password:
530 Login incorrect.
Login failed: 530 Login incorrect.
ftp>
The port number is optional, default is 21.
If ftp is invoked without arguments, the open command establishes the connection:
ftp> open 10.0.2.2
Connected to 10.0.2.2.
220 raspi2b FTP server (Version 6.4/OpenBSD/Linux-ftpd-0.17) ready.
Name (10.0.2.2:ftp): username
Password:
230-
230- The programs included with the Debian GNU/Linux system are free software;
230- the exact distribution terms for each program are described in the
230- individual files in /usr/share/doc/*/copyright.
230-
230- Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
230- permitted by applicable law.
230 User helge logged in.
Remote system is Unix/Linux, using binary mode
ftp>
The commands currently supported by ELKS ftp are:
ftp> help
directory: List remote directory
ls: List remote directory
get: Fetch remote file
mget: Fetch multiple remote files
put: Send file to remote
mput: Send multiple files to remote
cd: Change (remote) working directory
pwd: Show (remote) working directory
bin: Set file transfer type to BINARY
ascii: Set file transer type to ASCII
quit: Quit program
type: Show file transfer type.
prompt: Toggle interactive multifile transfers
debug: Set debug level 0-4 or 'off'.
bye: Leave program
!: Invoke shell to execute local command.
?: Alias for help.
help: Print this.
close: Close connection.
open: Open new connection, log in.
mkdir: Create remote directory.
delete: Delete remote file.
verbose: Toggle debug off or 1, default 1.
status: Show status, modes etc.
lcd: Change local working directory
glob: Toggle globbing
passive: Toggle passive/active mode
ftp>
Commands may be abbreviated to 3 characters.
Note the difference between bye and close: The former exits the program, the latter just closes the connection, and ftp is ready for a new open command. ^C will terminate the program.
ftp in QEMU
ftp (and the ELKS ftp server) take special measures to work seamlessly within QEMU. The qemu.sh script that comes with ELKS includes mapping necessary for the ftpd server to support passive connections:
# Simultaneous telnet, http and ftp forwarding
FWD="\
hostfwd=tcp:127.0.0.1:8080-10.0.2.15:80,\
hostfwd=tcp:127.0.0.1:2323-10.0.2.15:23,\
hostfwd=tcp::8020-:20,\
hostfwd=tcp::8021-:21,\
hostfwd=tcp::8041-:49821,\
hostfwd=tcp::8042-:49822,\
hostfwd=tcp::8043-:49823,\
hostfwd=tcp::8044-:49824,\
hostfwd=tcp::8045-:49825,\
hostfwd=tcp::8046-:49826,\
hostfwd=tcp::8047-:49827,\
hostfwd=tcp::8048-:49828,\
hostfwd=tcp::8049-:49829"
With this setup, the command to connect to ELKS from the QEMU host is ftp -P 8021 localhost.
Make sure QEMU=1 is enabled in either /bootopts or /etc/profile. The only limitation is that outbound ftp (from ELKS) requires passive mode (the default). Currently, when in QEMU-mode, the ftp-server will turn debug on, which means extra output to the console. This will eventually go away.
The FTP server (aka ´ftp daemon´) is normally started automatically with the net start command. It will accept connections on the standard ftp port and supports both passive and port (active) modes. It requires the external user to log in, but neither checks the username or the password at this time, so you can use whatever default comes up or just empty lines.
The server implements most of the regular ftp commands and will tell you if a command is not implemented. In most cases it will detect if the remote client is Linux or Unix, and set binary (image) mode automatically. The SITE command is supported and enables (among other things) a file transfer delay to be specified, which enables file transfers between ELKS systems to run reasonably smooth. In most cases, such delays (on client and server sides) are set automatically because the programs recognize the ELKS platform as a special case.
Likewise, the server will automatically detect QEMU and make adjustments accordingly (see above).
MORE ABOUT SPECIAL SERVER COMMANDS <<<<
urlget is a very flexible tool for transferring files to or from ELKS, and particularly well suited for inclusion into scripts. Four programs in one – urlget, httpget, ftpget and ftpput, supporting 3 different URLs and 3 protocols: HTTP, FTP and pure TCP (useful with netcat) - all requiring a server to talk to at the other end. Not a problem - if such a server is not already running, a python one-liner is normally all it takes to start your own. Which frequently turns out to be preferable even when such server is already running – because you get access to you own home directory. More about that below.
A summary of the command syntax:
urlget http://host:port/filepath
urlget http://username:passwd@host:port/filepath
urlget ftp://host/filepath
urlget ftp://username:passwd@host[:port]/filepath
urlget tcp://host:port/filepath
httpget [-h] [-d] [-p] host filepath
ftpget [-v] host[:port] filepath [username [passwd]]
ftpput [-v] host[:port] filepath [username [passwd]]
For all variants, the authentication can be omitted if acceptable to the server. If included, note that the username and password are sent in plaintext over the network. Also, port may be omitted in most cases, in which case the default port is used - 80 for http, 21 for ftp. For tcp, the port must be included as there is no standard.
For ftpget, if the filepath ends with a slash, a directory listing is delivered. The -v option extends the file listing to long format (ls -l). For http, the -h option will include HTTP-headers in the output, -d will discard the contents, which is meaningful only with the -h option (display headers only), and the -p option will post (send) data instead of fetching it. The data in this case is whatever follows a '?' in the filepath. This option is currently not very meaningful.
ftpput will create new directories as needed and if permitted. ftpput 10.0.2.2 /tmp/bin/sh name passed will create the tmp and bin directories (relative to the current directory) if they don't exist
The tcp-url sends the file name, but there is no standard way of handling such request at the other end, see below for a suggestion on how to use this variant.
If running QEMU, use the default qemu.sh script and make sure this line is 'uncommented': NET="-netdev user,id=mynet,$FWD -device ne2k_isa,irq=12,netdev=mynet"
urlget with http, httpget
- Start a web server on the host, like a simple python http-server:
- If you have python V3, run
python3 -m http.server 8000 --bind 127.0.0.1. If your version is 2 (check by runningpython -Vor resort to v3 explicitly if it exists, likepython3):python -m SimpleHTTPServer. If running Windows, the commands may be a little different, Google it to find out. - This will make the current directory available for file transfers.
- If you have python V3, run
I you want something more 'durable', install and run lighttpd (see below). If using the default install- and config-files, the files to transfer would be placed in a subdirectory of /var/www/html - such as /var/www/html/elks. In this case you don't need to specify the port number (8000 above), or just add '80' (default) for the record. Make sure to create the elkssubdirectory and set the access bits. This installation requires root access.
- Put the file(s) you need to transfer into that directory (or a subdirectory).
- Run
- [python3/python]
urlget http://10.0.2.2:8000/filename > local_filenameorurlget http://10.0.2.2:8000/dir1/dir2/filename > local_filename - [lighttpd, other servers]
urlget http://10.0.2.2/elks/filename > local_filename
- [python3/python]
urlget with ftp, ftpget
If an ftp server is already running, you're OK. Check the server config-files to determine if anonymous access is available, and what directory such access is limited to. If you will be using authentication (username and password), ensure your user is allowed incoming ftp access.
FTP:
If no ftp-server is running, apython-based server is the easy way out, just like the http-example above. pyftpdlib must be installed, google it if directions are needed. Run python3 -m pyftpdlib -p port -u someusername -P somepasswd (see the online docs for details). Any name and password will do, not limited to your local username/passwd, but they must match when making a connection. Anonymous authentication is supposed to work, but does not. The default port number is 2121. Also, be aware that this ftp-server does not support wildcards.
Assuming an FTP server running and your user is an allowed ftp-user, you can ftp files into ELKS like this:
urlget ftp://user:passwd@host:port/filename > localfile
for example
urlget ftp://testuser:[email protected]:21/testfile > localfile
The port number (in this case :21) is optional. 21 is the default and may be omitted. If the ftp server supports (allows) anonymous ftp, the username and password may be omitted. In that case the public (anonymous) ftp directory is where the file (testfile) will be looked for. Otherwise, if you provide your username and password (they are being sent over the network in clear text!), your home directory is the root directory for ftp. If test file ends with a slash /, then it is assumed to be a directory, and the slash means 'list this directory'. Example:
urlget ftp://user:[email protected]/
which will list the users home directory.
Similarly the syntax for ftpget is
ftpget [-v] host[:port] filepath [user [passwd]] > localfile
The -v (verbose) option is meaningful only when listing directories (filepath has a trailing '/').
TCP:
In many ways the simplest of all variants is 'raw tcp': All you need is netcat (nc) on the server, which exists on most Unix and Linux systems.
On the host, start nc so that it waits for connections and then transfers a file:
nc -l 2323 < elksout.txt
2323 is a randomly selected port number (> 1024), which you need when making the connection from ELKS:
urlget tcp://host:2323 > localfile
The port number is mandatory since there is no default. Notice that nc doesn't know when to terminate the connection, so you have to abort it with ^c.
ftpget and httpget
urlget may be linked to different names which will affect its behaviour. If you use ftpget it will use the ftp-protocol and the syntax is like this:
ftpget host[:port] filepath [user [passwd]] > localfile
for example
ftpget 10.0.2.2 filename username userpw > file
Like before, username and password may be omitted if anonymous ftp is supported by the server. And again, if the filename ends with a slash, the name is assumed to be a directory and its content will be displayed. If the command line option -v is used, the directory listing is verbose (like ls -l).
httpget follows the same rules, with syntax as specified above:
httpget [-h] [-d] [-p] host[:port] filepath > localfile
As mentioned above, files may be transferred out of an ELKS system either using ftp (ftpput) or the local http-server. The latter is non-trivial at first, easy when all is set up.
- Place the file/files to transfer in the
/var/wwwdirectory on ELKS. - [QEMU]Use curl or wget to fetch the files: [qemu]
curl http://localhost:8080/filename > filename. - [Physical machine]
curl http://10.0.2.15/filename > somename
In order to access files not in the httpd-directory, symbolic links are useful. For example, if you want to transfer the entire hard disk: cd /var/www; ln -s /dev/hda myfile, then use curl from the other end to fetch myfile. Be warned, this is going to take some time. As of this writing the transfer speed is about 27k bytes per second on a physical system (386/20, NE2K interface).
Using ftpget is simpler, in particular if a ftp-server is already running on the destination machine. The syntax is documented above and is identical to the ftpget variant.