-
Notifications
You must be signed in to change notification settings - Fork 16
FA 0.20
FA'Next 0.20 includes the following improvements:
-
use of .arc archive format (the same as FreeArc), extended with support of file attributes and deduplication info
-
large part of program was rewritten in Lua and is now shipped in the source form. Compared to FA'Next 0.11, amount of Lua code was increased from 500 to 3000 lines and now is about a half of entire program. Overall, C++ code now handles only low-end boring details (back-end), while Lua code handles all interaction with user (UI, cmdline/inifile processing). Lua 5.3.4, Penlight 1.4.1 and LuaFileSystem 1.6.3 are included into the executable.
-
encryption support, including all FreeArc options (-p -hp -kf -op -okf -ae) and algorithms (aes, blowfish, serpent, twofish, aes-128, aes/cfb, aes+serpent...)
-
Lizard compression algorithm and support of CELS external algorithms
Table of contents:
New commands (compatible with FreeArc):
lb|barelist bare list of filenames in the archive
v[erblist] verbosely list files in the archive
Now FA'Next supports all FreeArc archive listing operations and they display the same information as their FreeArc counterparts. Additionally, file attributes are displayed in the same format as in RAR.
New options, grouped by their purpose (most are compatible with FreeArc, remaining were borrowed from RAR):
-ai --no-attrs don't save/restore file attributes
-oc restore NTFS Compressed file attribute
-ep3 keep drive/rootchar in filenames, useful for multi-disk backups
-k --lock lock archive from further updates
-sfx[MODULE] add sfx MODULE ("freearc.sfx" by default)
-zFILE --arccmt=FILE read archive comment from FILE or stdin
--archive-comment=COMMENT input archive COMMENT in cmdline
--no-arc-ext don't add default extension to archive name
--no-crc don't compute/check CRC
-p[PASSWORD] --password=PASSWORD encrypt/decrypt compressed data using PASSWORD
-hp[PASSWORD] --headers-password=PASSWORD encrypt/decrypt archive headers and data using PASSWORD
-kfKEYFILE --keyfile=KEYFILE encrypt/decrypt using KEYFILE (with or without password)
-opPASSWORD --old-password=PASSWORD old PASSWORD used only for decryption
-okfKEYFILE --old-keyfile=KEYFILE old KEYFILE used only for decryption
-aeALGORITHM --encryption=ALGORITHM encryption ALGORITHM (aes, blowfish, serpent, twofish, aes+serpent...)
-dmMETHOD --dirmethod=METHOD compression method for archive directory
-lcN --LimitCompMem=N limit memory usage for compression to N mbytes (or N% of RAM)
-ldN --LimitDecompMem=N limit memory usage for decompression to N mbytes (or N% of RAM)
-mdN --dictionary=N set compression dictionary to N mbytes
-e[+]MASK include/exclude files based on file attributes (MASK is a number or [SHARD]+)
-taTIME --time-after=TIME select files modified after specified TIME (format: YYYYMMDDHHMMSS)
-tbTIME --time-before=TIME select files modified before specified TIME (format: YYYYMMDDHHMMSS)
-tnPERIOD --time-newer=PERIOD select files newer than specified time PERIOD (format: #d#h#m#s)
-toPERIOD --time-older=PERIOD select files older than specified time PERIOD (format: #d#h#m#s)
-diTYPES --display=TYPES select TYPES of information displayed ("haftnwk" by default)
-h --help display help on commands and options
-ioff --shutdown shutdown computer when operation completed
--queue queue operations across multiple FreeArc copies
Option -ds now can reverse sort order by using '-' suffix after the sorting symbol, f.e. --sort=ge-p-s sorts by group, then extension and path in descending order, and finally by size.
Options -n/-x now support listfiles: [email protected].
Since now FA'Next supports the same .arc format as original FreeArc, archives made by FreeArc can be listed/extracted by FA'Next and vice versa - archives created by FA'Next can be processed with FreeArc, SFX modules, unarc.exe and unarc.dll. Moreover, we added x64 versions of these modules, so now you can extract f.e. archives with 4GB lzma dictionary.
Option -sfx[MODULE] can be used in compression command to prepend self-extracting module to the archive being created. If supplied module name doesn't contain path chars, the module will be searched in the FA'Next executable directory. Default module name is freearc.sfx.
FA'Next extends .arc format by adding OS attributes (read-only, hidden... on Windows and rwx-rwx-rwx on Unix) to each file. The attributes are stored by compression commands, and restored by extraction ones. Archive listing commands show attributes in the same way as RAR does. Attributes stored on different OS (Unix/Windows) are displayed by archive listing commands, but cannot be restored on extraction. The attributes stored in archive ATM are ignored by old tools (including current versions of SFX/unarc modules). Option -ai AKA --no-attrs forces FA'Next to mimic old behavior - i.e. don't store file attributes when creating archive, and don't restore attributes on extraction. NTFS Compressed attribute is stored on Windows systems, but restored on extraction only when -oc option is specified.
Archives, created with -dup option, contain deduplication information required for extraction. Thus, old tools can't extract such archives, but can list their contents. Compression option --save-sha-hashes additionally store SHA256 checksum for each deduplicated chunk. These checksums are checked on extraction, unless --no-check option is specified.
Finally, option --no-crc disables CRC computation/check and can be used in any compression and extraction commands.
Options -k/--lock (lock archive from further updates), -ep3 (keep full pathname on creation and extraction), -zFILE (add archive comment from FILE) and --no-arc-ext (don't add default .arc extension, even when archive name has no extension) should be familiar to FreeArc users, while option --archive-comment=COMMENT (add archive COMMENT specified literally) is more intended for tools.
FA'Next 0.20 provides the same encryption functionality as FreeArc:
-p[PASSWORD] --password=PASSWORD encrypt/decrypt compressed data using PASSWORD
-hp[PASSWORD] --headers-password=PASSWORD encrypt/decrypt archive headers and data using PASSWORD
-kfKEYFILE --keyfile=KEYFILE encrypt/decrypt using KEYFILE (with or without password)
-opPASSWORD --old-password=PASSWORD old PASSWORD used only for decryption
-okfKEYFILE --old-keyfile=KEYFILE old KEYFILE used only for decryption
-aeALGORITHM --encryption=ALGORITHM encryption ALGORITHM (aes, blowfish, serpent, twofish, aes+serpent...)
Encryption is enabled by using any of -p, -hp, -kf options. Option -p provides password and requests encryption of only data, while -hp requests encryption of both data and archive directory. Option -kf provides keyfile (equivalent to password stored in the file). Password and keyfile may be used simultaneously, in this case password is prepended to keyfile contents. If option -hp doesn't include password, it only enables archive directory encryption, while password/keyfile should be supplied with -p/-kf options.
Options -op/-okf may be used to specify any number of passwords/keyfiles that will be used only for decryption of existing data. When FA'Next extracts encrypted data, it tries all passwords specified by -p/-op options, then all keyfiles specified by -kf/-okf and finally all combinations of any password with any keyfile.
ATM FA'Next cannot request passwords interactively, so using -p/-p?/-hp/-hp? without providing password/kefile in other way is impossible. Similarly, if data can't be decrypted with any password/keyfile provided, decompression fails.
Option -ae allows to specify encryption algorithm. Currently supported algorithms are:
- aes-128,192,256
- serpent-128,192,256
- twofish-128,192,256
- blowfish-64,72..448
Each algorithm can be used in CTR or CFB mode, f.e. aes-256/ctr. Keysize and/or mode can be omitted, in this case largest size and CTR mode are used: aes = aes-256 = aes/ctr = aes-256/ctr.
Finally, passwords are encrypted using PKCS5#2 algorithm. Number of algorithm iterations may be specified by ":n" parameter, i.e. "aes:n2000" and defaults to 1000.
Encryption algorithms can be chained similarly to TrueCrypt, f.e. "--encryption=aes+serpent/cfb+blowfish", or even "--encryption=aes+aes" which essentially increases key size to 2*256 bits.
We added the compression algorithm Lizard (former LZ5 v2) by Przemyslaw Skibinski, that can be competitive with LZ4 and ZSTD. There are compression levels from 'lizard:10' to 'lizard:49' (default is 17), providing various cspeed/dspeed/ratio tradeoffs. Since the compression dictionary is limited to 16 MB, it may be especially useful with 4x4, f.e. "-m=4x4:b64m:lizard:49".
FA'Next 0.20 also added support for CELS external compression libraries. CELS is the further development of CLS. CELS provides external compression libraries tight integration with FA'Next compression management code, that previously was available only to internal algorithms. This means that FA'Next can report and modify memory usage, dictionary/block size and other algorithm settings, tailoring compression algorithm to your computer and data being compressed. CELS libraries also can be used by unarc, sfx modules and unarc.dll.
Option -di provides flexible configuration of the program output. Its parameter is a list of symbols enabling each part of the output:
- h: program header like
FreeArc'Next x64 v0.20 (Mar 17 2017) - a: archive name and operation like
creating archive m:\a.arc - o: options read from environment/config like
using additional options: --logfile=c:\temp\fa.log --display=dnwl - c: compression method like
Compress with rep:512mb:c512+4x4:zstd:2:8mb, $compressed => rep:512mb:c512 - m: compression memory like
Compression memory 693mb + 8*1mb read-ahead buffers, decompression memory 528..674 MiB - f: compression/decompression totals like
Compressed: 21,000 -> 6,290 bytes. Ratio 29.95% - t: compression/decompression timing like
Compression time: cpu 203.27 sec/real 41.22 sec = 493%. Speed 58.1 MiB/s - l: the long progress indicator (see below); it's the only suboption absent in original FreeArc
- d: archive directory original and compressed size like
Archive directory: 2,284,922 -> 675,257 bytes(printed only when the directory size is >10KB) - w: warning messages like
WARNING: can't open file "C:\base\Multimedia\foobar2000\running" - n: total number of warnings at the end of operation like
There were 2 warning(s) - k:
All OKat the end of operation if there were no warnings/errors - e: extra empty line at the end of output
-
$: display profiling information on the screen -
#: write profiling information to the logfile -
%: display system memory info prior to operation
Also, -di+CHARS adds CHARS to the current -di config, while -di-CHARS removes these CHARS, so -di=haoc -di-hc -di+l is equivalent to -di=aol.
Default setting is -di=haftnwk displaying just essential information (default setting can be restored by -di--). Option -di without parameter enables all normal suboptions and is equivalent to '-di=haocmftdnwk'. Compare their output:
C:\test>fa create m:\a
FreeArc'Next x64 v0.20 (Mar 17 2017) creating archive m:\a.arc
Found 2,511,674,419 bytes in 221 folders and 2,301 files
Compressed: 2,511,674,419 -> 319,468,222 bytes. Ratio 12.72%
Compression time: cpu 203.32 sec/real 41.31 sec = 492%. Speed 58.0 MiB/s
All OK
C:\test>fa create m:\a -di
FreeArc'Next x64 v0.20 (Mar 17 2017) creating archive m:\a.arc using additional options: --logfile=c:\temp\fa.log
Found 2,511,674,419 bytes in 221 folders and 2,301 files
Compress with rep:512mb+dispack070+delta+4x4:lzma:96mb:normal:16:mc8, $obj => rep:512mb+delta+4x4:lzma:96mb:normal:16:mc8, $text => grzip:8mb:m1:l32:h15, $wav => tta, $bmp => mm+grzip:8mb:m1:l2048:h15:a, $compressed => rep:512mb+4x4:zstd:7:16m:h8m
Compression memory 3082mb + 8*1mb read-ahead buffers, decompression memory 735..2514 MiB. Prefetch 256mb using 1 thread
Compressed: 2,511,674,419 -> 319,468,222 bytes. Ratio 12.72%
Compression time: cpu 203.27 sec/real 41.22 sec = 493%. Speed 58.1 MiB/s
Archive directory: 76,531 -> 27,199 bytes
All OK
If suboption 'l' is enabled, long progress indicator displayed for disk scan, compression and extraction operations:
C:\test>fa create m:\a -di+l
FreeArc'Next x64 v0.20 (Mar 17 2017) creating archive m:\a.arc
Found 2,511,676,076 bytes in 221 folders and 2,301 files (RAM 3 MiB, I/O 0.000 sec, cpu 0.000 sec, real 0.009 sec)
Compressed: 2,511,676,076 -> 319,470,935: 12.72%. I/O 9.126 sec, cpu 11.8 MiB/s (202.318 sec), real 59.1 MiB/s (40.503 sec) = 500%
All OK
Without the 'l' suboption, the progress indicator sits on diet:
C:\base>fa create m:\a
FreeArc'Next x64 v0.20 (Mar 17 2017) creating archive m:\a.arc
Found 33,269,658,966 bytes in 36,046 folders and 505,290 files (1.70 sec)
11%: 3,643 -> 197 MiB: 5.41%. 113 MiB/s = 77% (32 sec). ETA 03:49
Once the compression/extraction operation is completed, the short progress indicator is replaced by summary lines produced by 'f' and 't' suboptions. If neither 'f' nor 't' are enabled, final output of the short progress indicator kept to show the result. If long progress indicator is enabled, 'ft' suboptions are ignored.
In -dup mode, extra information is displayed: deduplicated size in the long progress indicator and memory required for decompression in the archive directory line:
C:\base>fa create m:\a -dup -di=ld
Found 2,511,682,361 bytes in 221 folders and 2,301 files (RAM 3 MiB, I/O 0.016 sec, cpu 0.000 sec, real 0.015 sec)
Compressed: 2,511,682,361 -> 1,283,502,239 -> 406,444,362: 16.18%. RAM 17 MiB. I/O 3.448 sec, cpu 129 MiB/s (18.580 sec), real 504 MiB/s (4.753 sec) = 391%
Archive directory: 2,284,847 -> 676,763 bytes. Decompression memory: 218 + N*9 = 226..285 MiB
C:\base>fa.exe t m:\a -di=l
Testing 2,511,682,361 bytes in 221 folders and 2,301 files
Tested: 406,444,362 -> 1,283,502,239 -> 2,511,682,361: 16.18%. RAM 0 / 218 MiB. I/O 2.558 sec, cpu 462 MiB/s (5.179 sec), real 693 MiB/s (3.458 sec) = 150%
Operation logging (option --logfile) got a lot of improvements:
- each line in logfile supplied with PID (process ID) number, allowing to distinguish lines from multiple simultaneous runs
- log of each command starts with line that includes command start time, start directory and command line with sensitive information (e.g. passwords) removed
- logfile gets the same information as should be printed to the screen if all additional display options will be enabled
- the only exceptions are symbol '$' that prints profiling info to the screen ('#' outputs the same info to the logfile), and '%' that can output memory info only to the screen
- 'l' suboption also outputs long operation results to the logfile
- all info saved to logfile is UTF-8 encoded
The whole UI/logging is now implemented in Lua (file UI.lua), so you can further modify it to tailor your needs.
New Lua functionality can be split into 3 groups:
- services provided by the C++ code to the Lua code
- services provided by the Lua code to itself
- services provided by the Lua code to the program
Changes in the built-in libraries:
- Lua itself was updated to latest version 5.3.4
- added LuaFileSystem 1.6.3 library that provides filesystem functions absent in Lua itself
- added Penlight 1.4.1 library that is similar to Python built-in library
The filesystem operations, provided by all these libraries, isn't Unicode-aware on Windows - instead they use filenames in ANSI codepage. So, we added a few Unicode/UNC/LongName-aware operations:
- File.open(filename,mode) - replaces io.open
- File.exists(filename), Dir.exists(dirname) - return true if file/directory exists
- Dir.current() - return current directory
- SetTempDir(dirname) - set directory for temporary files, that may be created by external compressors
Pseudo-table archive contains the following fields with information about the current archive: deduplication_mode, locked, comment, rr_settings, sfx_size, host_os.
The main service now implemented in Lua is ParseCommandLine() - it receives from C++ code the list of command arguments and does the following:
- parses & checks cmdline, config files and environment variable FA_CFG
- executes user-supplied Lua code
- returns command name, list of filespecs to process and values of program options
At the return from this function, we are ready to execute the command.
Another new service is UI (implemented by UI.lua). Lua code displays all the information user sees as well as dumps its "hard copy" to the logfile.
Finally, every compression method that is going to be used for compression/extraction or displayed, is "filtered" through the Lua functions that can modify it. This is used by the following modules:
- Encryption.lua adds encryption algorithm to the method and generates encryption keys
- Compression.lua limits memory usage according to the -lc/-ld options
UI implementation uses the following resources:
- usual events such as
onArchiveStartused to inform user about the command progress. Event handlers are labeled 'UI' allowing user code to remove them - archive., command. and optvalue.* fields provide information about archive being processed and command being executed
- uiTransientMessage(msg), uiProgress(state), uiAskOverwrite(outname), uiWarning(message), uiError(message), uiProfiler(label) are special Lua UI events called from C++ code to handle UI-specific events
uiProgress(state) called by C++ code every 0.1 second during disk scan, compression and extraction operations. state is a table carrying all necessary information about the current operation progress. UI.lua uses this information to show short or long progress indicator while operation proceeds.
On Windows, UI uses the following services provided by C++ code, to improve user experience:
- EnvSetConsoleTitle(str)/EnvResetConsoleTitle() sets/resets title of the console window
- Taskbar_SetProgressValue(Completed,Total) displays progress indicator in the taskbar (where Completed and Total are 64-bit integers)
- Taskbar_Normal(), Taskbar_Error(), Taskbar_Pause(), Taskbar_Resume(), Taskbar_Done() changes the taskbar status (i.e. color of the taskbar progress indicator)
- WriteConsole(str) writes UTF8-encoded string to the console window associated with stderr (note that
printon Windows displays string in the current console encoding that is OEM codepage by default)
The following services provided by C++ code on Windows, aren't yet employed by UI.lua:
- SetConsoleCursorPosition(x,y), SetConsoleTextAttribute(attr) allows to create colorful, full-screen UI in the console window
- GetConsoleScreenBufferInfo() returns table describing the console properties (size, cursor position...)
These services may be used to extend UI with the following features:
- colorized output, f.e. showing warnings in orange and errors in red
- multi-line progress indicator, combining completeness of long progress indicator with support of 80-char wide consoles
- full-screen console UI, like one that was present in rar, arjz and ace at the DOS times. It will be useful for integration of FA'Next into console archive managers such as FAR.