diff --git a/.gitignore b/.gitignore index 2130984..76459f3 100644 --- a/.gitignore +++ b/.gitignore @@ -4,12 +4,12 @@ _site/ .sass-cache/ .jekyll-cache/ .jekyll-metadata - +_config.yml # Ruby .bundle/ .ruby-gemset .ruby-version - +*.gem Gemfile.lock # Mac/Windows System Files @@ -19,4 +19,4 @@ Icon Thumbs.db _config.yaml - +_config.yml diff --git a/Gemfile b/Gemfile index a78ce91..ce6ad5d 100644 --- a/Gemfile +++ b/Gemfile @@ -28,5 +28,3 @@ end # Performance-booster for watching directories on Windows gem "wdm", "~> 0.1.1", :platforms => [:mingw, :x64_mingw, :mswin] -#adding plugin for redirects: -gem 'jekyll-redirect-from' diff --git a/_config.yml b/_config.yml index 7330c89..29169c7 100644 --- a/_config.yml +++ b/_config.yml @@ -16,6 +16,3 @@ heading_anchors: true ga_tracking: G-4GBB05KF5M ga_tracking_anonymize_ip: true - -plugins: - - jekyll-redirect-from \ No newline at end of file diff --git a/_data/program.csv b/_data/program.csv new file mode 100644 index 0000000..ce1117d --- /dev/null +++ b/_data/program.csv @@ -0,0 +1,40 @@ +name,desc,inst,conf,use,docs +7-zip,7-zip is used in the lab to compress evidence and cases generated in Forensic Toolkit. It is also occasionally used to unpack collection material that is not supported by FTK.,[Download 7-zip](https://www.7-zip.org/), +apple3rtr,Apple /// Ready-to-Run (apple3rtr) is a software bundle used for MAME Apple emulation. It has been used by Digital Archives staff to view Apple II files.,For detailed installation instructions visit the [apple3rtr github repo](https://github.com/datajerk/apple3rtr).,,,https://github.com/datajerk/apple3rtr +AWS, Amazon Web Services (AWS) is a cloud computing platform offered by Amazon. AWS S3 features are used to store copies of born-digital material both before and after the collection has been processed by archivists.,,,,https://docs.aws.amazon.com/ +Bagger, Bagger is an applciation created by the U.S. Library of Congress to produce files packages following the Bagit specification. It has been used in the Digital Archives lab for transferring born-digital material.,,,,https://github.com/LibraryOfCongress/bagger +Bagit.py, Bagit.py is a command line tool used produce files packages following the Bagit specification, Check python version by entering ```python --version``` or ```python3 --version``` into terminal. Then install using pip by entering ```pip install bagit``` or ```pip3 install bagit```. Use the link below the main download on the VirtualBox download site to install the Extension Pack on your host.,,,https://github.com/LibraryOfCongress/bagit-python +CiderPress, CiderPress is a Windows utility for managing Apple II disk images. It has been used in the Digital Archives Lab for viewing Apple file formats.,,,,https://a2ciderpress.com +dbpoweramp, dbpoweramp is a program used in conjunction with [Iromlab](#Iromlab) for transferring born-digital material from optical media., +dd, dd is a command line utility used in the Digital Archives Lab to create disk images of optical media. In most cases transfers are made of optical media instead of disk images.,,,,https://linux.die.net/man/1/dd +Disk Utility, Disk Utility is a MacOS utility for disk volume management. Disk Utility used by Digital Archives staff for mounting/unmounting remvable media.,,,enter ```list disk``` or ```list volume``` to confirm media is mounted.,https://support.apple.com/guide/disk-utility/welcome/mac +diskpart, diskpart is Windows command line utility for disk partitioning. diskpart is used by Digital Archives staff for confimring removable media are properly mounted.,,,,https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/diskpart +disktype, disktype is a command line tool used by Digital Archives staff for extracting filesystem metadata like sector format for floppy disks, For Mac OS:
In terminal use homebrew to install by entering ```brew install disktype```,, In terminal navigate to the directory containing floppy disk image(s) and enter ```disktype a/disk/image```.,https://disktype.sourceforge.net/doc/ +DosBox, DosBox is an open-source emulator for the MS-DOS operating system. DosBox has been used by Digital Archives staff to view DOS executables and word processing files.,,,,https://www.dosbox.com/DOSBoxManual.html +DROID, DROID is file format identification tool developed by the UK National Archives.,,,,https://cdn.nationalarchives.gov.uk/documents/information-management/droid-user-guide.pdf +Emailchemy, Emailchemy has been used in the Digital Archives lab for migrating email fomats.,,,,https://www.weirdkid.com/products/emailchemy/doc/Emailchemy_User_Manual.pdf +ePADD, ePADD is an open-source program developed by Stanford University's Special Collections and University Archives. The software ha been used for appraisal and arrangement of email.,,,,https://github.com/ePADD/epadd/releases +Forensic Toolkit, Forensic Toolkit is a computer forensics software used in the Digital Archives Lab to appraise and arrange born-digital material. +FTK Imager, FTK Imager is a disk imaging program associated with Forensic Toolkit. The software has been used in the Digital Archives lab to image and extract files from digital carriers., +gtar, Gtar is a command line program used on Digital Archives MacOS workstations to compress Kryoflux stream files., For MacOS: Install via Homebrew by entering ```brew install gnu-tar``` in terminal.,,,https://www.gnu.org/software/tar/manual/tar.html +HFS Explorer, HFS Explorer is a program usd to view HFS formatted disks on Windows operating systems.,,,,https://catacombae.org/hfsexplorer/ +Iromlab, Iromlab is software for automated imaging of optical media using [Nimbie](#tools/nimbie) disc robots.,,,,https://catacombae.org/hfsexplorer/ +IsoBuster, IsoBuster is data recovery software that has been used in the Digital Archives lab to image and extract files from optical media., +Java, Java is a software package which has been a dependency for some Digital Archives tools like DROID and Bagger.,,, Check for the version of Java installed with commands ```which java``` or ```java -version``` in terminal on Mac or WSL on Windows., +MediaInfo, MediaInfo is an open-source program for extracting technical information from media files. The program has been used in the Digital Archives Lab to extract metadata from born-digital video., +Mediaconch, MediaConch is an open source software project developed by MediaArea and utilizing MediaInfo. It is used by Digital Archives staff to develop programatic policies for checking born-digital audiovisual material against preservation standards., +Mini Vmac, Mini vMac is an emulator for running early Macintosh software. +openssl, OpenSSL is a software library used to ssh.,,,,https://www.openssl.org/docs +Oracle VM VirtualBox, Virtualbox is open soure virtualization software. It has been used by Digital Archives staff to run teh BitCurator environment as a virtual machine., Download and install the [current VirtualBox release and Extension Pack](https://www.virtualbox.org/wiki/Downloads). +pip, Pip is package management software used for installing and upgrading Python modules,,,Pip should be installed with python3. Call pip by entering ```pip``` or ```pip3``` in terminal or command line., +Python3, Python3 is the latest version of the Python programming language, For Windows: Check for python versions by entering ```python --version``` on command line. Installers are avilable at python.org.
For MacOS: Check for python versions by entering ```python --version``` in terminal. Install your selected version of python with homebrew by entering ```brew install python@version.number```., +QuickTime Player, QuickTime is used in teh Digital Archives Lab to appraise Mcintosh formatted born-digital video., +QuickView Plus, QuickView Plus has been used in the Digital Archives Lab to view a range of legacy formats like word processing and photo formats., +rclone,rclone is a command line tool for managing and moving files between remote and local storage locations,For detailed installation and configuration instructions [see our dedicated rclone page](rclone.html).,,,https://rclone.org/docs/ +Sha1deep, Sha1deep is a command line tool for computing and cryptographic hashes. (M5deep | sha1deep | sha256deep are a set of programs to compute MD5 | SHA-1 | SHA-256 message digests on files.) Sha1deep is used by Digital Archives Staff to identify duplicate material across digital media in a collection by comparing checksums., For MacOS: Install via Homebrew by entering ```brew install m5deep```,,,https://linux.die.net/man/1/sha1deep +Sheepshaver, Sheepshaver is an Apple Macintosh emulator. It has been used in the Digital Archives Lab to view legacy Mac files., +Siegfried, Siegfried is a command line file format identification tool. Siegfried is used in the Digital Archives Lab to identify and validate file formats. Siegfried output also includes checksums., For Windows: Download [the latest Siegfried binary](https://github.com/richardlehane/siegfried/releases/download/v1.10.1/siegfried_1-10-1_win64.zip).
For MacOS: Install via homebrew by entering ```brew install richardlehane/digipres/siegfried``` into terminal,Current installations of siegfried use the config file to store preferred command options. Enter ```sf -setconf -csv -hash sha1 -z``` in terminal to set the preferred command options., Enter ```sf PATH > MediaID``` in terminal to create a report of file formats. Enter ```sf -help``` for a complete list of sf commands.,https://www.itforarchivists.com/siegfried/ +tree, tree is command line program for recursive directory listing and has been used in the Digital Archives Lab to identify contents of media including number of folders and files, For MacOS: Install via homebrew by entering ```brew install tree``` into terminal., +VLC, VLC is an open source media player used by Digital Archives staff to view and appraise born-digital audio and video., Download [the latest release of VLC Media Player](https://www.videolan.org/vlc/)., +WSL, Windows Subsystem for Linux (WSL) is a Windows feature used to run Linux environments on Windows operating systems. +WinUAE, WinUAE is an Amiga emulator and has been used in the Digital Archives Lab to view Amiga files., \ No newline at end of file diff --git a/_data/working.csv b/_data/working.csv index 7cb312c..da9f464 100644 --- a/_data/working.csv +++ b/_data/working.csv @@ -1,5 +1,5 @@ name,desc,use -movekflux (kryofluxmove.sh),This bash script is used by Digital Archives staff to move floppy disk images transferred using kryoflux to submission information packages.,Navigate to kryofluxOutput and run ```movekflux``` in terminal. Follow the script instructions to enter collection number and move floppy disk images. -makesips (SIPdir.sh),This bash script is used by Digital Archives staff to create consecutive numbers of submission information packages for material transferred from digital carriers., Navigate to collection directory in diskImages and run ```makesips``` in terminal. Follow script instructions to select collection number type and enter collection number and submission information package range. +kryofluxmove.sh,This bash script is used by Digital Archives staff to move floppy disk images transferred using kryoflux to folders with legacy CMS media IDs of the associated digital carrier., +SIPdir.sh,This bash script is used by Digital Archives staff to create consecutive numbers of submission information packages for material transferred from digital carriers., report_ftk_extents.py,This python script transforms XML reports exported from FTK after Processing Archivists have completed bookmarking into JSON files for import into ASpace.,The script takes two arguments: ```-f``` or ```--file``` followed by the path to the XML report to be transformed and ```-o``` or ```--output``` followed by the path the destination directory for JSON output. report_HDD_extents.py,This python script collects the extents of finding aid components created by a Processing Archivist when processing work is done at a workstation with a hard drive. The script then produces an extents JSON file for import into ASpace.,The script takes one argument: ```-d``` or ```--dir``` for the finding aid components directory on a hard drive.
Run the script following the syntax: ```python3 path/to/report_ftk_extents.py -d /path/to/collection/er/directory``` \ No newline at end of file diff --git a/accessioning/accessioning.md b/accessioning/accessioning.md index dc84c82..4550655 100644 --- a/accessioning/accessioning.md +++ b/accessioning/accessioning.md @@ -5,4 +5,8 @@ nav_order: 2 has_children: true --- # Accessioning -Accessioning consists of taking legal, administrative, and physical custody of digital material to be transferred to the lab. Accessioning born-digital material includes verifying that the acquisition has been properly documented, validating the material’s completeness and fixity, and the creating an inventory in CMS to support collection management and project planning. \ No newline at end of file +Accessioning consists of taking legal, administrative, and physical custody of digital material to be transferred to the lab. Accessioning born-digital material includes verifying that the acquisition has been properly documented, validating the material’s completeness and fixity, and then creating an object record in SPEC to support collection management and project planning. + +Accessioning of digital material is usually done by Collection Management. + +Processing archivists create SPEC object records for additional carriers found during processing. \ No newline at end of file diff --git a/accessioning/digitalmediaseparation/media/SPEC_DEMO.png b/accessioning/digitalmediaseparation/media/SPEC_DEMO.png new file mode 100644 index 0000000..15baa27 Binary files /dev/null and b/accessioning/digitalmediaseparation/media/SPEC_DEMO.png differ diff --git a/accessioning/digitalmediaseparation/media/acqid_floppy.jpg b/accessioning/digitalmediaseparation/media/acqid_floppy.jpg new file mode 100644 index 0000000..63179bd Binary files /dev/null and b/accessioning/digitalmediaseparation/media/acqid_floppy.jpg differ diff --git a/accessioning/digitalmediaseparation/media/separationsheet.png b/accessioning/digitalmediaseparation/media/separationsheet.png new file mode 100644 index 0000000..6d3d447 Binary files /dev/null and b/accessioning/digitalmediaseparation/media/separationsheet.png differ diff --git a/accessioning/inventorying-digital-media-accession.md b/accessioning/inventorying-digital-media-accession.md index c418a80..b5c5b41 100644 --- a/accessioning/inventorying-digital-media-accession.md +++ b/accessioning/inventorying-digital-media-accession.md @@ -1,11 +1,8 @@ --- title: Inventorying Digital Carriers (Accessioning) -title: Inventorying Digital Carriers (Accessioning) layout: default nav_order: 1 parent: Accessioning -redirect_from: - - https://nypl.github.io/digarch/accessioning/digitalmediaseparation.html --- # Digital Carrier Separation (Accessioning) diff --git a/staging/Archivist-Workstation-Processing.md b/staging/Archivist-Workstation-Processing.md index cc11a22..7726e9c 100644 --- a/staging/Archivist-Workstation-Processing.md +++ b/staging/Archivist-Workstation-Processing.md @@ -28,8 +28,8 @@ Archivists processing electronic records without using FTK or ePADD will receive The goal of the arrangement and description phase is to approximate the process in which an archivist works with a physical collection. By using -their workstation the archivist will be able to appraise records as they are contained -on their original media, create a set of intellectual components +their workstation the archivist will be able to appraise a copy of records with the same +structure as contained on their original media, create a set of intellectual components (arrangement), summarize the logical extents (size) and date ranges of the components, and enter them into ArchivesSpace. @@ -38,13 +38,17 @@ the components, and enter them into ArchivesSpace. * Double-click the drive icon to display the drive folders in Windows Explorer. * The folders will have the following structure: * \M12345_workingfiles - * \M12345-0001 - * \metadata - * \objects + * \ACQ_12345_54321 + * \metadata + * \objects + * \data (optional) * Note the metadata folder may contain metadata created during the transfer of the files. * Note a file directory listing of the files in .csv format, if present. * Navigate to the objects folder to view the files to be arranged. + +OR + * Navigate to the data folder in the objects folder to view the files to be arranged. @@ -58,13 +62,18 @@ the components, and enter them into ArchivesSpace. * Create a new folder on the drive you received at the same folder level as \M12345_workingfiles. * Use the naming convention ```CollectionID_FAcomponents``` to create a collection folder. -* Name the folder /M12345_FAcomponents, for example. -* Create a folder for each FA Component using ```CollectionID_ER_1```, ```CollectionID_ER_2```. -* Create an objects folder within each ER_# folder. ```CollectionID_ER_1\objects``` -* Move the files into the new FA Component objects folders as your process the files. -* Calculate the size of each FA Component by right clicking on the FA Component folder. -* Select Properties from the drop down menu. -* Note the file count and byte extent, when you are done processing. +* Name the folder \M12345_FAcomponents, for example. +* Create a folder for each heading represented in the electronic records. ```Correspondence``` +* Create a folder for each FA Component using ```ER # Title, dates```. +* Create an objects folder within each ER_# folder. ```ER 1 Files, 2012\objects``` +* Move the files into the new FA Component objects folders as your process the files. + +### Finishing up +* Email [Digital Archives](mailto:digitalarchives@nypl.org) when you are finished processing. +* Digital Archives will arrange to pick up the hard drive. +* Digital Archives will email you a JSON file for import into ASpace. +* Import the JSON file received from Digital Archives into ASpace. +* Email [Digital Archives](mailto:digitalarchives@nypl.org) when your arrangement is approved. ### Entering your collection in ArchivesSpace After arrangement is complete Processing Archivists notify Digital Archives and deliver the external hard drive containing FA Components to Digital Archives for review. Digital Archives staff then create a JSON file containing extents for import into ASpace. For detailed instruction on importing the extents JSON to ASpace review the [Importing FTK ERs in ArchiveSpace](https://docs.google.com/document/d/1BVMaDOzdcPFIht5yN5V16zmsnjMTwG_3e73HPTdVzSg/edit) guide. diff --git a/staging/FTK-Export.md b/staging/FTK-Export.md index c19894c..811253b 100644 --- a/staging/FTK-Export.md +++ b/staging/FTK-Export.md @@ -16,7 +16,18 @@ When a finding aid has been approved and the Bookmarks have been reviewed the co * Open the case for the collection and navigate to the Bookmark tab. * Note how many finding aid components you have in the Bookmarks. -* Windows subsystem for Linux and enter the following commands to build your folders: +* Open the WSL terminal and navigate to the mount point directory by entering ```cd /mnt``` + +* Any mounted drives should be accessible from the mount point directory. Drives you'll see in /mnt include: + * d - Sata Drive Bay + * f - Storage for FTK + * h - Open FTK cases + * i - Codemeter access key + * y - DigArchDiskStation + +* If the Y:\ drive is not visible in /mnt or /mnt/y appears to be empty then re-mount: + * Change to the top level directory by entering ```cd /``` + * Enter the command ```sudo mount -t drvfs Y: /mnt/y``` ```$ cd /mnt/y/Staging/faComponents``` diff --git a/staging/Loading-Archivist-Workstation.md b/staging/Loading-Archivist-Workstation.md index f16fd7b..be46065 100644 --- a/staging/Loading-Archivist-Workstation.md +++ b/staging/Loading-Archivist-Workstation.md @@ -17,7 +17,7 @@ Archivists processing electronic records without using FTK or ePADD will receive ## Confirming Archivist Workstation Processing -* Open CMS and navigate to the collection's media log. +* Open SPEC and navigate to the collection's object records. * Check that all the media in the collection has been transferred as files. * Consult with Digital Archives staff if there are disk images present or transfer is incomplete. diff --git a/staging/Navigating-FTK.md b/staging/Navigating-FTK.md index b1edda0..c127b48 100644 --- a/staging/Navigating-FTK.md +++ b/staging/Navigating-FTK.md @@ -46,9 +46,9 @@ the File List. - Sort by Name to identify similar file names. - Sort by Path to see the original arrangement on the disk images - including which files were in a which folder. + including which files were in which folder. - - Sort Modified date to see date ranges. + - Sort by Modified date to see date ranges. - Use the Category field instead of the Ext field when identifying formats if FTK does not recognize the correct extension. @@ -100,10 +100,10 @@ In the normal course of events you shouldn’t see the following formats when yo ![a computer icon of a file hierarchy tree with lines between squares representing files followed by the word unrecognized](FTK-Introduction/media/unrecognized%20FTK.PNG) * DVDs * VIDEO_TS -* Seeing the above formats means something has gone awry +* Seeing the above formats means something has gone awry. ## Unviewable Formats -You should also stop and talk to us before proceeding if you see unviewable files that you think are in scope. There are many files that FTK designates as unknown. For many of these files you are still able to view the complete contents of the file in natural or filtered view. Those files can be arranged as usual. If you encounter files where you cannot see the complete contents in any view but think the files are in scope please discuss these files with digital archives staff. In the case of image files with no view in FTK like Canon RAW we can stage these images so you can view them with software outside of FTK. Another common example is Quark files. You can sometimes see limited text for Quark files in filtered view but you can’t view any images. Since we can only partially review these files we need to be aware of them if they are being arranged in packages for access. +You should also stop and talk to us before proceeding if you see unviewable files that you think are in scope. There are many files that FTK designates as unknown. For many of these files you are still able to view the complete contents of the file in natural or filtered view. Those files can be arranged as usual. If you encounter files where you cannot see the complete contents in any view but think the files are in scope please discuss these files with Digital Archives staff. In the case of image files with no view in FTK like Canon RAW we can stage these images so you can view them with software outside of FTK. Another common example is Quark files. You can sometimes see limited text for Quark files in filtered view but you can’t view any images. Since we can only partially review these files we need to be aware of them if they are being arranged in packages for access. ## Stop and discuss with Digital Archives staff if you see… * Unknown formats with no Natural view and a limited Filtered view that are in scope @@ -118,9 +118,9 @@ There are some file types that either can’t be exported from FTK or don’t pr * Folders ![a computer icon of a yellow file folder](FTK-Introduction/media/ftkfolders.png) * Slack Space - ![computer icon of a white piece of paper with a yellow border on the bottom and red arrow pointing left](FTK-Introduction/media/ftkslackspace.png) + ![a computer icon of a white piece of paper with a yellow border on the bottom and red arrow pointing left](FTK-Introduction/media/ftkslackspace.png) * Deleted Files - ![computer icon of a white piece of paper with a red X over it](FTK-Introduction/media/ftkdeleted.png) + ![a computer icon of a white piece of paper with a red X over it](FTK-Introduction/media/ftkdeleted.png) * File systems ![ a computer icon of a file hierarchy tree with lines between squares representing files](FTK-Introduction/media/ftkfilesystem.png) * .DS_STORE @@ -137,8 +137,8 @@ There are some file types that either can’t be exported from FTK or don’t pr * Please check to see if any files have the label "Potentially Sensitive" and review to see if they contain sensitive material. - * When digital archives staff uploaded the collection's files into - FTK, they ran a search for Pii. If there were any hits, these + * When Digital Archives staff uploaded the collection's files into + FTK, they ran a search for PII. If there were any hits, these files were labeled "Potentially Sensitive". ## The Overview Tab @@ -234,4 +234,4 @@ numbers, social security numbers, credit card numbers, etc.). ![](dapi/media/image25.png) * Click the Search button. * Hits will display in the results window. -* Highlight hits to populate the File List. \ No newline at end of file +* Highlight hits to populate the File List. diff --git a/staging/Overview.md b/staging/Overview.md index 912e31f..5f98ba0 100644 --- a/staging/Overview.md +++ b/staging/Overview.md @@ -48,8 +48,6 @@ Send email to [digitalarchives@nypl.org](mailto:digitalarchives@nypl.org) when * [Reserve your FTK session](../using/using-lab-equipment#reserving-a-workstation-session) at least 24 hours in advance. * Click the FTK icon to open FTK and begin your session. * Select your collection from the bar on the left hand side of the screen. -**Information in the media log can assist with processing.** -* Open CMS and navigate to the collection's media log when appropriate. ## Appraisal Spend some time evaluating the electronic records before you begin bookmarking. You'll be using the same strategies you use for paper records for the most part but FTK has appraisal tools you should use as well. \ No newline at end of file diff --git a/tools/programs.md b/tools/programs.md new file mode 100644 index 0000000..292bd0a --- /dev/null +++ b/tools/programs.md @@ -0,0 +1,37 @@ +--- +title: Programs +layout: default +nav_order: 6 +parent: Software +grand_parent: Tools +has_children: False +--- + +# Programs: + +Programs are defined as any non-internally developed out-of-the-box software used in Digital Archives workflows. Many programs used by Digital Archives staff can be installed via Homebrew or by downloading installation files from the associated website. Programs requiring more intensive installation and configuration instructions have been given dedicated pages. + +{% for tool in site.data.program %} + +## {{tool.name}}: +{{tool.desc}} + +{% if tool.docs %} +[{{tool.name}} Official Documentation]({{tool.docs}}) +{% endif %} + +{% if tool.inst%} +### Installing {{tool.name}}: +{{tool.inst}} +{% endif %} + +{% if tool.conf%} +### Configuring {{tool.name}}: +{{tool.conf}} +{% endif %} + +{% if tool.use%} +### Using {{tool.name}}: +{{tool.use}} +{% endif %} +{% endfor %} \ No newline at end of file diff --git a/tools/rclone.md b/tools/rclone.md index a5d3d61..6bd9bf2 100644 --- a/tools/rclone.md +++ b/tools/rclone.md @@ -1,12 +1,13 @@ --- title: Rclone layout: default -nav_order: 1 +nav_order: 8 parent: Software grand_parent: Tools +has_children: False --- -# Using Rclone: +# Rclone: [*Rclone*](https://rclone.org/) is a command line tool for managing files on cloud storage, and is one tool used by Digital Archives for moving files between Digital Archives workstations, Google Shared Drives for Divisions, and long term storage. This page lists instructions for installing and configuring rclone. ## Installing Rclone: @@ -38,9 +39,9 @@ Once installed, you must configure Rclone which involves setting the remote loca - rclone will provide a list of scopes of access ranging from full access to all files to read-only access to file metadata. For our purposes enter the number corresponding to full access to all files. -- Rclone asks for a root folder id, this is optional and you can enter to select enter to choose default. +- Rclone asks for a root folder id, this is optional and you can select enter to choose default. -- Rclone asks for service account credentials, this is optional and you can enter to select enter to choose default. +- Rclone asks for service account credentials, this is optional and you can select enter to choose default. - Rclone asks you to edit advanced configuration, select “n” for No. Follow up prompt asks if you would like to use auto configuration, enter “y” for Yes. @@ -53,3 +54,13 @@ Once installed, you must configure Rclone which involves setting the remote loca - Rclone then asks if the configuration you’ve been doing is okay. Enter y for Yes and configuration is complete! To see all currently configured remotes (rclone term for any cloud storage locations) enter ```rclone listremotes``` into terminal. + +## Using Rclone with Google Drive: +When using rclone with google drive accessing files in "My Drive" and Shared Drives" differs from accessing any files in "Shared with me". To access the "Shared with me" subdirectory: + +* Create an rclone remote location for the drive the collection has been shared with. + +* Use the following syntax for accessing anything in the the “Shared With Me” subdirectory: + +```rclone copyto remote:shared/with/me/remote/directory —name-of-drive-shared-with-me /path/to/destination``` + diff --git a/tools/software.md b/tools/software.md index babbd6a..da0f46e 100644 --- a/tools/software.md +++ b/tools/software.md @@ -1,300 +1,16 @@ --- title: Software layout: default -nav_order: 3 +nav_order: 5 parent: Tools has_children: True --- - - -# Software -{: .no_toc } - ## Table of contents {: .no_toc .text-delta } -1. TOC -{:toc} - -## Types of Software -The following is a list of software used in born-digital workflows at NYPL. -### 7-zip -7-zip is used in the lab to compress evidence and cases generated in Forensic Toolkit. It is also occasionally used to unpack collection material that is not supported by FTK. -#### Install 7-zip binary -[Download 7-zip](https://www.7-zip.org/){:target="_blank"} - -### AWS -Amazon Web Services features S3 and Glacier are used to store copies of born-digital material both before and after the collection has been processed by archivists. - -Using the AWS dashboard requires a login. - -### Bagit.py -Bagit.py is used to transfer born-digital material to storage. Bagit creates file manifests that help to ensure transfers are complete and free of errors. - -#### Installation -Check python version -``` python --version ``` -or -``` python3 --version ``` -then install -``` pip install bagit ``` -or -``` pip3 install bagit ``` - -### BitCurator -BitCurator is a Linux distribution with a suite of open source digital forensics and data analysis tools used in the lab as both a virtual machine and as bootable OS alongside Windows. BitCurator is used for transfer and metadata extraction, especially for formats unrecognized in Windows and Mac environments. - -#### Installation -Download software -[BitCurator VM or Live CD](https://github.com/BitCurator/bitcurator-distro/wiki/Releases){:target="_blank"} - -[Current release of VirtualBox](https://www.virtualbox.org/wiki/Downloads){:target="_blank"} - -The VirtualBox Extension Pack must be installed on the host for shared folder support, USB 2.0/3.0 support, and support for some NVMe SSDs. Once you’ve installed VirtualBox, use the link below the main download on VirtualBox download site to install the Extension Pack on your host. - -Unpack the BitCurator Virtual Machine. - -Once you’ve installed VirtualBox and the VirtualBox extension pack, start up VirtualBox. - -Add Virtual Machine -From the menu bar, select the menu item “Machine -> Add…”, and navigate to the folder containing .vbox file that you extracted. Choose that file, and the Virtual Machine should appear in the list within the manager. - -### dBpoweramp -dBpoweramp is used in conjunction with Iromlab and the Nimbie to transfer born-digital material from optical media to storage. -### dd -The command line utility dd is used in the lab to create disk images of optical media. In most cases transfers are made of optical media instead of disk images. -#### Use -In the terminal type -``` dd if=/dev/sr0 of=/cygdrive/z/ingest/diskImages/CollID/MediaID/objects/MediaID.iso ``` -Hit return -Once complete the terminal will display bytes copied and return to a blank prompt. - -### Disk Utility -On MacOS Disk Utility is used in the lab to confirm that removable media is properly mounted. - -### DiskPart -On Windows DiskPart command line utility is used in the lab to confirm that removable media is properly mounted. -#### Use -Type -``` list disk ``` -or -``` list volume ``` -to confirm media is mounted. - -### disktype -Disktype is used on MacOS in the lab to extract filesystem metadata such as sector format for floppy disks. - -#### Installation -``` brew install disktype ``` -#### Use - ``` cd ``` - ``` diskype ``` - -### Filemaker Pro 17 -Filemaker is used by Preservation and Collection Services as a collection management system. Born-digital media is inventoried in Filemaker and PCP tracks projects through Filemaker. -#### Installation -Installing Filemaker requires license keys. Make sure you have the proper keys and permissions before installing. -#### Use - -### Forensic Toolkit -Forensic Toolkit is used in the lab to appraise, and arrange born-digital material. -#### Installation -Forensic Toolkit is licensed software and requires a dongle to open. The software can be downloaded from [AccessData](https://accessdata.com/product-download){:target="_blank"} -#### Use - -### gtar -Gtar is used on a Mac in the lab to compress KryoFlux stream files. It is called through the command line with the bash alias 'compress'. -#### Installation -``` brew install gnu-tar ``` -### Iromlab -Iromlab is used in conjunction with dBpoweramp and the Nimbie to transfer born-digital material from optical media to storage. -### Java -Java is required for tools like DROID and Bagger. -#### Use -Try commands -``` which java ``` -and -``` java -version ``` -if you are encountering java errors. You may need to put the path of the correct version of java in your bash profile. -In Windows you can use the search from the start menu to check the java version. Type 'about java' into search. Then click the About Java icon result. -### openssl -openssl is used in the lab to SSH between computers. SSH is used to run workflow scripts located on ARCHV Mac. -### Oracle VM VirtualBox -VirtualBox is used in the lab to run the BitCurator environment as a virtual machine. -#### Installation -Current release of [VirtualBox](https://www.virtualbox.org/wiki/Downloads){:target="_blank"} - -The VirtualBox Extension Pack must be installed on the host for shared folder support, USB 2.0/3.0 support, and support for some NVMe SSDs. Once you’ve installed VirtualBox, use the link below the main download on VirtualBox download site to install the Extension Pack on your host. -### pip -Pip is used in the lab to install and upgrade Python modules like bagit.py. -#### Use -Pip should be installed with python3. Call pip with -``` pip``` or ``` pip3 ``` -depending on your installation. -Check python version -``` python --version ``` -or -``` python3 --version ``` -### Python3 -Python3 is used in the lab to run a a variety of scripts, including bagit.py. -#### Installation -Windows: -Check python version -``` python --version ``` -Installers are available at [python.org](https://docs.python.org/3.8/using/windows.html#windows-full){:target="_blank"} - -Mac: -Check python version -``` python3 --version ``` -Replace the number below with the version you wish to install. -``` brew install python@3.8 ``` -BitCurator: -Check python version -``` python3 --version ``` -Replace the number below with the version you wish to install. -``` sudo apt-get install python3.8 ``` -### QuickTime Player -QuickTime is used in the lab to appraise Macintosh formatted born-digital video. - -### rclone -Rclone is used in the lab to transfer born-digital material between for Digital Archives workstations, Google Shared Drives for Divisions, and long term storage. -#### Installation -Windows: -[Download and install binary](https://rclone.org/downloads/){:target="_blank"} - -Mac: -```brew install rclone``` -### rsync -Rsync is used in the lab to transfer born-digital material when bagit.py is unable to complete a transfer. Rsync is used when transferring from a computer backup or a drive with a large number of system files. -#### Installation -Mac: -``` brew install rsync``` -### Siegfried -Siegfried is used in the lab to identify and validate file formats. Siegfried output also includes checksums, useful when rsync is used for transfer instead of bagit.py. -#### Installation -Windows: -[Download binary](https://github.com/richardlehane/siegfried/releases/download/v1.8.0/siegfried_1-8-0_win64.zip){:target="_blank"} -Mac: -```brew install richardlehane/digipres/siegfried``` - -#### Use -Current installations of siegfried use the config file to store preferred command options. -``` sf -setconf -csv -hash sha1 -z ``` -sets the preferred command options. -Find a complete list of sf commands with -``` sf -help ``` -### Sha1deep -Sha1deep is used in the lab to identify duplicate material across digital media in a collection. It is usually used in bash scripts. -#### Installation -Mac: -``` brew install m5deep ``` -#### Use -Here is an example of how to use sha1deep to transfer only unique files from media. -Write checksums of transfer volume to file -``` sha1deep -r [directory or volume] >> collnumber.sha1 ``` -Write path of unique files per directory or volume to file with mediaid -``` sha1deep -rx [path to checksums file] [directory or volume] >> [path to checksums for media] ``` -Remove ds_store from unique file list -```sed '/.DS_Store/d' [path to file list] > [file list]``` -change terminal path out of home directory -```cd ../../``` -Navigate to the media being transferred to avoid /Volumes/HD/folders being copied within media-id directory. -Make sure spaces are not escaped in paths. “ “ not “\ “ -Transfer unique files -```rsync -aPF --files-from=[path to file list] . [path to media]``` -### tree -Tree is used in bash scripts in the lab to identify contents of media including number of folders and files. -#### Installation -Mac: -```brew install tree``` -### VLC -VLC multimedia player is used in the lab to view and appraise born digital audio and video. -#### Installation -https://www.videolan.org/vlc/index.html -## Scripts -The following is list of scripts used in born-digital workflows at NYPL. The scripts are located in the [digarch_scripts](https://github.com/NYPL/digarch_scripts/blob/main/){:target="_blank"} Github repository. A link to each script is provided after the description. -### ft.sh -This script is used in the lab to create bagit transfers and Siegfried csvs for born-digital media. -[ft.sh ](https://github.com/NYPL/digarch_scripts/blob/main/Mac/ft.sh){:target="_blank"} -### kryofluxmove.sh -This script is used in the lab to move KryoFlux disk images to folders with the MediaIDs of the disks. -[kryofluxmove.sh](https://github.com/NYPL/digarch_scripts/blob/main/Mac/kryofluxmove.sh){:target="_blank"} -### metadata.sh -This script is used in the lab to move metadata from disk images to folders with the MediaIDs of the disks. -[metadata.sh](https://github.com/NYPL/digarch_scripts/blob/main/Mac/metadata.sh){:target="_blank"} -### Brunnhilde with disktype -This script is used in BitCurator to identify disk image type and pass the type to Brunnhilde to generate the correct output. -[Brunnhilde with disktype](https://github.com/NYPL/digarch_scripts/blob/main/BitCurator/brudt.sh){:target="_blank"} -### makesips script -This script is used to create a consecutive number of submission information packages for material from digital media. -[makesips script](https://github.com/NYPL/digarch_scripts/blob/main/Mac/SIPdir.sh){:target="_blank"} -Alternatively, ``mkdir`` command can be used to create SIPs. This works when SIPs aren't consecutively numbered. 0001 to 0009 require a different line from 0010 on. -```mkdir -p CollID/Media-000{1..9}/{metadata/submissionDocumentation,objects}``` -```mkdir -p CollID/Media-00{10..99}/{metadata/submissionDocumentation,objects}``` -```mkdir -p CollID/Media-000{1,5,7,9}/{metadata/submissionDocumentation,objects}``` -### moveimages script -This script is used in the lab to move a disk image into the objects folder of a SIP. -[moveimages script](https://github.com/NYPL/digarch_scripts/blob/main/Mac/kryofluxmove.sh){:target="_blank"} -### movemetadata script -This script is used in the lab to move metadata into the metadata folder of SIP. -[movemetadata script](https://github.com/NYPL/digarch_scripts/blob/main/Mac/metadata.sh){:target="_blank"} -### movephotograph script -This script is used in the lab to copy JPEGS of media from the photographs directory on the staging drive to the submissionDocumentation directory in the MediaID directory of SIPs. -[movephotograph script](https://github.com/NYPL/digarch_scripts/blob/main/Mac/movephotograph.sh){:target="_blank"} -### FACTools -This script is used in the lab to repackage Finding Aid Component packages. -[FACTools](https://github.com/NYPL/digarch_scripts/blob/main/Mac/qctools.sh){:target="_blank"} -### report ftk extents script -This script transforms XML reports exported from FTK after Processing Archivists have completed bookmarking into JSON files for import into ASpace. The script takes two arguments. -* ```-f``` or ```--file```: this argument accepts the path to the XML report to be transformed. -* ```-o``` or ```--output```: this arguments accepts the path to the destination directory for the transformed JSON. - -Run the script following the syntax: ```python3 path/to/report_ftk_extents.py -f /path/to/xml/report -o /path/to/json/destination/directory``` - -### report HDD extents script -This script collects the extents of finding aid components created by a Processing Archivist when processing work is done at a workstation with a hard drive. The script then produces an extents JSON file for import into ASpace. - -The script takes one argument: ```-d``` or ```--dir``` for the finding aid components directory on a hard drive. - -Run the script following the syntax: -```python3 path/to/report_ftk_extents.py -d /path/to/collection/er/directory``` +# Software +Digital Archives utilizes several software throughout our workflows. This page acts as a departmental glossary for these software and includes installation, detailed use instructions and links to official documentation whenever possible. Shorter contextual instructions for these software can be found on the associated workflow pages, consider this page our definitive guides. -### iterative scripts -Scripts are created per collection using a directory listing as input in a while loop. This allows previous move scripts to run per collection rather than one MediaID. -### CMS metadata import -This script is used in the lab to import metadata from transfers to the FileMaker Collection Management System. Currently, output from the Nimbie (Iromlab) is used. -## Previous and Occasional Software -The following is a list of software used in previous workflows. These tools can be used when transfer fails with current tools. -### Emailchemy -Emailchemy has been used in the lab to migrate email formats. -### ePADD -ePADD has been used in the lab to appraise and arrange email. -### DROID -DROID has been used in the lab to identify and validate file formats. -### MediaInfo -MediaInfo has been used in the lab to extract metadata from born-digital video. -### FTK Imager -FTK Imager has been used in the lab to image media and extract files from media. -### IsoBuster -IsoBuster has been used in the lab to image optical media and extract files from media. -### Bagger -Bagger has been used in the lab to transfer born-digital material. -## Legacy Format Appraisal -The following is a list of software used in the lab to view legacy formats not supported by FTK. -### WinUAE -WnUAE has been used in the lab to view Amiga files. -### apple3rtr -Apple3rtr has been used in the lab to view Apple II files. -### miniVmac -MiniVmac has been used in the lab to view legacy Mac files. -### DosBox -DosBox has been used in the to view DOS executables and word processing files. -### QuickView Plus -QuickView Plus has been used in the lab to view a range of legacy formats like word processing and photo formats. -### CiderPress -CiderPress has been used in the lab to view Apple file formats. -### HFS Explorer -HFS Explorer has been used in the lab to view files on HFS filesystems. -### Sheepshaver -Sheepshaver has been used in the lab to view legacy Mac files. \ No newline at end of file +* Navigate to [Programs](programs.html) for a glossary of out of the box tools used by Digital Archives staff. +* Navigate to [Working Scripts](working-scripts.html) for information on any scripts written by Digital Archives staff specific to our workflows. \ No newline at end of file diff --git a/transfers/cloud-file-transfer.md b/transfers/cloud-file-transfer.md new file mode 100644 index 0000000..888c909 --- /dev/null +++ b/transfers/cloud-file-transfer.md @@ -0,0 +1,79 @@ +--- +title: Cloud File Transfers +layout: default +nav_order: 8 +parent: Transfers +--- + +# Cloud File Transfers +{: .no_toc } + +## Table of contents +{: .no_toc .text-delta } + +1. TOC +{:toc} + +# Introduction +Born-digital collection material can be acquired through file transfer from cloud based storage locations. This page describes Digital Archives workflow for preparing for a cloud based file transfer, creating a destination package using ft_packager.py and transferring material using rclone. The instructions detailed on this page are specific to transfers from google drive, but are adaptable to other cloud storage contexts. + +# Before File Transfer +Before beginning a file transfer confirmation with Donors and Collections Management about what material is being acquired is essential. Transfer from cloud storage will often require the sharing of permissions or access to the storage location for the duration of the transfer, this should be communicated with the Donor. Digital Archives does not acquire or retain system files, hidden files, deleted files or files containing personally identifiable information. See our [Acquiring Born Digital Material](/sitevisits/acquiring-born-digital.html) page for more information. + +* Acquire link to cloud storage location from donor. Have google drive location shared with Digital Archives staff by donor. + +# Workstation Preparation +Born-digital material held in cloud storage is first transferred with rclone to a temporary working folder on a Digital Archives Lab workstation or a RAID connected to a Digital Archives Lab workstation. + +* Connect RAID to Lab Workstation or Login to Lab Workstation +* Open Terminal +* Create temporary working folder on RAID or on Lab Workstation by entering ```mkdir path/to/working/folder``` + * Note: Folder should be named following file transfer naming convention, ```ACQ_four-digit-acquisition-id_six-digit-spec-id```. Acquisiton, Collection and SPEC ID information can be found in the associated SPEC records. +* Navigate into temporary work folder by entering ```cd path/to/working/folder``` + + +# File transfer with Rclone +Rclone is a command line program for managing files on cloud storage. Using rclone for filetransfers requires configuring storage locations as saved remote locations. For detailed rclone installation and configuration instructions visit the dedicated [rclone](https://nypl.github.io/digarch/tools/rclone.html) page. + +* Confirm working folder created in the Workstation Preparation section is the current working directory in terminal. + +* Confirm you have access to the intended folder +```rclone ls remote:path/to/source``` + * For Google Shared Folders, try ```remote: --drive-root-folder-id ###``` where ### is the string at the end of the Google Drive Folder URL. +For example, `1yOaxTcgPl5zNwYQP2_k7SOW59l4DgXgd` from `https://drive.google.com/drive/folders/1yOaxTcgPl5zNwYQP2_k7SOW59l4DgXgd` + +* Transfer md5 checksum manifest for materials by entering +```rclone md5sum --exclude ".*" -P remote:path/to/source --output-file rclone-md5.txt``` + * `--exclude ".*"` excludes any files with a name beginning with ".", this is to exclude hidden system files + * `-P` visually tracks progress + * `--output-file rclone-md5.txt` saves checksums to a text file. + +* Transfer the materials payload by entering +```rclone copyto --exclude ".*" -P --log-level INFO --log-file=path/to/working/folder/rclone.log remote:path/to/source payload``` + * `--log-level INFO --log-file=rclone.log` saves the logs to defined file path + * `payload` is a directory that will be created by `rclone` to hold the files + +# Create Cloud File Transfer Package with package_cloud.py +After born-digital material, transfer log, and checksum manifest have been transferred to a temporary working folder, Digital Archives Staff repackage the files to meet file transfer specifications using package_cloud.py. Digital Archives specifications for file transfers described below: + +``` +/ACQ_four-digit-acquisition-id +└── /ACQ_four-digit-acquisition-id_six-digit-spec-id + ├── metadata + └── objects + +/ACQ_1234 +└── /ACQ_1234_123456 + ├── metadata + └── objects +``` + +The package_cloud.py script requires five inputs for repackaging: +* ```--payload```: The path to the payload folder created in the working directory, contains the actual born_digital material. +* ```md5```: The path to the rclone md5 text file generated during transfer. +* ```log```: The path to the rclone log generated during transfer. +* ```dest```: The path to the destination directory for package +* ```id```: The media id for the transferred material. + +An example run of package_cloud.py: +```python3 package_cloud.py --payload /path/to/working/payload --log /path/to/working/rclone.log --md5 /path/to/working/rclone-md5.txt --dest /DigArchDiskStation/Staging/ingest/filetransfers --id ACQ_0000_000111``` diff --git a/transfers/file-transfers.md b/transfers/file-transfers.md index f0e047b..df1a4c3 100644 --- a/transfers/file-transfers.md +++ b/transfers/file-transfers.md @@ -56,7 +56,7 @@ Note: If you search for WSL you may come across the WSL app with a penguin icon, * Change to fileTransfers directory by entering ```cd /mnt/y/Staging/ingest/fileTransfers``` -* Run [makesips script](https://nypl.github.io/digarch/tools/working-scripts.html#makesips-sipdirsh){:target="_blank"} to create a consecutive number of submission information packages for material from digital media. +* Run [makesips script](https://nypl.github.io/digarch/tools/software.html#makesips-script){:target="_blank"} to create a consecutive number of submission information packages for material from digital media. or @@ -85,7 +85,7 @@ On Mac: * Change into the acquisition directory. ```$ cd ACQ_acqID``` -* Run [makesips script](https://nypl.github.io/digarch/tools/working-scripts.html#makesips-sipdirsh){:target="_blank"} to create a consecutive number of submission information packages for material from digital media. +* Run [makesips script](https://nypl.github.io/digarch/tools/software.html#makesips-script){:target="_blank"} to create a consecutive number of submission information packages for material from digital media. Or diff --git a/transfers/floppy-disk-imaging.md b/transfers/floppy-disk-imaging.md index 01f6c2d..b8028d6 100644 --- a/transfers/floppy-disk-imaging.md +++ b/transfers/floppy-disk-imaging.md @@ -5,128 +5,199 @@ nav_order: 5 parent: Transfers --- -## Floppy Disks +# Floppy Disks +**Before a media object can be imaged it first must be recorded SPEC.** -**Before a media object can be imaged it first must be recorded in the collection’s media log in CMS. See [Verifying inventory in Media Log](/digarch/transfers/verify-inventory.html){:target="_blank"} for instructions.** +## Getting set up -### Image disks - -#### Getting set up - -* Connect the KryoFlux. - * Ensure the floppy drive is correctly connected to the Kryoflux. - * Attach the KryoFlux USB cable from the forensic workstation to the Kryoflux board. See a green power light glow on the Kryoflux board. - * Attach the AC power cable to the power supply. See the green LED on the power supply light up. Listen for a faint click from the disk drive. +### Connect the KryoFlux +* Ensure the floppy drive is correctly connected to the Kryoflux. +* Attach the KryoFlux USB cable from the forensic workstation to the Kryoflux board. +* See a green power light glow on the Kryoflux board. +* Attach the AC power cable to the power supply. +* See the green LED on the power supply light up. +* Listen for a faint click from the disk drive. **See the [KryoFlux page in the Tools section for more](https://nypl.github.io/digarch/tools/kryoflux).** -* Open CMS and locate the collection that you will be working with. - * Navigate to the electronic records view through the collection management screen. - * Click on the media number that you are going to image from the “other objects” list. (Check in with Digital Preservation staff if the media object is not listed in the CMS.) - -* Calibrate the KryoFlux. - * Open the KryoFlux software from the desktop "kryoflux-ui.jar". - * Click on the drive menu and ensure that ‘Drive 0’ is selected. - * Select ‘calibrate’ from the drive menu, then ‘yes’ from the pop-up window. When the calibration is complete you should see the message ‘calibration is successful, the maximum number of tracks available is: 83’. - * Calibrate only once when you begin a session, unless calibration fails. - -* Follow these steps if the KryoFlux is unable to communicate with the drive. - * Close the application. - * Disconnect the power source from the drive. - * Disconnect the USB cable from the KryoFlux board. - * Reconnect the USB cable to the KryoFlux board. - * Reconnect the power source to the drive. - * Try connecting a different drive if there is still no communication after the the previous steps. - -#### Imaging +### Calibrate the KryoFlux +* Open the KryoFlux software from the desktop "kryoflux-ui.jar". +* Click on the drive menu and ensure that ‘Drive 0’ is selected. +* Select ‘calibrate’ from the drive menu, then ‘yes’ from the pop-up window. +* When the calibration is complete you should see the message + `calibration is successful, the maximum number of tracks available is: 83` +* Calibrate only once when you begin a session, unless calibration fails. + +### Resolve communication with the drive. +Follow these steps if the KryoFlux is unable to communicate with the drive. +* Close the application. +* Disconnect the power source from the drive. +* Disconnect the USB cable from the KryoFlux board. +* Reconnect the USB cable to the KryoFlux board. +* Reconnect the power source to the drive. +* Try connecting a different drive if there is still no communication after the the previous steps. + +### Imaging * Insert the floppy disk into the drive. -* Enter the media object ID (for example, M11111-4444) in the "Enter name ..." text field in the Kryoflux imaging software. +* Enter the media object ID (for example, ACQ_11111_4444) in the "Enter name ..." text field in the Kryoflux imaging software. * Determine the filesystem format of the disk. - * “MFM sector image” is usually the correct format for PC disks, “Apple DOS 400/800k” is usually the correct format for Apple Macintosh disks. - * Test the disk if you're unsure about the format. Select a format in the dropdown underneath the name field and click "start" to begin imaging the disk. The sector grid displays color as the disk is imaged. + * “MFM sector image” is usually the correct format for PC disks + * “Apple DOS 400/800k” is usually the correct format for Apple Macintosh disks. + * Test the disk if you're unsure about the format. + * Select a format in the dropdown underneath the name field and select "start" to begin imaging the disk. + * The sector grid displays color as the disk is imaged. * Grey indicates the format is likely incorrect. Try another format. * Green, orange, or yellow indicates the correct sector format has been determined. * Red sectors indicate either the disk or the drive is damaged. -**See the section on [Problem disks](https://nypl.github.io/digarch/transfers/digital-forensic-imaging.html#problem-disks) for more on damaged media.** +* Select “Multiple” from the drop-down below the "Enter name ..." text field when you have determined the correct sector format. +* Select “KryoFlux stream image” and, while holding down the control key, select the correct sector format from the dropdown. +* Select ‘ok’ to close the popup. +* Click the start button to begin the imaging process. + The KryoFlux will now create a folder with the name of the id of the media that contains the stream image files and a sector image (again with the name of the media plus an “.001” file extension). Both the stream directory and the sector image file will be written to the kryofluxOutput directory on Staging. -* Select “Multiple” from the drop-down below the "Enter name ..." text field when you have determined the correct sector format. Select “KryoFlux stream image” and, while holding down the control key, select the correct sector format from the dropdown. Click ‘ok’ to close the popup. -* Click the start button to begin the imaging process. The KryoFlux will now create a folder with the name of the id of the media that contains the stream image files and a sector image (again with the name of the media plus an “.001” file extension). Both the stream directory and the sector image file will be written to the kryofluxOutput directory on Staging. +### Tracking +* Navigate to the [Tracking](https://drive.google.com/drive/folders/1tv4nr9Nq_c8wkqPpz_eQX7NKRRrlEisp?usp=share_link){:target="_blank"} folder in Google Drive. +* Find the spreadsheet for the acquisition that you will be working with. +* Copy [Tracking_TEMPLATE](https://docs.google.com/spreadsheets/d/1TwWMsrCf2hf5LzdtA6EG-2wcgFW_Uz750x-PZtFop90/edit?usp=sharing) to create a spreadsheet if one doesn't exist. +* Name the spreadsheet the acquisition ID of the acquisition that you will be working with. +* Check the media ID for the disk you are working with (for example, ACQ_1234_123456). +* Media IDs follow the naming convention ACQ_acqID_specObjectID. +* Enter acqID in the ref_acq_id field. +* Enter specObjectID in the object_id field. -#### Completing the imaging process +| ref_acq_id | object_id | +| -- | -- | +| 1234 | 123456 | -* Put the media back in the collection’s box and move or remove the pink “To image” flag as necessary if you are working on a large collection. Move the media to the “Small collections complete” box if you are working on a small collection without a box. -* Enter the following information about the image in CMS: +* Enter item in the type field. +* Enter digital carrier in the format_1 field. +* Enter floppy disk in the format_2 field. +* Enter floppy disk size in the format_3 field. (for example, 3.5 inch floppy) -Under ‘Image Information’ -interface: Kryoflux -imaging software: Kryoflux Imager -image successful: Yes | No -interpret successful: Yes | Yes w/Errors | No -* Use ‘yes w/errors’ if a few red sectors were observed with the KryoFlux software. +| type | format_1 | format_2 | format_3 | +| ------- | -------- |-------- | -------- | +| item | digital carrier | floppy disk | 3.5 inch floppy | -sector format: select the sector format from the dropdown -image format: raw -imaged by: your name +* Enter failed in the notes.transfer field if imaging fails. -* Select the current date under ‘Progress’ for the ‘Imaging done” field. +| notes.transfer | +| -- | +| failed | +* Enter the filesystem format in the notes.transfer field if that format isn't recognized by FTK. +* Amiga, ProDOS, CPM are formats not recognized in FTK. -### Metadata +| notes.transfer | +| -- | +| Amiga | -Deprecated -{: .label .label-red } +* Enter bad sectors in the notes.transfer field if a third or more of the sector blocks are red. -* Open Cygwin and enter the following commands: -* Connect to ARCHV Mac. -```$ ssh archv``` -* Change to the kryofluxOutput directory. -```$ kryofluxoutput``` -* Run disktype on the disk image to get the file system metadata. -```$ disktype M1111-0004.001``` -* Use the disk image size output by disktype if it is in kb. -* Get the size of the disk image using the du command if disktype outputs mb. -```$ du -ck M1111-0004.001``` -* Copy the size and file system information data into the media object’s CMS record. +| notes.transfer | +| -- | +| bad sectors | + +* Enter Y in the removed field if the disk is removed from the collection. + +| removed | +| -- | +| Y | + +### Completing the imaging process + +* Put the media back in the collection’s box and move or remove the pink “To image” flag as necessary if you are working on a large collection. +* Move the media to the “Small collections complete” box if you are working on a small collection without a box. + +### Using Digital Archives scripts +Make sure the setup instructions for Digital Archives [scripts](https://github.com/NYPL/digarch_scripts){:target="_blank"} have been completed before running the scripts in the next section. ### Create destination directories **These instructions show you how to create destination directories for a number of consecutive disks. Consider using a one-line command to create directories if the disks you are packaging do not have consecutive MediaID numbers.** -* Open Cygwin and enter the following commands: -* Connect to ARCHV Mac. -```$ ssh archv``` +On Mac: + +* Open Terminal. + +* Navigate to DigArchDiskStation. + +* Change into diskImages directory. +```$ cd /Volumes/DigArchDiskStation/Staging/ingest/diskImages``` + +* Create a directory for the acquisition if it does not exist. +```$ mkdir ACQ_acqID``` + +* Change into the acquisition directory. +```$ cd ACQ_acqID``` + +* Run [makesips script](https://nypl.github.io/digarch/tools/software.html#makesips-script){:target="_blank"} to create a consecutive number of submission information packages for material from digital media. + +Or -* Change to diskImage directory. -```$ diskimages``` +* Change to diskImages directory. -* Create a directory for your collection if it does not exist. -```$ mkdir M1111``` +```$ cd /Volumes/DigArchDiskStation/Staging/ingest/diskImages``` + * Enter ```mkdir``` command. +```mkdir -p ACQ_acqID/ACQ_acqID_specObjectID/{metadata,objects}``` -* Change into your collection directory. -```$ cd M1111``` +On Windows via WSL: -* Run the program to build structured directories. The program will ask you for your collection name and the first and last number of items of the directories you’d like to build. -```$ makesips``` +* Open WSL. +* If you do not see the Y:\ drive in /mnt or /mnt/y appears to be empty then it must be re-mounted by: + * Changing to the top level directory by entering ```cd ../``` + * Entering the command ```sudo mount drvfs Y: /mnt/y``` +* Change into diskImages directory. +```$ cd /mnt/y/Staging/ingest/diskImages``` + +* Create a directory for the acquisition if it does not exist. +```$ mkdir ACQ_acqID``` + +* Change into the acquisition directory. +```$ cd ACQ_acqID``` + +* Run [makesips script](https://nypl.github.io/digarch/tools/software.html#makesips-script){:target="_blank"} to create a consecutive number of submission information packages for material from digital media. + +Or + +* Change to diskImages directory. + +```$ cd /mnt/y/Staging/ingest/diskImages``` + * Enter ```mkdir``` command. +```mkdir -p ACQ_acqID/ACQ_acqID_specObjectID/{metadata,objects}``` +#### Directory structure + +* /ACQ_1234_123456 + * /metadata + * + * /objects ### Package images -* Open Cygwin and enter the following commands: -* Connect to ARCHV Mac. -```$ ssh archv``` +On Mac: + +* Open Terminal. + +* Navigate to DigArchDiskStation. + * Change to the kryofluxOutput directory. -```$ kryofluxoutput``` +```$ cd /Volumes/DigArchDiskStation/Staging/ingest/kryofluxoutput``` * Run the command to tar the stream files. ```$ compress``` * Run the program to move files from kryofluxOutput to the appropriate subdirectory in diskImages. ```$ moveimages``` * Move s0 and s1 images separately. -### Problem disks - -* Move disk images that are all errors or have unidentifiable file systems to the /problems directory in the Staging drive. -* Move logs for problem disk images to the /problems directory in the Staging drive. -* Move KryoFlux stream files folders to the /problems directory in the Staging drive. -* Select “Problem” from the “Issues” drop-down menu in the medialog and note the issue in the “Notes” field. +On Windows via WSL: + +* Open WSL. +* Navigate to DigArchDiskStation. +* Change to the kryofluxOutput directory. +```$ cd /mnt/y/Staging/ingest/kryofluxoutput``` +* Run the command to tar the stream files. +```$ compress``` +* Run the program to move files from kryofluxOutput to the appropriate subdirectory in diskImages. +```$ moveimages``` +* Move s0 and s1 images separately. diff --git a/transfers/iomega-disk-imaging.md b/transfers/iomega-disk-imaging.md index 47ee0be..d423bee 100644 --- a/transfers/iomega-disk-imaging.md +++ b/transfers/iomega-disk-imaging.md @@ -33,7 +33,7 @@ On Mac: * Change into your collection directory. ```$ cd ACQ_acqID``` -* Run [makesips script](https://nypl.github.io/digarch/tools/working-scripts.html#makesips-sipdirsh){:target="_blank"} to create a consecutive number of submission information packages for material from digital media. +* Run [makesips script](https://nypl.github.io/digarch/tools/software.html#makesips-script){:target="_blank"} to create a consecutive number of submission information packages for material from digital media. Or @@ -58,7 +58,7 @@ On Windows via WSL: * Change into the acquisition directory. ```$ cd ACQ_acqID``` -* Run [makesips script](https://nypl.github.io/digarch/tools/working-scripts.html#makesips-sipdirsh){:target="_blank"} to create a consecutive number of submission information packages for material from digital media. +* Run [makesips script](https://nypl.github.io/digarch/tools/software.html#makesips-script){:target="_blank"} to create a consecutive number of submission information packages for material from digital media. Or diff --git a/transfers/nimbie-transfers.md b/transfers/nimbie-transfers.md index b5ccc65..f2e4e36 100644 --- a/transfers/nimbie-transfers.md +++ b/transfers/nimbie-transfers.md @@ -11,7 +11,7 @@ This document describes how to use the [Nimbie Disc Auto-Loader]({% link tools/n This applies for the vast majority of CD's, DVD's. and BluRays with born-digital contents. -**Before a media object can be transferred it first must be recorded in the collection’s media log in CMS. See [Verifying inventory in Media Log](/digarch/transfers/verify-inventory.html){:target="_blank"} for instructions.** +**Before a media object can be transferred it first must be recorded SPEC.** Before beginning to work with the Nimbie, confirm that you have all of the collection's optical media by comparing discs to the media log in CMS. Using the Nimbie Robot consists of three steps. diff --git a/transfers/optical-media-imaging.md b/transfers/optical-media-imaging.md index 61334e2..2b39386 100644 --- a/transfers/optical-media-imaging.md +++ b/transfers/optical-media-imaging.md @@ -31,7 +31,7 @@ On Mac: * Change into the acquisition directory. ```$ cd ACQ_acqID``` -* Run [makesips script](https://nypl.github.io/digarch/tools/working-scripts.html#makesips-sipdirsh){:target="_blank"} to create a consecutive number of submission information packages for material from digital media. +* Run [makesips script](https://nypl.github.io/digarch/tools/software.html#makesips-script){:target="_blank"} to create a consecutive number of submission information packages for material from digital media. Or @@ -56,7 +56,7 @@ On Windows via WSL: * Change into the acquisition directory. ```$ cdA ACQ_acqID``` -* Run [makesips script](https://nypl.github.io/digarch/tools/working-scripts.html#makesips-sipdirsh){:target="_blank"} to create a consecutive number of submission information packages for material from digital media. +* Run [makesips script](https://nypl.github.io/digarch/tools/software.html#makesips-script){:target="_blank"} to create a consecutive number of submission information packages for material from digital media. Or diff --git a/transfers/transfers.md b/transfers/transfers.md index 79f15e1..3255bdd 100644 --- a/transfers/transfers.md +++ b/transfers/transfers.md @@ -21,7 +21,7 @@ The workflows vary based on media types and file types encountered but the gener 2. Acquiring basic metadata about the media and files. 3. Preparing the files for later archival processing. -**Before a media object can be imaged it first must be recorded in the collection’s media log in CMS.** +**Before a media object can be imaged it first must be recorded SPEC.** ### File Transfers diff --git a/transfers/using-tableaus.md b/transfers/using-tableaus.md index 54fb113..2344c63 100644 --- a/transfers/using-tableaus.md +++ b/transfers/using-tableaus.md @@ -21,7 +21,7 @@ These are instructions on how to connect drives using either the Tableau Ultraki ### Power cables * Connect the power supply to the forensic bridge. - - Device cables are available in the yellow case or labeled plastic drawers on the bakers rack. + - Device cables are available in the yellow case or cabinet by the cord board. * Connect the device to power. - Connect devices with external power as usual. - Connect uncased internal devices to power through the forensic bridge with a 3M drive power cable or a Molex to 3M drive power cable. diff --git a/transfers/verify-inventory.md b/transfers/verify-inventory.md index d5f26b5..12079cd 100644 --- a/transfers/verify-inventory.md +++ b/transfers/verify-inventory.md @@ -4,6 +4,12 @@ layout: default nav_order: 1 parent: Transfers --- +**Before a media object can be imaged it first must be recorded SPEC.** + +Deprecated +{: .label .label-red } + +|The following section is out of date. It needs to be revised with SPEC instructions.| ## Using Filemaker Collection Management System * Log in to CMS using your credentials.