Unable to delete files on full volume + KP

[GoogleCodeExporter]

What version of the product are you using? On what operating system?
74.2 on OS X Lion 10.7.4

Please provide any additional information below.

Unable to delete files on full volume + KP

Sort of related to issue 125: Kernel Panic when using rsync

After having rsynced some linux source tree files over ssh and this filled the 
volume, 
making it impossible to delete files and a kernel panic on 10.7.4. 
Trying to zpool export results in a KP so I reboot into ubuntu to try and 
import it from there,

$ sudo zpool status
[sudo] password for : 
  pool: nineteen
 state: UNAVAIL
status: One or more devices could not be opened.  There are insufficient
    replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-3C
 scrub: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    nineteen    UNAVAIL      0     0     0  insufficient replicas
      disk0s19  UNAVAIL      0     0     0  cannot open


j@K53E:~$ sudo zpool import -o version=8 nineteen
cannot import 'nineteen': pool may be in use from other system
use '-f' to import anyway
j@K53E:~$ sudo zpool import -f -o version=8 nineteen
cannot import 'nineteen': out of space
    Recovery is possible, but will result in some data loss.
    Returning the pool to its state as of 2013-12-27
    should correct the problem.  Recovery can be attempted
    by executing 'zpool import -F nineteen'.  A scrub of the pool
    is strongly recommended after recovery.
j@K53E:~$

man zpool
           -F

               Recovery mode for a non-importable pool. Attempt to return the pool to an importable state by discarding the last few  transactions.
               Not  all  damaged pools can be recovered by using this option. If successful, the data from the discarded transactions is irretriev‐
               ably lost. This option is ignored if the pool is importable or already imported.


j@K53E:~$ sudo zpool import -n -F -o version=8 nineteen
cannot import 'nineteen': pool may be in use from other system
use '-f' to import anyway

j@K53E:~$ sudo zpool import -n -F -f -o version=8 nineteen
Would be able to return nineteen to its state as of 2013-12-27.

j@K53E:~$ sudo zpool import -F -f -o version=8 nineteen
cannot import 'nineteen': out of space
    Recovery is possible, but will result in some data loss.
    Returning the pool to its state as of 2013-12-27
    should correct the problem.  Recovery can be attempted
    by executing 'zpool import -F nineteen'.  A scrub of the pool
    is strongly recommended after recovery.



I boot back into OS X and see that it boots normal and mounts fine, files can 
be read
and scrub reports no errors:


mbp81:Volumes user$ sudo zpool scrub nineteen
mbp81:Volumes user$ sudo zpool status
  pool: nineteen
 state: ONLINE
 scrub: scrub in progress, 0.14% done, 2h53m to go
config:

        NAME        STATE     READ WRITE CKSUM
        nineteen    ONLINE       0     0     0
          disk0s19  ONLINE       0     0     0

errors: No known data errors

mbp81:Volumes user$ sudo zpool status nineteen
Password:
  pool: nineteen
 state: ONLINE
 scrub: scrub completed with 0 errors on Mon Dec 30 22:13:28 2013
config:

        NAME        STATE     READ WRITE CKSUM
        nineteen    ONLINE       0     0     0
          disk0s19  ONLINE       0     0     0

errors: No known data errors



mbp81:Volumes user$ mv /Volumes/nineteen/music/test.mp3  
/Volumes/DATA/Downloads/
mv: /Volumes/nineteen/music/test.mp3: remove: No space left on device

mbp81:Volumes user$ rmdir  /Volumes/nineteen/.TemporaryItems/folders.0/
rmdir: /Volumes/nineteen/.TemporaryItems/folders.0/: No space left on device

# sudo zpool export nineteen
KP

panic (cpu 0 caller 0xffffff80002b597e): Preemption level underflow, possible 
cause unlocking an unlocked mutex or spinlock
Kernel Extensions in backtrace: com.bandlem.mac.zfs.fs(74.2) 
[ED82708C-....]@0xffffff7f80747000->0xffffff7f807affff
BSD process name corresponding to current thread: zpool
Mac OS 11E53          DKV 11.4.0    MacBookPro8,1



mbp81:~ user$ zfs get all  nineteen
NAME      PROPERTY       VALUE                  SOURCE
nineteen  type           filesystem             -
nineteen  creation       Thu Jul 11 22:54 2013  -
nineteen  used           31.0G                  -
nineteen  available      0                      -
nineteen  referenced     31.0G                  -
nineteen  compressratio  1.00x                  -
nineteen  mounted        yes                    -
nineteen  quota          none                   default
nineteen  reservation    none                   default
nineteen  recordsize     128K                   default
nineteen  mountpoint     /Volumes/nineteen      default
nineteen  sharenfs       off                    default
nineteen  checksum       on                     default
nineteen  compression    off                    default
nineteen  atime          off                    local
nineteen  devices        on                     default
nineteen  exec           on                     default
nineteen  setuid         on                     default
nineteen  readonly       off                    default
nineteen  zoned          off                    default
nineteen  snapdir        hidden                 default
nineteen  aclmode        groupmask              default
nineteen  aclinherit     secure                 default
nineteen  canmount       on                     default
nineteen  shareiscsi     off                    default
nineteen  xattr          on                     default
nineteen  copies         1                      default
nineteen  version        2                      -
mbp81:~ user$

mbp81:~ user$ zpool history -il nineteen
History for 'nineteen':
2013-07-11.22:54:06 zpool create nineteen disk0s19 [user root on:global]
2013-07-11.22:59:07 [internal property set txg:65] atime=0 dataset = 5 [user 
user on]
2013-07-11.22:59:08 zfs set atime=off nineteen [user user on:global]
2013-07-18.13:30:17 zpool import -f 4760846665821573081 [user root on:global]
2013-07-18.21:20:53 zpool scrub nineteen [user user on:global]
2013-07-18.22:04:16 zpool import -f 4760846665821573081 [user root on:global]
2013-07-19.13:30:15 zpool import -f 4760846665821573081 [user root on:global]
2013-07-21.15:06:43 [internal pool property set txg:164719] version 8 nineteen 
[user root on K53E]
2013-07-21.15:06:44 [internal pool import txg:164722] pool spa 8; zfs spa 23; 
zpl 4; uts K53E 3.5.0-23-generic #35~precise1-Ubuntu SMP Fri Jan 25 17:13:26 
UTC 2013 x$
2013-07-21.15:06:45 zpool import -f -o version=8 nineteen [user root on 
K53E:global]
2013-07-24.08:50:09 zpool export nineteen [user root on K53E:global]
2013-07-24.10:51:26 zpool import -f 4760846665821573081 [user root on:global]
2013-07-25.10:48:56 zpool import -f 4760846665821573081 [user root on:global]
2013-07-28.22:09:50 zpool import -f 4760846665821573081 [user root on:global]
2013-08-25.15:15:15 [internal pool property set txg:680220] version 8 nineteen 
[user root on K53E]
2013-08-25.15:15:15 [internal pool import txg:680223] pool spa 8; zfs spa 23; 
zpl 4; uts K53E 3.5.0-23-generic #35~precise1-Ubuntu SMP Fri Jan 25 17:13:26 
UTC 2013 x$
2013-08-25.15:15:16 zpool import -o version=8 -f nineteen [user root on 
K53E:global]
2013-08-27.08:58:14 zpool import -f 4760846665821573081 [user root on:global]
2013-08-27.20:55:06 zpool import -f 4760846665821573081 [user root on:global]
2013-09-04.21:10:39 zpool import -f 4760846665821573081 [user root on:global]
2013-09-10.23:04:06 zpool import -f 4760846665821573081 [user root on:global]
2013-09-17.19:57:25 zpool import -f 4760846665821573081 [user root on:global]
2013-09-20.16:50:26 zpool import -f 4760846665821573081 [user root on:global]
2013-09-25.21:08:58 zpool import -f 4760846665821573081 [user root on:global]
2013-10-06.12:27:02 zpool import -f 4760846665821573081 [user root on:global]
2013-10-06.13:12:27 zpool import -f 4760846665821573081 [user root on:global]
2013-10-22.07:42:01 zpool import -f 4760846665821573081 [user root on:global]
2013-10-23.10:42:24 zpool import -f 4760846665821573081 [user root on:global]
2013-10-26.18:56:44 [internal pool property set txg:1657272] version 8 nineteen 
[user root on K53E]
2013-10-26.18:56:45 [internal pool import txg:1657275] pool spa 8; zfs spa 23; 
zpl 4; uts K53E 3.5.0-23-generic #35~precise1-Ubuntu SMP Fri Jan 25 17:13:26 
UTC 2013 $
2013-10-26.18:56:45 zpool import -o version=8 -f nineteen [user root on 
K53E:global]
2013-10-27.10:27:18 zpool export nineteen [user root on K53E:global]
2013-10-27.14:28:55 zpool import -f 4760846665821573081 [user root on:global]
2013-10-30.08:55:28 zpool import -f 4760846665821573081 [user root on:global]
2013-11-02.18:17:30 zpool export nineteen [user root on:global]
2013-11-03.17:27:43 [internal pool property set txg:1756150] version 8 nineteen 
[user root on K53E]
2013-11-03.17:27:43 [internal pool import txg:1756153] pool spa 8; zfs spa 23; 
zpl 4; uts K53E 3.5.0-23-generic #35~precise1-Ubuntu SMP Fri Jan 25 17:13:26 
UTC 2013 $
2013-11-03.17:27:44 zpool import -o version=8 nineteen [user root on 
K53E:global]
2013-11-03.17:31:13 zpool export nineteen [user root on K53E:global]
2013-11-04.09:22:51 zpool import -f 4760846665821573081 [user root on:global]
2013-11-04.21:18:50 zpool export nineteen [user root on:global]
2013-11-05.08:40:20 zpool import -f 4760846665821573081 [user root on:global]
2013-11-06.16:20:54 zpool export nineteen [user root on:global]
2013-11-06.17:05:06 [internal pool property set txg:1788298] version 8 nineteen 
[user root on K53E]
2013-11-06.17:05:07 [internal pool import txg:1788301] pool spa 8; zfs spa 23; 
zpl 4; uts K53E 3.5.0-23-generic #35~precise1-Ubuntu SMP Fri Jan 25 17:13:26 
UTC 2013 $
2013-11-06.17:05:07 zpool import -o version=8 nineteen [user root on 
K53E:global]
2013-11-06.18:29:40 zpool import -f 4760846665821573081 [user root on:global]
2013-11-14.19:10:20 zpool export nineteen [user root on:global]
2013-11-14.19:11:51 zpool import -f 4760846665821573081 [user root on:global]
2013-11-16.13:10:42 zpool export nineteen [user root on:global]
2013-11-16.13:25:48 zpool import -f 4760846665821573081 [user root on:global]
2013-12-16.18:00:39 zpool import -f 4760846665821573081 [user root on:global]
2013-12-20.18:32:23 zpool export nineteen [user root on:global]
2013-12-21.19:47:11 zpool import -f 4760846665821573081 [user root on:global]
2013-12-27.08:21:25 zpool import -f 4760846665821573081 [user root on:global]
2013-12-30.21:58:24 zpool import -f 4760846665821573081 [user root on:global]
2013-12-30.22:00:08 zpool scrub nineteen [user root on:global]
2013-12-30.22:54:21 zpool import -f 4760846665821573081 [user root on:global]



system.log
 18:01:47 mbp81 fseventsd[58]: failed to unlink old log file /Volumes/nineteen/.fseventsd/000000000062bb40 (No space left on device)
 18:01:48 mbp81 diskarbitrationd[52]: unable to mount /dev/disk0s19 (status code 0x00000001).
 18:01:49 mbp81 fseventsd[58]: failed to store the uuid in /Volumes/nineteen/.fseventsd/fseventsd-uuid.  unlinking it.
 18:02:17: --- last message repeated 1 time ---


From ssh: sudo zfs umount nineteen   -> KP

Original issue reported on code.google.com by [email protected] on 2 Jan 2014 at 10:17