-
Notifications
You must be signed in to change notification settings - Fork 8
What happens during a proofreading test
The goal of a proofreading test is to make sure that a new proofreader knows everything they need to know for them to be able to improve the connectome, with minimal risk of making incorrect or accidental edits that would corrupt the connectome. This is typically done by an expert proofreader observing the new proofreader do some proofreading in the sandbox dataset, providing advice to help the new proofreader improve on anything they're struggling with or not doing in an ideal way, and then deciding whether the new proofreader is ready to start editing the main dataset or if they could use a bit more practice in the sandbox on some specific topics before being granted access.
Currently, these individuals are authorized to give proofreading tests:
- Jasper Phelps (community manager)
- Leila Elabbady (Tuthill lab)
- Li Guo and Durafshan Syed (Simpson lab)
- Philipp Schlegel (Jefferis lab)
- Laia Serratosa (Aelysia)
Before the test, the expert proofreader will find some neurons in the sandbox that would make for good proofreading practice. During the test, they'll send these neurons to the new proofreader, have the new proofreader share their screen, and the new proofreader should talk about what they see and how they would proofread the neuron and why, and then the new proofreader should proceed to proofread the neuron. The expert proofreader will ask questions to make sure the new proofreader understands general neuroanatomy concepts like axon/dendrite/cell body, and will ask the new proofreader to explain their thinking and procedure as they proofread, and will provide corrections and advice to help the new proofreader to help them start down the path toward becoming a skilled/expert proofreader.
The expert proofreader should provide exercises that cover the following major topics:
- Use of the merge tool and the split tool and [slightly more advanced] the find path tool.
- The most common causes of segmentation errors in FANC, which are knifemarks, fat globules, and glia. Knifemarks and fat globules are especially important to cover because they’re FANC-specific (i.e. they’re not in FAFB) and there’s tons of easy proofreading to do that just involves fixing errors caused by knifemarks or fat globules. (Sometimes cutting off glia can be skipped, if the other exercises already have the new proofreader show that they can cut a merge error caused by knifemarks and/or fat globules.)
Here are some proofreading tips that can/should be discussed with the new proofreader if time allows. Listed roughly in order of most important to least important:
- FANC's cut tool is slightly improved compared to the one used in FlyWire/FAFB. The FlyWire tutorial tells people to put down tons of red points in a row and a ton of blue points in a row. Our algorithm is smarter, and when you put just two red points far away from each other, everything on the line between those points automatically gets marked as red too. (Not a straight line, but a path through the neuron.) Same with blue points. So often putting a single pair of red points and a single pair of blue points is enough to accomplish a merge, if you’re good at knowing where to put the pair of points. A triangle of points is for sure sufficient in the large majority of cases.
- You don’t have to refresh your entire page to refresh a mesh after an edit, instead ctrl+shift+click on a neuron (in either 2D or 3D) will refresh just that mesh. This saves you from having to reload everything else in your scene like the image data and any other meshes that were already loaded.
- Uncheck the “Hide segment ID 0" button when traversing knifemarks to get an exact map of which regions of the image data are and aren’t segmented.
- When you’re looking at the 2D data and trying to figure out what’s happening, hide the segmentation layer by pressing 2 and consider hiding the crosshairs by pressing A, to make sure your view of the EM data is unobstructed.
- The L key assigns new colors to all the neurons.
- The Find Path tool has a precision mode that is worth using most of the time.
If time allows, discuss proofreading strategy, that is, how to spend time doing proofreading operations that have the biggest impact on the connectome, and how to avoid spending time on operations that have minimal impact:
- There are two goals in proofreading a neuron: to make the morphology of the neuron look generally correct, and have the correct collection of synapses associated with the neuron.
- The first few minutes of proofreading any particular neuron will be the most useful, as you will fix the major split and merge errors first, and then already the neuron’s morphology and connectivity will be mostly correct.
- Proofreading very thin short branches is usually not worth the time because those edits will not change the morphology or connectivity of the neuron very much – One edit that fixes a major problem of a neuron can have an impact of easily more than 10 edits that each fix a minor problem. So it’s better to move on to work on a new neuron once all the major problems with a neuron have been fixed, as long as there are other neurons in the dataset that still have major problems (which is likely going to be the case for a while longer).
- Some additional commentary on this topic can be found on the flywire slack here
If the new proofreader still needs to gain some confidence in some aspect of proofreading, or is still getting used to doing proofreading in a way that keeps the risk of making a mistake very low, they should be encouraged to continue practicing in the sandbox and should be encouraged to ask others in their lab for help learning to proofread. If the expert proofreader has time for it, they can send additional proofreading exercises to the new proofreader to help them have things to practice on, and the new proofreader can do those exercises when they have time and then send the finished neuron back to the expert proofreader for evaluation. The expert proofreader can decide to then pass the student based on these additional exercises without another interactive proofreading session, or can decide that it makes sense to have a second video call to do some more proofreading together to continue providing advice and assess progress.
If the new proofreader shows that they know how to use the various proofreading tools and they show that they know how to correctly see what’s going on in the EM images, the expert proofreader should tell them they pass the test and will be granted access soon. The new proofreader should be told that they can find links to access the real dataset on http://fanc.community/Neuroglancer-states-for-proofreading. Also tell them that they should feel free to post on #proofreading-help if they’d like a second set of eyes on an edit that they’re unsure of. Send the new proofreader messages on Slack tagging @neuron-information-bot and @proofreading-status-bot and tell them to send the message “help” to both of the bots right now, to read the responses from the bots soon, and try interacting with the bots as soon as they start proofreading.