Interested in collaborating?

[andrew-sayers]

Hey,

I've just released a new site for managing sleep diaries. My next big job is to add a logger to it, so I thought I'd reach out and see if we can work together.

I'm particularly interested in agreeing on a file format. It's long been my opinion that file formats are the hardest and most important part of a sleep diary, so a format developed among multiple independent stakeholders seems like the best solution.

In terms of timing - I've got a lot of notes written up (and many more ideas waiting to be written), but no code yet. From reading around the Circalog site, it seems like you're currently prototyping ideas to see what will work? If so, now's probably at a good time to work on a file format, and we can talk about sharing more work after that.

[lrq3000]

OMG YES! I'm so sorry I never saw your invitation to collaborate, I think we missed each others by bad luck. I also went to your github repos to invite you to collaborate, but I never noticed you invited me here on my repo as well, I'm so sorry!

It's been a long while since you posted your invitation, but anyway yes I would be very glad to collaborate on this. I read your site and how you discriminate various sleep patterns, it's awesome! I also have a draft unpublished method that I made on my own before finding your site, it's a different methodology but it would be great to have both methodology, maybe we can merge some or just offer the two different approaches and users will see what works best in practice.

So yes definitely if you have time and ever come by and are still interested, I would be very honoured to collaborate with you. I intend to restart this project in the not so long future, and I will definitely consider adding your methodologies and tools :D

[andrew-sayers]

Oh hey, good timing!

I ended up spending the whole of 2023 working through my own stuff, but am also planning to restart in the new year.

After a year of trying things, I'm now 99% sure my problems are about digesting some compound related to lactose, and that my sleep issues are just the most visible of a constellation of problems that last for about two weeks after consumption. Having largely finished hypothesis-forming, it's time to get back into making actual tools and gathering proper data.

I'm about to go see my family for Christmas, at which point I'll go silent until probably about mid-January. But whatever time I've got free will be spent reminding myself how this all works and reading around the subject. I have been concerned that nobody else can update the site I made, so would love to put something together where we both have the necessary permissions when one of us needs time off.

[lrq3000]

Dear Andrew, This is awesome to hear and I am so glad you are still excited about the prospect of collaborating together as I certainly am too :D Yes I would be happy to share permissions, I would love too to help you maintain your awesome projects! About lactose, this sounds like a very interesting hypothesis! Did you also consider FODMAPs more largely, since lactose is part of these? For me I started discovering my allergy to lactose and later to FODMAPs and it helped me a lot manage my digestive issues, but unfortunately not my circadirn rhythm issues, but hopefully for you it will be more helpful, and i think i remember at least one other member on reddit mentioning they found that their non24 was caused by lactose intolerance. If you'd like too we could arrange for a videocall to meet and exchange ideas :-) if yes contact me at ***@***.*** :D Warm regards and enjoy the holidays! Stephen 14 déc. 2023 16:49:15 Andrew Sayers ***@***.***>:

…

Oh hey, good timing! I ended up spending the whole of 2023 working through my own stuff, but am also planning to restart in the new year. After a year of trying things, I'm now 99% sure my problems are about digesting some compound related to lactose, and that my sleep issues are just the most visible of a constellation of problems that last for about two weeks after consumption. Having largely finished hypothesis-forming, it's time to get back into making actual tools and gathering proper data. I'm about to go see my family for Christmas, at which point I'll go silent until probably about mid-January. But whatever time I've got free will be spent reminding myself how this all works and reading around the subject. I have been concerned that nobody else can update the site I made, so would love to put something together where we both have the necessary permissions when one of us needs time off. — Reply to this email directly, view it on GitHub[#1 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AAIRFXSUGYHJWBTKNYN3QKLYJMNXXAVCNFSM5A6JRJY2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBVGYYDSNBWG43Q]. You are receiving this because you were assigned. [Image de pistage][https://github.com/notifications/beacon/AAIRFXXWUMOJ5AVCWRYOOPDYJMNXXA5CNFSM5A6JRJY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGON2Q4DVI.gif]

[andrew-sayers]

In that case, I'll be gathering ideas and get back to you in the new year! Depending how Christmas goes, I might come back with a long list or just a bunch of reminders for presents I should get the family them next year ;)

FODMAPs are an interesting angle, but this year I've had more luck examining effects than the causes. For example...

I've found that I react to even very small doses (e.g. eating three or four crisps that "may contain milk"), and my last test of the year showed that eating the peel of a particular type of new potato caused a strong reaction, but eating a thoroughly peeled potato from the same bag didn't affect me. My current hypothesis is that they glaze the potatoes with something, but at low enough doses that they don't bother writing it on the label. So I need to figure out a pattern based on unlisted ingredients, with a two week recovery time between tests, and a known-good diet in the mean time!

On the other hand, what's the first thought in your head when you wake up? Like "I need the toilet" or "I guess I'd better go do something"? I'm fairly sure my first thought is always "it is morning" when my sleep has been disturbed by lactose (or whatever it is), and never at any other time. Obviously I'll need a way of recording first thoughts in order to confirm that, but it could be an easy way to detect when I've had something I shouldn't have. And it would be interesting to see if that's a more common thought among non-24 people than the general population.

It sounds like the overlap between our interests is about helping people to rigourise the way they examine their sleep. Is that a fair assessment? Is there anything in particular you're planning to focus on next year?

[lrq3000]

@andrew-sayers Sorry for the delay in my reply, I had an unexpectedly very busy week!

Wow that's a crazy strong effect you get from just the skin of potatoes. In addition to the hypothesis you mentioned, it may be caused by a residue of pesticides? I don't think these get listed, and sometimes the farmer may have used a bit too much on some plants by mistake...

About recording first thoughts in the morning and other data, yes totally it could be very interesting. That's why I would like to allow for freeform inputs in Circalog, and even the possibility to design custom widgets with custom questions. I initially envisioned this feature to be more useful for clinicians handing a preconfigured device to patents with preconfigured questions, but users at home can also use this to define their own questions. But I did not make any code for that yet, I just defined the database schema! (But now there is GitHub Copilot so it may help speed up the process a bit ;-) )

It sounds like the overlap between our interests is about helping people to rigourise the way they examine their sleep. Is that a fair assessment? Is there anything in particular you're planning to focus on next year?

Yes, exactly! And I would also add that we both want to help people get an informal screening of their sleep patterns. I saw you defined rules to try to autodetect DSPD and non24 among others. I would like to implement them in webactogram, which I intend to be a en-mass screener for people who are curious whether they have a circadian rhythm disorder but don't know anything about them.

But going further, MaxiQS suggested on Reddit that there is a lack of a popular library for circadian rhythm analysis, and that we should make one. And I think he's totally right! You already made an awesome library for sleep diaries, What would you think if we generalize that into a circadian rhythm analysis library? Eg we could provide algorithms for the autodetection of circadian rhythm disorders patterns (which would then be used in webactogram and Circalog - but then this feature would be modularized, so other projects can also build on this to create new awesome circadian rhythm tools)!

In that case, I'll be gathering ideas and get back to you in the new year! Depending how Christmas goes, I might come back with a long list or just a bunch of reminders for presents I should get the family them next year ;)

Awesome! My mind is also racing with ideas! :D I'll be very happy to continue discussing with you next year! Or anytime later, as I hope to be able to dedicate more and more time to sleep research :-)

I hope you'll have a wonderful time with your family, Merry Christmas and my best wishes for the New Year :D

[lrq3000]

PS: I am aware we cannot do everything I suggested in just a couple of months lol, but these are ideas of the goal, and I'll follow you on what you prefer to work on, so just pick what you'd like to do :-)

[andrew-sayers]

I've got a few minutes while everyone's occupied, so here's a quick response - I'll try to write something more detailed another day. The old JavaScript library was becoming quite unwieldy last time I worked on it, and now might be a good time for a rewrite. But for a library to work with everything from MATLAB to Python to JavaScript, it will need to be written in C. Also, there's some tension between defining a "good" format vs. one that's compatible with other apps. For example, some apps represent dates in ways that break during DST changes, forcing us to choose between not supporting those apps or working around broken behaviour. How do you feel about each of those issues?

…

On 22 December 2023 15:23:21 EET, Stephen Karl Larroque ***@***.***> wrote: PS: I am aware we cannot do everything I suggested in just a couple of months lol, but these are ideas of the goal, and I'll follow you on what you prefer to work on, so just pick what you'd like to do :-) -- Reply to this email directly or view it on GitHub: #1 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[lrq3000]

Haha I know what it's like to type a quick response when you can, no worries ^^

About your JS lib, maybe it could be rewritten into typescript? And it could be a good moment to rewrite it as one module of a broader circadian rhythm module. You're right however that the broadest compatibility can only be achieve if such a library is written in C, but 1) I'm not a very good C dev, I wrote C in the past but it's clearly not my specialty, 2) I think such a library needs to target either Python or Javascript or both if possible, these are the two most used languages for server-side and client-side respectively. I think rewriting in TypeScript can work for us, there are several projects that aim to make Javascript modules usable in Python such as JS2Py. Note that I am still learning typescript but I can get up to speed hopefully.

There is also the possibility of writing in Python (which is the language I'm the most prolific in), and it can be converted nowadays into javascript using pyiodide or brython or other transpilers, but it won't be the same as coding in typescript directly of course. And if we want to include the lib in mobile apps, typescript is certainly the way to go (compatible with react-native and flutter - although flutter nowadays can also import python modules!).

About the format, there is not standard currently, at least you made one. I also defined one for Circalog but it's not meant to be an interoperable standard. IMHO we are free to choose what we consider important to track in digital sleep diaries. For apps that miss important features such as DST changes, this is indeed tricky, but I guess this should be managed by the user (eg, asking the user when importing how they want to manage this issue), it's not per se a problem of the standard, it's a problem of converting other formats to the standard.
I mean I understand that it can be helpful to specify in the specs how conversion should be managed ideally, but when the information is lacking, it's not possible to recreate it. So yeah, asking the user what to do is I think the only viable approach, making sure they are warned about the consequences of this issue.

[andrew-sayers]

It's mid-January, so here's what I've got so far. I'm hoping to make time during this month to talk things over and make plans. Hopefully February will be free enough to actually do something about them :)

Personas

I find it useful to write out personas when thinking about this kind of design problem. I'm not sure how useful it is to read them after the fact, but here's some archetypes I imagine this software being for...

Alice has gathered 20 years of data in SleepChart 1.0, and she finally has a weekend to throw the data into Python and start make the graphs she's been planning. SleepChart's binary format strikes her as a daunting timesink, as does a solution that forces her to learn about weird timezone issues - after all, SleepChart did everything she wanted without asking her about them. She wants to call something like sleep-diary converter sleep.diary sleep.csv, load the result into Python without spending time on extra modules, do the analysis she's already been planning, then look at software from the community another day.

Bob has also gathered 20 years of SleepChart data, and would have been happy using it forever, but his nephew keeps calling him a dinosaur for using an old Windows app, and has just gifted him Sleep As Android for Christmas. Bob doesn't care about numbers, he just wants to see a graph so he can decide how much of a lie-in to have tomorrow. He would like a website to take his old data, convert it as precisely as possible to Sleep As Android, and explain in plain English any unavoidable changes.

Carol hasn't used a sleep diary before, but wants to know why she's always tired. Her doctor has been running her through the basics (caffeine at night, doomscrolling before sleep etc.), and will need to be included in a conversation about balancing a sleeping disorder with her other issues. But Carol has a tendency to oversimplify her problems down to "find the name of the problem and get a doctor to prescribe the drug for it", so consultations tend to involve a lot of time debunking the latest thing she read on some random website. She and her doctor want something that helps her gather good data, then presents the analysis to the doctor for discussion.

The personas above make me think we need two different file formats - a baseline "interchange format" that can express every esoteric feature of all known sleep diary formats, with the goal of letting people migrate their data between arbitrary formats with as little loss as possible; and a more advanced "Circalog format" with the goal of expressing sleep-related data in the (single) way that best supports mathematical analysis, even if that involves normalising things in lossy ways. That means the interchange format can be as hairy and redundant as needed to support each little feature, while the Circalog format can jettison anything that would slow an analyst down.

Programming languages

The more I think about programming languages, the more important it is to split out "core" vs. "general" functionality. Here's a quick tour of reasons:

• "core" code needs to handle arbitrary input formats (from CSV to zipped XML to SleepChart 1.0's binary format), "general" code only needs to use APIs we control
• "core" code needs to smooth out issues with other libraries (especially parsing civil time - "Europe/London" etc. - which throws out plenty of unexpected surprises), "general" code should be free from such distractions
• "core" code needs to be thoroughly tested so we can confidently present information (e.g. in a paper), "general" code needs to be quick to write so we can iterate quickly on ideas (e.g. during hypothesis-forming)
• "core" code needs to be limited to a single programming language, "general" code needs to support whatever platform the next person prefers to use

It's much easier to maintain a distinction like this if you have some sort of natural barrier between the two types of code. My previous work did it with different JavaScript codebases, but I'd prefer use two different languages in future.

C

C code can be a pain to write, but every language (and even software like MATLAB) has extensive documentation about interfacing with C libraries. Python is a particularly good example, with many excellent combinations of low-level C libraries with easy-to-use Python libraries on top. That makes it an excellent choice for the core language, and should encourage us to expose APIs to other language(s) rather than bloat the core.

One issue with C is that civil time ("Europe/London" etc.) needs to be decoded with a timezone database. C implements this with platform-specific libraries, so a C implementation would need some kind of platform-specific "driver" for that. And once we've got a driver framework, we'll want to use it for e.g. JSON and XML parsing, so we can use browsers' built-in parsers. Not a fatal flaw, but worth keeping in mind.

This all makes me think C is the right choice for the core language, and a poor choice for a general one.

JavaScript / TypeScript

I looked at JS2Py, but it only supports ECMAScript 5.1. Core code needs to support binary formats (e.g. SleepChart 1.0), which requires ECMAScript 6. And I've found Node.JS dependencies cumbersome to maintain over the long term, so I'd rather not rely on that.

I've done some little projects that expose C APIs to JavaScript with WebAssembly. Even by C standards it can be fiddly to get right, but modern JS environments make it perfectly possible to write JS code with a C library underneath. So JS is a fine general language.

This means JavaScript (and TypeScript) can't be the core language. It's probably a good idea to support them as general languages because it'll be easier to make web stuff, but they don't offer anything special beyond that.

Python

It seems that Python can run in the browser now with pyodide, although I've just heard of it so I can't tell you if it's any good. One thing I have noticed is that timezone information is not supported, so a Python core doesn't solve the problem of needing drivers.

Python strikes me as the obvious choice for a general language, given how much scientific computing is done in it. But just as C encourages you to add APIs when you get to non-core functionality, Python encourages you to switch to C when things start getting technical.

So how about that for a plan - a C core that does all the low-level programming problems, with APIs in Python (and probably JavaScript) for general work?

[lrq3000]

Dear @andrew-sayers , just to let you know I will soon (upcoming days) provide a more detailed reply to your thoughtful comment, I really want to dedicate enough brainpower given how much efforts you put into it. Also I have great insights to share I recently got from my activities.

Until then for some context you can see one of the many things I have been up to lately (so maybe you'll excuse me a bit more for my late reply 🥺👉👈): https://www.reddit.com/r/N24/comments/1acmqp4/podcast_my_interview_in_the_science_magazine_zeit/

[lrq3000]

@andrew-sayers Another quick reply: if you are interested into looking more into FODMAPs and whether this may be pertinent to improve your issues (as it was for me), a very helpful ressource is the Monash University FODMAPs list app. Also to summarize, FODMAPs are subtypes of sugars. Since I am pretty much intolerant to all categories of FODMAPs (this is apparently rare, most people are only intolerant to one or two categories at most), I am still exploring ways to have a healthy balanced diet without FODMAPs exposure, but recently I found that avoiding foods with sugar, or at least trying to use food products with as minimal sugar as possible, usually yielded good results for me.

[andrew-sayers]

I took the opportunity last weekend to try another brand of potato - had a definite reaction and wouldn't have been able to reply this week anyway. It could be pesticide residue, and apparently farmers have been known to spray milk slurry on plants, so now I'm working out how to grow my own control potatoes.

Something that might interest you - I suspect some of my symptoms might caused by the abdominal muscles around my digestive system becoming distressed. This past week, I've got some relief by exercising just those muscles. If you'd like to try it, here's how:

1. lie face-down on the floor
2. push into a press-up position - arms and legs locked so they make a right-angle triangle with the floor
3. optionally move your hips a few centimetres up or down to exercise specific muscles
4. hold that position for about 10 seconds, or until your stomach muscles get tired
   • if any other muscles get tired, shift position until they're OK (e.g. make sure your arms aren't bent at all)

I've found my symptoms mostly go away for a few hours when I do that. I usually feel distress at the top of my abdomen at first, moving down to the base over the course of about 10 days, which so far this week has meant lowering my hips by as much as 2-3cm until I can feel it stretching out the muscle that's distressed today.

P.S. just listened to your podcast - good work!

[lrq3000]

@andrew-sayers Thank you very much for your kind words :-) And I'm so sorry again for my lateness!

So I have lots of things I would like to discuss with you, including some infos I prefer not to be made public just yet :-) Would you agree that we continue our discussion by e-mail maybe at (censored because of spam) ? And in fact I'd like to suggest that we do a video call sometime if you fancy the idea, so that we can bounce ideas much more dynamically :-)