fediverse => Bluesky bridging currently paused

[wbz0100]

Its delayed so much. Is this normal?
I think it's been lagging a lot since the last time it was disconnected.

Instead of being posted sequentially. And In severe cases, there is a delay of up to 10 hours.

[snarfed]

Yes! Not normal, we're badly backed up right now. Started around midnight PT last night, our send task queue peaked at over 3M tasks (!). Sorry for the trouble. It's down to 2.5M ish now, but still. 😕

I'm not sure what's going on yet. Coincidentally, I got pretty sick last night, and I'm still out of it, so I haven't been able to look yet. Hopefully soon.

[snarfed]

For my own notes, here's the repl code I'm using to clear out the send queue:

from google.cloud import tasks_v2
tasks_client = tasks_v2.CloudTasksClient()

from common import TASKS_LOCATION
from google.cloud.tasks_v2.types import Task

request = tasks_v2.ListTasksRequest(
    parent=tasks_client.queue_path(appengine_info.APP_ID, TASKS_LOCATION, 'send'),
    response_view=Task.View.FULL,
)

deleted = 0
for resp in tasks_client.list_tasks(request=request):
  if urllib.parse.parse_qs(resp.app_engine_http_request.body).get(b'user') == [b'ahBicmlkZ3ktZmVkZXJhdGVkci0LEgdBVFByb3RvIiBkaWQ6cGxjOm92eWFqYm5lZ3JpcjJkZHR3ZmhpamlvYgw']:
    deleted += 1
    print(deleted)
    tasks_client.delete_task(request=tasks_v2.DeleteTaskRequest(name=resp.name))

[snarfed]

We're almost entirely caught up, send task queue is under 10k now. Thanks for your patience all!

[imsodin]

Thanks for dealing with this so quickly!

For my own notes, here's the repl code I'm using to clear out the send queue:

Does that mean some posts during the incident were not replicated?
I am just asking to get an idea if some problems I just heared about might be related or not. I had a chat with someone who unfortunately started trying out the bridge more or less at the same time as these troubles happened.

[snarfed]

Hopefully we didn't drop any posts. Let me know if you see any that are missing and pass the checks in https://fed.brid.gy/docs#troubleshooting , happy to look.

[fadookie]

@snarfed thanks for handling this. Is it possible this is happening again? In my fedi -> bsky bridged profile I haven't seen any posts updating in over 9 hours (I'm not sure what the usual delay is): https://bsky.app/profile/Eliot-L.social.coop.ap.brid.gy

[wbz0100]

Umm... Looks like it's happening now.
A post from about 2 hours ago is publishing now.

This seems to happen constantly, and I'm not sure if it's because the post speed can't keep up with the processing speed, or if there's something wrong in the processing.

[EmilJacobs]

Can confirm, this issue isn't resolved.

[avocado1969]

I’m also not seeing fedi actions populate onto bsky since yesterday evening, even though my profile on fed.brid.gy shows the changes going though with the green check

[dakota-marshall]

Not to pile on, but I can confirm this is still happening aswell. I have severl public posts and replies that all seem to have the green check, but are not appearing on the Bluesky account.

I actually haven't had anything sync over since I bridged my account like 24 hours ago

[Ralph1798]

I followed a newly bridged account and posts from 350d+ started to appear in my Mastodon-timeline.
Interesting enough: no current posts, just the old ones.

[snarfed]

Not to pile on, but I can confirm this is still happening aswell. I have severl public posts and replies that all seem to have the green check, but are not appearing on the Bluesky account.

I actually haven't had anything sync over since I bridged my account like 24 hours ago

Yes! Sorry, reopening. The current Bluesky => fediverse delay is on us, fediverse => Bluesky delay is on Bluesky team. We're (all) working on it. 😬

[wbz0100]

I know you've been sick, so please take it easy and work slowly.

I think one of the causes may have been a delay in moving posts from X (formerly Twitter) to Bluesky.

[dakota-marshall]

Not to pile on, but I can confirm this is still happening aswell. I have severl public posts and replies that all seem to have the green check, but are not appearing on the Bluesky account.
I actually haven't had anything sync over since I bridged my account like 24 hours ago

Yes! Sorry, reopening. The current Bluesky => fediverse delay is on us, fediverse => Bluesky delay is on Bluesky team. We're (all) working on it. 😬

Thanks for the response! No one is dying, so this is such a non-issue in the grand scheme of things. Hope you feel better :) Thanks for such a cool tool :D

[Daft-Freak]

How much delay should I be expecting on the bluesky side? (I haven't seen a post I've made get through in the last 24 hours and am wondering if I have an actual problem...)

[snarfed]

@Daft-Freak you mean fediverse => Bluesky? They're doing a lot of work on their relay, bluesky-social/atproto#3036 , and they've temporarily paused ingesting commits from federated PDSes like Bridgy Fed. I don't have an ETA for when that will be lifted, sadly.

[Daft-Freak]

Ah okay, good to know!

[snarfed]

https://bsky.app/profile/bnewbold.net/post/3lbimsarmw22g
https://bsky.app/profile/bnewbold.net/post/3lbipqvij7k2m
https://bsky.app/profile/bnewbold.net/post/3lbirrqme2s2w

[wbz0100]

Maybe bsky to fedi has been normalized.
I think, but I don't think the reverse is yet, but I think they've blocked it.

[wbz0100]

Pause again. I don't know what the problem is.

[snarfed]

Bluesky => fediverse has had a couple bumps here and there, but should still be working. Fedi => Bluesky is still paused, sadly, I'm still waiting to hear more from the Bluesky team.

[moyitpro]

Bluesky => fediverse has had a couple bumps here and there, but should still be working. Fedi => Bluesky is still paused, sadly, I'm still waiting to hear more from the Bluesky team.

Just wondering, whatever Bluesky is doing also affect self-hosted PDS as well?

[lucajet]

Bluesky => fediverse has had a couple bumps here and there, but should still be working. Fedi => Bluesky is still paused, sadly, I'm still waiting to hear more from the Bluesky team.

Just wondering, whatever Bluesky is doing also affect self-hosted PDS as well?

I asked a friend with a selfhosted PDS and he said his new posts are seen from bsky.social users.
How is different ?

[snarfed]

Heard from the Bluesky team today, they hope to get this sorted out and fediverse => Bluesky bridging working again sometime this weekend.

Sorry for the downtime, everyone. Thank you for your patience!

[wbz0100]

That's great news! On a slightly different note, it would be nice to have a page that we could monitor to make sure it's working properly... But that would probably be too complicated, right?

[snarfed]

I asked a friend with a selfhosted PDS and he said his new posts are seen from bsky.social users.

True! Some (maybe most?) self-hosted PDSes are indeed working ok. Bridgy Fed is the largest self-hosted PDS, though, and it's over 200x the size of the next largest one: https://blue.mackuba.eu/directory/pdses

In general, the Bluesky team has been firefighting a lot this past week, I can't claim to know exactly what's working or not for them at any given moment.

[snarfed]

@wbz0100 I would love that! I already have decent observability and a dashboard, #1060 (comment) , but sadly it can't be exposed publicly, https://issuetracker.google.com/issues/354101899 . Maybe someday!

Feel free to follow #1546

[jlahtela]

I just set up this bridge and my account is not created for Bluesky yet... Just wondering will it "retry" or so once it starts work again?

[lucajet]

i just saw a 2 days old message of mine published on bsky side.
interesting with a small note

"Archived post
This post claims to have been created on 22 novembre 2024 alle ore 15:48, but was first seen by Bluesky on 24 novembre 2024 alle ore 08:39.
Bluesky cannot confirm the authenticity of the claimed date."

[wbz0100]

Sometimes I'll get one or two, but then I don't hear from them again. I guess they're not collecting them yet.

[lucajet]

Sometimes I'll get one or two, but then I don't hear from them again. I guess they're not collecting them yet.

yeah, thx, i kinda stopped interacting two days ago, no need to grow the queue when i can just pretend i was busy doing something else 🙂

[snarfed]

@jlahtela hopefully, once this is fixed on Bluesky's side, everything that's happened over the last four days will show up in Bluesky, including bridging new accounts. 🤞

[snarfed]

Filed bluesky-social/atproto#3094 to track on Bluesky's side.

[andrewrgross]

This is very interesting to read. I was wondering why I was seeing posts from a few days ago but nothing since. And this makes sense. I think a lot of stuff is popping right now (in a good way!), and I'm not surprised that troubleshooting will be needed.

Thanks for running this and the communication! I honestly feel like this could be a big step on the way to a less crappy internet.