vaapi rotation

[Gusar321]

It's a matter of setting the VADisplayAttribRotation attribute to VA_ROTATION_{NONE,90,180,270}. But I couldn't figure out where exactly to set the attribute and how to handle the width and height swapping required for 90/270 rotation.

[ghost]

I won't implement it. You might want to look at vo_rpi.c for a simple use of the rotation hints.

[Gusar321]

Looking at vo_rpi, I came up with the following for vaapi. The problem is, it also rotates subpictures (osd and subs) and makes a mess of them.

diff --git a/video/out/vo_vaapi.c b/video/out/vo_vaapi.c
index 5275d4d..1a45c10 100644
--- a/video/out/vo_vaapi.c
+++ b/video/out/vo_vaapi.c
@@ -78,6 +78,9 @@ struct priv {
     struct mp_image         *output_surfaces[MAX_OUTPUT_SURFACES];
     struct mp_image         *swdec_surfaces[MAX_OUTPUT_SURFACES];

+    int                      src_w;
+    int                      src_h;
+
     int                      output_surface;
     int                      visible_surface;
     int                      scaling;
@@ -146,6 +149,24 @@ static void resize(struct priv *p)
 {
     vo_get_src_dst_rects(p->vo, &p->src_rect, &p->dst_rect, &p->screen_osd_res);

+    p->src_w = p->src_rect.x1 - p->src_rect.x0;
+    p->src_h = p->src_rect.y1 - p->src_rect.y0;
+
+    int rotate[] = {VA_ROTATION_NONE,
+                    VA_ROTATION_90,
+                    VA_ROTATION_180,
+                    VA_ROTATION_270};
+
+    VADisplayAttribute attr;
+    attr.type = VADisplayAttribRotation;
+    attr.value = rotate[p->vo->params ? p->vo->params->rotate / 90 : 0];
+    vaSetDisplayAttributes(p->display, &attr, 1);
+
+    if (p->vo->params && (p->vo->params->rotate % 180) == 90) {
+        MPSWAP(int, p->src_rect.x0, p->src_rect.y0);
+        MPSWAP(int, p->src_w, p->src_h);
+    }
+
     // It's not clear whether this is needed; maybe not.
     //vo_x11_clearwindow(p->vo, p->vo->x11->window);

@@ -233,8 +254,8 @@ static bool render_to_screen(struct priv *p, struct mp_image *mpi)
                           p->vo->x11->window,
                           p->src_rect.x0,
                           p->src_rect.y0,
-                          p->src_rect.x1 - p->src_rect.x0,
-                          p->src_rect.y1 - p->src_rect.y0,
+                          p->src_w,
+                          p->src_h,
                           p->dst_rect.x0,
                           p->dst_rect.y0,
                           p->dst_rect.x1 - p->dst_rect.x0,
@@ -657,6 +678,7 @@ fail:
 const struct vo_driver video_out_vaapi = {
     .description = "VA API with X11",
     .name = "vaapi",
+    .caps = VO_CAP_ROTATE90,
     .preinit = preinit,
     .query_format = query_format,
     .reconfig = reconfig,

Edit: That was fun... I hacked the libva-intel-driver source to not apply rotation to subpictures. There might be a better way, but oh well, it works...

[ghost]

Can you send a git format-patch or a PR? Then it could be applied.

[Gusar321]

I can make a PR, but this code alone, without hacking the intel driver, the result is not pretty. Is it possible to set the attribute only for video but not subpictures from the mpv code? After I made the driver hack, I stopped looking into it.

[ghost]

Oh, I see. I'm afraid there's no easy way to make libass render in a rotated way. I also do not know of a better way to do OSD with pure vaapi. So maybe this can't be fixed.

[ghost]

It would be nice if a proper solution to this is found, but as it seems as though that is difficult, would it be possible to fallback on another video decoder that does support rotation whenever a video requests it on an unsupported backend?

[ghost]

At this point you can be lucky vo_vaapi is not deleted.

Rotation should work fine with vo_opengl + hwdec=vaapi.