[SPARK-53972][SS] Fix streaming query recentProgress regression in classic pyspark

### What changes were proposed in this pull request?
See #52688 for details. This PR is to backport the changes to Spark 4.0

### Why are the changes needed?

See #52688

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

See #52688

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #52698 from zifeif2/recent-progress-bug-4.0.

Authored-by: Ubuntu <[email protected]>
Signed-off-by: Anish Shrigondekar <[email protected]>

Author: Ubuntu

python/pyspark/sql/streaming/query.py

@@ -283,7 +283,10 @@ def recentProgress(self) -> List[StreamingQueryProgress]:
 
         >>> sq.stop()
         """
-        return [StreamingQueryProgress.fromJObject(p) for p in self._jsq.recentProgress()]
+        return [
+            StreamingQueryProgress.fromJson(json.loads(p.json()))
+            for p in self._jsq.recentProgress()
+        ]
 
     @property
     def lastProgress(self) -> Optional[StreamingQueryProgress]:
@@ -314,7 +317,7 @@ def lastProgress(self) -> Optional[StreamingQueryProgress]:
         """
         lastProgress = self._jsq.lastProgress()
         if lastProgress:
-            return StreamingQueryProgress.fromJObject(lastProgress)
+            return StreamingQueryProgress.fromJson(json.loads(lastProgress.json()))
         else:
             return None