Date: Accept all valid timezones from client, allow sub-second precision #5950

psafont · 2024-08-23T14:22:35Z

The Date module has a very convoluted way to deal with timezones because of its historic clients. While we can't remove all the issues, we can remove most of them.

Dates (timestamps, really) without timezones do not contain a frame of reference, and hence placing them in the UTC timeline can't be done accurately in the general case. This means that comparing timestamps gives incorrect results.
XMLRPC enforces dates to be encoded in ISO8601format. This means timestamps can lack a timezone.
Xapi uses xmlrpc's dates, adding this unfortunate limitation.
While Date allows these timestamps, it assumes that these are in the UTC timezone. On top of that, it refuses to process any timestamps that are in any timezone other than UTC (Z)
The Date module tries really hard to hide the timezone of timestamps that lack it when printing them. This means that timezoneless timestamps can persist in the database, for no good reason, as they are treated as UTC timestamps.
Host.localtime is the only call that returns timezoneless timestamps, all the other calls correctly return timestamps in the UTC timezone. Because the call on purpose does not want to return the current time in UTC, changing this might break clients not ready to process any timezone, mainly SDK-built ones CA-397409: Extend SDK deserialization support for xen-api dates #5802
Dates are stored as a tuple of date, hour, minutes, seconds. This has very limited precision, which might be unexpected.

This PR does the following mitigation / fixes:

Document all calls with Datetimes as parameters that the timestamps will be interpreted as UTC ones if they miss the timezone.
Remove the limitation to process any valid timezone
Timezoneless timstamps are immediately processed as UTC timestamps, as refusing these timestamps can break clients.

Issues that the PR does not fix

Host.localtime produces timestamps without timezones, this is needed as adding non-zero timestamps breaks the SDK clients.
The server does not reject timezoneless timestamp, this might break migrations, RPUs, or normal clients, so I've held back on this change.
Printed timestamp do not retain sub-second precision

Drafting as tests are ongoing

last-genius · 2024-08-23T14:49:12Z

ocaml/idl/datamodel_types.ml

@@ -26,12 +26,12 @@

 *)
 module Date = struct
-  open Xapi_stdext_date
+  module Date = Xapi_stdext_date.Date
  include Date


Would it be worth it removing this include to only be able to access Date explicitly?

include Date add the whole signature of Date to the current module, so I believe it's needed, unless I missed something.

lindig · 2024-08-23T15:35:45Z

ocaml/idl/datamodel.ml

      ; param_release= tampa_release
-      ; param_default= Some (VDateTime Date.never)
+      ; param_default= Some (VDateTime Date.epoch)


Is this just a different name but the same underlying date? Epoch is in the past but never could have been in the future.

It might be better to capture the intended meaning using never as an alias.

Date.never is 0. I don't know why None isn't used for many of these, currently epoch has the same semantic meaning as None in many of these. I couldn't find any instance where the 0 / never value means something special, it's for users to understand that when they see 1970 it means never, which is awkward.

None for the default value means that the parameter will not have a default value. But sometimes it is necessary for a parameter to have a default value (for backwards compatability). But yeah, I think 1970 is kind of awkward as a sign of "never", hopefully this value will actually gets filled when it is quried

ocaml/libs/clock/date.ml

Vincent-lau · 2024-08-28T12:57:45Z

ocaml/idl/datamodel.ml

      ; param_release= tampa_release
-      ; param_default= Some (VDateTime Date.never)
+      ; param_default= Some (VDateTime Date.epoch)


None for the default value means that the parameter will not have a default value. But sometimes it is necessary for a parameter to have a default value (for backwards compatability). But yeah, I think 1970 is kind of awkward as a sign of "never", hopefully this value will actually gets filled when it is quried

ocaml/idl/datamodel_types.ml

Vincent-lau · 2024-08-28T13:52:43Z

ocaml/libs/clock/date.ml

@@ -152,19 +162,19 @@ let is_later ~than t = Ptime.is_later ~than:(to_ptime than) (to_ptime t)

 let diff a b = Ptime.diff (to_ptime a) (to_ptime b)

-let compare_print_tz a b =
+let compare_tz a b =


so this function makes input with timezones always bigger than those that does not? Is it possible to for a to contain a negative number such as Some -1?

That's correct, remember that not having a timezone is not the same as being in the UTC. The order is arbitrary.

emm, I am a bit confused, so in the doc of the parameter you said when no timezones are provided, UTC is assumed. So do we assume UTC when there are no time zones?

In the dates that are read from strings, but those do have a timezone now. The ones that are produced with localtime do not. (we should be able to change that soon, when the SDK accepts any valid timezone)

ocaml/libs/clock/date.mli

ocaml/idl/datamodel.ml

psafont · 2024-08-29T13:10:19Z

BST is now passing with the regression fixed around the dates in xapi_xenopsd. This happened because now dates can be more precise than seconds, but dates stored in the database are clamped to full second, making the comparison unstable.

I tried finding other comparisons like this one by looking into how the Datetime fields are used, but couldn't find any other like this one. In particular, Task has a lot of Datetime fields.

psafont · 2024-09-13T12:38:55Z

The Java SDK failure happens on master as well, it's unrelated to this PR.
XenRT tests are looking green, this is good to go.

And replace its users Signed-off-by: Pau Ruiz Safont <[email protected]>

Previously the comments and code talked about "timezone printing", but that is only a consequence of a more fundamental problem: datetimes without timezone do not share a frame of reference with datetimes that have a timezone, nor among themselves. This means that timezoneless shouldn't be compared with other timezones as assuming a timezone can be incorrect. Datetimes with timezone all share a frame of reference: the unix epoch. Now the code uses the exact offset in seconds, which also enables it to accept datetimes that have an offset different than 0. Comparisons with timezoneless datetimes are not forbidden for the time being. Instead UTC is assumed, like before, even if it might be the wrong guess. Signed-off-by: Pau Ruiz Safont <[email protected]>

ocaml/idl/datamodel_types.ml

ocaml/libs/clock/date.ml

This allows to maintain sub-second precision, and only convert to datetimes when converting to strings. Now timezone conversion is more explicit. Datetimes without timezone are assumed to be UTC. While rare in practice, this is not safe in general, so print a warning to stdout whenever this happens. Signed-off-by: Pau Ruiz Safont <[email protected]>

Datetimes received as call parameters can lack timezone. In that case the host assumed the datetime is in UTC. This is not always safe to do, but returning an error will break clients. Instead keep doing the assumption and enforce proper documentation of the parameters in the datamodel. Signed-off-by: Pau Ruiz Safont <[email protected]>

Signed-off-by: Pau Ruiz Safont <[email protected]>

Previously the dates without timezone were assumed to use UTC when being converted to unix time, but the datatype maintained the lack of timezone when being printed. This means that the problematic timezoneless timestamps were kept and could be produced by the hosts. Since the dates are assumed to be UTC, add the timezone when they are parsed for the first time. Now their timezone is displayed at all times. The output of Localtime is kept without a timezone because clients are not prepared to accept arbitrary timezones in dates. (Both the SDK and XO(Lite)) Signed-off-by: Pau Ruiz Safont <[email protected]>

psafont changed the title ~~Date: Accept all valid timezones from client, add sub-second precision~~ Date: Accept all valid timezones from client, allow sub-second precision Aug 23, 2024

psafont force-pushed the private/paus/pdates branch from 2a8daf6 to e41943e Compare August 23, 2024 14:36

last-genius reviewed Aug 23, 2024

View reviewed changes

lindig reviewed Aug 23, 2024

View reviewed changes

psafont force-pushed the private/paus/pdates branch from 433de8a to 1d8874d Compare August 27, 2024 08:25

Vincent-lau reviewed Aug 28, 2024

View reviewed changes

psafont force-pushed the private/paus/pdates branch from 645f423 to 648be29 Compare August 29, 2024 13:18

psafont force-pushed the private/paus/pdates branch 3 times, most recently from 905f8e6 to 27cc91c Compare September 13, 2024 12:30

psafont marked this pull request as ready for review September 13, 2024 12:38

lindig approved these changes Sep 13, 2024

View reviewed changes

psafont added 2 commits September 13, 2024 14:58

xapi-stdext-date: Mark deprecated bindings as such

347f6e3

And replace its users Signed-off-by: Pau Ruiz Safont <[email protected]>

psafont force-pushed the private/paus/pdates branch from 27cc91c to 9d76583 Compare September 13, 2024 13:58

edwintorok reviewed Sep 16, 2024

View reviewed changes

ocaml/idl/datamodel_types.ml Show resolved Hide resolved

edwintorok reviewed Sep 16, 2024

View reviewed changes

ocaml/libs/clock/date.ml Outdated Show resolved Hide resolved

edwintorok approved these changes Sep 16, 2024

View reviewed changes

psafont added 4 commits September 16, 2024 15:12

datamodel: document how timezone-less datetimes are handled

0f9b1df

Signed-off-by: Pau Ruiz Safont <[email protected]>

psafont force-pushed the private/paus/pdates branch from d68f9f3 to 96195f7 Compare September 16, 2024 14:12

psafont enabled auto-merge September 16, 2024 14:13

psafont disabled auto-merge September 16, 2024 14:17

psafont enabled auto-merge September 16, 2024 14:17

psafont disabled auto-merge September 16, 2024 14:17

psafont added this pull request to the merge queue Sep 16, 2024

Merged via the queue into xapi-project:master with commit 1d32682 Sep 16, 2024
15 checks passed

psafont deleted the private/paus/pdates branch September 16, 2024 14:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Date: Accept all valid timezones from client, allow sub-second precision #5950

Date: Accept all valid timezones from client, allow sub-second precision #5950

psafont commented Aug 23, 2024 •

edited

Loading

last-genius Aug 23, 2024 •

edited

Loading

psafont Aug 23, 2024

lindig Aug 23, 2024

lindig Aug 23, 2024

psafont Aug 23, 2024 •

edited

Loading

Vincent-lau Aug 28, 2024

Vincent-lau Aug 28, 2024

Vincent-lau Aug 28, 2024

psafont Aug 28, 2024

Vincent-lau Aug 29, 2024

psafont Aug 29, 2024

psafont commented Aug 29, 2024

psafont commented Sep 13, 2024 •

edited

Loading

Date: Accept all valid timezones from client, allow sub-second precision #5950

Date: Accept all valid timezones from client, allow sub-second precision #5950

Conversation

psafont commented Aug 23, 2024 • edited Loading

last-genius Aug 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

psafont Aug 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

psafont commented Aug 29, 2024

psafont commented Sep 13, 2024 • edited Loading

psafont commented Aug 23, 2024 •

edited

Loading

last-genius Aug 23, 2024 •

edited

Loading

psafont Aug 23, 2024 •

edited

Loading

psafont commented Sep 13, 2024 •

edited

Loading