[stdlib][SR-9438] Re-implement integer-to-string conversion

[xwu]

The referenced bug discusses several drawbacks of the status quo, which this PR attempts to address:

• Fast path most common usage, Int64 or UInt64, base 10 or 16
• Outline cold path
• Write contents directly into pre-allocated String buffer, small when appropriate
• Stop relying on C functions _{u}int64ToString, implementing purely in Swift

(The diff is kind of garbage this time; the relevant new code is in a contiguous chunk starting here.) Let's see if this improves performance at all.

Resolves SR-9438.

[xwu]

@swift-ci please smoke test

[xwu]

@swift-ci please smoke test macOS platform

[xwu]

@milseman Hehe, this has uncovered a bug in equality checking for _SmallString:

23:16:55 stdout>>> check failed at /Users/buildnode/jenkins/workspace/swift-PR-macos-smoke-test/branch-main/swift/test/stdlib/PrintInteger.swift, line 42
23:16:55 stdout>>> stacktrace:
23:16:55 stdout>>> #0: /Users/buildnode/jenkins/workspace/swift-PR-macos-smoke-test/branch-main/swift/stdlib/private/StdlibUnittest/StringConvertible.swift:137
23:16:55 stdout>>> expected: any of ["42"]
23:16:55 stdout>>> actual: "42"

I can confirm that, with this patch [edit: without the workaround now applied; see below], I can reduce the problem to literally:

print(String(42)) // "42"
print("42")       // "42"
print("42" == String(42)) // false

I've figured out the problem, I think.

The _unsafeUninitializedCapacity initializer requires users to return the initialized count and to leave the remaining bytes uninitialized. This is what I do here. To wit:

• Bytes are written from the end of the buffer towards the start as the string representation is being built up.
• Once the final byte sequence for the string has been obtained, it is move-initialized to the start of the buffer (as per discussions on the forums, it is permitted to used moveInitialize with overlapping source and destination in this way)
• This leaves the unused bytes uninitialized, as required by the documentation.

All of this works perfectly when the string isn't small. Digging through the sources, I understand that a _SmallString's buffer is backed by a tuple of (UInt64, UInt64). Although unused bytes are deinitialized by me, they aren't zero immediately prior to deinitialization; it should be the job of _SmallString to initialize unused bytes to zero before rebinding the buffer to its original type. (In fact, isn't it undefined behavior to rebind memory and read from it after it's been deinitialized?)

These unused bytes could also be masked before comparison, but that doesn't occur either. Since neither course of action is taken, the result is that two apparently equal strings compared as not equal.

Overall, I think _SmallString is assuming here that leaving the unused bytes uninitialized means leaving them alone entirely, but those aren't the same thing. In this case, those bytes are initialized to certain values, then deinitialized before the initializer returns, and _SmallString doesn't expect that at all.

[Edit]
I'm working around the bug for now by initializing the unused capacity to zero, then deinitializing, but I think the UB issue still applies on _SmallString's end.

[milseman]

Overall, I think _SmallString is assuming here that leaving the unused bytes uninitialized means leaving them alone entirely, but those aren't the same thing. In this case, those bytes are initialized to certain values, then deinitialized before the initializer returns, and _SmallString doesn't expect that at all.

Distinguishing between uninitialized vs 0-initialized is pretty silly for _SmallString, which is designed to primarily live inside registers and only holds trivial values. We should just always fill it with 0s, that is, any _SmallString has no bits set between the last code unit and the discriminator.

CC @atrick for the typed access stuff.

I can confirm that, with this patch [edit: without the workaround now applied; see below], I can reduce the problem to literally...

That is bad. @Catfish-Man , did you encounter this anywhere when you were testing the uninitialized init?

I'll work on adding the all-bits-are-zero-until-discriminator assertion to _invariantCheck and see what I can write that can reproduce it.

[xwu]

Distinguishing between uninitialized vs 0-initialized is pretty silly for _SmallString, which is designed to primarily live inside registers and only holds trivial values. We should just always fill it with 0s, that is, any _SmallString has no bits set between the last code unit and the discriminator.

Agreed about the distinction.

My point here is that the zero-initialization needs to be assured by _SmallString itself after f() has executed here. This is because the user-facing documentation for unsafeUninitializedCapacity tells users they must leave the unused capacity uninitialized, and one legitimate way to assure that is to do whatever you want with the full buffer capacity, then deinitialize the bytes that are unused. The user can't leave the unused capacity zero-initialized themselves after using it because (a) that may not be what a non-small string expects; and (b) the documentation tells the user that they must leave the unused capacity uninitialized.

AFAICT, to fix this bug, it's just a matter of writing (rawPtr + initializedCount).initialize(repeating: 0, count: unusedCapacity) before rebinding the memory to the type of self._storage.

[xwu]

@swift-ci please smoke test macOS platform

[xwu]

@swift-ci please smoke test Linux platform

[xwu]

@swift-ci benchmark

[xwu]

@swift-ci benchmark

[xwu]

@milseman I've reduced and filed the _SmallString bug as SR-14424.

[xwu]

@swift-ci benchmark

[swift-ci]

Build failed before running benchmark.

[xwu]

@swift-ci benchmark

[swift-ci]

Performance: -O

Regression	OLD	NEW	DELTA	RATIO
Dictionary2	810	1045	+29.0%	0.78x
LazilyFilteredArrayContains	14600	17600	+20.5%	0.83x
ArrayAppendLatin1Substring	15912	19152	+20.4%	0.83x (?)
StringFromLongWholeSubstringGeneric	5	6	+20.0%	0.83x
ArrayAppendUTF16Substring	15804	18828	+19.1%	0.84x
ArrayAppendAsciiSubstring	15840	18864	+19.1%	0.84x
RangeIterationSigned	171	200	+17.0%	0.86x
Dictionary2OfObjects	2130	2370	+11.3%	0.90x (?)
StringInterpolationSmall	1900	2060	+8.4%	0.92x (?)
Improvement	OLD	NEW	DELTA	RATIO
IndexPath.Min	510	85	-83.3%	6.00x
IndexPath.Max	506	85	-83.2%	5.95x
FlattenListLoop	2516	1636	-35.0%	1.54x (?)
FlattenListFlatMap	6638	5324	-19.8%	1.25x (?)
DataSetCountMedium	330	270	-18.2%	1.22x
PrefixWhileSequence	386	351	-9.1%	1.10x (?)
ObjectiveCBridgeStubToNSStringRef	119	110	-7.6%	1.08x (?)
DictionaryKeysContainsCocoa	29	27	-6.9%	1.07x (?)
DropLastAnySeqCRangeIter	639	595	-6.9%	1.07x (?)
ObjectiveCBridgeFromNSSetAnyObjectToStringForced	89000	83000	-6.7%	1.07x (?)
RandomShuffleLCG2	480	448	-6.7%	1.07x
Array2D	7216	6736	-6.7%	1.07x (?)
String.replaceSubrange.RepChar	2734	2554	-6.6%	1.07x (?)

Code size: -O

Regression	OLD	NEW	DELTA	RATIO
BitCount.o	1522	2530	+66.2%	0.60x
MonteCarloPi.o	1498	2490	+66.2%	0.60x
SevenBoom.o	1527	2520	+65.0%	0.61x
RangeIteration.o	1527	2504	+64.0%	0.61x
Fibonacci.o	1553	2546	+63.9%	0.61x
NSDictionaryCastToSwift.o	1562	2554	+63.5%	0.61x
Join.o	1577	2576	+63.3%	0.61x
XorLoop.o	1602	2594	+61.9%	0.62x
ByteSwap.o	1616	2616	+61.9%	0.62x
ProtocolDispatch2.o	1625	2623	+61.4%	0.62x
PointerArithmetics.o	1630	2622	+60.9%	0.62x
DeadArray.o	1705	2650	+55.4%	0.64x
Ackermann.o	1828	2821	+54.3%	0.65x
LinkedList.o	2207	3226	+46.2%	0.68x
RecursiveOwnedParameter.o	2150	3138	+46.0%	0.69x
Memset.o	2203	3203	+45.4%	0.69x
Integrate.o	2358	3351	+42.1%	0.70x
ArraySubscript.o	2382	3380	+41.9%	0.70x
Calculator.o	2625	3618	+37.8%	0.73x
PopFrontGeneric.o	2662	3666	+37.7%	0.73x
StrComplexWalk.o	2642	3634	+37.5%	0.73x
MonteCarloE.o	2865	3865	+34.9%	0.74x
ArrayLiteral.o	3125	4139	+32.4%	0.76x
DictionaryBridge.o	3236	4236	+30.9%	0.76x
ChainedFilterMap.o	3189	4141	+29.9%	0.77x
RangeAssignment.o	3369	4369	+29.7%	0.77x
RC4.o	3489	4487	+28.6%	0.78x
NopDeinit.o	3575	4575	+28.0%	0.78x
OpenClose.o	3584	4577	+27.7%	0.78x
PopFront.o	3738	4706	+25.9%	0.79x
ProtocolConformance.o	3967	4967	+25.2%	0.80x
DictionaryOfAnyHashableStrings.o	7824	9696	+23.9%	0.81x
TwoSum.o	4160	5142	+23.6%	0.81x
ObjectAllocation.o	4291	5278	+23.0%	0.81x
StrToInt.o	4615	5615	+21.7%	0.82x
Exclusivity.o	4434	5364	+21.0%	0.83x
Walsh.o	4772	5770	+20.9%	0.83x
StringInterpolation.o	6698	8018	+19.7%	0.84x
DictionaryBridgeToObjC.o	5065	6049	+19.4%	0.84x
BinaryFloatingPointProperties.o	5080	6019	+18.5%	0.84x
RangeReplaceableCollectionPlusDefault.o	5442	6442	+18.4%	0.84x
HTTP2StateMachine.o	5441	6425	+18.1%	0.85x
DictTest2.o	10677	12549	+17.5%	0.85x
StringRemoveDupes.o	5807	6805	+17.2%	0.85x
Combos.o	5974	6974	+16.7%	0.86x
CString.o	6335	7335	+15.8%	0.86x
RomanNumbers.o	6870	7868	+14.5%	0.87x
RangeOverlaps.o	6626	7564	+14.2%	0.88x
SortLettersInPlace.o	8226	9242	+12.4%	0.89x
ObjectiveCNoBridgingStubs.o	8204	9204	+12.2%	0.89x
LazyFilter.o	7944	8886	+11.9%	0.89x
DictionaryCopy.o	8438	9422	+11.7%	0.90x
StringTests.o	7505	8370	+11.5%	0.90x
DictionaryKeysContains.o	8438	9405	+11.5%	0.90x
BucketSort.o	8751	9751	+11.4%	0.90x
DictTest3.o	14973	16613	+11.0%	0.90x
SortIntPyramids.o	9072	10039	+10.7%	0.90x
COWTree.o	11328	12328	+8.8%	0.92x
LuhnAlgoEager.o	11370	12370	+8.8%	0.92x
LuhnAlgoLazy.o	11370	12370	+8.8%	0.92x
Mirror.o	11134	12086	+8.6%	0.92x
DictionaryGroup.o	11735	12687	+8.1%	0.92x
Queue.o	12119	13055	+7.7%	0.93x
DictionaryRemove.o	13288	14256	+7.3%	0.93x
DictTest.o	13564	14532	+7.1%	0.93x
NibbleSort.o	14489	15489	+6.9%	0.94x
DictionaryCompactMapValues.o	13853	14805	+6.9%	0.94x
ObjectiveCBridgingStubs.o	15221	16205	+6.5%	0.94x
ChaCha.o	15699	16699	+6.4%	0.94x
ReduceInto.o	13787	14650	+6.3%	0.94x
RemoveWhere.o	16107	17113	+6.2%	0.94x
DictionarySwap.o	17049	17937	+5.2%	0.95x
SortLargeExistentials.o	19854	20854	+5.0%	0.95x
DictionarySubscriptDefault.o	18775	19679	+4.8%	0.95x
Prefix.o	15720	16464	+4.7%	0.95x
RGBHistogram.o	21138	22106	+4.6%	0.96x
Hash.o	21943	22911	+4.4%	0.96x
DictOfArraysToArrayOfDicts.o	22825	23825	+4.4%	0.96x
Substring.o	21092	22012	+4.4%	0.96x
DropWhile.o	16324	17004	+4.2%	0.96x
ArrayAppend.o	24681	25705	+4.1%	0.96x
PrefixWhile.o	15677	16309	+4.0%	0.96x
Prims.o	25155	26155	+4.0%	0.96x
PrimsSplit.o	25207	26207	+4.0%	0.96x
DropFirst.o	17332	18012	+3.9%	0.96x
SequenceAlgos.o	20860	21604	+3.6%	0.97x
TestsUtils.o	28915	29907	+3.4%	0.97x
StringSplitting.o	35958	36974	+2.8%	0.97x
DropLast.o	21310	21894	+2.7%	0.97x
Suffix.o	21980	22580	+2.7%	0.97x
DictTest4.o	15250	15658	+2.7%	0.97x
WordCount.o	37457	38324	+2.3%	0.98x
DictTest4Legacy.o	15664	16024	+2.3%	0.98x
MapReduce.o	27417	28004	+2.1%	0.98x
CSVParsing.o	55241	56065	+1.5%	0.99x

Performance: -Osize

Regression	OLD	NEW	DELTA	RATIO
MapReduceLazyCollectionShort	64	85	+32.8%	0.75x
Dictionary2	950	1230	+29.5%	0.77x
StringInterpolationSmall	1800	2170	+20.6%	0.83x
StringFromLongWholeSubstringGeneric	5	6	+20.0%	0.83x
PointerArithmetics	31400	37100	+18.2%	0.85x
StringEqualPointerComparison	171	200	+17.0%	0.86x
Join	222	257	+15.8%	0.86x (?)
ConvertFloatingPoint.MockFloat64Exactly	8	9	+12.5%	0.89x (?)
Dictionary2OfObjects	2475	2760	+11.5%	0.90x (?)
BinaryFloatingPointPropertiesBinade	28	31	+10.7%	0.90x (?)
DropWhileAnyCollectionLazy	276	305	+10.5%	0.90x (?)
PrefixWhileAnyCollectionLazy	176	194	+10.2%	0.91x
ConvertFloatingPoint.MockFloat64Exactly2	12	13	+8.3%	0.92x (?)
ClosedRangeOverlapsClosedRange	89	96	+7.9%	0.93x (?)
Improvement	OLD	NEW	DELTA	RATIO
IndexPath.Max	511	85	-83.4%	6.01x
IndexPath.Min	510	85	-83.3%	6.00x
DataSetCountMedium	330	280	-15.2%	1.18x (?)
ObjectiveCBridgeStubToNSStringRef	131	113	-13.7%	1.16x (?)
ObjectiveCBridgeStubDateAccess	257	228	-11.3%	1.13x
DropWhileAnySeqCRangeIterLazy	305	276	-9.5%	1.11x
DropWhileAnySeqCntRangeLazy	305	276	-9.5%	1.11x (?)
DropFirstAnySeqCRangeIter	199	181	-9.0%	1.10x
DropFirstAnySeqCntRange	199	181	-9.0%	1.10x
DropLastCountableRange	12	11	-8.3%	1.09x (?)
ObjectiveCBridgeFromNSArrayAnyObjectForced	5020	4620	-8.0%	1.09x (?)
Fibonacci	201	185	-8.0%	1.09x
RandomShuffleLCG2	448	416	-7.1%	1.08x (?)
DictionaryKeysContainsCocoa	28	26	-7.1%	1.08x (?)
Array2D	7520	6992	-7.0%	1.08x (?)
FindString.Loop1.Substring	499	465	-6.8%	1.07x

Code size: -Osize

Regression	OLD	NEW	DELTA	RATIO
MonteCarloPi.o	1319	2259	+71.3%	0.58x
XorLoop.o	1368	2308	+68.7%	0.59x
RangeIteration.o	1395	2334	+67.3%	0.60x
Fibonacci.o	1392	2328	+67.2%	0.60x
NSDictionaryCastToSwift.o	1436	2376	+65.5%	0.60x
Join.o	1567	2585	+65.0%	0.61x
BitCount.o	1464	2404	+64.2%	0.61x
ByteSwap.o	1492	2432	+63.0%	0.61x
PointerArithmetics.o	1536	2476	+61.2%	0.62x
SevenBoom.o	1558	2498	+60.3%	0.62x
Ackermann.o	1662	2602	+56.6%	0.64x
DeadArray.o	1569	2450	+56.2%	0.64x
ProtocolDispatch2.o	1873	2813	+50.2%	0.67x
Memset.o	1899	2835	+49.3%	0.67x
LinkedList.o	2055	2995	+45.7%	0.69x
Integrate.o	2187	3123	+42.8%	0.70x
ArraySubscript.o	2409	3349	+39.0%	0.72x
Calculator.o	2440	3384	+38.7%	0.72x
StrComplexWalk.o	2590	3530	+36.3%	0.73x
TwoSum.o	2593	3520	+35.8%	0.74x
PopFrontGeneric.o	2711	3651	+34.7%	0.74x
ArrayLiteral.o	2794	3738	+33.8%	0.75x
MonteCarloE.o	2826	3753	+32.8%	0.75x
RC4.o	3054	3994	+30.8%	0.76x
DictionaryOfAnyHashableStrings.o	5386	7004	+30.0%	0.77x
ChainedFilterMap.o	3059	3968	+29.7%	0.77x
DictionaryBridge.o	3226	4172	+29.3%	0.77x
RangeAssignment.o	3225	4165	+29.1%	0.77x
PopFront.o	3429	4343	+26.7%	0.79x
OpenClose.o	3629	4569	+25.9%	0.79x
ProtocolConformance.o	3714	4651	+25.2%	0.80x
StringRemoveDupes.o	3746	4690	+25.2%	0.80x
ObjectAllocation.o	4068	4980	+22.4%	0.82x
NopDeinit.o	4255	5195	+22.1%	0.82x
Walsh.o	4209	5136	+22.0%	0.82x
DictTest2.o	7924	9639	+21.6%	0.82x
Exclusivity.o	4049	4925	+21.6%	0.82x
DictionaryBridgeToObjC.o	4200	5108	+21.6%	0.82x
StrToInt.o	4404	5344	+21.3%	0.82x
RangeReplaceableCollectionPlusDefault.o	4715	5655	+19.9%	0.83x
StringInterpolation.o	6473	7647	+18.1%	0.85x
RomanNumbers.o	5216	6156	+18.0%	0.85x
HTTP2StateMachine.o	5118	6037	+18.0%	0.85x
BinaryFloatingPointProperties.o	5091	5972	+17.3%	0.85x
CString.o	5847	6787	+16.1%	0.86x
Combos.o	5976	6916	+15.7%	0.86x
RangeOverlaps.o	5602	6483	+15.7%	0.86x
DictionaryCopy.o	7170	8079	+12.7%	0.89x
ObjectiveCNoBridgingStubs.o	7619	8562	+12.4%	0.89x
SortLettersInPlace.o	7628	8568	+12.3%	0.89x
LazyFilter.o	7280	8161	+12.1%	0.89x
StringTests.o	6848	7665	+11.9%	0.89x
DictionaryKeysContains.o	7758	8653	+11.5%	0.90x
BucketSort.o	8336	9274	+11.3%	0.90x
SortIntPyramids.o	8922	9831	+10.2%	0.91x
ReduceInto.o	8187	9003	+10.0%	0.91x
DictionaryGroup.o	10299	11266	+9.4%	0.91x
Mirror.o	9882	10794	+9.2%	0.92x
DictionaryRemove.o	10579	11507	+8.8%	0.92x
DictTest.o	10682	11602	+8.6%	0.92x
COWTree.o	10968	11908	+8.6%	0.92x
RemoveWhere.o	13419	14448	+7.7%	0.93x
LuhnAlgoEager.o	12327	13267	+7.6%	0.93x
LuhnAlgoLazy.o	12327	13267	+7.6%	0.93x
Queue.o	11993	12874	+7.3%	0.93x
ChaCha.o	12714	13642	+7.3%	0.93x
DictionaryCompactMapValues.o	12248	13130	+7.2%	0.93x
DictTest3.o	12212	13058	+6.9%	0.94x
NibbleSort.o	14157	15103	+6.7%	0.94x
ObjectiveCBridgingStubs.o	13796	14710	+6.6%	0.94x
DictionarySubscriptDefault.o	13600	14477	+6.4%	0.94x
DictionarySwap.o	15254	16100	+5.5%	0.95x
DictTest4.o	11383	11942	+4.9%	0.95x
RGBHistogram.o	19296	20231	+4.8%	0.95x
DictTest4Legacy.o	11622	12181	+4.8%	0.95x
ArrayAppend.o	22193	23246	+4.7%	0.95x
Hash.o	18449	19323	+4.7%	0.95x
DictOfArraysToArrayOfDicts.o	20147	21085	+4.7%	0.96x
SortLargeExistentials.o	20469	21411	+4.6%	0.96x
PrefixWhile.o	13807	14442	+4.6%	0.96x
Substring.o	19239	20105	+4.5%	0.96x
Prefix.o	14936	15608	+4.5%	0.96x
TestsUtils.o	23152	24169	+4.4%	0.96x
DropFirst.o	15451	16089	+4.1%	0.96x
Prims.o	22899	23838	+4.1%	0.96x
PrimsSplit.o	22951	23890	+4.1%	0.96x
DropWhile.o	15400	15980	+3.8%	0.96x
DropLast.o	19190	19768	+3.0%	0.97x
MapReduce.o	20371	20935	+2.8%	0.97x
StringSplitting.o	35171	36140	+2.8%	0.97x
SequenceAlgos.o	20492	21042	+2.7%	0.97x
WordCount.o	35449	36255	+2.3%	0.98x
Suffix.o	24751	25308	+2.3%	0.98x
CSVParsing.o	52214	53144	+1.8%	0.98x

Performance: -Onone

Regression	OLD	NEW	DELTA	RATIO
Join	185	255	+37.8%	0.73x
Dictionary2	1555	1865	+19.9%	0.83x (?)
NSStringConversion.LongUTF8	1527	1696	+11.1%	0.90x (?)
StringWordBuilderReservingCapacity	2630	2900	+10.3%	0.91x (?)
StringWordBuilder	2680	2950	+10.1%	0.91x (?)
StringInterpolation	12500	13700	+9.6%	0.91x (?)
NSStringConversion.Rebridge	454	490	+7.9%	0.93x (?)
Improvement	OLD	NEW	DELTA	RATIO
IndexPath.Min	661	252	-61.9%	2.62x
IndexPath.Max	657	253	-61.5%	2.60x
String.data.Medium	183	163	-10.9%	1.12x (?)
NSStringConversion.MutableCopy.Long	1821	1626	-10.7%	1.12x (?)
FloatingPointPrinting_Double_interpolated	80200	71800	-10.5%	1.12x (?)
FloatingPointPrinting_Float80_interpolated	116200	106600	-8.3%	1.09x (?)
DataSetCountMedium	1110	1020	-8.1%	1.09x (?)
NSStringConversion.MutableCopy.Medium	1600	1478	-7.6%	1.08x (?)
ObjectiveCBridgeFromNSSetAnyObjectToStringForced	92500	85500	-7.6%	1.08x (?)

Code size: -swiftlibs

Regression	OLD	NEW	DELTA	RATIO
libswiftSwiftPrivateLibcExtras.dylib	16384	32768	+100.0%	0.50x
Improvement	OLD	NEW	DELTA	RATIO
libswiftFoundation.dylib	1294336	1245184	-3.8%	1.04x

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

[milseman]

Small string fix is in #36667

[xwu]

Closing in favor of #85180