When updating a (partially) sorted model, take the known runs for the
existing sort and apply them to the new sort. That way, we don't have to
check the whole model again.
Benchmarks:
appending half the items to a model of strings
old new
512,000 items 437ms 389ms
1,024,000 items 1006ms 914ms
appending 10% of the items to a model of strings
old new
512,000 items 206ms 132ms
1,024,000 items 438ms 301ms
appending 1 item to a model of strings
old new
64,000 items 1.8ms 0.00ms
512,000 items --- 0.01ms
In particular, add a GtkTreeModel of GtkDirectoryList that enumerates
the full tree of everything below G_TEST_SRCDIR.
You can set it to ~ or / to get lots of files.
Use this to test numeric sorting, string sorting, the treelistrow
sorter and the multisorter.
And while doing all of that, try to use something that is still
realistic.
* "max time"
Reports the max time taken for all main loop iterations + initial
set.
This is useful to judge the performance of incremental search models.
* "changes"
Reports the sum of all changes during items-changed signal emissions.
Uses MAX(added, removed) as the number to not double-count items that
were moved.
This is useful to judge the accuracy of the models by comparing the
number among them.
Previously, the sort was not stable when items were added/removed while
sorting or the sort algorithm was changed.
Now the sort looks at the item position (via the key's location in the
keys array) to make sure each comparison stays stable with respect to
this position.
This massively speeds up sorting with expensive sort functions that it's
the most worthwhile optimization of this whole branch.
It's slower for simple sort functions though.
It's also quite a lot slower when the model doesn't support sort keys
(like GtkCustomSorter), but all the other sorters do support keys.
Of course, this depends on the number of items in the model - the number
of comparisons scales O(N * log N) while the overhead for key handling
scales O(N).
So as the log N part grows, generating keys gets more and more
beneficial.
Benchmarks:
initial sort of a GFileInfo model with display-name keys
items keys
8,000 items 715ms 50ms
64,000 items --- 554ms
initial sort of a GFileInfo model with complex keys
items keys
64,000 items 340ms 295ms
128,000 items 641ms 605ms
removing half a GFileInfo model with display-name keys
(no comparisons, just key freeing overhead of a complex sorter)
items keys
512,000 items 14ms 21ms
2,048,000 items 40ms 62ms
removing half a GFileInfo model with complex keys
(no comparisons, just key freeing overhead of a complex sorter)
items keys
512,000 items 90ms 237ms
2,048,000 items 247ms 601ms
GtkSortKeys is an immutable struct that can be used to manage "sort
keys" for items.
Sort keys are memory that is created specifically for sorting. Because
sorting involves lots of comparisons, it's a good idea to prepare the
data relevant for sorting in advance and sort on that data.
In measurements with a PropertyExpression on a string sorter, it's about
??? faster
Instead of one item keeping the item + its position and sorting that
list, keep the items in 1 array and put the positions into a 2nd array.
This is generally slower while sorting, but allows multiple improvements:
1. We can replace items with keys
This allows avoiding multiple slow lookups when using complex
comparisons
2. We can keep multiple position arrays
This allows doing a sorting in the background without actually
emitting items-changed() until the array is completely sorted.
3. The main list tracks the items in the original model
So only a single memmove() is necessary there, while the old version
had to upgrade the position in every item.
Benchmarks:
sorting a model of simple strings
old new
256,000 items 256ms 268ms
512,000 items 569ms 638ms
sorting a model of file trees, directories first, by size
old new
64,000 items 350ms 364ms
128,000 items 667ms 691ms
removing half the model
old new
512,000 items 24ms 15ms
1,024,000 items 49ms 25ms
1. Run step() for a while to avoid very short steps
This way, we batch items-changed() emissions.
2. Track the change region accurately
This way, we can avoid invalidating the whole list if our step just
touched a small part of a huge list.
As this is a merge sort, this is a common occurence when we're buys
merging chunks: The rest of the model outside those chunks isn't
changed.
Note that the tracking is accurate: It determines the minimum change
region in the model.
This will be important, because the testsuite is going to test this.
... and use it in the SortListModel
Setting runs allows declaring already sorted regions so the sort does
not attempt to sort them again.
This massively speeds up partial inserts where we can reuse the sorted
model as a run and only resort the newly inserted parts.
Benchmarks:
appending half the model
qsort timsort
128,000 items 94ms 69ms
256,000 items 202ms 143ms
512,000 items 488ms 328ms
appending 1 item
qsort timsort
8,000 items 1.5ms 0.0ms
16,000 items 3.1ms 0.0ms
...
512,000 items --- 1.8ms
Simply replace the old qsort() call with a timsort() call.
This is ultimately relevant because timsort is a LOT faster in merging
to already sorted lists (think items-chaged adding some items) or
reversing an existing list (think columnview sort order changes).
Benchmarks:
initially sorting the model
qsort timsort
128,000 items 124ms 111ms
256,000 items 264ms 250ms
The model now tracks the original positions on top of just the items so that
it can remove items in an items-changed emission.
It now takes twice as much memory but removes items much faster.
Benchmarks:
Removing 50% of a model:
before after
250,000 items 135ms 10ms
500,000 items 300ms 25ms
Removing 1 item:
4,000 items 2.2ms 0ms
8,000 items 4.6ms 0ms
500,000 items --- 0.01ms
This is the dumbest possible sortmodel using an array:
Just grab all the items, put them in the array, qsort() the array.
Some benchmarks (setting a new model):
125,000 items - old: 549ms
new: 115ms
250,000 items - new: 250ms
This performance can not be kept for simple additions and removals
though.
