As pointed out by several users, introduction of parallel operations on
vectors severely impacts solver performance on small problems, with time
consumption increasing with the number of threads.
In order to minimize overhead of trying to execute small tasks using a
large number of threads, task scheduling mechanism was changed to avoid
scheduling all tasks at once.
However, there is still a large difference in exectuion time because the
main thread always launches the next thread before starting doing the
work. This leads to several orders of magnitude slowdown when going from
a single-threaded execution (which follows a fast-forward path to a
single loop over all indices, without any synchronization involved)
to a two-thread execution:
/bin/parallel_vector_operations_benchmark
-------------------------------------------
Benchmark Time
-------------------------------------------
SetZero/128 12.8 ns
SetZeroParallel/128/1 16.6 ns
SetZeroParallel/128/2 2211 ns
In order to eliminate this effect, we limit the block-size of parallel
execution of vector operations to 2^16 elements (thus, starting parallel
execution only for vectors of at least 2^17 elements).
Threshold of 2^16 elements was choosen by evaluating thresholds from
2^10 to 2^20 (only powers of 2), with 2^14..2^20 significantly reducing
worst-case runtime degradation.
Details can be found in discussion of the issue at
https://github.com/ceres-solver/ceres-solver/issues/1016
Change-Id: I555c882d63ee53323ceb426743b970f989b65503
1. Add a version history
2. Update copyright years across the code base
3. Run format_all.sh
4. Update version strings from 2.1.0 to 2.2.0 in the docs and
elsewhere.
Change-Id: I46d8d479d54bd6002d532785e67342106e73c9ac
Main focus of this change is to parallelize remaining operations (most of them
are operations on vectors) in code-path utilized with iterative Schur
complement.
Parallelization is handled using lazy evaluation of Eigen expressions.
On linux pc with intel 8176 processor parallelization of vector operations has
the following effect:
Running ./bin/parallel_vector_operations_benchmark
Run on (112 X 3200.32 MHz CPU s)
CPU Caches:
L1 Data 32 KiB (x56)
L1 Instruction 32 KiB (x56)
L2 Unified 1024 KiB (x56)
L3 Unified 39424 KiB (x2)
Load Average: 3.30, 8.41, 11.82
-----------------------------------
Benchmark Time
-----------------------------------
SetZero 10009532 ns
SetZeroParallel/1 10024139 ns
...
SetZeroParallel/16 877606 ns
Negate 4978856 ns
NegateParallel/1 5145413 ns
...
NegateParallel/16 721823 ns
Assign 10731408 ns
AssignParallel/1 10749944 ns
...
AssignParallel/16 1829381 ns
D2X 15214399 ns
D2XParallel/1 15623245 ns
...
D2XParallel/16 2687060 ns
DivideSqrt 8220050 ns
DivideSqrtParallel/1 9088467 ns
...
DivideSqrtParallel/16 905569 ns
Clamp 3502010 ns
ClampParallel/1 4507897 ns
...
ClampParallel/16 759576 ns
Norm 4426782 ns
NormParallel/1 4442805 ns
...
NormParallel/16 430290 ns
Dot 9023276 ns
DotParallel/1 9031304 ns
...
DotParallel/16 1157267 ns
Axpby 14608289 ns
AxpbyParallel/1 14570825 ns
...
AxpbyParallel/16 2672220 ns
-----------------------------------
Multi-threading of vector operations in ISC and program evaluation results into
the following improvement:
Running ./bin/evaluation_benchmark
--------------------------------------------------------------------------------------
Benchmark this 2fd81de
--------------------------------------------------------------------------------------
Residuals<problem-13682-4456117-pre.txt>/1 4136 ms 4292 ms
Residuals<problem-13682-4456117-pre.txt>/2 2919 ms 2670 ms
Residuals<problem-13682-4456117-pre.txt>/4 2065 ms 2198 ms
Residuals<problem-13682-4456117-pre.txt>/8 1458 ms 1609 ms
Residuals<problem-13682-4456117-pre.txt>/16 1152 ms 1227 ms
ResidualsAndJacobian<problem-13682-4456117-pre.txt>/1 19759 ms 20084 ms
ResidualsAndJacobian<problem-13682-4456117-pre.txt>/2 10921 ms 10977 ms
ResidualsAndJacobian<problem-13682-4456117-pre.txt>/4 6220 ms 6941 ms
ResidualsAndJacobian<problem-13682-4456117-pre.txt>/8 3490 ms 4398 ms
ResidualsAndJacobian<problem-13682-4456117-pre.txt>/16 2277 ms 3172 ms
Plus<problem-13682-4456117-pre.txt>/1 339 ms 322 ms
Plus<problem-13682-4456117-pre.txt>/2 220 ms
Plus<problem-13682-4456117-pre.txt>/4 128 ms
Plus<problem-13682-4456117-pre.txt>/8 78.0 ms
Plus<problem-13682-4456117-pre.txt>/16 49.8 ms
ISCRightMultiplyAndAccumulate<problem-13682-4456117-pre.txt>/1 2434 ms 2478 ms
ISCRightMultiplyAndAccumulate<problem-13682-4456117-pre.txt>/2 2706 ms 2688 ms
ISCRightMultiplyAndAccumulate<problem-13682-4456117-pre.txt>/4 1430 ms 1548 ms
ISCRightMultiplyAndAccumulate<problem-13682-4456117-pre.txt>/8 742 ms 883 ms
ISCRightMultiplyAndAccumulate<problem-13682-4456117-pre.txt>/16 438 ms 555 ms
ISCRightMultiplyAndAccumulateDiag<problem-13682-4456117-pre.txt>/1 2438 ms 2481 ms
ISCRightMultiplyAndAccumulateDiag<problem-13682-4456117-pre.txt>/2 2565 ms 2790 ms
ISCRightMultiplyAndAccumulateDiag<problem-13682-4456117-pre.txt>/4 1434 ms 1551 ms
ISCRightMultiplyAndAccumulateDiag<problem-13682-4456117-pre.txt>/8 765 ms 892 ms
ISCRightMultiplyAndAccumulateDiag<problem-13682-4456117-pre.txt>/16 435 ms 559 ms
JacobianSquaredColumnNorm<problem-13682-4456117-pre.txt>/1 1278 ms
JacobianSquaredColumnNorm<problem-13682-4456117-pre.txt>/2 1555 ms
JacobianSquaredColumnNorm<problem-13682-4456117-pre.txt>/4 833 ms
JacobianSquaredColumnNorm<problem-13682-4456117-pre.txt>/8 459 ms
JacobianSquaredColumnNorm<problem-13682-4456117-pre.txt>/16 250 ms
JacobianScaleColumns<problem-13682-4456117-pre.txt>/1 1468 ms
JacobianScaleColumns<problem-13682-4456117-pre.txt>/2 1871 ms
JacobianScaleColumns<problem-13682-4456117-pre.txt>/4 957 ms
JacobianScaleColumns<problem-13682-4456117-pre.txt>/8 528 ms
JacobianScaleColumns<problem-13682-4456117-pre.txt>/16 294 ms
End-to-end improvements with bundle_adjuster invoked with
./bin/bundle_adjuster --num_threads 28 --num_iterations 40 \
--linear_solver iterative_schur \
--preconditioner jacobi --input
---------------------------------------------
Problem this 2fd81de
---------------------------------------------
problem-13682-4456117-pre.txt 508.6 892.7
problem-1778-993923-pre.txt 763.8 1129.9
problem-1723-156502-pre.txt 6.3 14.4
problem-356-226730-pre.txt 76.3 116.2
problem-257-65132-pre.txt 38.6 52.0
Change-Id: Ie31cc5015f13fa479c16ffb5ce48c9b880990d49
Since c++11, we can depend on C++ threads always being available.
With the recent work on the performance of CXX threading, the
additional complexity of maintaining multiple backends for some
minor performance delta is not worth it
https://github.com/ceres-solver/ceres-solver/issues/886
Change-Id: Idee480b22a498daec9c4366da8589aa58eaf36a1
The constructor now takes a LinearSolver::Options as input
and uses that to compute the partitioning once and uses it
for its lifetime.
Change-Id: I9ef30df0b60f8fa91c8b5601c397b2d9314a2cc7
Parallel for with user-supplied [cumulative] iteration costs allows to
get performance improvements on problems with significantly different
time requirements per parallel loop iteration.
One of those problems is left multiplication with block-sparse matrix.
Using number of non-zero values per column block, we partition column
blocks into contiguous sets with approximately equal number of
operations to be performed.
Change-Id: I4a862a10a586cdfbec22e8168a3423537039abc2
Implemented templated invocation routines for ParallelFor backends
in order to improve loop body inlining.
Several modifications of ParallelFor implementation using CXX threads:
- Index order changed from interleaved to sequential
- Static task scheduling replaced with dynamic (controlled by
kWorkBlocksPerThread)
- Changed index retrieval to atomic
Modifications of OpenMP backend:
- Changed loop scheduling to guided
Changing index order from interleaved to sequential in parallel seem
to significantly improve run-times of parallel loops, for example in
evaluation of jacobian and residuals.
Other modifications provide minor improvements for unbalanced
sub-problem lengths and parallel for loops with small number of
computation per operation.
Single-threaded performance was improved by avoiding costs of
wrapping parallel loop bodies in std::function.
On BAL dataset the following improvements in time consumed for
evaluation of residuals or jacobian and residuals were observed:
OLD NEW OLD/NEW
dataset threads r J r J r J
problem-257-65132-pre.txt 1 0.025 0.079 0.025 0.074 1.016 1.056
problem-257-65132-pre.txt 2 0.030 0.062 0.022 0.050 1.333 1.246
problem-257-65132-pre.txt 4 0.023 0.052 0.014 0.034 1.592 1.515
problem-257-65132-pre.txt 8 0.015 0.035 0.010 0.025 1.477 1.401
problem-257-65132-pre.txt 16 0.011 0.027 0.008 0.019 1.365 1.377
problem-356-226730-pre.txt 1 0.150 0.442 0.147 0.412 1.017 1.070
problem-356-226730-pre.txt 2 0.155 0.322 0.100 0.281 1.542 1.145
problem-356-226730-pre.txt 4 0.129 0.291 0.089 0.196 1.439 1.485
problem-356-226730-pre.txt 8 0.091 0.184 0.066 0.139 1.381 1.319
problem-356-226730-pre.txt 16 0.070 0.148 0.055 0.110 1.272 1.340
problem-1723-156502-pre.txt 1 0.084 0.243 0.082 0.229 1.023 1.063
problem-1723-156502-pre.txt 2 0.088 0.188 0.055 0.154 1.589 1.222
problem-1723-156502-pre.txt 4 0.072 0.159 0.049 0.108 1.475 1.475
problem-1723-156502-pre.txt 8 0.050 0.105 0.037 0.077 1.348 1.368
problem-1723-156502-pre.txt 16 0.038 0.083 0.030 0.062 1.269 1.344
problem-1778-993923-pre.txt 1 0.621 1.777 0.609 1.667 1.018 1.065
problem-1778-993923-pre.txt 2 0.621 1.273 0.415 1.199 1.494 1.061
problem-1778-993923-pre.txt 4 0.514 1.140 0.361 0.786 1.421 1.449
problem-1778-993923-pre.txt 8 0.365 0.808 0.277 0.559 1.319 1.443
problem-1778-993923-pre.txt 16 0.279 0.608 0.223 0.441 1.252 1.379
problem-13682-4456117-pre.txt 1 3.877 10.726 3.738 10.082 1.037 1.063
problem-13682-4456117-pre.txt 2 3.310 7.170 2.423 6.448 1.366 1.111
problem-13682-4456117-pre.txt 4 3.070 6.344 2.064 4.474 1.486 1.417
problem-13682-4456117-pre.txt 8 2.051 4.612 1.527 3.133 1.343 1.472
problem-13682-4456117-pre.txt 16 1.549 3.453 1.218 2.488 1.271 1.387
Run time in seconds for a single evaluation, using evaluation_benchmark
numactl -N 0 -m 0 ./bin/evaluation_benchmark --bal_root ${path_to_BAL}
Evaluation was performed on 28-core CPU.
Note: performance when running across numa-nodes degrades in both old
and proposed implementations, thus the test was executed limiting memory
and compute resources allocation to a single numa-node.
Change-Id: Ia195580bdab9d05c95ac983bfe37b045eecfaf49
covariance.h was using SUITE_SPARSE even when SUITESPARSE
was disabled because it did not have config.h included in it
so it did not see that CERES_NO_SUITESPARSE was defined.
Add more config.h includes to files that are using these
configuration macros.
Change-Id: I6b1d2c2bd9e559de40a6332cd6be85ad4da3377b
Currently, the logic for exporting symbols is rather complicated: when
tests are enabled internal symbols are exported in addition to the
public symbols. Such logic causes several problems. (1) Test binaries
link against a Ceres build that is different from the final release
since fewer optimizations are applied if more symbols are exported. (2)
Also, some toolchains hide symbols by default breaking the existing
logic eventually causing linker errors.
Since internal symbols are not intended to be used outside of the
project, we can compile them into object files and use exactly the same
binary code both for the final build and the tests without relying on
conditionals.
By default, all symbols are now hidden unless annotated as public.
Internal symbols are explicitly marked as not being exported in case
users chose not to hide symbols by default.
Change-Id: I589dd10be2f6f438508783cf99d141af0120057b
- Previously we were only bounding num_threads_used based on whether
CERES_NO_THREADS was defined, meaning that we could erroneously report
a value larger than the number of threads actually used.
Change-Id: I7373c0c968f9be268c8b7ab0b9561ae31700fda6
Previously, the thread ID was acquired and released on every iteration
of the for loop. The C++11 concurrent queue implementation is much
slower than TBB's version and consequently this was a huge bottleneck.
This introduces another ParallelFor API which takes the thread ID as a
parameter in the evaluation function. This allows us to acquire and
release the thread ID for each block of work which drastically improves
the performance.
This change brings us on par with OpenMP and TBB. See below for a
timing comparison. Note: in this example this CLs C++11 version is
faster to compute the residuals because TBB still must acquire the
thread ID on every iteration, which has some overhead.
Tested by building and running tests for no threading, OpenMP, TBB, and
C++11 threads. Also ran bazel tests.
./bin/bundle_adjuster --input=problem-744-543562-pre.txt --num_threads=8
C++11 @Head
Time (in seconds):
Residual only evaluation 7.819692 (5)
Jacobian & residual evaluation 11.606063 (6)
Linear solver 47.860195 (5)
Minimizer 70.877072
Total 90.806338
---------------------------------------------------
C++11 (This CL)
Time (in seconds):
Residual only evaluation 1.217500 (5)
Jacobian & residual evaluation 5.796112 (6)
Linear solver 44.080873 (5)
Minimizer 54.635524
Total 77.640072
---------------------------------------------------
OpenMP
Time (in seconds):
Residual only evaluation 0.797023 (5)
Jacobian & residual evaluation 5.633916 (6)
Linear solver 43.280020 (5)
Minimizer 53.199058
Total 76.250861
---------------------------------------------------
TBB
Time (in seconds):
Residual only evaluation 1.911095 (5)
Jacobian & residual evaluation 5.557807 (6)
Linear solver 44.074680 (5)
Minimizer 55.002688
Total 78.052687
---------------------------------------------------
No Threads
Time (in seconds):
Residual only evaluation 2.939212 (5)
Jacobian & residual evaluation 18.519874 (6)
Linear solver 74.017837 (5)
Minimizer 98.980080
Total 122.216391
Change-Id: I3af959b0771bbdfe8cad8c13896191d6ac903181
A Ceres Context holds common global state that can be re-used within
Ceres. The Context current contains a thread pool if compiling with
C++11 threading support. Threads are expensive to create and destroy so
it is good to maintain across multiple Ceres solves.
Tested by compiling with and without TBB support and ran unit tests. Ran
bazel as well.
Change-Id: I82f598dfae642aa0e81a6039dc174608a5e8dbfb
This is in preparation for adding support for a c++11 based parallel
for implementation. The parallel for abstraction does not have the
ability to constrain the total number of threads in nested for loops.
This is solved by distributing the number of threads evenly between
the nested for loops. Adds a TODO to consolidate the next for loops
into a single loop that can be properly split between threads.
Tested by building with TBB and running tests.
Change-Id: I546973b9a4d19b9cdd53caff55d1c80bac8ea953
Dmitriy Korchemkin