.. Copyright (C) 2018 Nikunj Gupta Copyright (C) 2007-2017 Hartmut Kaiser SPDX-License-Identifier: BSL-1.0 Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) .. _starting_hpx: ========================== Starting the |hpx| runtime ========================== In order to write an application that uses services from the |hpx| runtime system, you need to initialize the |hpx| library by inserting certain calls into the code of your application. Depending on your use case, this can be done in 3 different ways: * :ref:`Minimally invasive `: Re-use the ``main()`` function as the main |hpx| entry point. * :ref:`Balanced use case `: Supply your own main |hpx| entry point while blocking the main thread. * :ref:`Most flexibility `: Supply your own main |hpx| entry point while avoiding blocking the main thread. * :ref:`Suspend and resume `: As above but suspend and resume the |hpx| runtime to allow for other runtimes to be used. .. _minimal: Re-use the ``main()`` function as the main |hpx| entry point ============================================================ This method is the least intrusive to your code. However, it provides you with the smallest flexibility in terms of initializing the |hpx| runtime system. The following code snippet shows what a minimal |hpx| application using this technique looks like: .. code-block:: c++ #include int main(int argc, char* argv[]) { return 0; } The only change to your code you have to make is to include the file ``hpx/hpx_main.hpp``. In this case the function ``main()`` will be invoked as the first |hpx| thread of the application. The runtime system will be initialized behind the scenes before the function ``main()`` is executed and will automatically stop after ``main()`` has returned. For this method to work you must link your application to the |cmake| target ``HPX::wrap_main``. This is done automatically if you are using the provided macros (:ref:`using_hpx_cmake_macros`) to set up your application, but must be done explicitly if you are using targets directly (:ref:`using_hpx_cmake_targets`). All |hpx| API functions can be used from within the ``main()`` function now. .. note:: The function ``main()`` does not need to expect receiving ``argc`` and ``argv`` as shown above, but could expose the signature ``int main()``. This is consistent with the usually allowed prototypes for the function ``main()`` in C++ applications. All command line arguments specific to |hpx| will still be processed by the |hpx| runtime system as usual. However, those command line options will be removed from the list of values passed to ``argc``/\ ``argv`` of the function ``main()``. The list of values passed to ``main()`` will hold only the commandline options that are not recognized by the |hpx| runtime system (see the section :ref:`commandline` for more details on what options are recognized by |hpx|). .. note:: In this mode all one-letter shortcuts that are normally available on the |hpx| command line are disabled (such as ``-t`` or ``-l`` see :ref:`commandline`). This is done to minimize any possible interaction between the command line options recognized by the |hpx| runtime system and any command line options defined by the application. The value returned from the function ``main()`` as shown above will be returned to the operating system as usual. .. important:: To achieve this seamless integration, the header file ``hpx/hpx_main.hpp`` defines a macro:: #define main hpx_startup::user_main which could result in unexpected behavior. .. important:: To achieve this seamless integration, we use different implementations for different operating systems. In case of Linux or macOS, the code present in ``hpx_wrap.cpp`` is put into action. We hook into the system function in case of Linux and provide alternate entry point in case of macOS. For other operating systems we rely on a macro:: #define main hpx_startup::user_main provided in the header file ``hpx/hpx_main.hpp``. This implementation can result in unexpected behavior. .. caution:: We make use of an *override* variable ``include_libhpx_wrap`` in the header file ``hpx/hpx_main.hpp`` to swiftly choose the function call stack at runtime. Therefore, the header file should *only* be included in the main executable. Including it in the components will result in multiple definition of the variable. .. _medium: Supply your own main |hpx| entry point while blocking the main thread ===================================================================== With this method you need to provide an explicit main-thread function named ``hpx_main`` at global scope. This function will be invoked as the main entry point of your |hpx| application on the console :term:`locality` only (this function will be invoked as the first |hpx| thread of your application). All |hpx| API functions can be used from within this function. The thread executing the function :cpp:func:`hpx::init` will block waiting for the runtime system to exit. The value returned from ``hpx_main`` will be returned from :cpp:func:`hpx::init` after the runtime system has stopped. The function :cpp:func:`hpx::finalize` has to be called on one of the |hpx| localities in order to signal that all work has been scheduled and the runtime system should be stopped after the scheduled work has been executed. This method of invoking |hpx| has the advantage of the user being able to decide which version of :cpp:func:`hpx::init` to call. This allows to pass additional configuration parameters while initializing the |hpx| runtime system. .. code-block:: c++ #include int hpx_main(int argc, char* argv[]) { // Any HPX application logic goes here... return hpx::finalize(); } int main(int argc, char* argv[]) { // Initialize HPX, run hpx_main as the first HPX thread, and // wait for hpx::finalize being called. return hpx::init(argc, argv); } .. note:: The function ``hpx_main`` does not need to expect receiving ``argc``/``argv`` as shown above, but could expose one of the following signatures:: int hpx_main(); int hpx_main(int argc, char* argv[]); int hpx_main(hpx::program_options::variables_map& vm); This is consistent with (and extends) the usually allowed prototypes for the function ``main()`` in C++ applications. The header file to include for this method of using |hpx| is ``hpx/hpx_init.hpp``. There are many additional overloads of :cpp:func:`hpx::init` available, such as the ability to provide your own entry-point function instead of ``hpx_main``. Please refer to the function documentation for more details (see: ``hpx/hpx_init.hpp``). .. _flexible: Supply your own main |hpx| entry point while avoiding blocking the main thread ============================================================================== With this method you need to provide an explicit main thread function named ``hpx_main`` at global scope. This function will be invoked as the main entry point of your |hpx| application on the console :term:`locality` only (this function will be invoked as the first |hpx| thread of your application). All |hpx| API functions can be used from within this function. The thread executing the function :cpp:func:`hpx::start` will *not* block waiting for the runtime system to exit, but will return immediately. The function :cpp:func:`hpx::finalize` has to be called on one of the |hpx| localities in order to signal that all work has been scheduled and the runtime system should be stopped after the scheduled work has been executed. This method of invoking |hpx| is useful for applications where the main thread is used for special operations, such a GUIs. The function :cpp:func:`hpx::stop` can be used to wait for the |hpx| runtime system to exit and should at least be used as the last function called in ``main()``. The value returned from ``hpx_main`` will be returned from :cpp:func:`hpx::stop` after the runtime system has stopped. .. code-block:: c++ #include int hpx_main(int argc, char* argv[]) { // Any HPX application logic goes here... return hpx::finalize(); } int main(int argc, char* argv[]) { // Initialize HPX, run hpx_main. hpx::start(argc, argv); // ...Execute other code here... // Wait for hpx::finalize being called. return hpx::stop(); } .. note:: The function ``hpx_main`` does not need to expect receiving ``argc``/``argv`` as shown above, but could expose one of the following signatures:: int hpx_main(); int hpx_main(int argc, char* argv[]); int hpx_main(hpx::program_options::variables_map& vm); This is consistent with (and extends) the usually allowed prototypes for the function ``main()`` in C++ applications. The header file to include for this method of using |hpx| is ``hpx/hpx_start.hpp``. There are many additional overloads of :cpp:func:`hpx::start` available, such as the option for users to provide their own entry point function instead of ``hpx_main``. Please refer to the function documentation for more details (see: ``hpx/hpx_start.hpp``). .. _explicit_startup_function: Supply your own explicit startup function as the main |hpx| entry point ======================================================================= There is also a way to specify any function (besides ``hpx_main``) to be used as the main entry point for your |hpx| application: .. code-block:: c++ #include int application_entry_point(int argc, char* argv[]) { // Any HPX application logic goes here... return hpx::finalize(); } int main(int argc, char* argv[]) { // Initialize HPX, run application_entry_point as the first HPX thread, // and wait for hpx::finalize being called. return hpx::init(&application_entry_point, argc, argv); } .. note:: The function supplied to :cpp:func:`hpx::init` must have one of the following prototypes: int application_entry_point(int argc, char* argv[]); int application_entry_point(hpx::program_options::variables_map& vm); .. note:: If ``nullptr`` is used as the function argument, |hpx| will not run any startup function on this locality. .. _suspend_resume: Suspending and resuming the |hpx| runtime ========================================= In some applications it is required to combine |hpx| with other runtimes. To support this use case, |hpx| provides two functions: :cpp:func:`hpx::suspend` and :cpp:func:`hpx::resume`. :cpp:func:`hpx::suspend` is a blocking call which will wait for all scheduled tasks to finish executing and then put the thread pool OS threads to sleep. :cpp:func:`hpx::resume` simply wakes up the sleeping threads so that they are ready to accept new work. :cpp:func:`hpx::suspend` and :cpp:func:`hpx::resume` can be found in the header ``hpx/hpx_suspend.hpp``. .. code-block:: c++ #include #include int main(int argc, char* argv[]) { // Initialize HPX, don't run hpx_main hpx::start(nullptr, argc, argv); // Schedule a function on the HPX runtime hpx::post(&my_function, ...); // Wait for all tasks to finish, and suspend the HPX runtime hpx::suspend(); // Execute non-HPX code here // Resume the HPX runtime hpx::resume(); // Schedule more work on the HPX runtime // hpx::finalize has to be called from the HPX runtime before hpx::stop hpx::post([]() { hpx::finalize(); }); return hpx::stop(); } .. note:: :cpp:func:`hpx::suspend` does not wait for :cpp:func:`hpx::finalize` to be called. Only call :cpp:func:`hpx::finalize` when you wish to fully stop the |hpx| runtime. .. warning:: :cpp:func:`hpx::suspend` only waits for local tasks, i.e. tasks on the current locality, to finish executing. When using :cpp:func:`hpx::suspend` in a multi-locality scenario the user is responsible for ensuring that any work required from other localities has also finished. |hpx| also supports suspending individual thread pools and threads. For details on how to do that, see the documentation for :cpp:class:`hpx::threads::thread_pool_base`. Automatically suspending worker threads --------------------------------------- The previous method guarantees that the worker threads are suspended when you ask for it and that they stay suspended. An alternative way to achieve the same effect is to tweak how quickly |hpx| suspends its worker threads when they run out of work. The following configuration values make sure that |hpx| idles very quickly: .. code-block:: ini hpx.max_idle_backoff_time = 1000 hpx.max_idle_loop_count = 0 They can be set on the command line using ``--hpx:ini=hpx.max_idle_backoff_time=1000`` and ``--hpx:ini=hpx.max_idle_loop_count=0``. See :ref:`launching_and_configuring` for more details on how to set configuration parameters. After setting idling parameters the previous example could now be written like this instead: .. code-block:: c++ #include int main(int argc, char* argv[]) { // Initialize HPX, don't run hpx_main hpx::start(nullptr, argc, argv); // Schedule some functions on the HPX runtime // NOTE: run_as_hpx_thread blocks until completion. hpx::run_as_hpx_thread(&my_function, ...); hpx::run_as_hpx_thread(&my_other_function, ...); // hpx::finalize has to be called from the HPX runtime before hpx::stop hpx::post([]() { hpx::finalize(); }); return hpx::stop(); } In this example each call to :cpp:func:`hpx::run_as_hpx_thread` acts as a "parallel region". .. _hpx_main_implementation: Working of ``hpx_main.hpp`` =========================== In order to initialize |hpx| from ``main()``, we make use of linker tricks. It is implemented differently for different operating systems. The method of implementation is as follows: * :ref:`Linux `: Using linker ``--wrap`` option. * :ref:`Mac OSX `: Using the linker ``-e`` option. * :ref:`Windows `: Using ``#define main hpx_startup::user_main`` .. _hpx_main_implementation_linux: Linux implementation -------------------- We make use of the Linux linker ``ld``\ 's ``--wrap`` option to wrap the ``main()`` function. This way any calls to ``main()`` are redirected to our own implementation of main. It is here that we check for the existence of ``hpx_main.hpp`` by making use of a shadow variable ``include_libhpx_wrap``. The value of this variable determines the function stack at runtime. The implementation can be found in ``libhpx_wrap.a``. .. important:: It is necessary that ``hpx_main.hpp`` be not included more than once. Multiple inclusions can result in multiple definition of ``include_libhpx_wrap``. .. _hpx_main_implementation_osx: Mac OSX implementation ---------------------- Here we make use of yet another linker option ``-e`` to change the entry point to our custom entry function ``initialize_main``. We initialize the |hpx| runtime system from this function and call main from the initialized system. We determine the function stack at runtime by making use of the shadow variable ``include_libhpx_wrap``. The implementation can be found in ``libhpx_wrap.a``. .. important:: It is necessary that ``hpx_main.hpp`` be not included more than once. Multiple inclusions can result in multiple definition of ``include_libhpx_wrap``. .. _hpx_main_implementation_windows: Windows implementation ---------------------- We make use of a macro ``#define main hpx_startup::user_main`` to take care of the initializations. This implementation could result in unexpected behaviors.