View previous topic :: View next topic |
Author |
Message |
Marcofras n00b
Joined: 29 May 2023 Posts: 26
|
Posted: Sat Nov 16, 2024 9:13 pm Post subject: parallalization and taskset |
|
|
is it possible have parallalization with taskset?
Code: |
m@localhost ~/Desktop/bitmap $ time taskset -c 1 ./desc && time taskset -c 2 ./desc && time taskset -c 3 ./desc && time taskset -c 4 ./desc
real 0m0.019s
user 0m0.013s
sys 0m0.006s
real 0m0.014s
user 0m0.013s
sys 0m0.001s
real 0m0.012s
user 0m0.011s
sys 0m0.001s
real 0m0.012s
user 0m0.012s
sys 0m0.000s
|
four instance of the program are executed simultaniously with save time? |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 22680
|
Posted: Sat Nov 16, 2024 11:32 pm Post subject: |
|
|
Why are you using taskset at all? If you want four instances, run four instances: Code: | for a in {0..3}; do ./desc & done; wait | Let the scheduler sort out where to run them. |
|
Back to top |
|
|
Marcofras n00b
Joined: 29 May 2023 Posts: 26
|
Posted: Sun Nov 17, 2024 11:07 am Post subject: |
|
|
With your method i have no guarante its parallelized in different core with taskset instead i have guarante. Or not? |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 22680
|
Posted: Sun Nov 17, 2024 3:48 pm Post subject: |
|
|
With taskset, you guarantee that each task either runs on the specified core or, if that core is not available, does not run at all. Without taskset, you let the scheduler pick which core to run each one on, and it will try to run as many things in parallel as you have cores available. Why do you think you need to force particular core assignments for particular processes? |
|
Back to top |
|
|
Marcofras n00b
Joined: 29 May 2023 Posts: 26
|
Posted: Sun Nov 17, 2024 4:45 pm Post subject: |
|
|
Are you sure that its sufficient & for have parallalization?
I read somewhere that its not sure this |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 22680
|
Posted: Sun Nov 17, 2024 5:09 pm Post subject: |
|
|
& tells the shell to place the command in the background. Where did you read that this would not work? Have you tried the command I gave you? What went wrong? |
|
Back to top |
|
|
Marcofras n00b
Joined: 29 May 2023 Posts: 26
|
Posted: Sun Nov 17, 2024 6:31 pm Post subject: |
|
|
Im under c++ and that command no works, i use a series of system(./desc &) or something similar is it correct? Will it works? |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 22680
|
Posted: Sun Nov 17, 2024 7:39 pm Post subject: |
|
|
If you are using system from C++, why is your opening post expressed as shell? Rather than asking if something will work after I told you it will, why not try it and report whether it works?
I think you should provide a Minimum Reproducible Example demonstrating your problem. |
|
Back to top |
|
|
Marcofras n00b
Joined: 29 May 2023 Posts: 26
|
Posted: Sun Nov 17, 2024 7:48 pm Post subject: |
|
|
Indeed i was wrong im tryng in shell before c++, dont you think its the same? I dont make c++ code yet, do you think system(./desc &) will work and save time with parallelization i wish? |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 22680
|
Posted: Sun Nov 17, 2024 8:00 pm Post subject: |
|
|
I gave you a loop that should work. I later reiterated that I expect it to work. Yes, system should work fine, since, according to its manual page, that is delegating back to the shell, which is almost always the wrong thing to do in a C++ program. If you need further help, please demonstrate a specific problem. |
|
Back to top |
|
|
Marcofras n00b
Joined: 29 May 2023 Posts: 26
|
Posted: Sun Nov 17, 2024 8:31 pm Post subject: |
|
|
I will make the code c++ in days.
Why its wrong thing which is the correct form? Maybe thread or something similar? |
|
Back to top |
|
|
GDH-gentoo Veteran
Joined: 20 Jul 2019 Posts: 1703 Location: South America
|
Posted: Sun Nov 17, 2024 8:45 pm Post subject: |
|
|
Because a C++ program can do things like spawning processes by using the libc directly (or through some convenience library) instead of invoking the shell. _________________
NeddySeagoon wrote: | I'm not a witch, I'm a retired electronics engineer |
Ionen wrote: | As a packager I just don't want things to get messier with weird build systems and multiple toolchains requirements though |
Last edited by GDH-gentoo on Sun Nov 17, 2024 8:46 pm; edited 1 time in total |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3435
|
Posted: Sun Nov 17, 2024 8:45 pm Post subject: |
|
|
Marcofras, you're bouncing around too much. Pause for a minute and focus.
What are you _actually_ trying to do? _________________ Make Computing Fun Again |
|
Back to top |
|
|
Marcofras n00b
Joined: 29 May 2023 Posts: 26
|
Posted: Mon Nov 18, 2024 11:04 am Post subject: |
|
|
parallalization, because i had no success with thread (in the past) im opting to divide the program in several subprograms and run parallelized, it seems more easy. |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3435
|
Posted: Mon Nov 18, 2024 12:02 pm Post subject: |
|
|
Eh... This is clearly a case of https://en.wikipedia.org/wiki/XY_problem.
Regarding your original question:
> is it possible have parallalization with taskset?
The only correct answer is: "Yes but no". Unfortunately, on top of being correct, it is also completely useless.
Parallelization where? Why did you start with shell and then you're suddenly going to cpp? This part is confusing.
More importantly: What is the thing you're trying to write there? What is the problem you're trying to solve? What is standing in your way?
> I think you should provide a Minimum Reproducible Example demonstrating your problem.
Yeah, I second that. Although this technique is primarily used for reporting bugs, it does help with communication in general.
Overkill is still a kill. _________________ Make Computing Fun Again |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 22680
|
Posted: Mon Nov 18, 2024 12:30 pm Post subject: |
|
|
Yes, a Minimum Reproducible Example might be overkill here. I asked for it because OP's questions implied to me that he had directly experienced shell background not to exhibit the desired parallelism that I had stated would work. If he was experiencing that lack, we could try to understand why. If he has not tried the solutions given to him, I would like for him to do so or explain why not (particularly since this should be easy to test, due to being independent of the content of ./desc), before we put more work into assuring him again that his doubts are misplaced. |
|
Back to top |
|
|
Marcofras n00b
Joined: 29 May 2023 Posts: 26
|
Posted: Mon Nov 18, 2024 2:41 pm Post subject: |
|
|
the original problem is i need parallalization for save time, i tried thread in desc that parallelize the function shap but it doesn' t work and freeze in ruuntime
Code: |
thread th1(shap, px, 1);
thread th2(shap, px, 2);
thread th3(shap, px, 3);
thread th4(shap, px, 4);
thread th5(shap, px, 5);
thread th6(shap, px, 6);
thread th7(shap, px, 7);
thread th8(shap, px, 8);
thread th9(shap, px, 9);
thread th10(shap, px, 10);
thread th11(shap, px, 11);
thread th12(shap, px, 12);
thread th13(shap, px, 13);
thread th14(shap, px, 14);
thread th15(shap, px, 15);
thread th16(shap, px, 16);
thread th17(shap, px, 17);
thread th18(shap, px, 18);
th1.join();
th2.join();
th3.join();
th4.join();
th5.join();
th6.join();
th7.join();
th8.join();
th9.join();
th10.join();
th11.join();
th12.join();
th13.join();
th14.join();
th15.join();
th16.join();
th17.join();
th18.join();
|
i cant reveal other for the code because its a big stuff.
So, i opted for build several indipendent program and launch with taskset or something similar (&) but you all tell me its not correct. |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 22680
|
Posted: Mon Nov 18, 2024 2:59 pm Post subject: |
|
|
That does not appear to be a MRE. None of those identifiers are defined, and it is not legal to use a statement at global scope to call join. A Minimal Reproducible Example does not need to be your full and secret code, and by its very definition, generally should not be. We want it to be Minimal. Sharing your full secret code is very likely not minimal.
What I told you is that using system from C++ is "almost always the wrong thing to do." GDH-gentoo then elaborated my point for you.
Putting all that together, I will again ask you to show us a MRE that exhibits your problem. As a counter to your claim that this hangs, I offer this MRE that is not defective. It exactly mirrors the fragment you showed, and fills in the missing bits as needed to make it compile. t.cpp: | // g++-14 -O2 -Wall -Wextra t.cpp
#include <thread>
#include <cstdio>
static void shap(int, int seq) {
printf("Thread %i ...\n", seq);
}
int main()
{
const int px = 1;
printf("Starting threads ...\n");
std::thread th1(shap, px, 1);
std::thread th2(shap, px, 2);
std::thread th3(shap, px, 3);
std::thread th4(shap, px, 4);
std::thread th5(shap, px, 5);
std::thread th6(shap, px, 6);
std::thread th7(shap, px, 7);
std::thread th8(shap, px, 8);
std::thread th9(shap, px, 9);
std::thread th10(shap, px, 10);
std::thread th11(shap, px, 11);
std::thread th12(shap, px, 12);
std::thread th13(shap, px, 13);
std::thread th14(shap, px, 14);
std::thread th15(shap, px, 15);
std::thread th16(shap, px, 16);
std::thread th17(shap, px, 17);
std::thread th18(shap, px, 18);
printf("Joining threads ...\n");
th1.join();
th2.join();
th3.join();
th4.join();
th5.join();
th6.join();
th7.join();
th8.join();
th9.join();
th10.join();
th11.join();
th12.join();
th13.join();
th14.join();
th15.join();
th16.join();
th17.join();
th18.join();
printf("Done\n");
} | Possible output (depending on scheduling): Code: | $ ./a.out
Starting threads ...
Thread 1 ...
Thread 2 ...
Thread 3 ...
Thread 4 ...
Thread 5 ...
Thread 6 ...
Thread 7 ...
Thread 8 ...
Thread 9 ...
Thread 10 ...
Thread 11 ...
Thread 12 ...
Thread 13 ...
Thread 14 ...
Thread 15 ...
Thread 16 ...
Thread 17 ...
Joining threads ...
Thread 18 ...
Done | From this, using threads and joining them is not inherently wrong. If your program hangs, it must be because some part you refused to share is incorrect. |
|
Back to top |
|
|
Marcofras n00b
Joined: 29 May 2023 Posts: 26
|
Posted: Mon Nov 18, 2024 3:23 pm Post subject: |
|
|
i made the same thing of you but it freeze |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3435
|
Posted: Mon Nov 18, 2024 3:40 pm Post subject: |
|
|
Works for me.
Adding a 1 sec sleep shows the parallel execution even better. If we delay print, we get to see the order randomized by scheduler due to system load:
Code: | $ ./a.out
Starting threads ...
Joining threads ...
Thread 1 ...
Thread 15 ...
Thread 5 ...
Thread 6 ...
Thread 4 ...
Thread 2 ...
Thread 10 ...
Thread 7 ...
Thread 18 ...
Thread 13 ...
Thread 14 ...
Thread 3 ...
Thread 9 ...
Thread 8 ...
Thread 11 ...
Thread 16 ...
Thread 12 ...
Thread 17 ...
Done
|
Meanwhile, running it through taskset to lock all threads to a single core results in almost all single-digit tasks running before almost all double-digits, almost every time. The output produced by threads running concurrently on a single core is far less random than output from parallel threads running on 4 cores. _________________ Make Computing Fun Again |
|
Back to top |
|
|
Marcofras n00b
Joined: 29 May 2023 Posts: 26
|
Posted: Mon Nov 18, 2024 3:45 pm Post subject: |
|
|
the order is not important in my case i want use system() and shell its more easy for me.
suggestions? |
|
Back to top |
|
|
GDH-gentoo Veteran
Joined: 20 Jul 2019 Posts: 1703 Location: South America
|
Posted: Mon Nov 18, 2024 3:55 pm Post subject: |
|
|
Marcofras wrote: | the order is not important in my case i want use system() and shell its more easy for me.
suggestions? |
Yes. Don't just do things in a program "because it is easier", learn to do paralellization properly in C++. Threads are difficult because concurrent access to shared memory can create all sorts of problems without doing it properly, but I'm not convinced so far that you can't just spawn processes with standard POSIX function calls to solve your problem, for example, whatever that problem is. _________________
NeddySeagoon wrote: | I'm not a witch, I'm a retired electronics engineer |
Ionen wrote: | As a packager I just don't want things to get messier with weird build systems and multiple toolchains requirements though |
|
|
Back to top |
|
|
Genone Retired Dev
Joined: 14 Mar 2003 Posts: 9612 Location: beyond the rim
|
Posted: Mon Nov 18, 2024 3:59 pm Post subject: |
|
|
Have you looked into just using fork() for your issue?
Using system() is almost always a bad idea. |
|
Back to top |
|
|
Marcofras n00b
Joined: 29 May 2023 Posts: 26
|
Posted: Mon Nov 18, 2024 4:14 pm Post subject: |
|
|
i dont knew fork i read it can call a program from the filesystem, will it be parallellized with time saving?
i dont understand well how to use:
Code: |
int id = fork(prog);
|
?
do i have to put fork in the begin of function shap? |
|
Back to top |
|
|
pingtoo Veteran
Joined: 10 Sep 2021 Posts: 1249 Location: Richmond Hill, Canada
|
Posted: Mon Nov 18, 2024 4:33 pm Post subject: |
|
|
Marcofras wrote: | i dont knew fork i read it can call a program from the filesystem, will it be parallellized with time saving?
i dont understand well how to use:
Code: |
int id = fork(prog);
|
? | Where did you read lead you to think "fork" can call a program? I ask because this is not true.
If you don't mind please share some information about your experience in programming C language in general and more specifically on programming C language on Linux.
I found in this thread you have been given out advise in various degree of knowledge on programming on Linux. I think it could lead to confuse because these advises assume different level of understanding.
Your question in "parallelism" assume it will have time saving. this is not always true in my experience. So it will also be nice if you can share more information in the problem space. For example how large is the to be process data set. And where the data set located. (i.e. locally on the process machine or remotely in a database). |
|
Back to top |
|
|
|