Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
parallalization and taskset
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
Marcofras
n00b
n00b


Joined: 29 May 2023
Posts: 26

PostPosted: Sat Nov 16, 2024 9:13 pm    Post subject: parallalization and taskset Reply with quote

is it possible have parallalization with taskset?
Code:

m@localhost ~/Desktop/bitmap $ time taskset -c 1 ./desc  && time taskset -c 2 ./desc  && time taskset -c 3 ./desc  && time taskset -c 4 ./desc

real   0m0.019s
user   0m0.013s
sys   0m0.006s

real   0m0.014s
user   0m0.013s
sys   0m0.001s

real   0m0.012s
user   0m0.011s
sys   0m0.001s

real   0m0.012s
user   0m0.012s
sys   0m0.000s


four instance of the program are executed simultaniously with save time?
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22680

PostPosted: Sat Nov 16, 2024 11:32 pm    Post subject: Reply with quote

Why are you using taskset at all? If you want four instances, run four instances:
Code:
for a in {0..3}; do ./desc & done; wait
Let the scheduler sort out where to run them.
Back to top
View user's profile Send private message
Marcofras
n00b
n00b


Joined: 29 May 2023
Posts: 26

PostPosted: Sun Nov 17, 2024 11:07 am    Post subject: Reply with quote

With your method i have no guarante its parallelized in different core with taskset instead i have guarante. Or not?
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22680

PostPosted: Sun Nov 17, 2024 3:48 pm    Post subject: Reply with quote

With taskset, you guarantee that each task either runs on the specified core or, if that core is not available, does not run at all. Without taskset, you let the scheduler pick which core to run each one on, and it will try to run as many things in parallel as you have cores available. Why do you think you need to force particular core assignments for particular processes?
Back to top
View user's profile Send private message
Marcofras
n00b
n00b


Joined: 29 May 2023
Posts: 26

PostPosted: Sun Nov 17, 2024 4:45 pm    Post subject: Reply with quote

Are you sure that its sufficient & for have parallalization?
I read somewhere that its not sure this
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22680

PostPosted: Sun Nov 17, 2024 5:09 pm    Post subject: Reply with quote

& tells the shell to place the command in the background. Where did you read that this would not work? Have you tried the command I gave you? What went wrong?
Back to top
View user's profile Send private message
Marcofras
n00b
n00b


Joined: 29 May 2023
Posts: 26

PostPosted: Sun Nov 17, 2024 6:31 pm    Post subject: Reply with quote

Im under c++ and that command no works, i use a series of system(./desc &) or something similar is it correct? Will it works?
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22680

PostPosted: Sun Nov 17, 2024 7:39 pm    Post subject: Reply with quote

If you are using system from C++, why is your opening post expressed as shell? Rather than asking if something will work after I told you it will, why not try it and report whether it works?

I think you should provide a Minimum Reproducible Example demonstrating your problem.
Back to top
View user's profile Send private message
Marcofras
n00b
n00b


Joined: 29 May 2023
Posts: 26

PostPosted: Sun Nov 17, 2024 7:48 pm    Post subject: Reply with quote

Indeed i was wrong im tryng in shell before c++, dont you think its the same? I dont make c++ code yet, do you think system(./desc &) will work and save time with parallelization i wish?
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22680

PostPosted: Sun Nov 17, 2024 8:00 pm    Post subject: Reply with quote

I gave you a loop that should work. I later reiterated that I expect it to work. Yes, system should work fine, since, according to its manual page, that is delegating back to the shell, which is almost always the wrong thing to do in a C++ program. If you need further help, please demonstrate a specific problem.
Back to top
View user's profile Send private message
Marcofras
n00b
n00b


Joined: 29 May 2023
Posts: 26

PostPosted: Sun Nov 17, 2024 8:31 pm    Post subject: Reply with quote

I will make the code c++ in days.
Why its wrong thing which is the correct form? Maybe thread or something similar?
Back to top
View user's profile Send private message
GDH-gentoo
Veteran
Veteran


Joined: 20 Jul 2019
Posts: 1703
Location: South America

PostPosted: Sun Nov 17, 2024 8:45 pm    Post subject: Reply with quote

Because a C++ program can do things like spawning processes by using the libc directly (or through some convenience library) instead of invoking the shell.
_________________
NeddySeagoon wrote:
I'm not a witch, I'm a retired electronics engineer :)
Ionen wrote:
As a packager I just don't want things to get messier with weird build systems and multiple toolchains requirements though :)


Last edited by GDH-gentoo on Sun Nov 17, 2024 8:46 pm; edited 1 time in total
Back to top
View user's profile Send private message
szatox
Advocate
Advocate


Joined: 27 Aug 2013
Posts: 3435

PostPosted: Sun Nov 17, 2024 8:45 pm    Post subject: Reply with quote

Marcofras, you're bouncing around too much. Pause for a minute and focus.
What are you _actually_ trying to do?
_________________
Make Computing Fun Again
Back to top
View user's profile Send private message
Marcofras
n00b
n00b


Joined: 29 May 2023
Posts: 26

PostPosted: Mon Nov 18, 2024 11:04 am    Post subject: Reply with quote

parallalization, because i had no success with thread (in the past) im opting to divide the program in several subprograms and run parallelized, it seems more easy.
Back to top
View user's profile Send private message
szatox
Advocate
Advocate


Joined: 27 Aug 2013
Posts: 3435

PostPosted: Mon Nov 18, 2024 12:02 pm    Post subject: Reply with quote

Eh... This is clearly a case of https://en.wikipedia.org/wiki/XY_problem.
Regarding your original question:
> is it possible have parallalization with taskset?
The only correct answer is: "Yes but no". Unfortunately, on top of being correct, it is also completely useless.

Parallelization where? Why did you start with shell and then you're suddenly going to cpp? This part is confusing.
More importantly: What is the thing you're trying to write there? What is the problem you're trying to solve? What is standing in your way?

> I think you should provide a Minimum Reproducible Example demonstrating your problem.
Yeah, I second that. Although this technique is primarily used for reporting bugs, it does help with communication in general.
Overkill is still a kill.
_________________
Make Computing Fun Again
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22680

PostPosted: Mon Nov 18, 2024 12:30 pm    Post subject: Reply with quote

Yes, a Minimum Reproducible Example might be overkill here. I asked for it because OP's questions implied to me that he had directly experienced shell background not to exhibit the desired parallelism that I had stated would work. If he was experiencing that lack, we could try to understand why. If he has not tried the solutions given to him, I would like for him to do so or explain why not (particularly since this should be easy to test, due to being independent of the content of ./desc), before we put more work into assuring him again that his doubts are misplaced.
Back to top
View user's profile Send private message
Marcofras
n00b
n00b


Joined: 29 May 2023
Posts: 26

PostPosted: Mon Nov 18, 2024 2:41 pm    Post subject: Reply with quote

the original problem is i need parallalization for save time, i tried thread in desc that parallelize the function shap but it doesn' t work and freeze in ruuntime
Code:

thread th1(shap, px, 1);
                                 thread th2(shap, px, 2);
                                 thread th3(shap, px, 3);
                                 thread th4(shap, px, 4);
                                 thread th5(shap, px, 5);
                                 thread th6(shap, px, 6);
                                 thread th7(shap, px, 7);
                                 thread th8(shap, px, 8);
                                 thread th9(shap, px, 9);
                                 thread th10(shap, px, 10);
                                 thread th11(shap, px, 11);
                                 thread th12(shap, px, 12);
                                 thread th13(shap, px, 13);
                                 thread th14(shap, px, 14);
                                 thread th15(shap, px, 15);
                                 thread th16(shap, px, 16);
                                 thread th17(shap, px, 17);
                                 thread th18(shap, px, 18);
                                 
                                 th1.join();
                                 th2.join();
                                 th3.join();
                                 th4.join();
                                 th5.join();
                                 th6.join();
                                 th7.join();
                                 th8.join();
                                 th9.join();
                                 th10.join();
                                 th11.join();
                                 th12.join();
                                 th13.join();
                                 th14.join();
                                 th15.join();
                                 th16.join();
                                 th17.join();
                                 th18.join();

i cant reveal other for the code because its a big stuff.
So, i opted for build several indipendent program and launch with taskset or something similar (&) but you all tell me its not correct.
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22680

PostPosted: Mon Nov 18, 2024 2:59 pm    Post subject: Reply with quote

That does not appear to be a MRE. None of those identifiers are defined, and it is not legal to use a statement at global scope to call join. A Minimal Reproducible Example does not need to be your full and secret code, and by its very definition, generally should not be. We want it to be Minimal. Sharing your full secret code is very likely not minimal.

What I told you is that using system from C++ is "almost always the wrong thing to do." GDH-gentoo then elaborated my point for you.

Putting all that together, I will again ask you to show us a MRE that exhibits your problem. As a counter to your claim that this hangs, I offer this MRE that is not defective. It exactly mirrors the fragment you showed, and fills in the missing bits as needed to make it compile.
t.cpp:
// g++-14 -O2 -Wall -Wextra t.cpp
#include <thread>
#include <cstdio>

static void shap(int, int seq) {
   printf("Thread %i ...\n", seq);
}

int main()
{
   const int px = 1;
   printf("Starting threads ...\n");
   std::thread th1(shap, px, 1);
   std::thread th2(shap, px, 2);
   std::thread th3(shap, px, 3);
   std::thread th4(shap, px, 4);
   std::thread th5(shap, px, 5);
   std::thread th6(shap, px, 6);
   std::thread th7(shap, px, 7);
   std::thread th8(shap, px, 8);
   std::thread th9(shap, px, 9);
   std::thread th10(shap, px, 10);
   std::thread th11(shap, px, 11);
   std::thread th12(shap, px, 12);
   std::thread th13(shap, px, 13);
   std::thread th14(shap, px, 14);
   std::thread th15(shap, px, 15);
   std::thread th16(shap, px, 16);
   std::thread th17(shap, px, 17);
   std::thread th18(shap, px, 18);
   printf("Joining threads ...\n");
   th1.join();
   th2.join();
   th3.join();
   th4.join();
   th5.join();
   th6.join();
   th7.join();
   th8.join();
   th9.join();
   th10.join();
   th11.join();
   th12.join();
   th13.join();
   th14.join();
   th15.join();
   th16.join();
   th17.join();
   th18.join();
   printf("Done\n");
}
Possible output (depending on scheduling):
Code:
$ ./a.out
Starting threads ...
Thread 1 ...
Thread 2 ...
Thread 3 ...
Thread 4 ...
Thread 5 ...
Thread 6 ...
Thread 7 ...
Thread 8 ...
Thread 9 ...
Thread 10 ...
Thread 11 ...
Thread 12 ...
Thread 13 ...
Thread 14 ...
Thread 15 ...
Thread 16 ...
Thread 17 ...
Joining threads ...
Thread 18 ...
Done
From this, using threads and joining them is not inherently wrong. If your program hangs, it must be because some part you refused to share is incorrect.
Back to top
View user's profile Send private message
Marcofras
n00b
n00b


Joined: 29 May 2023
Posts: 26

PostPosted: Mon Nov 18, 2024 3:23 pm    Post subject: Reply with quote

i made the same thing of you but it freeze
Back to top
View user's profile Send private message
szatox
Advocate
Advocate


Joined: 27 Aug 2013
Posts: 3435

PostPosted: Mon Nov 18, 2024 3:40 pm    Post subject: Reply with quote

Works for me.
Adding a 1 sec sleep shows the parallel execution even better. If we delay print, we get to see the order randomized by scheduler due to system load:
Code:
$  ./a.out
Starting threads ...
Joining threads ...
Thread 1 ...
Thread 15 ...
Thread 5 ...
Thread 6 ...
Thread 4 ...
Thread 2 ...
Thread 10 ...
Thread 7 ...
Thread 18 ...
Thread 13 ...
Thread 14 ...
Thread 3 ...
Thread 9 ...
Thread 8 ...
Thread 11 ...
Thread 16 ...
Thread 12 ...
Thread 17 ...
Done

Meanwhile, running it through taskset to lock all threads to a single core results in almost all single-digit tasks running before almost all double-digits, almost every time. The output produced by threads running concurrently on a single core is far less random than output from parallel threads running on 4 cores.
_________________
Make Computing Fun Again
Back to top
View user's profile Send private message
Marcofras
n00b
n00b


Joined: 29 May 2023
Posts: 26

PostPosted: Mon Nov 18, 2024 3:45 pm    Post subject: Reply with quote

the order is not important in my case i want use system() and shell its more easy for me.
suggestions?
Back to top
View user's profile Send private message
GDH-gentoo
Veteran
Veteran


Joined: 20 Jul 2019
Posts: 1703
Location: South America

PostPosted: Mon Nov 18, 2024 3:55 pm    Post subject: Reply with quote

Marcofras wrote:
the order is not important in my case i want use system() and shell its more easy for me.
suggestions?

Yes. Don't just do things in a program "because it is easier", learn to do paralellization properly in C++. Threads are difficult because concurrent access to shared memory can create all sorts of problems without doing it properly, but I'm not convinced so far that you can't just spawn processes with standard POSIX function calls to solve your problem, for example, whatever that problem is.
_________________
NeddySeagoon wrote:
I'm not a witch, I'm a retired electronics engineer :)
Ionen wrote:
As a packager I just don't want things to get messier with weird build systems and multiple toolchains requirements though :)
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9612
Location: beyond the rim

PostPosted: Mon Nov 18, 2024 3:59 pm    Post subject: Reply with quote

Have you looked into just using fork() for your issue?

Using system() is almost always a bad idea.
Back to top
View user's profile Send private message
Marcofras
n00b
n00b


Joined: 29 May 2023
Posts: 26

PostPosted: Mon Nov 18, 2024 4:14 pm    Post subject: Reply with quote

i dont knew fork i read it can call a program from the filesystem, will it be parallellized with time saving?
i dont understand well how to use:
Code:

int id = fork(prog);

?
do i have to put fork in the begin of function shap?
Back to top
View user's profile Send private message
pingtoo
Veteran
Veteran


Joined: 10 Sep 2021
Posts: 1249
Location: Richmond Hill, Canada

PostPosted: Mon Nov 18, 2024 4:33 pm    Post subject: Reply with quote

Marcofras wrote:
i dont knew fork i read it can call a program from the filesystem, will it be parallellized with time saving?
i dont understand well how to use:
Code:

int id = fork(prog);

?
Where did you read lead you to think "fork" can call a program? I ask because this is not true.

If you don't mind please share some information about your experience in programming C language in general and more specifically on programming C language on Linux.

I found in this thread you have been given out advise in various degree of knowledge on programming on Linux. I think it could lead to confuse because these advises assume different level of understanding.

Your question in "parallelism" assume it will have time saving. this is not always true in my experience. So it will also be nice if you can share more information in the problem space. For example how large is the to be process data set. And where the data set located. (i.e. locally on the process machine or remotely in a database).
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum