| View previous topic :: View next topic |
| Author |
Message |
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1223
Location: Richmond Hill, Canada
|
 Posted: Thu Oct 31, 2024 4:59 pm Post subject: Gentoo Dtrace |
 |
|
When enable dtrace, those kernel configuration options must be static link in to kernel? Can they be set to as module? This question is conceptual as in I know this can be find from kconfig. But question is about is it better to have it static therefor lesser limitation or it does not matter.
And is there additional kernel stuff (files) required on file system in order for dtrace to function? (I mean beside user space dtraec utilities) |
|
| Back to top |
|
 |
sam_
Developer
Joined: 14 Aug 2020
Posts: 1946
|
| Posted: Fri Nov 01, 2024 3:59 am Post subject: |
|
|
Thanks for playing with it! Really nice to see all the interest in it so far.
The kernel configuration options the ebuild marks (see https://wiki.gentoo.org/wiki/DTrace#Kernel_configuration) must be on, of course. As for whether they need to be a module or built-in "modules": there is better support for them being real modules for now. There are patches in gentoo-sources (not yet in the default config in gentoo-kernel so not in gentoo-kernel-bin but the patches are there too) which better support modules being built-in. The issue is obtaining range information when they're in the kernel image rather than separate modules. The only downside there is not being able to probe things in those affected modules.
No other stuff should be needed other than correct kernel configuration and dev-debug/dtrace installed.
TL;DR: Enable the kernel config options the ebuild mentions, prefer modules rather than built-in for now, install dev-debug/dtrace. |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1223
Location: Richmond Hill, Canada
|
| Posted: Fri Nov 01, 2024 3:07 pm Post subject: |
|
|
sam_,
Thank you for the information. |
|
| Back to top |
|
|
CaptainBlood
Advocate
Joined: 24 Jan 2010
Posts: 3839
|
| Posted: Fri Nov 01, 2024 7:01 pm Post subject: |
|
|
Nice question and clarification from both of you.
Hence currently installing...
Thks 4 ur attention, interest & support.
_________________
USE="-* ..." in /etc/portage/make.conf here, i.e. a countermeasure to portage implicit braces, belt & diaper paradigm
LT: "I've been doing a passable imitation of the Fontana di Trevi, except my medium is mucus. Sooo much mucus. " |
|
| Back to top |
|
|
CaptainBlood
Advocate
Joined: 24 Jan 2010
Posts: 3839
|
| Posted: Sat Nov 02, 2024 12:33 pm Post subject: |
|
|
Kernel config base is highly constrained here:
Only what is functionnaly required.
Mostly following app-admin/kernel-hardening-checker too.
Kernel config requirements aren't defined at ebuild level.
Thus emerging here throws: | Code: | >>> Running pre-merge checks for dev-debug/dtrace-2.0.1.1-r2
* Determining the location of the kernel source code
* Found kernel source directory:
* /usr/src/linux
* Could not detect kernel version.
* Please ensure that /usr/src/linux points to a complete set of Linux sources.
* Unable to calculate Linux Kernel version for build, attempting to use running version
* Checking for suitable kernel configuration options ...
* CONFIG_DEBUG_INFO_BTF: is not set when it should be.
* CONFIG_KALLSYMS_ALL: is not set when it should be.
* CONFIG_CUSE: is not set when it should be.
* CONFIG_TRACING: is not set when it should be.
* CONFIG_UPROBES: is not set when it should be.
* CONFIG_UPROBE_EVENTS: is not set when it should be.
* CONFIG_FTRACE: is not set when it should be.
* CONFIG_FTRACE_SYSCALLS: is not set when it should be.
* CONFIG_DYNAMIC_FTRACE: is not set when it should be.
* CONFIG_FUNCTION_TRACER: is not set when it should be.
* CONFIG_FPROBE: is not set when it should be.
* CONFIG_GCC_PLUGIN_RANDSTRUCT: should not be set. But it is. [ !! ] |
which is quite informative what should be changed here.
Thks 4 ur attention, interest & support.
_________________
USE="-* ..." in /etc/portage/make.conf here, i.e. a countermeasure to portage implicit braces, belt & diaper paradigm
LT: "I've been doing a passable imitation of the Fontana di Trevi, except my medium is mucus. Sooo much mucus. " |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1223
Location: Richmond Hill, Canada
|
| Posted: Sat Nov 02, 2024 2:19 pm Post subject: |
|
|
I think it was said in the Gentoo Wiki for Dtrace that harden kernel not supported.
But in any case does the package dev-debug/dtrace actually build? |
|
| Back to top |
|
|
CaptainBlood
Advocate
Joined: 24 Jan 2010
Posts: 3839
|
| Posted: Sat Nov 02, 2024 2:56 pm Post subject: |
|
|
| pingtoo wrote: | | I think it was said in the Gentoo Wiki for Dtrace that harden kernel not supported. |
| Code: | eselect profile list
[45] default/linux/amd64/23.0/split-usr (stable) * |
However I admit kernel config is fully manual here.
| pingtoo wrote: | | But in any case does the package dev-debug/dtrace actually build? |
Yes it does build, although
CONFIG_DEBUG_INFO_BTF wasn't configured yet.
As far as 6.11.6 is concerned here, diff is as follows: | Code: | diff config-2024-11-02_14:51:17 config-2024-11-02_17:20:29
117c117
< # CONFIG_BPF_SYSCALL is not set
---
> CONFIG_BPF_SYSCALL=y
118a119,120
> CONFIG_BPF_UNPRIV_DEFAULT_OFF=y
> # CONFIG_BPF_PRELOAD is not set
147a150,153
> CONFIG_TASKS_RCU_GENERIC=y
> CONFIG_NEED_TASKS_RCU=y
> CONFIG_TASKS_RUDE_RCU=y
> CONFIG_TASKS_TRACE_RCU=y
188a195
> # CONFIG_CGROUP_BPF is not set
247c254,257
< # CONFIG_KALLSYMS is not set
---
> CONFIG_KALLSYMS=y
> # CONFIG_KALLSYMS_SELFTEST is not set
> CONFIG_KALLSYMS_ALL=y
> CONFIG_KALLSYMS_ABSOLUTE_PERCPU=y
259a270
> CONFIG_TRACEPOINTS=y
648a660
> CONFIG_UPROBES=y
782,783c794
< CONFIG_GCC_PLUGINS=y
< CONFIG_GCC_PLUGIN_LATENT_ENTROPY=y
---
> # CONFIG_GCC_PLUGINS is not set
820c831
< CONFIG_UNUSED_KSYMS_WHITELIST="/usr/src/linux-6.11.6-gentoo-dtrc/KSYMS_WHITELIST.tMF5"
---
> CONFIG_UNUSED_KSYMS_WHITELIST="/usr/src/linux-6.11.6-gentoo-dtrc/KSYMS_WHITELIST.cXgE"
1019a1031,1032
> CONFIG_NET_EGRESS=y
> CONFIG_NET_XGRESS=y
1031a1045
> # CONFIG_XDP_SOCKETS is not set
1063a1078
> CONFIG_NETFILTER_BPF_LINK=y
1178a1194
> # CONFIG_NET_DROP_MONITOR is not set
1251a1268,1270
> CONFIG_NET_SOCK_MSG=y
> CONFIG_PAGE_POOL=y
> # CONFIG_PAGE_POOL_STATS is not set
1625a1645
> # CONFIG_NETKIT is not set
3936c3956
< # CONFIG_CUSE is not set
---
> CONFIG_CUSE=m
4123,4127d4142
< CONFIG_GCC_PLUGIN_STACKLEAK=y
< # CONFIG_GCC_PLUGIN_STACKLEAK_VERBOSE is not set
< CONFIG_STACKLEAK_TRACK_MIN_SIZE=100
< # CONFIG_STACKLEAK_METRICS is not set
< # CONFIG_STACKLEAK_RUNTIME_DISABLE is not set
4141,4145c4156
< # CONFIG_RANDSTRUCT_NONE is not set
< CONFIG_RANDSTRUCT_FULL=y
< # CONFIG_RANDSTRUCT_PERFORMANCE is not set
< CONFIG_RANDSTRUCT=y
< CONFIG_GCC_PLUGIN_RANDSTRUCT=y
---
> CONFIG_RANDSTRUCT_NONE=y
4374a4386,4387
> CONFIG_BINARY_PRINTF=y
>
4512a4526
> CONFIG_DEBUG_INFO=y
4514,4515c4528,4529
< CONFIG_DEBUG_INFO_NONE=y
< # CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT is not set
---
> # CONFIG_DEBUG_INFO_NONE is not set
> CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
4517a4532,4542
> # CONFIG_DEBUG_INFO_REDUCED is not set
> CONFIG_DEBUG_INFO_COMPRESSED_NONE=y
> # CONFIG_DEBUG_INFO_COMPRESSED_ZLIB is not set
> # CONFIG_DEBUG_INFO_COMPRESSED_ZSTD is not set
> # CONFIG_DEBUG_INFO_SPLIT is not set
> CONFIG_DEBUG_INFO_BTF=y
> CONFIG_PAHOLE_HAS_SPLIT_BTF=y
> CONFIG_PAHOLE_HAS_LANG_EXCLUDE=y
> CONFIG_DEBUG_INFO_BTF_MODULES=y
> # CONFIG_MODULE_ALLOW_BTF_MISMATCH is not set
> # CONFIG_GDB_SCRIPTS is not set
4572a4598
> # CONFIG_DEBUG_PAGE_REF is not set
4688a4715
> CONFIG_NOP_TRACER=y
4689a4717
> CONFIG_RETHOOK=y
4690a4719,4720
> CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
> CONFIG_HAVE_FUNCTION_GRAPH_RETVAL=y
4702a4733,4739
> CONFIG_BUILDTIME_MCOUNT_SORT=y
> CONFIG_TRACE_CLOCK=y
> CONFIG_RING_BUFFER=y
> CONFIG_EVENT_TRACING=y
> CONFIG_CONTEXT_SWITCH_TRACER=y
> CONFIG_TRACING=y
> CONFIG_GENERIC_TRACER=y
4704c4741,4785
< # CONFIG_FTRACE is not set
---
> CONFIG_FTRACE=y
> # CONFIG_BOOTTIME_TRACING is not set
> CONFIG_FUNCTION_TRACER=y
> # CONFIG_FUNCTION_GRAPH_TRACER is not set
> CONFIG_DYNAMIC_FTRACE=y
> CONFIG_DYNAMIC_FTRACE_WITH_REGS=y
> CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y
> CONFIG_DYNAMIC_FTRACE_WITH_ARGS=y
> CONFIG_FPROBE=y
> # CONFIG_FUNCTION_PROFILER is not set
> # CONFIG_STACK_TRACER is not set
> # CONFIG_IRQSOFF_TRACER is not set
> # CONFIG_SCHED_TRACER is not set
> # CONFIG_HWLAT_TRACER is not set
> # CONFIG_OSNOISE_TRACER is not set
> # CONFIG_TIMERLAT_TRACER is not set
> # CONFIG_MMIOTRACE is not set
> CONFIG_FTRACE_SYSCALLS=y
> # CONFIG_TRACER_SNAPSHOT is not set
> CONFIG_BRANCH_PROFILE_NONE=y
> # CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
> # CONFIG_BLK_DEV_IO_TRACE is not set
> CONFIG_FPROBE_EVENTS=y
> CONFIG_PROBE_EVENTS_BTF_ARGS=y
> CONFIG_UPROBE_EVENTS=y
> CONFIG_BPF_EVENTS=y
> CONFIG_DYNAMIC_EVENTS=y
> CONFIG_PROBE_EVENTS=y
> CONFIG_FTRACE_MCOUNT_RECORD=y
> CONFIG_FTRACE_MCOUNT_USE_CC=y
> # CONFIG_SYNTH_EVENTS is not set
> # CONFIG_USER_EVENTS is not set
> # CONFIG_HIST_TRIGGERS is not set
> # CONFIG_TRACE_EVENT_INJECT is not set
> # CONFIG_TRACEPOINT_BENCHMARK is not set
> # CONFIG_RING_BUFFER_BENCHMARK is not set
> # CONFIG_TRACE_EVAL_MAP_FILE is not set
> # CONFIG_FTRACE_RECORD_RECURSION is not set
> # CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING is not set
> # CONFIG_FTRACE_STARTUP_TEST is not set
> # CONFIG_FTRACE_SORT_STARTUP_TEST is not set
> # CONFIG_RING_BUFFER_STARTUP_TEST is not set
> # CONFIG_RING_BUFFER_VALIDATE_TIME_DELTAS is not set
> # CONFIG_PREEMPTIRQ_DELAY_TEST is not set
> # CONFIG_RV is not set |
Plz it could be some activated settings aren't required, and are only a result of kernel cascading activation.
CONFIG_BPF_SYSCALL, which seems required here, isn't explicitly mentioned in the wiki.
However it may be cascaded by another advised setting.
Thks 4 ur attention, interest & support.
_________________
USE="-* ..." in /etc/portage/make.conf here, i.e. a countermeasure to portage implicit braces, belt & diaper paradigm
LT: "I've been doing a passable imitation of the Fontana di Trevi, except my medium is mucus. Sooo much mucus. " |
|
| Back to top |
|
|
CaptainBlood
Advocate
Joined: 24 Jan 2010
Posts: 3839
|
| Posted: Sat Nov 02, 2024 3:17 pm Post subject: |
|
|
On openrc, manual | Code: | | rc-update add dtprobed default |
wasn't required here.
From wiki | Code: | root #dtrace -l
...
As of August 2024, on an ~amd64 system with linux-6.6, around 125000 probes are registered. If the number is substantially lower than that, it's possible some required kernel config options are not enabled. |
Something must be wrong here as: | Code: | dtrace -l|wc -l
172 |
Indeed wiki's first basic sanity check fails here as | Code: | dtrace -n 'BEGIN { exit(0); }'
dtrace: invalid probe specifier BEGIN { exit(0); }: "/usr/lib64/dtrace/6.10/ip.d", line 248: skc_v6_daddr is not a member of struct sock_common |
My 2 cent is that ip6 kernel stack is expected...
Thks 4 ur attention, interest & support.
_________________
USE="-* ..." in /etc/portage/make.conf here, i.e. a countermeasure to portage implicit braces, belt & diaper paradigm
LT: "I've been doing a passable imitation of the Fontana di Trevi, except my medium is mucus. Sooo much mucus. " |
|
| Back to top |
|
|
nullnix
n00b
Joined: 04 Nov 2024
Posts: 1
Location: Ely, UK
|
| Posted: Mon Nov 04, 2024 3:16 pm Post subject: |
|
|
| CaptainBlood wrote: | On openrc, manual | Code: | | rc-update add dtprobed default |
wasn't required here.
|
Yeah, we try to start that automatically because it's really an implementation detail that users shouldn't need to be concerned by (and because it needs to be running before any processes with USDT probes start if you want to be able to probe them).
| CaptainBlood wrote: | From wiki | Code: | root #dtrace -l
...
As of August 2024, on an ~amd64 system with linux-6.6, around 125000 probes are registered. If the number is substantially lower than that, it's possible some required kernel config options are not enabled. |
Something must be wrong here as: | Code: | dtrace -l|wc -l
172 |
Indeed wiki's first basic sanity check fails here as | Code: | dtrace -n 'BEGIN { exit(0); }'
dtrace: invalid probe specifier BEGIN { exit(0); }: "/usr/lib64/dtrace/6.10/ip.d", line 248: skc_v6_daddr is not a member of struct sock_common |
My 2 cent is that ip6 kernel stack is expected...
|
It's certainly the case that operation with unusual kernel configs is very little tested so far, and operation without IPv6 (or IPv4, for that matter) is likely even less tested. (Indeed, my local test systems are IPv6-only these days specifically so I can make sure IPv6 doesn't break, though they do have IPv4 compiled in as well.)
I'd recommend trying to compile IPv6 in and see what happens. If it still fails, try running with DTRACE_DEBUG=t set in the environment and post the spray of output it prints on stderr. There may be extra error messages hiding in there that we need to expose as non-debugging messages.
More generally, we should probably add some way to test kernel configs in translators and skip them if specific things aren't set (or disable bits of them, in this case). Right now we can test for the presence of providers, but since both ipv4 and ipv6 are implemented via the "ip" provider, this isn't much help...
More generally yet, if you only have hundreds of probes, something else is wrong: the fbt provider alone should be providing tens of thousands, so that's not initializing properly for some reason.
What does
| Code: | cat /sys/kernel/debug/tracing/available_filter_functions | wc -l
|
report on your kernel? I get a figure around 60000... |
|
| Back to top |
|
|
CaptainBlood
Advocate
Joined: 24 Jan 2010
Posts: 3839
|
| Posted: Mon Nov 04, 2024 7:08 pm Post subject: |
|
|
| nullnix wrote: | | I'd recommend trying to compile IPv6 in and see what happens. If it still fails, try running with DTRACE_DEBUG=t set in the environment and post the spray of output it prints on stderr. There may be extra error messages hiding in there that we need to expose as non-debugging messages. |
dpaste
Obviously more work required beyond solely IPv6 activation... In progress.
| nullnix wrote: | | Code: | cat /sys/kernel/debug/tracing/available_filter_functions | wc -l
|
report on your kernel? I get a figure around 60000... |
| Code: | LC_MESSAGES=C cat /sys/kernel/debug/tracing/available_filter_functions|wc -l
cat: /sys/kernel/debug/tracing/available_filter_functions: No such file or directory
0
|
Any idea more than welcome.
Thks 4 ur attention, interest & support.
_________________
USE="-* ..." in /etc/portage/make.conf here, i.e. a countermeasure to portage implicit braces, belt & diaper paradigm
LT: "I've been doing a passable imitation of the Fontana di Trevi, except my medium is mucus. Sooo much mucus. " |
|
| Back to top |
|
|
CaptainBlood
Advocate
Joined: 24 Jan 2010
Posts: 3839
|
| Posted: Mon Nov 04, 2024 8:55 pm Post subject: |
|
|
Oracle Dtrace tutorial refers to About DTrace Providers as kernel modules which seems to be missing in the dpaste above. | Code: | libdtrace DEBUG 1730746374: creating provider fbt
libdtrace DEBUG 1730746374: loaded 0 probes for fbt
libdtrace DEBUG 1730746374: creating provider sdt
libdtrace DEBUG 1730746374: loaded 0 probes for sdt
libdtrace DEBUG 1730746374: creating provider syscall
libdtrace DEBUG 1730746374: loaded 0 probes for syscall |
However | Code: | libdtrace DEBUG 1730746374: creating provider profile
libdtrace DEBUG 1730746374: loaded 13 probes for profile |
seems provided.
dtrace and fasttrap modules aren't reported for some reason.
I don't know how critical these are but I must have done something wrong.
Any idea?
Thks 4 ur attention, interest & support.
_________________
USE="-* ..." in /etc/portage/make.conf here, i.e. a countermeasure to portage implicit braces, belt & diaper paradigm
LT: "I've been doing a passable imitation of the Fontana di Trevi, except my medium is mucus. Sooo much mucus. " |
|
| Back to top |
|
|
CaptainBlood
Advocate
Joined: 24 Jan 2010
Posts: 3839
|
| Posted: Tue Nov 05, 2024 1:52 am Post subject: |
|
|
These missing modules are likely to be oracle specific, available with sys-kernel/uek-sources.
Thks 4 ur attention, interest & support.
_________________
USE="-* ..." in /etc/portage/make.conf here, i.e. a countermeasure to portage implicit braces, belt & diaper paradigm
LT: "I've been doing a passable imitation of the Fontana di Trevi, except my medium is mucus. Sooo much mucus. " |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Gentoo Chat
