steffers
n00b
Joined: 03 Nov 2007
Posts: 12
|
 Posted: Sat Mar 06, 2021 2:51 pm Post subject: Experience with ceph on aarch64 |
 |
|
Hi,
I had a heavily patched ceph-10.2.10 on my various rpi's running on gentoo 32bit.
Well I created that binary package quite some time back and the dependencies within it increase the difficulties of updating anything for these systems...
Lucky enough I saw that the current ceph ebuilds are ~arm64 keywords and I tried to give it a try.
First I decided to stick with the 32bit system (because I was to lazy to configure a new kernel then). With some keyword acceptance:
| Code: |
sys-cluster/ceph ~arm64
dev-libs/rocksdb ~arm64
dev-python/pecan ~arm64
sys-apps/nvme-cli ~arm64
sci-libs/scikit-learn ~arm64
dev-python/logutils arm64
dev-python/prettytable arm64
# Not sure if the following are needed
dev-lang/yasm x86
dev-python/ordereddict ~arm arm
sys-auth/oath-toolkit ~arm arm
|
a small change in the ebuild:
| Code: |
--- /usr/portage/sys-cluster/ceph/ceph-15.2.8.ebuild 2020-12-21 21:39:14.000000000 +0100
+++ /usr/local/portage/sys-cluster/ceph.bak/ceph-15.2.8-r1.ebuild 2021-03-06 15:28:58.173139395 +0100
@@ -68,7 +68,7 @@
sys-libs/libcap-ng:=
sys-libs/ncurses:0=
sys-libs/zlib:=
- sys-process/numactl:=
+ numa? ( sys-process/numactl:= )
x11-libs/libpciaccess:=
babeltrace? ( dev-util/babeltrace )
fuse? ( sys-fs/fuse:0= )
@@ -203,6 +203,7 @@
"${FILESDIR}/ceph-15.2.4-system-uring.patch"
"${FILESDIR}/ceph-15.2.5-missing-includes.patch"
"${FILESDIR}/ceph-15.2.5-glibc-2.32.patch"
+ "${FILESDIR}/ceph-15.2.8-arm32build.patch"
)
check-reqs_export_vars() {
|
and a small patch to the build:
| Code: |
diff -Naur ceph-15.2.8-orig/src/mds/PurgeQueue.cc ceph-15.2.8/src/mds/PurgeQueue.cc
--- ceph-15.2.8-orig/src/mds/PurgeQueue.cc 2020-12-16 18:29:50.000000000 +0100
+++ ceph-15.2.8/src/mds/PurgeQueue.cc 2021-03-01 07:57:26.654460816 +0100
@@ -503,7 +503,7 @@
in_flight[expire_to] = item;
logger->set(l_pq_executing, in_flight.size());
- files_high_water = std::max(files_high_water, in_flight.size());
+ files_high_water = std::max(files_high_water, static_cast<uint64_t>(in_flight.size()));
logger->set(l_pq_executing_high_water, files_high_water);
auto ops = _calculate_ops(item);
ops_in_flight += ops;
@@ -581,7 +581,7 @@
logger->set(l_pq_executing_ops_high_water, ops_high_water);
in_flight.erase(expire_to);
logger->set(l_pq_executing, in_flight.size());
- files_high_water = std::max(files_high_water, in_flight.size());
+ files_high_water = std::max(files_high_water, static_cast<uint64_t>(in_flight.size()));
logger->set(l_pq_executing_high_water, files_high_water);
return;
}
@@ -659,7 +659,7 @@
in_flight.erase(iter);
logger->set(l_pq_executing, in_flight.size());
- files_high_water = std::max(files_high_water, in_flight.size());
+ files_high_water = std::max(files_high_water, static_cast<uint64_t>(in_flight.size()));
logger->set(l_pq_executing_high_water, files_high_water);
dout(10) << "in_flight.size() now " << in_flight.size() << dendl;
diff -Naur ceph-15.2.8-orig/src/pybind/ceph_argparse.py ceph-15.2.8/src/pybind/ceph_argparse.py
--- ceph-15.2.8-orig/src/pybind/ceph_argparse.py 2020-12-16 18:29:50.000000000 +0100
+++ ceph-15.2.8/src/pybind/ceph_argparse.py 2021-02-26 06:57:25.428574224 +0100
@@ -1319,7 +1319,10 @@
# otherwise it will keep polling until timeout or thread stops.
# wait for INT32_MAX, as python 3.6.8 use int32_t to present the
# timeout in integer when converting it to nanoseconds
- timeout = (1 << (32 - 1)) - 1
+ #timeout = (1 << (32 - 1)) - 1
+ # On arm32 this leads to a nearly immediately thrown timeout exception...
+ # The following works...
+ timeout = (1 << (29 - 1)) - 1
t = RadosThread(func, *args, **kwargs)
# allow the main thread to exit (presumably, avoid a join() on this
diff -Naur ceph-15.2.8-orig/src/tools/rbd_mirror/image_replayer/snapshot/Replayer.cc ceph-15.2.8/src/tools/rbd_mirror/image_replayer/snapshot/Replayer.cc
--- ceph-15.2.8-orig/src/tools/rbd_mirror/image_replayer/snapshot/Replayer.cc 2020-12-16 18:29:50.000000000 +0100
+++ ceph-15.2.8/src/tools/rbd_mirror/image_replayer/snapshot/Replayer.cc 2021-02-25 16:47:45.405117124 +0100
@@ -234,7 +234,7 @@
json_spirit::mObject root_obj;
root_obj["replay_state"] = replay_state;
- root_obj["remote_snapshot_timestamp"] = remote_snap_info->timestamp.sec();
+ root_obj["remote_snapshot_timestamp"] = static_cast<uint64_t>(remote_snap_info->timestamp.sec());
auto matching_remote_snap_id = util::compute_remote_snap_id(
m_state_builder->local_image_ctx->image_lock,
@@ -248,8 +248,8 @@
// use the timestamp from the matching remote image since
// the local snapshot would just be the time the snapshot was
// synced and not the consistency point in time.
- root_obj["local_snapshot_timestamp"] =
- matching_remote_snap_it->second.timestamp.sec();
+ root_obj["local_snapshot_timestamp"] = static_cast<uint64_t>(
+ matching_remote_snap_it->second.timestamp.sec());
}
matching_remote_snap_it = m_state_builder->remote_image_ctx->snap_info.find(
@@ -257,7 +257,7 @@
if (m_remote_snap_id_end != CEPH_NOSNAP &&
matching_remote_snap_it !=
m_state_builder->remote_image_ctx->snap_info.end()) {
- root_obj["syncing_snapshot_timestamp"] = remote_snap_info->timestamp.sec();
+ root_obj["syncing_snapshot_timestamp"] = static_cast<uint64_t>(remote_snap_info->timestamp.sec());
root_obj["syncing_percent"] = static_cast<uint64_t>(
100 * m_local_mirror_snap_ns.last_copied_object_number /
static_cast<float>(std::max<uint64_t>(1U, m_local_object_count)));
|
I got a ceph to build for arm32 and at least some of the resulting binaries where usable... I had working mon's and osd's.
But the mgr did segfault and without it current ceph clusters does not seems to be very usable.
Another thing that did not work was that the ceph created on my amd64 PC seemed to be unable to talk to that cluster.
Well luckily my rpi's are all 3b 3b+ or 4 and thus able to run a 64bit gentoo... So I created a kernel and a system and everything was working very well at once.
My test rpi was a 3b+
Nearly no USE changes:
| Code: |
*/* CPU_FLAGS_ARM: thumb vfp neon vfpv3
*/* LLVM_TARGETS: ARM WebAssembly -AArch64 -BPF -Hexagon -Lanai -MSP430 -Mips -NVPTX -PowerPC -Sparc -SystemZ -XCore
*/* PYTHON_TARGETS: python3_8 -python2_7
dev-python/numpy lapack
#dev-util/google-perftools optimisememory minimal
dev-util/google-perftools minimal
|
and in make.conf:
| Code: |
USE="${ARCH} lz4"
ACCEPT_KEYWORDS="${ARCH} ~${ARCH}"
CFLAGS="-Os -march=armv8-a+crc -mtune=cortex-a53 -ftree-vectorize -pipe -fomit-frame-pointer"
CXXFLAGS="${CFLAGS}"
FFLAGS="${CFLAGS}"
FCFLAGS="${FFLAGS}"
|
I was able to create a binary package with no problems at all using cross and chroot building. Sadly ceph-mon segfaulted on both the qemu chroot and the rpi 3b+.
Currently I try rebuilding ceph with O2 instead of Os. Already did this with
dev-util/google-perftools
sys-devel/gcc
sys-libs/libunwind
sys-libs/glibc
But no change so far.... A backtrace of the segmentation fault is:
| Code: |
#0 0x0000007f76a29a14 in ?? () from /usr/lib64/libunwind.so.8
#1 0x0000007f76a2a3bc in _ULaarch64_is_signal_frame () from /usr/lib64/libunwind.so.8
#2 0x0000007f76a2a974 in _ULaarch64_step () from /usr/lib64/libunwind.so.8
#3 0x0000007f77330f38 in ?? () from /usr/lib64/libtcmalloc.so.4
#4 0x0000007f77331330 in GetStackTrace(void**, int, int) () from /usr/lib64/libtcmalloc.so.4
#5 0x0000007f77321eb0 in tcmalloc::PageHeap::GrowHeap(unsigned long) () from /usr/lib64/libtcmalloc.so.4
#6 0x0000007f773221ac in tcmalloc::PageHeap::New(unsigned long) () from /usr/lib64/libtcmalloc.so.4
#7 0x0000007f7731fec8 in tcmalloc::CentralFreeList::Populate() () from /usr/lib64/libtcmalloc.so.4
#8 0x0000007f773200f0 in tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) ()
from /usr/lib64/libtcmalloc.so.4
#9 0x0000007f773201a0 in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) ()
from /usr/lib64/libtcmalloc.so.4
#10 0x0000007f77323c3c in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long)) () from /usr/lib64/libtcmalloc.so.4
#11 0x0000007f773348d0 in tcmalloc::allocate_full_malloc_oom(unsigned long) () from /usr/lib64/libtcmalloc.so.4
#12 0x0000007f770e91e4 in ?? () from /usr/lib/gcc/aarch64-unknown-linux-gnu/10.2.0/libstdc++.so.6
#13 0x0000007f8048386c in ?? () from /lib/ld-linux-aarch64.so.1
#14 0x0000007f80483964 in ?? () from /lib/ld-linux-aarch64.so.1
#15 0x0000007f80475fc8 in ?? () from /lib/ld-linux-aarch64.so.1
Backtrace stopped: not enough registers or memory available to unwind further
|
The last lines of LD_DEBUG=all are...
| Code: |
4863: binding file /usr/lib64/libunwind.so.8 [0] to /lib64/libc.so.6 [0]: normal symbol `dl_iterate_phdr' [GLIBC_2.17]
4863: symbol=memcpy; lookup in file=ceph-mon [0]
4863: symbol=memcpy; lookup in file=/usr/lib64/ceph/libceph-common.so.2 [0]
4863: symbol=memcpy; lookup in file=/usr/lib64/libleveldb.so.1 [0]
4863: symbol=memcpy; lookup in file=/usr/lib64/librocksdb.so.6 [0]
4863: symbol=memcpy; lookup in file=/usr/lib64/libtcmalloc.so.4 [0]
4863: symbol=memcpy; lookup in file=/lib64/libpthread.so.0 [0]
4863: symbol=memcpy; lookup in file=/lib64/libdl.so.2 [0]
4863: symbol=memcpy; lookup in file=/usr/lib/gcc/aarch64-unknown-linux-gnu/10.2.0/libstdc++.so.6 [0]
4863: symbol=memcpy; lookup in file=/lib64/libm.so.6 [0]
4863: symbol=memcpy; lookup in file=/usr/lib/gcc/aarch64-unknown-linux-gnu/10.2.0/libgcc_s.so.1 [0]
4863: symbol=memcpy; lookup in file=/lib64/libc.so.6 [0]
4863: binding file /usr/lib64/libunwind.so.8 [0] to /lib64/libc.so.6 [0]: normal symbol `memcpy' [GLIBC_2.17]
Segmentation fault
|
|
|
Gentoo on ARM
