View previous topic :: View next topic |
Author |
Message |
manphiz n00b
Joined: 01 Dec 2021 Posts: 26
|
Posted: Tue Dec 14, 2021 12:37 am Post subject: Cannot find /bin/bash when chrooting during crossdev |
|
|
Hi,
I've been trying to create a mips64el n32 chroot for loongson2f by following the instructions in https://wiki.gentoo.org/wiki/Cross_build_environment. I've made some progress in creating the crossdev using "crossdev -t mips64el-unknown-linux-gnu", cross-compiling @system under /usr/mips64-unknown-linux-gnu, and compiling most of the essential packages except Python-3.9.9, which AIUI doesn't support cross compiling and needs to be done in chroot.
I've also compiled qemu on the host and the static qemu for mips. One difference is that my Gentoo is using systemd so I followed the instructions in https://wiki.gentoo.org/wiki/Embedded_Handbook/General/Full to create the bits required by systemd so that it can start systemd-binfmt, more specifically I've prepared the binfmt-support for both mips and mipsel under /etc/binfmt.d. When trying to chroot to /usr/mips64el-unknown-linux-gnu I was hit by this error:
Quote: | $ cd /usr/mips64el-unknown-linux-gnu/ && sudo chroot . /bin/bash --login
chroot: failed to run command ‘/bin/bash’: No such file or directory
|
I tried to search for this error but found most of the cases are related to installing Gentoo using stage3.
Any suggestions? Let me know if more info is needed that may help debugging. Thanks! |
|
Back to top |
|
|
mattst88 Developer
Joined: 28 Oct 2004 Posts: 422
|
Posted: Tue Dec 14, 2021 3:02 am Post subject: |
|
|
Obvious question: is there a /bin/bash in that directory? That is, is there a /usr/mips64el-unknown-linux-gnu/bin/bash? _________________ My Wiki page |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9815 Location: almost Mile High in the USA
|
Posted: Tue Dec 14, 2021 3:21 am Post subject: |
|
|
And then the next maybe not so obvious question, are /lib/libc.so.6, libtinfo, libtinfow, readline; and the dynamic library loader (ld-linux-mips64el*) there in the chroot? _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
manphiz n00b
Joined: 01 Dec 2021 Posts: 26
|
Posted: Tue Dec 14, 2021 10:01 pm Post subject: |
|
|
mattst88 wrote: | Obvious question: is there a /bin/bash in that directory? That is, is there a /usr/mips64el-unknown-linux-gnu/bin/bash? |
Yes it's there:
Code: | $ ls -l /usr/mips64el-unknown-linux-gnu/bin/bash
-rwxr-xr-x 1 root root 996268 Dec 12 21:44 /usr/mips64el-unknown-linux-gnu/bin/bash
$ ldd /usr/mips64el-unknown-linux-gnu/bin/bash
not a dynamic executable
|
eccerr0r wrote: | And then the next maybe not so obvious question, are /lib/libc.so.6, libtinfo, libtinfow, readline; and the dynamic library loader (ld-linux-mips64el*) there in the chroot? |
Some of them are there, but others don't.
Code: |
$ ls -l /usr/mips64el-unknown-linux-gnu/lib/libc.so.6
-rwxr-xr-x 1 root root 1969420 Dec 12 21:04 /usr/mips64el-unknown-linux-gnu/lib/libc.so.6
$ ldd /usr/mips64el-unknown-linux-gnu/lib/libc.so.6
not a dynamic executable
$ sudo find /usr/mips64el-unknown-linux-gnu -name "*libtinfo*" # No return
$ ls -l /usr/mips64el-unknown-linux-gnu/usr/lib/libreadline*
lrwxrwxrwx 1 root root 16 Dec 12 19:56 /usr/mips64el-unknown-linux-gnu/usr/lib/libreadline.so -> libreadline.so.8
lrwxrwxrwx 1 root root 18 Dec 12 19:56 /usr/mips64el-unknown-linux-gnu/usr/lib/libreadline.so.8 -> libreadline.so.8.1
-rwxr-xr-x 1 root root 308920 Dec 12 19:56 /usr/mips64el-unknown-linux-gnu/usr/lib/libreadline.so.8.1
$ ldd /usr/mips64el-unknown-linux-gnu/usr/lib/libreadline.so.8.1
not a dynamic executable
$ sudo find /usr/mips64el-unknown-linux-gnu -name "ld*"
/usr/mips64el-unknown-linux-gnu/sbin/ldconfig
/usr/mips64el-unknown-linux-gnu/usr/sbin/ldattach
/usr/mips64el-unknown-linux-gnu/usr/bin/lddtree
/usr/mips64el-unknown-linux-gnu/usr/bin/ld.gold
/usr/mips64el-unknown-linux-gnu/usr/bin/ldd
/usr/mips64el-unknown-linux-gnu/usr/bin/ld.bfd
/usr/mips64el-unknown-linux-gnu/usr/bin/ld
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/info/ld.info
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/man/man1/ld.1
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/uk/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/zh_TW/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/vi/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/ru/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/tr/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/fi/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/es/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/da/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/it/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/zh_CN/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/de/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/ja/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/pt_BR/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/sv/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/bg/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/fr/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/id/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/sr/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/ga/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/bash-completion/completions/ldattach
/usr/mips64el-unknown-linux-gnu/usr/mips64el-unknown-linux-gnu/binutils-bin/2.37_p1/ld.gold
/usr/mips64el-unknown-linux-gnu/usr/mips64el-unknown-linux-gnu/binutils-bin/2.37_p1/ld.bfd
/usr/mips64el-unknown-linux-gnu/usr/mips64el-unknown-linux-gnu/binutils-bin/2.37_p1/ld
/usr/mips64el-unknown-linux-gnu/usr/mips64el-unknown-linux-gnu/bin/ld.gold
/usr/mips64el-unknown-linux-gnu/usr/mips64el-unknown-linux-gnu/bin/ld.bfd
/usr/mips64el-unknown-linux-gnu/usr/mips64el-unknown-linux-gnu/bin/ld
/usr/mips64el-unknown-linux-gnu/usr/mips64el-unknown-linux-gnu/lib/ldscripts
/usr/mips64el-unknown-linux-gnu/usr/lib/binutils/mips64el-unknown-linux-gnu/2.37_p1/ldscripts
/usr/mips64el-unknown-linux-gnu/tmp/portage/dev-lang/python-3.9.9/work/Python-3.9.9/Modules/ld_so_aix
/usr/mips64el-unknown-linux-gnu/tmp/portage/dev-lang/python-3.9.9/work/Python-3.9.9/Modules/ld_so_aix.in
/usr/mips64el-unknown-linux-gnu/etc/ld.so.conf
/usr/mips64el-unknown-linux-gnu/etc/ld.so.conf.d
/usr/mips64el-unknown-linux-gnu/lib32/ld.so.1
/usr/mips64el-unknown-linux-gnu/lib/ld.so.1
|
|
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54575 Location: 56N 3W
|
Posted: Tue Dec 14, 2021 10:18 pm Post subject: |
|
|
manphiz,
Code: | $ cd /usr/mips64el-unknown-linux-gnu/ && sudo chroot . /bin/bash --login |
That's a bit convoluted.
What happens if you do that one command at a time, being root before the chroot command ?
I like to see which step breaks. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9815 Location: almost Mile High in the USA
|
Posted: Tue Dec 14, 2021 10:35 pm Post subject: |
|
|
I'm not sure you can use the host ldd on non native binaries though theoretically qemu should handle it, and the file is too small to be statically linked anyway so it's lying to you.
My guess is you still have qemu problems, binfmt problems, or your ld-linux-* is missing (should have been built with glibc), unfortunately I ran out of string over here... _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3407
|
Posted: Tue Dec 14, 2021 11:38 pm Post subject: |
|
|
Quote: | My guess is you still have qemu problems, binfmt problems, or your ld-linux-* is missing (should have been built with glibc), unfortunately I ran out of string over here... | I have recently set qemu-user for arch64 + musl chroot.
I build qemu with USE=static-user, copied it into chroot, and started qemu-binfmt service (that was the tricky part for me... Finding arch definitions).
@manphiz,
I suppose you missed step 2 from my list above: copying qemu binary to chroot. It would result in the very error message you posted. (Bash doesn't exist. Actually, no, it's interpreter that's missing there) |
|
Back to top |
|
|
manphiz n00b
Joined: 01 Dec 2021 Posts: 26
|
Posted: Tue Dec 14, 2021 11:48 pm Post subject: |
|
|
NeddySeagoon wrote: | manphiz,
Code: | $ cd /usr/mips64el-unknown-linux-gnu/ && sudo chroot . /bin/bash --login |
That's a bit convoluted.
What happens if you do that one command at a time, being root before the chroot command ?
I like to see which step breaks. |
I tried this step by step and confirmed that it's the chroot command that failed:
Code: | gentoo-vm /usr/mips64el-unknown-linux-gnu # chroot . /bin/bash --login
chroot: failed to run command ‘/bin/bash’: No such file or directory
|
|
|
Back to top |
|
|
manphiz n00b
Joined: 01 Dec 2021 Posts: 26
|
Posted: Tue Dec 14, 2021 11:50 pm Post subject: |
|
|
szatox wrote: | Quote: | My guess is you still have qemu problems, binfmt problems, or your ld-linux-* is missing (should have been built with glibc), unfortunately I ran out of string over here... | I have recently set qemu-user for arch64 + musl chroot.
I build qemu with USE=static-user, copied it into chroot, and started qemu-binfmt service (that was the tricky part for me... Finding arch definitions).
@manphiz,
I suppose you missed step 2 from my list above: copying qemu binary to chroot. It would result in the very error message you posted. (Bash doesn't exist. Actually, no, it's interpreter that's missing there) |
Thanks szatox. I was following the steps in https://wiki.gentoo.org/wiki/Cross_build_environment. More specifically the following steps:
Quote: |
1) install qemu on the host
root #emerge --ask app-emulation/qemu
2) prepare qemu for the target (in this case for the arm architecture)
root #QEMU_USER_TARGETS="arm" QEMU_SOFTMMU_TARGETS="arm" USE="static-user static-libs" emerge --ask --buildpkg --oneshot qemu
3) install qemu to the build environment
root #cd /usr/<TARGET> && ROOT=$PWD/ emerge --ask --usepkgonly --oneshot --nodeps qemu
4) A first test:
root #/etc/init.d/qemu-binfmt start && cd /usr/<TARGET> && chroot . /bin/bash --login
|
I thought step 3 should perform the installation of qemu-user into the chroot, no? What should be the correct way to achieve this? |
|
Back to top |
|
|
pingtoo Veteran
Joined: 10 Sep 2021 Posts: 1214 Location: Richmond Hill, Canada
|
Posted: Wed Dec 15, 2021 12:10 am Post subject: |
|
|
manphiz,
please share output for following
Code: | ls /proc/sys/fs/binfmt_misc |
you should have /proc/sys/fs/binfmt_misc/qemu-mips64el file, please
Code: | cat /proc/sys/fs/binfmt_mis/qemu-mips64el |
expect, (ignore those mark as <example>, I don't know what they should be on your machine)
Quote: |
enabled
interpreter /usr/bin/qemu-mips64el
flags: <example>
offset <example>
magic <example>
mask <example> |
finally you should have a file in /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el
and before chroot
Code: | file /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el | expect to see something like below
Code: | /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el: ELF 64-bit LSB executable, ..., statically linked, for GNU/Linux ... |
|
|
Back to top |
|
|
manphiz n00b
Joined: 01 Dec 2021 Posts: 26
|
Posted: Wed Dec 15, 2021 7:36 am Post subject: |
|
|
pingtoo wrote: | manphiz,
please share output for following
Code: | ls /proc/sys/fs/binfmt_misc |
you should have /proc/sys/fs/binfmt_misc/qemu-mips64el file
[..snip..] |
Thanks pingtoo! Looks like this reveals the problem as I don't have the user qemu for mips64el:
Code: | $ ls -l /proc/sys/fs/binfmt_misc/
total 0
-rw-r--r-- 1 root root 0 Dec 13 16:14 mips
-rw-r--r-- 1 root root 0 Dec 13 16:14 mipsel
--w------- 1 root root 0 Dec 13 15:40 register
-rw-r--r-- 1 root root 0 Dec 13 15:29 status
|
However trying to build user qemu for mips64el failed for me. The command I used was:
Code: | $ QEMU_USER_TARGETS="mips64el" QEMU_SOFTMMU_TARGETS="mips64el" USE="static-user static-libs" emerge --ask --buildpkg --oneshot qemu
|
The build log failed with the following log:
Quote: | ../configure --prefix=/usr --sysconfdir=/etc --bindir=/usr/bin --libdir=/usr/lib64 --datadir=/usr/share --docdir=/usr/share/doc/qemu-6.0.0-r54/html --mandir=/usr/share/man --localstatedi
r=/var --disable-bsd-user --disable-containers --disable-guest-agent --disable-strip --disable-tcg-interpreter --disable-werror --disable-gcrypt --python=/usr/bin/python3.9 --cc=x86_64-p
c-linux-gnu-gcc --cxx=x86_64-pc-linux-gnu-g++ --host-cc=x86_64-pc-linux-gnu-gcc --disable-debug-info --disable-debug-tcg --enable-docs --enable-gettext --disable-plugins --enable-attr --
disable-brlapi --disable-linux-aio --disable-bzip2 --disable-capstone --disable-cap-ng --disable-curl --disable-fdt --disable-fuse --disable-glusterfs --disable-gnutls --disable-nettle -
-disable-gtk --disable-rdma --disable-libiscsi --disable-linux-io-uring --disable-jemalloc --disable-vnc-jpeg --disable-kvm --disable-lzo --disable-mpath --disable-curses --disable-libnf
s --disable-numa --disable-opengl --disable-vnc-png --disable-rbd --disable-vnc-sasl --disable-sdl --disable-sdl-image --disable-seccomp --disable-slirp --disable-smartcard --disable-sna
ppy --disable-spice --disable-libssh --disable-libudev --disable-libusb --disable-usb-redir --disable-vde --disable-vhost-net --disable-vhost-user-fs --disable-virtiofsd --disable-virglr
enderer --disable-virtfs --disable-vnc --disable-vte --disable-xen --disable-xen-pci-passthrough --disable-xfsctl --disable-xkbcommon --disable-zstd --disable-libxml2 --enable-linux-user
--disable-system --disable-blobs --disable-tools --target-list=mips64el-linux-user --static --disable-pie
ERROR: sizeof(size_t) doesn't match GLIB_SIZEOF_SIZE_T.
You probably need to set PKG_CONFIG_LIBDIR
to point to the right pkg-config files for your
build target
|
Looks like "mips" or "mipsel" could succeed. Probably the mips64el used "n32" ABI instead of "n64" which somehow confuses the sizeof check?
Any suggestions? |
|
Back to top |
|
|
pingtoo Veteran
Joined: 10 Sep 2021 Posts: 1214 Location: Richmond Hill, Canada
|
Posted: Wed Dec 15, 2021 1:14 pm Post subject: |
|
|
This is bit of beyond me. Let me do some research see if I can find something about this. |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54575 Location: 56N 3W
|
Posted: Wed Dec 15, 2021 5:05 pm Post subject: |
|
|
manphiz,
I'm trying to recreate your problem here.
Code: | USE=static-user QEMU_SOFTMMU_TARGETS="mips64el x86_64" QEMU_USER_TARGETS="mips64el" emerge app-emulation/qemu -1av |
made me a static
Code: | $ lddtree /usr/bin/qemu-mips64el
/usr/bin/qemu-mips64el (interpreter => None |
Code: | crossdev -t mips64el-unknown-linux-gnu | was uneventful.
I've changed the profile in /usr/mips64el-unknown-linux-gnu/etc/portage from /var/db/repos/gentoo/profiles/embedded to /var/db/repos/gentoo/profiles/default/linux/mips/17.0/n32.
embedded is not useful and you seem to want an n32 system.
Its presently cross compiling @system, which is 487 packages and I have my binfmt service to fix, since this is a new install and I've never set it up.
I expect cross compiling @system to fail, but I've followed this path a few times now. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
szatox Advocate
Joined: 27 Aug 2013 Posts: 3407
|
Posted: Wed Dec 15, 2021 5:22 pm Post subject: |
|
|
manphiz wrote: |
Quote: |
1) install qemu on the host
root #emerge --ask app-emulation/qemu
2) prepare qemu for the target (in this case for the arm architecture)
root #QEMU_USER_TARGETS="arm" QEMU_SOFTMMU_TARGETS="arm" USE="static-user static-libs" emerge --ask --buildpkg --oneshot qemu
3) install qemu to the build environment
root #cd /usr/<TARGET> && ROOT=$PWD/ emerge --ask --usepkgonly --oneshot --nodeps qemu
4) A first test:
root #/etc/init.d/qemu-binfmt start && cd /usr/<TARGET> && chroot . /bin/bash --login
|
I thought step 3 should perform the installation of qemu-user into the chroot, no? What should be the correct way to achieve this? |
Oh, man, that's so overengineered I just used which qemu-<target> to find the binary and cp to copy it into chroot. Your approach is not _wrong_ though.
Anyway, you messed up qemu user target. You built qemu with user-target ARM but your chroot is MIPS64EL. Which is still in line with my earlier "it's the interpreter that's missing"
Qemu does support mips64el, so just fix that flag and enjoy your brand slow chroot! |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54575 Location: 56N 3W
|
Posted: Wed Dec 15, 2021 6:26 pm Post subject: |
|
|
szatox,
That's what the guide says.
The OP said Quote: | However trying to build user qemu for mips64el failed for me. The command I used was:
Code:
$ QEMU_USER_TARGETS="mips64el" QEMU_SOFTMMU_TARGETS="mips64el" USE="static-user static-libs" emerge --ask --buildpkg --oneshot qemu |
I didn't read the guide, so I didn't use USE=static-libs and it built here.
I don't have enough of the target root built yet to try a chroot but poking about with file looks promising.
-- edit --
Code: | qemu-mips64el -L /usr/mips64el-unknown-linux-gnu/ /usr/mips64el-unknown-linux-gnu/bin/bash
qemu-mips64el: /usr/mips64el-unknown-linux-gnu/bin/bash: Invalid ELF image for this architecture |
Code: | /usr/mips64el-unknown-linux-gnu # file bin/bash
bin/bash: ELF 32-bit LSB pie executable, MIPS, N32 MIPS-III version 1 (SYSV), dynamically linked, interpreter /lib32/ld.so.1, for GNU/Linux 3.2.0, stripped |
That suggests that qemu-mips64el cannot execute 32 bit binaries. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
manphiz n00b
Joined: 01 Dec 2021 Posts: 26
|
Posted: Thu Dec 16, 2021 9:46 am Post subject: |
|
|
szatox wrote: | [..snip..]
Anyway, you messed up qemu user target. You built qemu with user-target ARM but your chroot is MIPS64EL. Which is still in line with my earlier "it's the interpreter that's missing"
Qemu does support mips64el, so just fix that flag and enjoy your brand slow chroot! |
Thanks szatox. As NeddySeagoon pointed out, I was copying the texts from the wiki page. I did used the mips64el flags in the actual commands.
NeddySeagoon wrote: | [..snip..]
I didn't read the guide, so I didn't use USE=static-libs and it built here.
I don't have enough of the target root built yet to try a chroot but poking about with file looks promising.
|
Thanks NeddySeagoon! Dropping USE="static-user" made the qemu build succeed. Now I have /usr/mips64el-unknown-linux-gnu/usr/bin/{qemu-mips64el,qemu-system-mips64el} as pingtoo pointed out earlier.
Code: | # file /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el
/usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, stripped
# file /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-system-mips64el
/usr/mips64el-unknown-linux-gnu/usr/bin/qemu-system-mips64el: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, stripped
|
I've also find the correct code to register mips64el for binfmt in https://chromium.googlesource.com/external/qemu/+/stable-0.13/qemu-binfmt-conf.sh. Now my /proc/sys/fs/binfmt_misc/ looks like this
Code: | # ls /proc/sys/fs/binfmt_misc/
mips mips64el mipsel register status
|
I've also restarted systemd-bimfmt
Code: | # systemctl restart systemd-binfmt
# systemctl status systemd-binfmt
● systemd-binfmt.service - Set Up Additional Binary Formats
Loaded: loaded (/lib/systemd/system/systemd-binfmt.service; static)
Active: active (exited) since Thu 2021-12-16 01:47:11 PST; 4s ago
Docs: man:systemd-binfmt.service(8)
man:binfmt.d(5)
https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html
https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
Process: 3338118 ExecStart=/lib/systemd/systemd-binfmt (code=exited, status=0/SUCCESS)
Main PID: 3338118 (code=exited, status=0/SUCCESS)
CPU: 3ms
Dec 16 01:47:11 gentoo-vm systemd[1]: Starting Set Up Additional Binary Formats...
Dec 16 01:47:11 gentoo-vm systemd[1]: Finished Set Up Additional Binary Formats.
|
However, I'm still unable to chroot to /usr/mips64el-unknown-linux-gnu, and the error message is the same.
Code: | # chroot . /bin/bash --login
chroot: failed to run command ‘/bin/bash’: No such file or directory
# file /usr/mips64el-unknown-linux-gnu/bin/bash
/usr/mips64el-unknown-linux-gnu/bin/bash: ELF 32-bit LSB pie executable, MIPS, N32 MIPS-III version 1 (SYSV), dynamically linked, interpreter /lib32/ld.so.1, for GNU/Linux 3.2.0, stripped
|
The only thing I know I could be missing from previous suggests is the "ld-linux*" related binaries under /usr/mips64el-unknown-linux-gnu. Other ld related binaries are there. I have tried to re-emerge glibc and still cannot find that file anywhere.
Any suggestions are appreciated.
EDIT: Added systemd restarting on binfmt.
Update: Just saw NeddySeagoon's update. If qemu-mips64el really cannot run n32 binaries, is this a limitation of qemu? I assume mipsel won't work either as it will only work for o32 binaries. |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54575 Location: 56N 3W
|
Posted: Thu Dec 16, 2021 10:09 am Post subject: |
|
|
manphiz,
Code: | # file /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el
/usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, stripped |
That's a problem. It says dynamically linked. It has to be statically linked as its the only code inside the chroot that will be x86-64. It can't load libraries for outside the chroot.
That's what static-user does. You need
Code: | file /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el
/usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, stripped |
Your /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el cannot work. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
manphiz n00b
Joined: 01 Dec 2021 Posts: 26
|
Posted: Thu Dec 16, 2021 3:47 pm Post subject: |
|
|
NeddySeagoon wrote: | manphiz,
Code: | # file /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el
/usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, stripped |
That's a problem. It says dynamically linked. It has to be statically linked as its the only code inside the chroot that will be x86-64. It can't load libraries for outside the chroot.
That's what static-user does. You need
Code: | file /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el
/usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, stripped |
Your /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el cannot work. |
Thanks Neddy! I tried to compile using your previous command:
Code: | $ time USE=static-user QEMU_SOFTMMU_TARGETS="mips64el x86_64" QEMU_USER_TARGETS="mips64el" emerge app-emulation/qemu -1av
|
However it resulted in the same error I saw before:
Code: | [..snip..]
Found ninja-1.10.2 at /usr/bin/ninja
../configure --prefix=/usr --sysconfdir=/etc --bindir=/usr/bin --libdir=/usr/lib64 --datadir=/usr/share --docdir=/usr/share/doc/qemu-6.0.0-r54/html --mandir=/usr/share/man --localstatedir=/var --disable-bsd-user --disable-containers --disable-guest-agent --disable-strip --disable-tcg-interpreter --disable-werror --disable-gcrypt --python=/usr/bin/python3.9 --cc=x86_64-pc-linux-gnu-gcc --cxx=x86_64-pc-linux-gnu-g++ --host-cc=x86_64-pc-linux-gnu-gcc --disable-debug-info --disable-debug-tcg --enable-docs --enable-gettext --disable-plugins --enable-attr --disable-brlapi --disable-linux-aio --disable-bzip2 --disable-capstone --disable-cap-ng --disable-curl --disable-fdt --disable-fuse --disable-glusterfs --disable-gnutls --disable-nettle --disable-gtk --disable-rdma --disable-libiscsi --disable-linux-io-uring --disable-jemalloc --disable-vnc-jpeg --disable-kvm --disable-lzo --disable-mpath --disable-curses --disable-libnfs --disable-numa --disable-opengl --disable-vnc-png --disable-rbd --disable-vnc-sasl --disable-sdl --disable-sdl-image --disable-seccomp --disable-slirp --disable-smartcard --disable-snappy --disable-spice --disable-libssh --disable-libudev --disable-libusb --disable-usb-redir --disable-vde --disable-vhost-net --disable-vhost-user-fs --disable-virtiofsd --disable-virglrenderer --disable-virtfs --disable-vnc --disable-vte --disable-xen --disable-xen-pci-passthrough --disable-xfsctl --disable-xkbcommon --disable-zstd --disable-libxml2 --enable-linux-user --disable-system --disable-blobs --disable-tools --target-list=mips64el-linux-user --static --disable-pie
ERROR: sizeof(size_t) doesn't match GLIB_SIZEOF_SIZE_T.
You probably need to set PKG_CONFIG_LIBDIR
to point to the right pkg-config files for your
build target
* ERROR: app-emulation/qemu-6.0.0-r54::gentoo failed (configure phase):
* configure failed
*
* Call stack:
* ebuild.sh, line 127: Called src_configure
* environment, line 4219: Called qemu_src_configure 'user'
* environment, line 4077: Called die
* The specific snippet of code:
* ../configure "${conf_opts[@]}" || die "configure failed";
*
* If you need support, post the output of `emerge --info '=app-emulation/qemu-6.0.0-r54::gentoo'`,
* the complete build log and the output of `emerge -pqv '=app-emulation/qemu-6.0.0-r54::gentoo'`.
* The complete build log is located at '/var/tmp/portage/app-emulation/qemu-6.0.0-r54/temp/build.log'.
* The ebuild environment file is located at '/var/tmp/portage/app-emulation/qemu-6.0.0-r54/temp/environment'.
* Working directory: '/var/tmp/portage/app-emulation/qemu-6.0.0-r54/work/qemu-6.0.0/user-build'
* S: '/var/tmp/portage/app-emulation/qemu-6.0.0-r54/work/qemu-6.0.0'
|
Any idea on how to fix this is appreciated |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54575 Location: 56N 3W
|
Posted: Thu Dec 16, 2021 4:02 pm Post subject: |
|
|
manphiz,
Make friends with wgetpaste and put the entire build log onto a pastebin please.
Code: | * The complete build log is located at '/var/tmp/portage/app-emulation/qemu-6.0.0-r54/temp/build.log'. |
I'm all testing here so I have app-emulation/qemu-6.1.0-r1. You have the newest stable, which is 6.0.0-r54.
I wonder if that matters.
Your version builds here with Code: | USE="static-user -doc" QEMU_SOFTMMU_TARGETS="mips64el x86_64" QEMU_USER_TARGETS="mips64el" emerge =app-emulation/qemu-6.0.0-r54 -1av | nte USE=-doc is to avoid a documentation related build error.
The complete build log may help. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54575 Location: 56N 3W
|
Posted: Thu Dec 16, 2021 9:01 pm Post subject: |
|
|
manphiz,
Some progress. I was getting Code: | Invalid ELF image for this architecture | errors, so I built all the mips* targets for qemu.
Now I can chroot to a statically linked busybox.
Outside the chroot Code: | # ps -Alf | grep mips
4 S root 25964 11599 0 80 0 - 55383 do_sys 20:42 pts/1 00:00:00 /usr/bin/qemu-mipsn32el /bin/busybox sh
0 S root 25970 27949 0 80 0 - 2026 pipe_r 20:43 pts/5 00:00:00 grep --colour=auto mips |
It picked qemu-mipsn32el to run the code.
That's not enough though. Anything that needs the linking loader returns
Code: | / # gcc -V
/usr/bin/gcc: error while loading shared libraries: /lib32/libc.so.6: cannot read file data |
Chrooting to bash does not work yet. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
pingtoo Veteran
Joined: 10 Sep 2021 Posts: 1214 Location: Richmond Hill, Canada
|
Posted: Thu Dec 16, 2021 10:38 pm Post subject: |
|
|
manphiz,
I too successful build qemu with mips64el. I am use gcc-10 and build with older qemu-6.0.0-r2. however I doubt those matter since the error happen at early configuration stage. I found a online error report similar to your error, even thought it is not same arch (for m68k) but may indicate your environment have same setup.
qemu emerge command line I used Code: | USE="static-user -gtk -vte" QEMU_SOFTMMU_TARGETS="mips64el" QEMU_USER_TARGETS="mips64el" emerge app-emulation/qemu -1kgav --binpkg-respect-use=y |
my glib build Code: | [binary R ] dev-libs/glib-2.68.3-r1-2:2::gentoo USE="elf mime static-libs xattr -dbus -debug -fam -gtk-doc (-selinux) (-sysprof) -systemtap -test -utils" 0 KiB |
my qemu build Code: | [binary R ] app-emulation/qemu-6.0.0-r2-1::gentoo USE="aio bzip2 caps curl doc fdt filecaps fuse jpeg lzo ncurses nfs nls oss pin-upstream-blobs png seccomp slirp static-user vhost-net vnc xattr xfs zstd -accessibility -alsa (-capstone) -debug (-glusterfs) -gnutls -gtk (-infiniband) (-io-uring) -iscsi -jack -jemalloc -multipath -numa -opengl -plugins -pulseaudio -python (-rbd) -sasl -sdl -sdl-image (-selinux) (-smartcard) -snappy -spice -ssh -static -systemtap -test -udev -usb -usbredir -vde -vhost-user-fs (-virgl) -virtfs -vte (-xen)" PYTHON_TARGETS="python3_9 (-python3_10) -python3_8" QEMU_SOFTMMU_TARGETS="mips64el -aarch64 -alpha -arm -avr -cris -hppa -i386 -lm32 -m68k -microblaze -microblazeel -mips -mips64 -mipsel -moxie -nios2 -or1k -ppc -ppc64 -riscv32 -riscv64 -rx -s390x -sh4 -sh4eb -sparc -sparc64 -tricore -unicore32 -x86_64 -xtensa -xtensaeb" QEMU_USER_TARGETS="mips64el -aarch64 -aarch64_be -alpha -arm -armeb -cris -hexagon -hppa -i386 -m68k -microblaze -microblazeel -mips -mips64 -mipsel -mipsn32 -mipsn32el -nios2 -or1k -ppc -ppc64 -ppc64abi32 -ppc64le -riscv32 -riscv64 -s390x -sh4 -sh4eb -sparc -sparc32plus -sparc64 -x86_64 -xtensa -xtensaeb" 0 KiB |
|
|
Back to top |
|
|
eccerr0r Watchman
Joined: 01 Jul 2004 Posts: 9815 Location: almost Mile High in the USA
|
Posted: Fri Dec 17, 2021 2:36 am Post subject: |
|
|
hmm. I tried setting up qemu-arm to auto run binaries and tried to chroot into a raspbian rootfs. Well, didn't work, same outcome as the OP so must be doing something wrong too. If I tried running without chroot, it gives me an error that it can't find the dynamic loader (ld-linux-arm), which is expected as there's only the native ld-linux installed there on root. Not sure how much actually ran there.
now I don't know if there are any static binaries in the raspbian rootfs to play with to see if there's any sign of life... _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
|
manphiz n00b
Joined: 01 Dec 2021 Posts: 26
|
Posted: Fri Dec 17, 2021 9:12 am Post subject: |
|
|
Thanks guys! So I figured out that to properly build qemu with USE="static-user static" I need to build all its dependencies with "static-libs". Now I have this
Code: | gentoo-vm /usr/mips64el-unknown-linux-gnu # file /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips*
/usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, stripped
/usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mipsn32el: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, stripped
gentoo-vm /usr/mips64el-unknown-linux-gnu # file /usr/mips64el-unknown-linux-gnu/bin/bash
/usr/mips64el-unknown-linux-gnu/bin/bash: ELF 32-bit LSB pie executable, MIPS, N32 MIPS-III version 1 (SYSV), dynamically linked, interpreter /lib32/ld.so.1, for GNU/Linux 3.2.0, stripped
gentoo-vm /usr/mips64el-unknown-linux-gnu # file /usr/mips64el-unknown-linux-gnu/lib32/ld.so.1
/usr/mips64el-unknown-linux-gnu/lib32/ld.so.1: ELF 32-bit LSB shared object, MIPS, N32 MIPS-III version 1 (SYSV), static-pie linked, not stripped
|
However I still cannot chroot due to the same issue:
Code: | gentoo-vm /usr/mips64el-unknown-linux-gnu # chroot . /bin/bash --login
chroot: failed to run command ‘/bin/bash’: No such file or directory
|
I wonder whether I need to build bash to be static? However it doesn't seem to have a static USE flag. I'm still confused. |
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54575 Location: 56N 3W
|
Posted: Fri Dec 17, 2021 11:56 am Post subject: |
|
|
manphiz,
Lets rewind a little. Your stated objective was to make a Quote: | mips64el n32 chroot for loongson2f |
I tried that with the Code: | default/linux/mips/17.0/n32 | profile and got the same errors as your report.
So I built qemu with all the mips targets, all static-user and it picked
Code: | # ps -Alf | grep mips
4 S root 25964 11599 0 80 0 - 55383 do_sys 20:42 pts/1 00:00:00 /usr/bin/qemu-mipsn32el /bin/busybox sh
0 S root 25970 27949 0 80 0 - 2026 pipe_r 20:43 pts/5 00:00:00 grep --colour=auto mips | to run a statically linked busybox.
That's the qemu-mipsn32el emulator, not qemu-mipsn64el as I was expecting.
It looked promising but running dynamically liked programs failed with
Code: | / # gcc -V
/usr/bin/gcc: error while loading shared libraries: /lib32/libc.so.6: cannot read file data
|
Next up, try rebuilding it all for -march=loongson2f. QEMU didn't like that. GCC appears to plant opcodes that qemu does not have.
Digging a bigger hole ... the Loongson2f is a 64 bit machine and poking about in the old stage3 files, there is a multilib offering.
So choose the Code: | default/linux/mips/17.0/multilib/n32 | profile in the chroot and cross compile it again.
Now glibc won't build. :)
I don't have a Loongson2f to play with so lets start from the other end. The old stages worked. They probably still work.
Which stage3 would you start with and why, to install when those stages were current?
They will probably run in a qemu chroot today and that would tell which version of the mips emulator you need.
Right now, the number of unknowns is growing and we need something to prune them. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
manphiz n00b
Joined: 01 Dec 2021 Posts: 26
|
Posted: Fri Dec 17, 2021 11:44 pm Post subject: |
|
|
Thanks Neddy. It looks to me that maybe qemu doesn't handle mips64el n32 ABI correctly. I'll see whether I can get the old Loongson2f image work in qemu-mips64el.
Also, can you show me which command you used to chroot to "/usr/mips64el-unknown-linux-gcc"? I suspect that my binfmt may not be set up correctly which could explain why my chroot failed. |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|