Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Cannot find /bin/bash when chrooting during crossdev
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Gentoo on Alternative Architectures
View previous topic :: View next topic  
Author Message
manphiz
n00b
n00b


Joined: 01 Dec 2021
Posts: 26

PostPosted: Tue Dec 14, 2021 12:37 am    Post subject: Cannot find /bin/bash when chrooting during crossdev Reply with quote

Hi,

I've been trying to create a mips64el n32 chroot for loongson2f by following the instructions in https://wiki.gentoo.org/wiki/Cross_build_environment. I've made some progress in creating the crossdev using "crossdev -t mips64el-unknown-linux-gnu", cross-compiling @system under /usr/mips64-unknown-linux-gnu, and compiling most of the essential packages except Python-3.9.9, which AIUI doesn't support cross compiling and needs to be done in chroot.

I've also compiled qemu on the host and the static qemu for mips. One difference is that my Gentoo is using systemd so I followed the instructions in https://wiki.gentoo.org/wiki/Embedded_Handbook/General/Full to create the bits required by systemd so that it can start systemd-binfmt, more specifically I've prepared the binfmt-support for both mips and mipsel under /etc/binfmt.d. When trying to chroot to /usr/mips64el-unknown-linux-gnu I was hit by this error:
Quote:
$ cd /usr/mips64el-unknown-linux-gnu/ && sudo chroot . /bin/bash --login
chroot: failed to run command ‘/bin/bash’: No such file or directory


I tried to search for this error but found most of the cases are related to installing Gentoo using stage3.

Any suggestions? Let me know if more info is needed that may help debugging. Thanks!
Back to top
View user's profile Send private message
mattst88
Developer
Developer


Joined: 28 Oct 2004
Posts: 422

PostPosted: Tue Dec 14, 2021 3:02 am    Post subject: Reply with quote

Obvious question: is there a /bin/bash in that directory? That is, is there a /usr/mips64el-unknown-linux-gnu/bin/bash?
_________________
My Wiki page
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 9826
Location: almost Mile High in the USA

PostPosted: Tue Dec 14, 2021 3:21 am    Post subject: Reply with quote

And then the next maybe not so obvious question, are /lib/libc.so.6, libtinfo, libtinfow, readline; and the dynamic library loader (ld-linux-mips64el*) there in the chroot?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
manphiz
n00b
n00b


Joined: 01 Dec 2021
Posts: 26

PostPosted: Tue Dec 14, 2021 10:01 pm    Post subject: Reply with quote

mattst88 wrote:
Obvious question: is there a /bin/bash in that directory? That is, is there a /usr/mips64el-unknown-linux-gnu/bin/bash?


Yes it's there:

Code:
$ ls -l /usr/mips64el-unknown-linux-gnu/bin/bash
-rwxr-xr-x 1 root root 996268 Dec 12 21:44 /usr/mips64el-unknown-linux-gnu/bin/bash
$ ldd /usr/mips64el-unknown-linux-gnu/bin/bash
        not a dynamic executable


eccerr0r wrote:
And then the next maybe not so obvious question, are /lib/libc.so.6, libtinfo, libtinfow, readline; and the dynamic library loader (ld-linux-mips64el*) there in the chroot?


Some of them are there, but others don't.

Code:

$ ls -l /usr/mips64el-unknown-linux-gnu/lib/libc.so.6
-rwxr-xr-x 1 root root 1969420 Dec 12 21:04 /usr/mips64el-unknown-linux-gnu/lib/libc.so.6
$ ldd /usr/mips64el-unknown-linux-gnu/lib/libc.so.6
        not a dynamic executable
$ sudo find /usr/mips64el-unknown-linux-gnu -name "*libtinfo*"  # No return
$ ls -l /usr/mips64el-unknown-linux-gnu/usr/lib/libreadline*
lrwxrwxrwx 1 root root     16 Dec 12 19:56 /usr/mips64el-unknown-linux-gnu/usr/lib/libreadline.so -> libreadline.so.8
lrwxrwxrwx 1 root root     18 Dec 12 19:56 /usr/mips64el-unknown-linux-gnu/usr/lib/libreadline.so.8 -> libreadline.so.8.1
-rwxr-xr-x 1 root root 308920 Dec 12 19:56 /usr/mips64el-unknown-linux-gnu/usr/lib/libreadline.so.8.1
$ ldd /usr/mips64el-unknown-linux-gnu/usr/lib/libreadline.so.8.1
        not a dynamic executable
$ sudo find /usr/mips64el-unknown-linux-gnu -name "ld*"
/usr/mips64el-unknown-linux-gnu/sbin/ldconfig
/usr/mips64el-unknown-linux-gnu/usr/sbin/ldattach
/usr/mips64el-unknown-linux-gnu/usr/bin/lddtree
/usr/mips64el-unknown-linux-gnu/usr/bin/ld.gold
/usr/mips64el-unknown-linux-gnu/usr/bin/ldd
/usr/mips64el-unknown-linux-gnu/usr/bin/ld.bfd
/usr/mips64el-unknown-linux-gnu/usr/bin/ld
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/info/ld.info
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/man/man1/ld.1
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/uk/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/zh_TW/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/vi/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/ru/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/tr/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/fi/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/es/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/da/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/it/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/zh_CN/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/de/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/ja/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/pt_BR/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/sv/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/bg/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/fr/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/id/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/sr/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/binutils-data/mips64el-unknown-linux-gnu/2.37_p1/locale/ga/LC_MESSAGES/ld.mo
/usr/mips64el-unknown-linux-gnu/usr/share/bash-completion/completions/ldattach
/usr/mips64el-unknown-linux-gnu/usr/mips64el-unknown-linux-gnu/binutils-bin/2.37_p1/ld.gold
/usr/mips64el-unknown-linux-gnu/usr/mips64el-unknown-linux-gnu/binutils-bin/2.37_p1/ld.bfd
/usr/mips64el-unknown-linux-gnu/usr/mips64el-unknown-linux-gnu/binutils-bin/2.37_p1/ld
/usr/mips64el-unknown-linux-gnu/usr/mips64el-unknown-linux-gnu/bin/ld.gold
/usr/mips64el-unknown-linux-gnu/usr/mips64el-unknown-linux-gnu/bin/ld.bfd
/usr/mips64el-unknown-linux-gnu/usr/mips64el-unknown-linux-gnu/bin/ld
/usr/mips64el-unknown-linux-gnu/usr/mips64el-unknown-linux-gnu/lib/ldscripts
/usr/mips64el-unknown-linux-gnu/usr/lib/binutils/mips64el-unknown-linux-gnu/2.37_p1/ldscripts
/usr/mips64el-unknown-linux-gnu/tmp/portage/dev-lang/python-3.9.9/work/Python-3.9.9/Modules/ld_so_aix
/usr/mips64el-unknown-linux-gnu/tmp/portage/dev-lang/python-3.9.9/work/Python-3.9.9/Modules/ld_so_aix.in
/usr/mips64el-unknown-linux-gnu/etc/ld.so.conf
/usr/mips64el-unknown-linux-gnu/etc/ld.so.conf.d
/usr/mips64el-unknown-linux-gnu/lib32/ld.so.1
/usr/mips64el-unknown-linux-gnu/lib/ld.so.1
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54586
Location: 56N 3W

PostPosted: Tue Dec 14, 2021 10:18 pm    Post subject: Reply with quote

manphiz,

Code:
$ cd /usr/mips64el-unknown-linux-gnu/ && sudo chroot . /bin/bash --login


That's a bit convoluted.
What happens if you do that one command at a time, being root before the chroot command ?

I like to see which step breaks.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 9826
Location: almost Mile High in the USA

PostPosted: Tue Dec 14, 2021 10:35 pm    Post subject: Reply with quote

I'm not sure you can use the host ldd on non native binaries though theoretically qemu should handle it, and the file is too small to be statically linked anyway so it's lying to you.

My guess is you still have qemu problems, binfmt problems, or your ld-linux-* is missing (should have been built with glibc), unfortunately I ran out of string over here...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
szatox
Advocate
Advocate


Joined: 27 Aug 2013
Posts: 3449

PostPosted: Tue Dec 14, 2021 11:38 pm    Post subject: Reply with quote

Quote:
My guess is you still have qemu problems, binfmt problems, or your ld-linux-* is missing (should have been built with glibc), unfortunately I ran out of string over here...
I have recently set qemu-user for arch64 + musl chroot.

I build qemu with USE=static-user, copied it into chroot, and started qemu-binfmt service (that was the tricky part for me... Finding arch definitions).

@manphiz,
I suppose you missed step 2 from my list above: copying qemu binary to chroot. It would result in the very error message you posted. (Bash doesn't exist. Actually, no, it's interpreter that's missing there)
Back to top
View user's profile Send private message
manphiz
n00b
n00b


Joined: 01 Dec 2021
Posts: 26

PostPosted: Tue Dec 14, 2021 11:48 pm    Post subject: Reply with quote

NeddySeagoon wrote:
manphiz,

Code:
$ cd /usr/mips64el-unknown-linux-gnu/ && sudo chroot . /bin/bash --login


That's a bit convoluted.
What happens if you do that one command at a time, being root before the chroot command ?

I like to see which step breaks.


I tried this step by step and confirmed that it's the chroot command that failed:

Code:
gentoo-vm /usr/mips64el-unknown-linux-gnu # chroot . /bin/bash --login
chroot: failed to run command ‘/bin/bash’: No such file or directory
Back to top
View user's profile Send private message
manphiz
n00b
n00b


Joined: 01 Dec 2021
Posts: 26

PostPosted: Tue Dec 14, 2021 11:50 pm    Post subject: Reply with quote

szatox wrote:
Quote:
My guess is you still have qemu problems, binfmt problems, or your ld-linux-* is missing (should have been built with glibc), unfortunately I ran out of string over here...
I have recently set qemu-user for arch64 + musl chroot.

I build qemu with USE=static-user, copied it into chroot, and started qemu-binfmt service (that was the tricky part for me... Finding arch definitions).

@manphiz,
I suppose you missed step 2 from my list above: copying qemu binary to chroot. It would result in the very error message you posted. (Bash doesn't exist. Actually, no, it's interpreter that's missing there)


Thanks szatox. I was following the steps in https://wiki.gentoo.org/wiki/Cross_build_environment. More specifically the following steps:

Quote:

1) install qemu on the host
root #emerge --ask app-emulation/qemu

2) prepare qemu for the target (in this case for the arm architecture)
root #QEMU_USER_TARGETS="arm" QEMU_SOFTMMU_TARGETS="arm" USE="static-user static-libs" emerge --ask --buildpkg --oneshot qemu

3) install qemu to the build environment
root #cd /usr/<TARGET> && ROOT=$PWD/ emerge --ask --usepkgonly --oneshot --nodeps qemu

4) A first test:
root #/etc/init.d/qemu-binfmt start && cd /usr/<TARGET> && chroot . /bin/bash --login


I thought step 3 should perform the installation of qemu-user into the chroot, no? What should be the correct way to achieve this?
Back to top
View user's profile Send private message
pingtoo
Veteran
Veteran


Joined: 10 Sep 2021
Posts: 1268
Location: Richmond Hill, Canada

PostPosted: Wed Dec 15, 2021 12:10 am    Post subject: Reply with quote

manphiz,

please share output for following

Code:
ls /proc/sys/fs/binfmt_misc


you should have /proc/sys/fs/binfmt_misc/qemu-mips64el file, please

Code:
cat /proc/sys/fs/binfmt_mis/qemu-mips64el


expect, (ignore those mark as <example>, I don't know what they should be on your machine)
Quote:

enabled
interpreter /usr/bin/qemu-mips64el
flags: <example>
offset <example>
magic <example>
mask <example>


finally you should have a file in /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el

and before chroot

Code:
file /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el
expect to see something like below
Code:
/usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el: ELF 64-bit LSB executable, ..., statically linked, for GNU/Linux ...
Back to top
View user's profile Send private message
manphiz
n00b
n00b


Joined: 01 Dec 2021
Posts: 26

PostPosted: Wed Dec 15, 2021 7:36 am    Post subject: Reply with quote

pingtoo wrote:
manphiz,

please share output for following

Code:
ls /proc/sys/fs/binfmt_misc


you should have /proc/sys/fs/binfmt_misc/qemu-mips64el file
[..snip..]


Thanks pingtoo! Looks like this reveals the problem as I don't have the user qemu for mips64el:
Code:
$ ls -l /proc/sys/fs/binfmt_misc/
total 0
-rw-r--r-- 1 root root 0 Dec 13 16:14 mips
-rw-r--r-- 1 root root 0 Dec 13 16:14 mipsel
--w------- 1 root root 0 Dec 13 15:40 register
-rw-r--r-- 1 root root 0 Dec 13 15:29 status


However trying to build user qemu for mips64el failed for me. The command I used was:
Code:
$ QEMU_USER_TARGETS="mips64el" QEMU_SOFTMMU_TARGETS="mips64el" USE="static-user static-libs" emerge --ask --buildpkg --oneshot qemu


The build log failed with the following log:
Quote:
../configure --prefix=/usr --sysconfdir=/etc --bindir=/usr/bin --libdir=/usr/lib64 --datadir=/usr/share --docdir=/usr/share/doc/qemu-6.0.0-r54/html --mandir=/usr/share/man --localstatedi
r=/var --disable-bsd-user --disable-containers --disable-guest-agent --disable-strip --disable-tcg-interpreter --disable-werror --disable-gcrypt --python=/usr/bin/python3.9 --cc=x86_64-p
c-linux-gnu-gcc --cxx=x86_64-pc-linux-gnu-g++ --host-cc=x86_64-pc-linux-gnu-gcc --disable-debug-info --disable-debug-tcg --enable-docs --enable-gettext --disable-plugins --enable-attr --
disable-brlapi --disable-linux-aio --disable-bzip2 --disable-capstone --disable-cap-ng --disable-curl --disable-fdt --disable-fuse --disable-glusterfs --disable-gnutls --disable-nettle -
-disable-gtk --disable-rdma --disable-libiscsi --disable-linux-io-uring --disable-jemalloc --disable-vnc-jpeg --disable-kvm --disable-lzo --disable-mpath --disable-curses --disable-libnf
s --disable-numa --disable-opengl --disable-vnc-png --disable-rbd --disable-vnc-sasl --disable-sdl --disable-sdl-image --disable-seccomp --disable-slirp --disable-smartcard --disable-sna
ppy --disable-spice --disable-libssh --disable-libudev --disable-libusb --disable-usb-redir --disable-vde --disable-vhost-net --disable-vhost-user-fs --disable-virtiofsd --disable-virglr
enderer --disable-virtfs --disable-vnc --disable-vte --disable-xen --disable-xen-pci-passthrough --disable-xfsctl --disable-xkbcommon --disable-zstd --disable-libxml2 --enable-linux-user
--disable-system --disable-blobs --disable-tools --target-list=mips64el-linux-user --static --disable-pie

ERROR: sizeof(size_t) doesn't match GLIB_SIZEOF_SIZE_T.
You probably need to set PKG_CONFIG_LIBDIR
to point to the right pkg-config files for your
build target


Looks like "mips" or "mipsel" could succeed. Probably the mips64el used "n32" ABI instead of "n64" which somehow confuses the sizeof check?

Any suggestions?
Back to top
View user's profile Send private message
pingtoo
Veteran
Veteran


Joined: 10 Sep 2021
Posts: 1268
Location: Richmond Hill, Canada

PostPosted: Wed Dec 15, 2021 1:14 pm    Post subject: Reply with quote

This is bit of beyond me. Let me do some research see if I can find something about this.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54586
Location: 56N 3W

PostPosted: Wed Dec 15, 2021 5:05 pm    Post subject: Reply with quote

manphiz,

I'm trying to recreate your problem here.

Code:
USE=static-user QEMU_SOFTMMU_TARGETS="mips64el x86_64" QEMU_USER_TARGETS="mips64el" emerge app-emulation/qemu -1av

made me a static
Code:
$ lddtree /usr/bin/qemu-mips64el
/usr/bin/qemu-mips64el (interpreter => None


Code:
crossdev -t mips64el-unknown-linux-gnu
was uneventful.

I've changed the profile in /usr/mips64el-unknown-linux-gnu/etc/portage from /var/db/repos/gentoo/profiles/embedded to /var/db/repos/gentoo/profiles/default/linux/mips/17.0/n32.
embedded is not useful and you seem to want an n32 system.

Its presently cross compiling @system, which is 487 packages and I have my binfmt service to fix, since this is a new install and I've never set it up.
I expect cross compiling @system to fail, but I've followed this path a few times now.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
szatox
Advocate
Advocate


Joined: 27 Aug 2013
Posts: 3449

PostPosted: Wed Dec 15, 2021 5:22 pm    Post subject: Reply with quote

manphiz wrote:

Quote:

1) install qemu on the host
root #emerge --ask app-emulation/qemu

2) prepare qemu for the target (in this case for the arm architecture)
root #QEMU_USER_TARGETS="arm" QEMU_SOFTMMU_TARGETS="arm" USE="static-user static-libs" emerge --ask --buildpkg --oneshot qemu

3) install qemu to the build environment
root #cd /usr/<TARGET> && ROOT=$PWD/ emerge --ask --usepkgonly --oneshot --nodeps qemu

4) A first test:
root #/etc/init.d/qemu-binfmt start && cd /usr/<TARGET> && chroot . /bin/bash --login


I thought step 3 should perform the installation of qemu-user into the chroot, no? What should be the correct way to achieve this?

Oh, man, that's so overengineered :lol: I just used which qemu-<target> to find the binary and cp to copy it into chroot. Your approach is not _wrong_ though.
Anyway, you messed up qemu user target. You built qemu with user-target ARM but your chroot is MIPS64EL. Which is still in line with my earlier "it's the interpreter that's missing"
Qemu does support mips64el, so just fix that flag and enjoy your brand slow chroot!
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54586
Location: 56N 3W

PostPosted: Wed Dec 15, 2021 6:26 pm    Post subject: Reply with quote

szatox,

That's what the guide says.

The OP said
Quote:
However trying to build user qemu for mips64el failed for me. The command I used was:
Code:
$ QEMU_USER_TARGETS="mips64el" QEMU_SOFTMMU_TARGETS="mips64el" USE="static-user static-libs" emerge --ask --buildpkg --oneshot qemu


I didn't read the guide, so I didn't use USE=static-libs and it built here.
I don't have enough of the target root built yet to try a chroot but poking about with file looks promising.

-- edit --

Code:
qemu-mips64el -L /usr/mips64el-unknown-linux-gnu/ /usr/mips64el-unknown-linux-gnu/bin/bash
qemu-mips64el: /usr/mips64el-unknown-linux-gnu/bin/bash: Invalid ELF image for this architecture


Code:
/usr/mips64el-unknown-linux-gnu # file bin/bash
bin/bash: ELF 32-bit LSB pie executable, MIPS, N32 MIPS-III version 1 (SYSV), dynamically linked, interpreter /lib32/ld.so.1, for GNU/Linux 3.2.0, stripped


That suggests that qemu-mips64el cannot execute 32 bit binaries.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
manphiz
n00b
n00b


Joined: 01 Dec 2021
Posts: 26

PostPosted: Thu Dec 16, 2021 9:46 am    Post subject: Reply with quote

szatox wrote:
[..snip..]
Anyway, you messed up qemu user target. You built qemu with user-target ARM but your chroot is MIPS64EL. Which is still in line with my earlier "it's the interpreter that's missing"
Qemu does support mips64el, so just fix that flag and enjoy your brand slow chroot!


Thanks szatox. As NeddySeagoon pointed out, I was copying the texts from the wiki page. I did used the mips64el flags in the actual commands.

NeddySeagoon wrote:
[..snip..]
I didn't read the guide, so I didn't use USE=static-libs and it built here.
I don't have enough of the target root built yet to try a chroot but poking about with file looks promising.


Thanks NeddySeagoon! Dropping USE="static-user" made the qemu build succeed. Now I have /usr/mips64el-unknown-linux-gnu/usr/bin/{qemu-mips64el,qemu-system-mips64el} as pingtoo pointed out earlier.

Code:
# file /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el
/usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, stripped

# file /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-system-mips64el
/usr/mips64el-unknown-linux-gnu/usr/bin/qemu-system-mips64el: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, stripped


I've also find the correct code to register mips64el for binfmt in https://chromium.googlesource.com/external/qemu/+/stable-0.13/qemu-binfmt-conf.sh. Now my /proc/sys/fs/binfmt_misc/ looks like this
Code:
# ls /proc/sys/fs/binfmt_misc/
mips  mips64el  mipsel  register  status


I've also restarted systemd-bimfmt
Code:
# systemctl restart systemd-binfmt
# systemctl status systemd-binfmt
● systemd-binfmt.service - Set Up Additional Binary Formats
     Loaded: loaded (/lib/systemd/system/systemd-binfmt.service; static)
     Active: active (exited) since Thu 2021-12-16 01:47:11 PST; 4s ago
       Docs: man:systemd-binfmt.service(8)
             man:binfmt.d(5)
             https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html
             https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
    Process: 3338118 ExecStart=/lib/systemd/systemd-binfmt (code=exited, status=0/SUCCESS)
   Main PID: 3338118 (code=exited, status=0/SUCCESS)
        CPU: 3ms

Dec 16 01:47:11 gentoo-vm systemd[1]: Starting Set Up Additional Binary Formats...
Dec 16 01:47:11 gentoo-vm systemd[1]: Finished Set Up Additional Binary Formats.


However, I'm still unable to chroot to /usr/mips64el-unknown-linux-gnu, and the error message is the same.
Code:
# chroot . /bin/bash --login
chroot: failed to run command ‘/bin/bash’: No such file or directory
# file /usr/mips64el-unknown-linux-gnu/bin/bash
/usr/mips64el-unknown-linux-gnu/bin/bash: ELF 32-bit LSB pie executable, MIPS, N32 MIPS-III version 1 (SYSV), dynamically linked, interpreter /lib32/ld.so.1, for GNU/Linux 3.2.0, stripped


The only thing I know I could be missing from previous suggests is the "ld-linux*" related binaries under /usr/mips64el-unknown-linux-gnu. Other ld related binaries are there. I have tried to re-emerge glibc and still cannot find that file anywhere.

Any suggestions are appreciated.

EDIT: Added systemd restarting on binfmt.

Update: Just saw NeddySeagoon's update. If qemu-mips64el really cannot run n32 binaries, is this a limitation of qemu? I assume mipsel won't work either as it will only work for o32 binaries.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54586
Location: 56N 3W

PostPosted: Thu Dec 16, 2021 10:09 am    Post subject: Reply with quote

manphiz,

Code:
# file /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el
/usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, stripped


That's a problem. It says dynamically linked. It has to be statically linked as its the only code inside the chroot that will be x86-64. It can't load libraries for outside the chroot.
That's what static-user does. You need
Code:
file /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el
/usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, stripped


Your /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el cannot work.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
manphiz
n00b
n00b


Joined: 01 Dec 2021
Posts: 26

PostPosted: Thu Dec 16, 2021 3:47 pm    Post subject: Reply with quote

NeddySeagoon wrote:
manphiz,

Code:
# file /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el
/usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, stripped


That's a problem. It says dynamically linked. It has to be statically linked as its the only code inside the chroot that will be x86-64. It can't load libraries for outside the chroot.
That's what static-user does. You need
Code:
file /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el
/usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, stripped


Your /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el cannot work.


Thanks Neddy! I tried to compile using your previous command:

Code:
$ time USE=static-user QEMU_SOFTMMU_TARGETS="mips64el x86_64" QEMU_USER_TARGETS="mips64el" emerge app-emulation/qemu -1av


However it resulted in the same error I saw before:
Code:
[..snip..]

Found ninja-1.10.2 at /usr/bin/ninja
../configure --prefix=/usr --sysconfdir=/etc --bindir=/usr/bin --libdir=/usr/lib64 --datadir=/usr/share --docdir=/usr/share/doc/qemu-6.0.0-r54/html --mandir=/usr/share/man --localstatedir=/var --disable-bsd-user --disable-containers --disable-guest-agent --disable-strip --disable-tcg-interpreter --disable-werror --disable-gcrypt --python=/usr/bin/python3.9 --cc=x86_64-pc-linux-gnu-gcc --cxx=x86_64-pc-linux-gnu-g++ --host-cc=x86_64-pc-linux-gnu-gcc --disable-debug-info --disable-debug-tcg --enable-docs --enable-gettext --disable-plugins --enable-attr --disable-brlapi --disable-linux-aio --disable-bzip2 --disable-capstone --disable-cap-ng --disable-curl --disable-fdt --disable-fuse --disable-glusterfs --disable-gnutls --disable-nettle --disable-gtk --disable-rdma --disable-libiscsi --disable-linux-io-uring --disable-jemalloc --disable-vnc-jpeg --disable-kvm --disable-lzo --disable-mpath --disable-curses --disable-libnfs --disable-numa --disable-opengl --disable-vnc-png --disable-rbd --disable-vnc-sasl --disable-sdl --disable-sdl-image --disable-seccomp --disable-slirp --disable-smartcard --disable-snappy --disable-spice --disable-libssh --disable-libudev --disable-libusb --disable-usb-redir --disable-vde --disable-vhost-net --disable-vhost-user-fs --disable-virtiofsd --disable-virglrenderer --disable-virtfs --disable-vnc --disable-vte --disable-xen --disable-xen-pci-passthrough --disable-xfsctl --disable-xkbcommon --disable-zstd --disable-libxml2 --enable-linux-user --disable-system --disable-blobs --disable-tools --target-list=mips64el-linux-user --static --disable-pie

ERROR: sizeof(size_t) doesn't match GLIB_SIZEOF_SIZE_T.
       You probably need to set PKG_CONFIG_LIBDIR
       to point to the right pkg-config files for your
       build target

 * ERROR: app-emulation/qemu-6.0.0-r54::gentoo failed (configure phase):
 *   configure failed
 *
 * Call stack:
 *     ebuild.sh, line  127:  Called src_configure
 *   environment, line 4219:  Called qemu_src_configure 'user'
 *   environment, line 4077:  Called die
 * The specific snippet of code:
 *       ../configure "${conf_opts[@]}" || die "configure failed";
 *
 * If you need support, post the output of `emerge --info '=app-emulation/qemu-6.0.0-r54::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=app-emulation/qemu-6.0.0-r54::gentoo'`.
 * The complete build log is located at '/var/tmp/portage/app-emulation/qemu-6.0.0-r54/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/app-emulation/qemu-6.0.0-r54/temp/environment'.
 * Working directory: '/var/tmp/portage/app-emulation/qemu-6.0.0-r54/work/qemu-6.0.0/user-build'
 * S: '/var/tmp/portage/app-emulation/qemu-6.0.0-r54/work/qemu-6.0.0'


Any idea on how to fix this is appreciated :)
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54586
Location: 56N 3W

PostPosted: Thu Dec 16, 2021 4:02 pm    Post subject: Reply with quote

manphiz,

Make friends with wgetpaste and put the entire build log onto a pastebin please.

Code:
* The complete build log is located at '/var/tmp/portage/app-emulation/qemu-6.0.0-r54/temp/build.log'.

I'm all testing here so I have app-emulation/qemu-6.1.0-r1. You have the newest stable, which is 6.0.0-r54.
I wonder if that matters.

Your version builds here with
Code:
USE="static-user -doc" QEMU_SOFTMMU_TARGETS="mips64el x86_64" QEMU_USER_TARGETS="mips64el" emerge =app-emulation/qemu-6.0.0-r54 -1av
nte USE=-doc is to avoid a documentation related build error.

The complete build log may help.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54586
Location: 56N 3W

PostPosted: Thu Dec 16, 2021 9:01 pm    Post subject: Reply with quote

manphiz,

Some progress. I was getting
Code:
Invalid ELF image for this architecture
errors, so I built all the mips* targets for qemu.
Now I can chroot to a statically linked busybox.

Outside the chroot
Code:
# ps -Alf | grep mips
4 S root     25964 11599  0  80   0 - 55383 do_sys 20:42 pts/1    00:00:00 /usr/bin/qemu-mipsn32el /bin/busybox sh
0 S root     25970 27949  0  80   0 -  2026 pipe_r 20:43 pts/5    00:00:00 grep --colour=auto mips

It picked qemu-mipsn32el to run the code.

That's not enough though. Anything that needs the linking loader returns
Code:
/ # gcc -V
/usr/bin/gcc: error while loading shared libraries: /lib32/libc.so.6: cannot read file data


Chrooting to bash does not work yet.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
pingtoo
Veteran
Veteran


Joined: 10 Sep 2021
Posts: 1268
Location: Richmond Hill, Canada

PostPosted: Thu Dec 16, 2021 10:38 pm    Post subject: Reply with quote

manphiz,

I too successful build qemu with mips64el. I am use gcc-10 and build with older qemu-6.0.0-r2. however I doubt those matter since the error happen at early configuration stage. I found a online error report similar to your error, even thought it is not same arch (for m68k) but may indicate your environment have same setup.

qemu emerge command line I used
Code:
USE="static-user -gtk -vte" QEMU_SOFTMMU_TARGETS="mips64el" QEMU_USER_TARGETS="mips64el" emerge app-emulation/qemu -1kgav --binpkg-respect-use=y


my glib build
Code:
[binary   R    ] dev-libs/glib-2.68.3-r1-2:2::gentoo  USE="elf mime static-libs xattr -dbus -debug -fam -gtk-doc (-selinux) (-sysprof) -systemtap -test -utils" 0 KiB


my qemu build
Code:
[binary   R    ] app-emulation/qemu-6.0.0-r2-1::gentoo  USE="aio bzip2 caps curl doc fdt filecaps fuse jpeg lzo ncurses nfs nls oss pin-upstream-blobs png seccomp slirp static-user vhost-net vnc xattr xfs zstd -accessibility -alsa (-capstone) -debug (-glusterfs) -gnutls -gtk (-infiniband) (-io-uring) -iscsi -jack -jemalloc -multipath -numa -opengl -plugins -pulseaudio -python (-rbd) -sasl -sdl -sdl-image (-selinux) (-smartcard) -snappy -spice -ssh -static -systemtap -test -udev -usb -usbredir -vde -vhost-user-fs (-virgl) -virtfs -vte (-xen)" PYTHON_TARGETS="python3_9 (-python3_10) -python3_8" QEMU_SOFTMMU_TARGETS="mips64el -aarch64 -alpha -arm -avr -cris -hppa -i386 -lm32 -m68k -microblaze -microblazeel -mips -mips64 -mipsel -moxie -nios2 -or1k -ppc -ppc64 -riscv32 -riscv64 -rx -s390x -sh4 -sh4eb -sparc -sparc64 -tricore -unicore32 -x86_64 -xtensa -xtensaeb" QEMU_USER_TARGETS="mips64el -aarch64 -aarch64_be -alpha -arm -armeb -cris -hexagon -hppa -i386 -m68k -microblaze -microblazeel -mips -mips64 -mipsel -mipsn32 -mipsn32el -nios2 -or1k -ppc -ppc64 -ppc64abi32 -ppc64le -riscv32 -riscv64 -s390x -sh4 -sh4eb -sparc -sparc32plus -sparc64 -x86_64 -xtensa -xtensaeb" 0 KiB
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 9826
Location: almost Mile High in the USA

PostPosted: Fri Dec 17, 2021 2:36 am    Post subject: Reply with quote

hmm. I tried setting up qemu-arm to auto run binaries and tried to chroot into a raspbian rootfs. Well, didn't work, same outcome as the OP so must be doing something wrong too. If I tried running without chroot, it gives me an error that it can't find the dynamic loader (ld-linux-arm), which is expected as there's only the native ld-linux installed there on root. Not sure how much actually ran there.

now I don't know if there are any static binaries in the raspbian rootfs to play with to see if there's any sign of life...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
manphiz
n00b
n00b


Joined: 01 Dec 2021
Posts: 26

PostPosted: Fri Dec 17, 2021 9:12 am    Post subject: Reply with quote

Thanks guys! So I figured out that to properly build qemu with USE="static-user static" I need to build all its dependencies with "static-libs". Now I have this

Code:
gentoo-vm /usr/mips64el-unknown-linux-gnu # file /usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips*
/usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mips64el:  ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, stripped
/usr/mips64el-unknown-linux-gnu/usr/bin/qemu-mipsn32el: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, stripped
gentoo-vm /usr/mips64el-unknown-linux-gnu # file /usr/mips64el-unknown-linux-gnu/bin/bash
/usr/mips64el-unknown-linux-gnu/bin/bash: ELF 32-bit LSB pie executable, MIPS, N32 MIPS-III version 1 (SYSV), dynamically linked, interpreter /lib32/ld.so.1, for GNU/Linux 3.2.0, stripped
gentoo-vm /usr/mips64el-unknown-linux-gnu # file /usr/mips64el-unknown-linux-gnu/lib32/ld.so.1
/usr/mips64el-unknown-linux-gnu/lib32/ld.so.1: ELF 32-bit LSB shared object, MIPS, N32 MIPS-III version 1 (SYSV), static-pie linked, not stripped


However I still cannot chroot due to the same issue:

Code:
gentoo-vm /usr/mips64el-unknown-linux-gnu # chroot . /bin/bash --login
chroot: failed to run command ‘/bin/bash’: No such file or directory


I wonder whether I need to build bash to be static? However it doesn't seem to have a static USE flag. I'm still confused.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54586
Location: 56N 3W

PostPosted: Fri Dec 17, 2021 11:56 am    Post subject: Reply with quote

manphiz,

Lets rewind a little. Your stated objective was to make a
Quote:
mips64el n32 chroot for loongson2f

I tried that with the
Code:
default/linux/mips/17.0/n32
profile and got the same errors as your report.

So I built qemu with all the mips targets, all static-user and it picked
Code:
# ps -Alf | grep mips
4 S root     25964 11599  0  80   0 - 55383 do_sys 20:42 pts/1    00:00:00 /usr/bin/qemu-mipsn32el /bin/busybox sh
0 S root     25970 27949  0  80   0 -  2026 pipe_r 20:43 pts/5    00:00:00 grep --colour=auto mips
to run a statically linked busybox.
That's the qemu-mipsn32el emulator, not qemu-mipsn64el as I was expecting.
It looked promising but running dynamically liked programs failed with
Code:
/ # gcc -V
/usr/bin/gcc: error while loading shared libraries: /lib32/libc.so.6: cannot read file data


Next up, try rebuilding it all for -march=loongson2f. QEMU didn't like that. GCC appears to plant opcodes that qemu does not have.

Digging a bigger hole ... the Loongson2f is a 64 bit machine and poking about in the old stage3 files, there is a multilib offering.
So choose the
Code:
default/linux/mips/17.0/multilib/n32
profile in the chroot and cross compile it again.
Now glibc won't build. :)

I don't have a Loongson2f to play with so lets start from the other end. The old stages worked. They probably still work.
Which stage3 would you start with and why, to install when those stages were current?

They will probably run in a qemu chroot today and that would tell which version of the mips emulator you need.
Right now, the number of unknowns is growing and we need something to prune them.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
manphiz
n00b
n00b


Joined: 01 Dec 2021
Posts: 26

PostPosted: Fri Dec 17, 2021 11:44 pm    Post subject: Reply with quote

Thanks Neddy. It looks to me that maybe qemu doesn't handle mips64el n32 ABI correctly. I'll see whether I can get the old Loongson2f image work in qemu-mips64el.

Also, can you show me which command you used to chroot to "/usr/mips64el-unknown-linux-gcc"? I suspect that my binfmt may not be set up correctly which could explain why my chroot failed.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo on Alternative Architectures All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum