| View previous topic :: View next topic |
| Author |
Message |
Dominique_71
Veteran
Joined: 17 Aug 2005
Posts: 1895
Location: Switzerland (Romandie)
|
 Posted: Sat Nov 09, 2024 11:00 am Post subject: BASH: issue with grep and $[...]<.> in variables |
 |
|
I make a BASH script that scan a file with a list of files and create a temporary file with all the lines of theses files. These lines have 2 parts "key binding w/without modifiers command". That works fine.
From that, it create a preference file for later editing. As the original files can change over time, it have to compare each line of the temporary file with the preference file and copy into it only the new/changed lines according to an algorithm that begun to mostly works.
Extract of the temporary file:
| Code: | key XF86AudioStop A $[Mod1] Music-Speed normal mplayer
key XF86AudioPlay A $[Mod1]2 Music-Speed normal mplayer
Key XF86AudioRaiseVolume A $[Mod0] Change-Volume-Up $[infostore.Vol_Step]
Key XF86AudioRaiseVolume A $[Mod0]2 Change-Volume-Up $[infostore.Vol_Step]
|
Extract of the script:
| Code: | getcmd() {
cmds=$(echo $1 | grep -v -e "^#" -e "^$" | sed -e "s: * : :g" -e "s: * : :g" -e "s:.*Key:Key:" -e "s:'$::" | cut -d " " -f 5-)
#echo C: "${cmds}"
}
#getcmd
getkey() {
keys=$(echo $1 | grep -v -e "^#" -e "^$" | sed -e "s: * : :g" -e "s: * : :g" -e "s:.*Key:Key:" -e "s:'$::" | cut -d " " -f 1-4)
#echo K: "${keys}"
}
keyfuncexist() {
grep -F -e "$1" -e "$2" "$3"
# 1>/dev/null && echo found
}
write_bindingsfile() {
while read myline ; do
getcmd "${myline}"
getkey "${myline}"
# true | true == identical line exist; do nothing.
# if ! findidenticalline "${myline}" "${BindingsUserFile}"; then
echo C:"${cmds}" F:"${keys}" L:"${myline}"
keyfuncexist "${keys}" "${cmds}" "${BindingsUserFile}"
# fi
done < "${BindingsTmpFile}"
}
|
keyfuncexist don't work as expected. As example, if
cmds="key XF86AudioStop A $[Mod1] Music-Speed normal mplayer", the grep -F will return both
| Quote: | key XF86AudioStop A $[Mod1] Music-Speed normal mplayer
key XF86AudioPlay A $[Mod1]2 Music-Speed normal mplayer |
when I want only the exact match, the first line. If I remove the -F, it is even worst because grep want to interpret the $[...]. So I have no clue about what to do here. Any help will be appreciated.
_________________
"Confirm You are a robot." - the singularity |
|
| Back to top |
|
 |
Goverp
Advocate
Joined: 07 Mar 2007
Posts: 2179
|
| Posted: Sat Nov 09, 2024 2:26 pm Post subject: |
|
|
Dominique_71,
grep -e foo -e bah
searchs for strings matching either foo or bah.
I note your file contents has key spelt sometimes with a capital K, others with lower case k. Is that intentional? Should you specify "-i" in your greps?
keyfuncexist with those parameters will match either lines containing either "key" or "XF86AudioStop" in file "A". You probably wanted | Code: | | grep -F -e "$1 $2" "$3" |
which matches the string "$1 $2", i.e. "key XF86AudioStop".
_________________
Greybeard |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9612
Location: beyond the rim
|
| Posted: Sat Nov 09, 2024 2:47 pm Post subject: |
|
|
| I'd check if you can reimplement that in another language where you don't have to deal with 3+ different semantics for $ and other special characters that will sooner or later lead to problems with quoting/escaping. |
|
| Back to top |
|
|
Dominique_71
Veteran
Joined: 17 Aug 2005
Posts: 1895
Location: Switzerland (Romandie)
|
| Posted: Sun Nov 10, 2024 12:02 am Post subject: |
|
|
| Goverp wrote: | Dominique_71,
grep -e foo -e bah
searchs for strings matching either foo or bah. |
Yep, I saw that, but didn't really pay attention to this and instead, put it on my todo list.
| Quote: | | I note your file contents has key spelt sometimes with a capital K, others with lower case k. Is that intentional? Should you specify "-i" in your greps? |
Thanks for pointing that out. I will try it and check the original bindings file and make their case to be consistent, that even if fvwm don't care about the case.
| Quote: | keyfuncexist with those parameters will match either lines containing either "key" or "XF86AudioStop" in file "A". You probably wanted | Code: | | grep -F -e "$1 $2" "$3" |
which matches the string "$1 $2", i.e. "key XF86AudioStop". |
It is a check for identical line, it is commented into the extract. The script must follow that logic which simplify into the script, it is why I need to also test the bindings and the commands:
| Code: | # key | func | action
#-------|-------|--------
# true | true | line exist - do nothing
# true | false | key exist - copy commented line
# false | true | key don't exist - copy line
# false | false | line don't exist - copy line
|
I found a way to make grep to work with this:
| Code: | keyexist() {
tmpv=$(sed -e 's:\[:\\\[:g' -e 's:\]:\\\]:g' <<<"$1")
grep "^$tmpv " "$2" 1>/dev/null
}
funcexist() {
tmpv=$(sed -e 's:\[:\\\[:g' -e 's:\]:\\\]:g' <<<"$1")
grep " $tmpv$" "$2" 1>/dev/null
}
|
I am almost done implementing the logic and will post the result later.
_________________
"Confirm You are a robot." - the singularity |
|
| Back to top |
|
|
RumpletonBongworth
Tux's lil' helper
Joined: 17 Jun 2024
Posts: 82
|
| Posted: Sun Nov 10, 2024 2:18 am Post subject: |
|
|
My advice is as follows.
Firstly, jettison all of the code that you have shown thus far.
Secondly, forget about sed and grep (for now).
Thirdly, consider that bash supports arrays. Not only that, but it supports associative arrays. While normal bash arrays map integers to strings, associative arrays map strings to strings, making them directly relevant to your problem domain.
The following code is for demonstrative purposes and may help to put you on the right track.
| Code: | #!/bin/bash
# Synthesize some input data for testing.
mockdata='key XF86AudioStop A $[Mod1] Music-Speed normal mplayer
key XF86AudioPlay A $[Mod1]2 Music-Speed normal mplayer
Key XF86AudioRaiseVolume A $[Mod0] Change-Volume-Up $[infostore.Vol_Step]
Key XF86AudioRaiseVolume A $[Mod0]2 Change-Volume-Up $[infostore.Vol_Step]'
# Bash supports associative arrays. Use them!
declare -A cmd_by key_by
# Inputs may be treated as a series of fields.
while read -r firstcol key cmd; do
if [[ $firstcol == [Kk]ey && $key && $cmd ]]; then
cmd_by[$key]=$cmd # dictionary of commands by key name
key_by[$cmd]=$key # dictionary of keys by command name
fi
done <<<"$mockdata"
# Given an arbitrary key name, is it defined?
key='XF86AudioPlay'
if [[ ${cmd_by[$key]} ]]; then
# Yes it is. Show the command defined for it.
echo "$key => ${cmd_by[$key]}"
fi
# Given an arbitrary command definition, is it defined?
cmd='A $[Mod1] Music-Speed normal mplayer'
if [[ ${key_by[$cmd]} ]]; then
# Yes it is. Show the key for which the command is defined.
echo "$cmd => ${key_by[$cmd]}"
fi
# If you need to iterate over all the keys of an array, it is straightforward.
for key in "${!cmd_by[@]}"; do
echo "$key => ${cmd_by[$key]}"
done |
|
|
| Back to top |
|
|
Dominique_71
Veteran
Joined: 17 Aug 2005
Posts: 1895
Location: Switzerland (Romandie)
|
| Posted: Sun Nov 10, 2024 10:11 am Post subject: |
|
|
| Genone wrote: | | I'd check if you can reimplement that in another language where you don't have to deal with 3+ different semantics for $ and other special characters that will sooner or later lead to problems with quoting/escaping. |
The freedesktop application menu with full support for the additional categories of the spec was added in FVWM-Crystal 3.0.5 back in 2011. Crystal was the first desktop to fully support that spec and for that, it contain a more than thousand lines bash script that scan the desktop files into /usr/share/applications/ and populate the application menu of that desktop.
From that time, most of the maintenance of that script was adding new sub categories into a case statement. That's not worst than with other languages like python or even C which deprecates/change/remove semantics and mnemonics. I think the only language that don't do that is assembler. With it, if You want other mnemonics, you change the hardware.
For me in fvwm-crystal, the issue is more with the different grep implementations than with the $ semantics. On linux, we have the gnu grep, when FreeBSD don't have it.
_________________
"Confirm You are a robot." - the singularity |
|
| Back to top |
|
|
Dominique_71
Veteran
Joined: 17 Aug 2005
Posts: 1895
Location: Switzerland (Romandie)
|
| Posted: Sun Nov 10, 2024 12:29 pm Post subject: |
|
|
I got something that begin to works as expected:
| Code: | #!/bin/bash
# 1st run: Generate preference files for the key or mouse bindings
# from the file list into $FVWM_USERDIR/tmp/<Key|Mouse>BindingsList
# and $FVWM_USERDIR/components/bindings if any.
# Maybe TODO: When done, remove the files into $FVWM_USERDIR/components/bindings.
# Usage:
# MakeBindingsFile <Key|Mouse>
# Mouse bindings are a wip.
usage() {
echo "$(basename $0) usage:"
echo " $(basename $0) Key|Mouse"
exit 1
}
if [ x"" == x"$1" ]; then
usage
fi
if [ $1 != Key ]; then
if [ $1 != Mouse ]; then
usage
fi
fi
# Preferences directoey and file
BindingsPrefDir="${FVWM_USERDIR}/preferences/bindings"
BindingsUserFile="${BindingsPrefDir}/${1}Bindings"
mkdir -p "${BindingsPrefDir}"
# path of temporary generated files
TmpDir="${FVWM_USERDIR}/tmp"
mkdir -p "${TmpDir}"
# default bindings files list; generated by crystal at Start and Restart
BindingsFilesList="${TmpDir}/${1}BindingsList"
# temporary bindings list
BindingsTmpFile="${TmpDir}/${1}Bindings"
# We must know if it is the first run
FRun="0"
if [ -f "${BindingsUserFile}" ]; then
FRun="0"
else
FRun="1"
fi
# make a bindings list from the files list
find_bindings() {
echo -n '' > "${BindingsTmpFile}"
while read myline; do
echo "## From ${myline}: {{{1" >> "${BindingsTmpFile}"
cat "${myline}" | sed -e "s: * : :g" -e "s: * : :g" >> "${BindingsTmpFile}"
done < "${BindingsFilesList}"
}
# get the key and function strings associated to a binding
getkey() {
keys=$(echo $1 | grep -i -v -e "^#" -e "^$" | sed -e "s: * : :g" -e "s: * : :g" -e "s:.*Key:Key:" -e "s:'$::" | cut -d " " -f 1-4)
}
getfunc() {
funcs=$(echo $1 | grep -i -v -e "^#" -e "^$" | sed -e "s: * : :g" -e "s: * : :g" -e "s:.*Key:Key:" -e "s:'$::" | cut -d " " -f 5-)
}
identicalline() {
grep -x -F "$1" "$2" 1>/dev/null || echo "$1"
}
findidenticalline() {
grep -x -F "$1" "$2" 1>/dev/null
}
keyexist() {
tmpv=$(sed -e 's:\[:\\\[:g' -e 's:\]:\\\]:g' <<<"$1")
grep "^$tmpv " "$2" 1>/dev/null
}
funcexist() {
tmpv=$(sed -e 's:\[:\\\[:g' -e 's:\]:\\\]:g' <<<"$1")
grep " $tmpv$" "$2" 1>/dev/null
}
write_bindingsfile() {
# echo -n "" > "${BindingsUserFile}"
# We want only 1 vim line at the end of the file.
sed -i 's/# vim:ft=fvwm/# New line additions:/' "${BindingsUserFile}"
while read myline ; do
# if non empty line
if [[ -n "${myline}" ]]; then
# skip wim lines
if [[ "# vim:ft=fvwm" != "${myline}" ]]; then
case "${myline}" in
\#*)
# copy commented lines as it only if they dont exist
identicalline "${myline}" "${BindingsUserFile}" >> "${BindingsUserFile}"
;;
*)
getkey "${myline}"
getfunc "${myline}"
# key | func | action
#-------|-------|--------
# true | true | line exist - do nothing
# true | false | key exist - copy commentd line
# false | true | key don't exist - copy line
# false | false | line don't exist - copy line
# TODO: after adding a line, info and ask for editing
# true | true == identical line exist; do nothing.
if ! findidenticalline "${myline}" "${BindingsUserFile}"; then
echo C:"${funcs}" F:"${keys}" L:"${myline}"
# ! identical && key exist => true false
if keyexist "${keys}" "${BindingsUserFile}" ; then
echo "#${myline}" >> "${BindingsUserFile}"
echo "Add commented line with existing key binding and a different function:"
echo " ${myline}"
else
echo "${myline}" >> "${BindingsUserFile}"
echo "Add new keybinding: ${myline}"
fi
fi
;;
esac
fi
else
# copy empty lines first time
if [[ "${FRun}" == "1" ]]; then
echo "${myline}" >> "${BindingsUserFile}"
fi
fi
done < "${BindingsTmpFile}"
echo "# vim:ft=fvwm" >> "${BindingsUserFile}"
}
# main()
if [ -f "${BindingsTmpFile}" ]; then
write_bindingsfile "${1}"
else
if [ -f "${BindingsFilesList}" ]; then
find_bindings "${1}"
write_bindingsfile "${1}"
else
echo "For $(basename $0) to run, restart FVWM-Crystal."
rm -f "${BindingsTmpFile}"
exit 2
fi
fi
FvwmCommand "AT 'FVWM-Crystal Bindings Editor' ${EDITOR} ${BindingsUserFile}"
|
Edit: With the 'true false' condition, when an existing key combination bind to a different action, the existing one can be an user preference. It is why the script add the new one as a commented line. I never lost a single user preference with Crystal, and I want it to continue that way.
_________________
"Confirm You are a robot." - the singularity |
|
| Back to top |
|
|
Dominique_71
Veteran
Joined: 17 Aug 2005
Posts: 1895
Location: Switzerland (Romandie)
|
| Posted: Mon Nov 11, 2024 9:31 am Post subject: |
|
|
| Genone wrote: | | I'd check if you can reimplement that in another language where you don't have to deal with 3+ different semantics for $ and other special characters that will sooner or later lead to problems with quoting/escaping. |
But You are not necessarily wrong, because bash can be a hell with special characters and not only the $. As example with that script, if I try the sed '<blabla>' <<<"$var" in the console, it expand the [...] into the variable when the script don't do that.
That script allow me to waterproof the algorithm used to generate the user preference file. Making it to work with the Mouse bindings showed me another issue that I fixed. That issue is important because, even if its cause was into the system files, users can make similar things or worse. That show I must do a complete copy of these original system and user binding files, and use it as reference in all cases.
When I am done with that algorithm, I will consider what I will do with it and if I want to have a gui. fvwmscript can do that, but it is very time consuming to do it well with complicated things. And it don't scale with different font size preferences, which complicate further the scripts with no really good way to do that: The script and widget sizes must be calculated in some relatively arbitrary way and passed to the script at launch. Another option will be to add a supplementary dependency like PyGtk and do the whole thing in python.
_________________
"Confirm You are a robot." - the singularity
Last edited by Dominique_71 on Mon Nov 11, 2024 10:03 am; edited 3 times in total |
|
| Back to top |
|
|
Dominique_71
Veteran
Joined: 17 Aug 2005
Posts: 1895
Location: Switzerland (Romandie)
|
| Posted: Mon Nov 11, 2024 9:44 am Post subject: |
|
|
| RumpletonBongworth wrote: | My advice is as follows.
Thirdly, consider that bash supports arrays. Not only that, but it supports associative arrays. While normal bash arrays map integers to strings, associative arrays map strings to strings, making them directly relevant to your problem domain.
The following code is for demonstrative purposes and may help to put you on the right track. |
Thanks for the advice and smart and concise explanation and code example. I will take a deep look at it later.
_________________
"Confirm You are a robot." - the singularity |
|
| Back to top |
|
|
RumpletonBongworth
Tux's lil' helper
Joined: 17 Jun 2024
Posts: 82
|
| Posted: Mon Nov 18, 2024 11:02 pm Post subject: |
|
|
| Dominique_71 wrote: | | But You are not necessarily wrong, because bash can be a hell with special characters and not only the $. As example with that script, if I try the sed '<blabla>' <<<"$var" in the console, it expand the [...] into the variable when the script don't do that. |
This doesn't make sense. Not even history expansion in an interactive shell could interfere with the code (precisely as it is written there). Do you have a reproducible example that demonstrates a difference in behaviour between an interactive and a non-interactive shell? |
|
| Back to top |
|
|
|
Portage & Programming
