| View previous topic :: View next topic |
| Author |
Message |
supermihi
Guru
Joined: 09 Feb 2005
Posts: 348
|
 Posted: Mon Mar 28, 2005 9:52 pm Post subject: HOWTO: set up a local cddb/freedb server |
 |
|
HOWTO set up a local cddb/freedb server
to obtain automated Audio CD information without a permanent internet connection
Navigation:
0. Introduction
1. Pro's/Con's
2. Required Software
3. Installing the Database
4. Optional: setting up the webserver for http access
5. Installing the server software
6. Setting up xinetd
7. Client setup
8. Additional hints
0. Introduction
The CDDB protocol allows automated recognition of audio CDs by a special checksum. This checksum is sent to a server (e.g. freedb.org), that holds a huge database to find what album you are using. Then the server sends you information about track titles, album artist etc. This can be used for very comfortable automated cd ripping.
BUT:
Are you ripping a lot of your home music CD collection? Are you using a dialup internet connection being angry that you have to dial to your ISP a few seconds long for every disc to get the track names etc. filled in automatically? Or do you just think that there's nothing more cool around than your own, local cddb server providing information on hundreds of thousands of CDs? Well, then this is the right place for you - get ready to start!
*** WARNING! This setup is useful for local / home network use only!!! This document does NOT cover how to setup an official mirror of freedb. Security is ALMOST NOT covered here!! ***
1. Pro's/Con's:
There are many benefits from a local cddb/freedb server:
- you don't need an open internet connection for obtaining disc information from freedb
- you don't have to worry about security - nothing is transferred (unencrypted) over the internet, noone will ever know which kind of music you hear
- you can serve a local network with cddb infos
- you can tell everyone that you have your own cddb server running
- can be run with xinet.d - no performance issues
- there's a mailinglist available that tells you about database updates, so you don't have to worry about them
However, there are also some things you should keep in mind:
- you have to download the initial database, which is currently 315 megs (tar.rar) / 375 megs (tar.bz2), so you at least need a good friend who has a cheap connection
- you shoud download the updates that come about every month (don't need this if your collection contains only older discs)
- you will need some disc space - at least 2GB, depending on what filesystem you use
- unfortunately, there's currently no ebuild available for the server software
If you still think that you just WANT a local cddb server, continue!
2. Required Software
a) Software available in portage:
I recommend the use of xinetd which can start the cddb server only when it's needed:
| Code: | root # emerge xinetd
root # /etc/init.d/xinetd start
root # rc-update add xinetd default |
- to unpack the rar-packed database (which is much smaller in download size), you'll need rar or unrar.
| Code: | | root # emerge unrar |
Filesystem considerations:
Since the database consits of hundreds of thousands of very small files, the filesystem you are using for storing the data has a great influence on how much disk space will actually be used (covering several hundreds of percents from reiser4 to vfat). I *highly* recommend to use reiser4, which uses disc space extremely efficiently with small files, or at least reiserfs. Reiser4 is not yet in the official kernel, you'll either need to patch your kernel or use one of the many kernel versions that include reiser4, like the mm-sources, nitro-sources etc. Also you will need the userspace utilities, which are part of portage:
| Code: | | emerge reiser4progs |
**TIP** (thanks to Sanguinary) If you don't want to repartition your hdd, you could use the loop-mounting to create a filesystem on a single, large file in an existing filesystem. Then you can format this one with reiser4 or reiserfs (v3).
b) Software you have to download by hand:
- the cddb server software (currently v.1.5.1PL2) which is available at the freedb homepage: http://www.freedb.org/modules.php?name=Sections&sop=viewarticle&artid=7
- the cddb database which is also available at freedb.org: http://www.freedb.org/modules.php?name=Sections&sop=viewarticle&artid=12
3. Installing the database
First you should, like mentioned above, set up a partition that uses the reiser4 filesystem (if that's not possible for you, e.g. if you're using amd64, reiserfs (v3) will also do a good job, but far not as good as reiser4). What I did is to put my whole /var directory into a reiser4 partition, since there are a lot of small files.
Then unpack the database:
| Code: | root # mkdir /var/db/cddb
root # rar p -inul freedb-complete-<latest version>.tar.rar | tar x -C /var/db/cddb/
|
if you took the bz2 version:
| Code: | | root # tar xjf freedb-complete-<latest version>.tar.bz2 -C /var/db/cddb |
This will take a while! 
4. Optional: setting up the webserver for http access
The cddbd daemon can be also accessed via normal http using cgi-scripts. This is useful if you're having firewall issues or if you use a ripper that only supports this way (e.g. grip).
Therefore you need an apache webserver running:
| Code: | root # emerge apache
root # /etc/init.d/apache2 start
root # rc-update add apache default |
If you already have a webserver running on your machine, you will have to specify a cgi-path where cddbd can install its scripts. Otherwise, the default configuration will do fine (you might want to check the config file for security reasons).
5. Installing the server software
a). Extract the tarball
| Code: | root # tar -xvzf ccdbd-1.5.1PL2.tar.gz
root # cd cddbd-1.5.1PL2 |
b) Run the config script
| Code: | | root # sh config.sh |
You can leave all of the questions to their defaults.
c) Compile the daemon.
d) Install using the install script
| Code: | | root # sh install.sh |
The script will ask some questions that you should not leave as default:
CD database directory: whatever you extracted the db to (in this howto /var/db/cddb)
enable server access via HTTP? see section 4
cddb cgi directory: This will show up if you answered the previous question with yes. In the default configuration of apache on gentoo, this should be set to /var/www/localhost/cgi-bin/
enable logging of server accesses: this probably isn't neccesary for a local db, though it's good for testing reasons
All other options can be left as they are by default. The creation of the "fuzzy matching hash file" will also take a while, so grab a cup of coffee and think about your next steps.
6. Setting up xinetd
The "extended internet services daemon" xinetd lets you start services only when they are needed, thereby increasing both performance and security of your machine.
(NOTE: this is only for the direct cddbd access. If you also choosed HTTP acces to your server, you have to make sure that apache2 is running!)
Two steps are needed to get xinetd working with cddb:
1. Edit /etc/services
The file /etc/services sets up matches between ports and services.
Adding a line
lets xinetd know that any incoming tcp request on port 8880 should be served by cddbp.
Now we should tell xinetd what cddbp is. Therefore we create a file /etc/xinetd.d/cddbp with the following content:
| Code: | service cddbp
{
disable = no
socket_type = stream
protocol = tcp
wait = no
user = root
server = /usr/local/bin/cddbd # set this to wherever you installed the binary
} |
Now reload the xinetd configuration and test if all goes well:
| Code: | root # /etc/init.d/xinetd reload
root # telnet localhost 8880
|
you should get some response now - congratulations! You now have your own cddb server.
7. Client setup
To set up your clients using native cddb protocol on port 8880 (not http access), simply fill in the hostname or IP your local cddbd server is running at (localhost if it's the same machine the client runs on).
If you act as an internet gateway, it should also be possible to redirect outgoing CDDB requests from the local net to your server with iptables.
If you're using HTTP access, server is same as above and cgi-path is "cgi-bin/cddb.cgi".
8. Additional hints
a) If you set up HTTP access and used the default apache config, you'll need to change the permissions of the cddb.cgi script:
| Code: | root # chmod a+rx /var/www/cgi-bin/cddb.cgi
root # chmod u+s /var/www/cgi-bin/cddb.cgi
|
You can try if it works by accessing the following URL in your browser: http://localhost/cgi-bin/cddb.cgi?cmd=stat&hello=me+host+none+1.0&proto=1
b) Be sure to register to the fdb-announce@freedb.org mailinglist, so you get mails everytime an update of the database gets released.
c) Further information can be found in the CDDBD_HOWTO file in your extracted server software package.
That's it! Please post comments, suggestions etc!!
_________________
"You may say I'm a dreamer, but I'm not the only one."
Last edited by supermihi on Sun Jan 08, 2006 6:20 pm; edited 1 time in total |
|
| Back to top |
|
 |
beandog
Bodhisattva
Joined: 04 May 2003
Posts: 2072
Location: /usa/utah
|
| Posted: Sun May 15, 2005 6:29 pm Post subject: |
|
|
Thanks for the great guide -- this is a good condensed alternative to the very lengthy, but well documented, HOWTO. 
I'm just having one issue with my server. I'm getting errors when I access the cddb.cgi. Here's what my apache error_log is spitting out:
| Code: | | [error] [client 127.0.0.1] Premature end of script headers: cddb.cgi |
I've tried your user permissions on the cgi, I've tried setting it to 777, and tried it in both /var/www/localhost/cgi-bin, /home/public_html/cddb and /home/cddb/public_html/cgi-bin and nothing works.
Other than that, it works fine if I access it over port 8880. Any ideas?
_________________
If it ain't broke, tweak it. dvds | blurays | blog | wiki |
|
| Back to top |
|
|
supermihi
Guru
Joined: 09 Feb 2005
Posts: 348
|
| Posted: Mon May 16, 2005 6:46 am Post subject: |
|
|
not really, since I don't know anything about cgi / apache. Sorry!
_________________
"You may say I'm a dreamer, but I'm not the only one." |
|
| Back to top |
|
|
Sanguinary
n00b
Joined: 17 Oct 2004
Posts: 16
|
| Posted: Sun Jan 08, 2006 6:11 pm Post subject: Re: HOWTO: set up a local cddb/freedb server |
|
|
| supermihi wrote: |
Filesystem considerations:
Since the database consits of hundreds of thousands of very small files, the filesystem you are using for storing the data has a great influence on how much disk space will actually be used (covering several hundreds of percents from reiser4 to vfat). I *highly* recommend to use reiser4, which uses disc space extremely efficiently with small files, or at least reiserfs. Reiser4 is not yet in the official kernel, you'll either need to patch your kernel or use one of the many kernel versions that include reiser4, like the mm-sources, nitro-sources etc.
|
Resurrecting an old thread here, but I used this to put a local freedb server on an old Cobalt Qube 2, and my experience taught me two things:
1. As a practical matter, you can't put the freedb database on an ext3 filesystem. The December 2005 version of the database is something like 1.5 million little text files. I gave up trying to untar the thing onto an ext3 filesystem when it took more than 36 hours without getting halfway through (and this wasn't on the Qube, so it wasn't a matter of the miniscule processing power that system has).
2. If you don't have reiserfs on your system and you don't want to repartition, loopback is your friend. Just as an illustration, I created and mounted a 5 gig file with a reiser3 filesystem, and untarring the same database took about 25 minutes. |
|
| Back to top |
|
|
supermihi
Guru
Joined: 09 Feb 2005
Posts: 348
|
| Posted: Sun Jan 08, 2006 6:18 pm Post subject: |
|
|
Good tip! Thanks, I'll mention that in the tutorial.
_________________
"You may say I'm a dreamer, but I'm not the only one." |
|
| Back to top |
|
|
Gruffi
Apprentice
Joined: 15 Aug 2003
Posts: 209
Location: Antwerpen - Flanders - Belgium
|
| Posted: Sun Nov 15, 2009 4:19 pm Post subject: |
|
|
You can also use squashfs. (emerge squashfs-tools)
The whole database was 11GB on ext3 but compressed with squashfs it was 950MB
| Code: | cd /mnt/storage/cddb
mksquashfs -always-use-fragments -no-duplicates . ../cddb.squashed
rm -r /mnt/storage/cddb/* #deleting the files takes forever
mount -o loop -t squashfs /mnt/storage/cddb.squashed /mnt/storage/cddb |
works just fine, mksquashfs took 80 minutes tho 
And thanks for the how-to! 
_________________
... and we will show Microsoft, that they cannot take whatever they want. And that Free Software is our software! |
|
| Back to top |
|
|
jlpoole
Guru
Joined: 01 Nov 2005
Posts: 489
Location: Salem, OR
|
| Posted: Wed Sep 21, 2011 2:59 am Post subject: |
|
|
Here's a gotcha if you're not famliar with xinet: it appears xinet does not tolerate comments that start after column #1. For example, the above FAQ recommended:
| Code: | | server = /usr/local/bin/cddbd # set this to wherever you installed the binary |
I left the comment in (having just copy/pasted the recommend file contents) as is since /usr/local/bin/cddbd was indeed the server executable. Leaving the comment "# set this..." in the file apparently causes xinetd to quietly reject the entry. When I attempted to telnet to the server, as recommended, the connection failed:
| Code: | themis cddbd-1.5.2 # telnet localhost:8880
telnet: could not resolve localhost:8880/telnet: Name or service not known
themis cddbd-1.5.2 #
|
I then modified /etc/xinet.d/cddbp by removing the comment text as follows:
| Code: | themis cddbd-1.5.2 # cat /etc/xinetd.d/cddbp
service cddbp
{
disable = no
socket_type = stream
protocol = tcp
wait = no
user = apache
server = /usr/local/bin/cddbd
}
themis cddbd-1.5.2 #
|
Note: I also replaced root with apache since I wanted the cgi script to work. Using the modified file, telnet successfully worked.
| Code: | themis cddbd-1.5.2 # telnet localhost 8880
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
201 themis CDDBP server v1.5.2PL0 ready at Tue Sep 20 19:39:56 2011
exit
Connection closed by foreign host.
themis cddbd-1.5.2 # |
Lastly, don't even think about unzipping the database (which is now about 800 MBs) thinking you'll copy the unzipped files to their final destination -- I have a pretty powerful laptop and after several hours, canceled the "cp -r ..." and just moved the tar.bz2 to the directory where I wanted the database to reside and unzipped it there. I think I may investigate the suggestion of using the reiserfs on a file (rather than create another partition), but that's for another adventure.
Also, if you are on ext3 and have the database staged, remember when you "updatedb" to exclude the directory, as in:
| Code: | | updatedb -e /usr/local/cddb |
Finally, just for statistics sake, here's the output from my apache session (which worked right after the install):
| Code: | 210 OK, status information follows (until terminating `.')
Server status:
current proto: 1
max proto: 6
interface: http
gets: no
puts: no
updates: no
posting: no
validation: accepted
quotes: no
strip ext: no
secure: yes
current users: 1
max users: 100
Database entries: 3088945
Database entries by category:
newage: 121127
rock: 799682
folk: 250973
data: 70241
jazz: 183896
misc: 967927
classical: 279550
blues: 159920
reggae: 42189
country: 88599
soundtrack: 124841
. |
Yeah! Classical music is alive and well represented, e.g. near 280,000 entries. |
|
| Back to top |
|
|
jlpoole
Guru
Joined: 01 Nov 2005
Posts: 489
Location: Salem, OR
|
| Posted: Wed Sep 21, 2011 3:04 am Post subject: |
|
|
| Also, people may wonder why I went to the trouble to set up a local server. Basically, I wanted to be able to create MP3s and wave format files and found abcde to best fit the bill for my needs. Unfortunately, when it tries to connect to the cddb server that everyone else connects to, if it doesn't get the information sought, it just proceeds using generic naming of the files. The failure to get the appropriate naming due to a server being oversubscribed caused me to undertake the trouble of having my own local server so my files would always be suitably named. When I checked the statistics of the main free server, there were 100 connections -- the limit, so I was competing for a limited resource. |
|
| Back to top |
|
|
farneko
n00b
Joined: 25 Sep 2022
Posts: 1
|
| Posted: Sun Sep 25, 2022 7:25 am Post subject: |
|
|
After picking up a bunch of CD's at a garage sale for cheap; A TimeLife 70's collection, complete... I popped them in my media player PC to try them out. It runs windows 10, Winamp (showing my age, there  ), where it sends track name/artist for all my ripped, etc music & audio books - to my Raspberry Pi-based wall info center for display.
It's been some time since I placed a new-to-me cd in, just to find that, of course; Generic track labels came up. CDDBv1/FreeDB has long been depreciated and taken offline. Bummer.
>>More and more cloud-based services to follow<<
So, I decided to do something about it. And since I have a Pi4 running Buster debian, I took a look on archive.org for a FreeDB copy... and found it!
Following the guide above, I had it up and running. Big thanks!
I took it a couple steps farther, and got the 'submissions' system/part in cgi running too. Based on Lighttpd & Sendmail, submissions are sent from anything (software) using the old CDDBv1 & FreeDB to my Pi via DNS (or optionally, hosts file) pointers.
cddb.cddb.com & freedb.freedb.org are being redirected to my pi via my caching & forwarding DNS on my nas.
Querying and submitting are working flawlessly with software that hasn't received server responses since the 2000's (EAC, Audiograbber, Winamp 2.77, etc)! The DB is current up to very early 2020. So, baring a lightning strike, it won't go offline randomly some future day. If such happens, a restore from a compressed VHD file to cf will have me up and going again.
A couple of (okay, many!) things I had to do, was;
First, I duplicated my 32Gb sd card to a 64Gb, as the database takes over 18Gb un-tarred. After I did that, I made a 21Gb xfs partition with the free space, and then added it to my fstab file. I chose xfs, since there's rumblings of riserFs being pulled in the future.
I cloned the cf with winimage's vhd create & restore, then used parted live cd via virtualbox to do the partition expansion and creation.
When all was said and done, only 36% of the Inode table is being used by the 18.1Gb database (3.93M entries) on it's private xfs partition.
The fstab entry looks like this;
| Quote: | | /dev/mmcblk0p8 /usr/local/cddb xfs defaults 0 2 |
Once the storage real estate was made, and the pi's os partition received some more storage space in the process, I got to work on the software side. Here's as much additional work as my notes recall:
| Code: | | sudo apt-get install sendmail |
| Code: | chmod a+rx /usr/lib/cgi-bin/submit.cgi
chmod u+s /usr/lib/cgi-bin/submit.cgi |
Add the 'mail' user to the 'sudo' group;
| Code: | | sudo usermod -aG sudo mail |
Edit my /etc/sudoers file with visudo to turn off the password challenge when sendmail submissions are made;
| Quote: | | mail ALL=(ALL) NOPASSWD:ALL |
The sendmail MTA uses the user 'mail' to execute cddbd when the email is piped to it via aliases
my /etc/mail/aliases looks like this:
| Quote: | | freedb-submit: "| sudo /usr/local/bin/cddbd -m" |
I had to go the sudo route, because the cddbd daemon wants to change user/group permissions on database files as it runs 
Remember to run 'sudo newaliases' after editing the sendmail aliases file!
On a side note for troubleshooting, you should test it out before untarring the database. Odds are you wont have a CD that's not already in the database, so there's no good way to be sure by that point. You can use
| Quote: | | freedb-submit: /usr/local/cddb/test.txt |
in place to test if sendmail is receiving the http/cgi submission, and that the mail user is working (be sure to 'sudo chown -R mail /usr/local/cddb' first). If when an app makes a submission, and the test.txt file is made, you can then 'sudo -u mail /bin/bash' into the mail user, and then manually 'cat /usr/local/cddb/test.txt | sudo /usr/local/bin/cddbd -m' and see if that works. Hopefully, you will see a file appear in a genre subfolder; eq 'rock' or 'classical' (oh yeah, mkdir these, since cddbd doesn't itself)
'/usr/local/cddbd/server/log' & '/var/log/mail.info' are your best friends
For folks using lighttpd like me, here's what the applicable part of my /etc/lighttpd/conf-enabled/10-cgi.conf looks like:
| Quote: | $HTTP["url"] =~ "^/cgi-bin/" {
cgi.assign = ( "" => "" )
alias.url += ( "/cgi-bin/" => "/usr/lib/cgi-bin/" )
alias.url += ( "/~cddb/" => "/usr/lib/cgi-bin/" )
}
$HTTP["url"] =~ "^/~cddb/" {
cgi.assign = ( "" => "")
alias.url += ( "/~cddb/" => "/usr/lib/cgi-bin/" )
} |
For reference, here's part of what my installation responses of cddbd were:
| Quote: |
Enter cddbd binary directory
[/usr/local/bin]:
Enter cddbd man directory (for cddbd.1)
[/usr/local/man/man1]:
Enter cddbd work directory
[/usr/local/cddbd]:
Enter CD database directory
[/usr/local/cddb]:
Do you want to enable server access via HTTP? [y]
Enter cddbd CGI directory
[/usr/lib/cgi-bin]:
Do you want to accept submissions via http? [y]
Enter the path to sendmail.
[/usr/sbin/sendmail]:
Enter submit e-mail address.
[freedb-submit@freedb.org]:
Do you want configure your site as a master database hub? [y]
Enter your SMTP server host
[localhost]:
Enter your full email address
[root@marquee]:
Enter the hostname used for banner message etc.
[marquee]:
|
I kept the rest as defaults; root user, file masks, etc.
A thing to be aware of, is to check your /usr/local/cddbd/access file, and set the postdir to the same directory as cddbdir (the actual db location). In there, I also set dup_policy to 'always', but the latter might not be necessary:
| Quote: | postdir: /usr/local/cddb
dup_policy: always
.
.
host_perms: s default connect post update get put nopasswd
|
By default, based on the answers I stated above, the submissions were being placed into a 'post' subdirectory of the cddbd work directory, and not showing up in the database. I pulled my hair for an hour since everything was running smoothly and no errors, yet nothing appearing in my blank test database. If the postdir is something else than the DB directory, a command has to be run regularly (via cron) to 'post' the submissions to the actual database.
The host_perms second line might not be needed, but it's working for me. Notice the 's'... I took that out of the default existing line and added this one to allow connections for 'submissions'. It's documented in the beginning of the access file.
Additionally, I made a cddb.service in systemd to start it up on boot, as well as give me start/stop control;
/etc/systemd/system/cddb.service:
| Quote: | [Unit]
Description=CDDB Service
Requires=network.target
[Service]
Type=Simple
User=root
ExecStart=/usr/local/bin/cddbd.sh
[Install]
WantedBy=multi-user.target
|
And the /usr/local/bin/cddbd.sh file;
| Quote: | #!/bin/bash
/usr/local/bin/cddbd -s default
sleep infinity |
I had to use the 'sleep infinity' in bash script workaround, because systemd would immediately terminate the service, once cddbd forked into the background, as it does in 'standalone server mode'.
Finally, a bit about winamp. My cause for all this...resurrection of abandoned service;
I'm using 5.666? So in order to get track/artist cddbv1/freedb support, I have two choices:
"CD Reader" plugin, or Nullsoft CD/LineIn plugin v1.71
The latter is found in winamp 2.77 and earlier, and works on 5.x without issue.
So, I'm partying like it's...2001!
Dave? Confidence in the mission has been regained. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Documentation, Tips & Tricks
