| View previous topic :: View next topic |
| Author |
Message |
hielvc
Advocate
Joined: 19 Apr 2002
Posts: 2805
Location: Oceanside, Ca
|
|
| Back to top |
|
 |
b1f30
Apprentice
Joined: 16 Nov 2005
Posts: 262
Location: USA
|
 Posted: Wed Mar 21, 2007 1:37 pm Post subject: |
 |
|
Wow - that is quite a change that I was not aware of. Does this obsolete http-replicator?
Yeesh...
_________________
H T T P : / / W W W . B I N A R Y F R E E D O M . I N F O / |
|
| Back to top |
|
|
dahoste
Tux's lil' helper
Joined: 01 Dec 2005
Posts: 138
Location: Maryland, USA
|
| Posted: Wed Mar 21, 2007 5:14 pm Post subject: |
|
|
| Quote: | | Wow - that is quite a change that I was not aware of. Does this obsolete http-replicator? |
um... I don't see how it would. http-replicator is a means of proxy-caching gentoo packages. It doesn't alter, nor particularly care what's in, any of the packages it caches/proxies. The need that http-replicator satisfies isn't going away. The 'undesirable change' mentioned in the previous post (and related thread) sounds like a hiccup in the transitional phase of the GLEP44 proposal. The GLEP44 itself says this:
| Quote: | | It is important to note that this proposal only deals with a change of the format of the digest and Manifest system. |
So, no, once the dust settles, we'll all appreciate having http-replicator as much as we currently do. Unless I'm really missing something about all of this. |
|
| Back to top |
|
|
b1f30
Apprentice
Joined: 16 Nov 2005
Posts: 262
Location: USA
|
| Posted: Thu Mar 22, 2007 2:19 pm Post subject: |
|
|
| dahoste wrote: | | Quote: | | Wow - that is quite a change that I was not aware of. Does this obsolete http-replicator? |
um... I don't see how it would. http-replicator is a means of proxy-caching gentoo packages. It doesn't alter, nor particularly care what's in, any of the packages it caches/proxies. The need that http-replicator satisfies isn't going away. The 'undesirable change' mentioned in the previous post (and related thread) sounds like a hiccup in the transitional phase of the GLEP44 proposal. The GLEP44 itself says this:
| Quote: | | It is important to note that this proposal only deals with a change of the format of the digest and Manifest system. |
So, no, once the dust settles, we'll all appreciate having http-replicator as much as we currently do. Unless I'm really missing something about all of this. |
The problem isn't so much http-replicator, but perhaps it's indexing tool, repcacheman, which seems to fail a lot due to not being able to find MD5's for a lot of packages now - at least that's what I've observed from my own experience within the past month now, and from what I've read from previous posts in this thread.
_________________
H T T P : / / W W W . B I N A R Y F R E E D O M . I N F O / |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Thu Mar 22, 2007 8:29 pm Post subject: |
|
|
repcacheman uses portage functions to do it's job. It just needs to call the new portage functions and all is well. I'm working on the new changes now. The only real change is It needs to call the sha1 checksum rather than the md5 checksum.
Also, repcacheman isn't really required at all. Truth is I don't even run it all the time.
It does several things for you that aren't strictly needed to make http-replicator work.
1. It creates the cache dir upon install. It does this only upon initial install so I don't have to have the ebuild or users do it. mkdir /var/cache/http-replicator, chown portage:portage /var/cache/http-replicator if you had to do it manually.
2. It transfers your initial distdir to the cache dir. Allows your already populated distdir to populate the cache dir. Nice, but not required.
3. It deletes duplicate files only on the http-replicator server. I just delete all the distdir files on the server. The most I lose is one or two files that were only available thru FTP. With a good mirror, I can go months without using FTP at all so I don't loose anything. Check your logs, how many times has repcacheman actually added new files?
So repcacheman will soon work with the small but increasing number of manifest2 packages. Until then, http-replicator works just fine |
|
| Back to top |
|
|
lorenct
n00b
Joined: 20 May 2004
Posts: 31
|
| Posted: Thu Apr 26, 2007 4:15 am Post subject: http-replicator not remaining running after reboot |
|
|
For a while I have had a problem with http-replicator not remaining running after a reboot on my central LAN server.
I would have to SSH in to the system to restart it manual for it remain running.
Then I noticed once when SSH'ng into the system before it was completely up and I saw the http-replicator process (python script) running on tty1. A short time later the process was gone and /sbin/agetty from /etc/inittab process was running on tty1 (when performing a ps -ef command). I know that /etc/init.d/http-replicator script is supposed to run the python script as a daemon, but it is possible that the script is not forking as a daemon fast enough before being "smacked-down" by the /sbin/agetty process and terminating. (I do not know for sure, but that is what it seems like is happening to me).
So I changed the following in /etc/init.d/http-replicator which seemed to resolve the problem for me:
FROM:
| Code: |
start() {
ebegin "Starting Http-Replicator"
start-stop-daemon --start --pidfile /var/run/http-replicator.pid --name http-replicator \
--startas /usr/bin/http-replicator -- -s -f --pid /var/run/http-replicator.pid --daemon $DAEMON_OPTS
eend $? "Failed to start Http-Replicator"
}
|
TO:
| Code: |
start() {
ebegin "Starting Http-Replicator"
start-stop-daemon --start --background --pidfile /var/run/http-replicator.pid --name http-replicator \
--startas /usr/bin/http-replicator -- -s -f --pid /var/run/http-replicator.pid --daemon $DAEMON_OPTS
eend $? "Failed to start Http-Replicator"
}
|
NOTICE: The addition of the --background option to the start case.
Just through I would share this with the rest of you incase you have encountered the same problem...
_________________
--
That's my story and I am sticking to it... |
|
| Back to top |
|
|
dalek
Veteran
Joined: 19 Sep 2003
Posts: 1353
Location: Mississippi USA
|
| Posted: Thu Apr 26, 2007 5:25 am Post subject: |
|
|
I would also like to add something to the mix, since you are already under the hood fixing things. I'm on a really slow dial-up and sometimes I have to stop replicator for various reasons. I notice that when I restart it and restart the emerge process which will try to continue the download, it has trouble reconnecting to replicator. Replicator will start to download right away as it should but emerge can't seem to get back in sync.
I run into things like this on the really large packages. If you need me to I can post what I get so you can see it.
Maybe one of these days I will get DSL or something. Solve a lot of problems.
_________________
My rig: Gigabyte GA-970A-UD3P mobo, AMD FX-8350 Eight-Core CPU, ZALMAN CNPS10X Performa CPU cooler,
G.SKILL 32GB DDR3 PC3 12800 Memory Nvidia GTX-650 video card LG W2253 Monitor
60TBs of hard drive space using LVM
Cooler Master HAF-932 Case |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Mon May 07, 2007 2:21 am Post subject: |
|
|
| dalek wrote: | | I have to stop replicator for various reasons. |
How are you stopping the download /replicator? Are you using the init scripts, killing a process or just shutting down the system?
What mirror are you using? |
|
| Back to top |
|
|
dalek
Veteran
Joined: 19 Sep 2003
Posts: 1353
Location: Mississippi USA
|
| Posted: Mon May 07, 2007 3:02 am Post subject: |
|
|
I do use the init scripts to stop it. /etc/init.d/http-replicator stop. The mirror varies, just whichever one it picks I guess. It seems to do the same on about all of them though.
I do have a big download tonight if you want me to post what it does, or email it to you over pm. If I hadn't logged out of KDE I could post you one now.
| Code: | root@smoker / # uptime
22:01:56 up 26 days, 19:45, 1 user, load average: 1.10, 1.05, 1.08
root@smoker / #
|
I try not to reboot to much, mostly when the power goes out.
_________________
My rig: Gigabyte GA-970A-UD3P mobo, AMD FX-8350 Eight-Core CPU, ZALMAN CNPS10X Performa CPU cooler,
G.SKILL 32GB DDR3 PC3 12800 Memory Nvidia GTX-650 video card LG W2253 Monitor
60TBs of hard drive space using LVM
Cooler Master HAF-932 Case |
|
| Back to top |
|
|
Scen
Retired Dev
Joined: 29 Jul 2003
Posts: 2470
Location: Padova, Italy
|
| Posted: Mon May 07, 2007 9:30 am Post subject: Re: http-replicator not remaining running after reboot |
|
|
| lorenct wrote: | | Just through I would share this with the rest of you incase you have encountered the same problem... |
You rule, boy 
I've searched this solution without results for several months!
I'll open ASAP a bug report to fix this issue!
Thanks a lot again! 
_________________
I was born in a deep forest/I wish I could live here all my life/I am made from stones and roots/My home, these woods and roads
All my life I loved this sound/Of the woods all around/Eagles flies where the winds blows free
Journey is my destiny |
|
| Back to top |
|
|
Griffon26
Retired Dev
Joined: 12 Nov 2002
Posts: 51
Location: The Netherlands
|
| Posted: Mon May 07, 2007 5:06 pm Post subject: Re: http-replicator not remaining running after reboot |
|
|
| Scen wrote: |
I've searched this solution without results for several months!
|
Then why didn't you report it before today? I'm the maintainer of http-replicator, but I don't read the forums. When you find a problem, please submit a bug so i know about it. |
|
| Back to top |
|
|
Scen
Retired Dev
Joined: 29 Jul 2003
Posts: 2470
Location: Padova, Italy
|
| Posted: Mon May 07, 2007 6:36 pm Post subject: |
|
|
You're right.... 
/me flagellates himself 
I've read your e-mail, i send you more information ASAP
_________________
I was born in a deep forest/I wish I could live here all my life/I am made from stones and roots/My home, these woods and roads
All my life I loved this sound/Of the woods all around/Eagles flies where the winds blows free
Journey is my destiny |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Tue May 08, 2007 11:09 pm Post subject: |
|
|
| dalek wrote: | | I do use the init scripts to stop it. /etc/init.d/http-replicator stop. The mirror varies |
OK, that eliminates some possible problems. Then I'll have to say I'm afraid you've found a "feature", not a bug. Well, actually a lack of a feature 
Scen and Griffon26, I'll save you some time chasing this down!
The feature is that replicator supports resuming on the client end but not on the internet end. If replicator is stopped or killed it will delete the partial download from the cache!!!! Think about that for a second and your problem will be clear.
Here is how to test it. Start a long fetch, (any file will probably do on dialup  openoffice-bin for the rest of us) and then shutdown http-replicator before the download is finished. Then look and see if the partial file is in the cache. It won't be there, but portage will keep the partial download in the distfile dir!!
Then you restart replicator and then restart your long fetch. Replicator will try and honor portage's request to resume the download but it has to start the download from the beginning over the net!! This means portage will probably timeout or you will give up before the download from the net catches up with the partial download portage still has.
So, in it's current level of development, http-replicator is not designed for nor can it handle all situations, shutting it down in the middle of a download is not a supported use at this time (developers needed). Resuming on the net side along with FTP support was a planned part of the next release of http-replicator. Unfortunately, that release was bogged down in a total rewrite.
dalek to suit your dialup needs, it would be better to bypass replicator for the interrupted long download, then move the file in the cache when complete. Just do this
| Code: |
http_proxy="" emerge -f longdownload
|
restarting is as often as necessary and when it completes
|
|
| Back to top |
|
|
dalek
Veteran
Joined: 19 Sep 2003
Posts: 1353
Location: Mississippi USA
|
| Posted: Wed May 09, 2007 1:28 am Post subject: |
|
|
OK. The solution you posted is what I have been doing, sort of. My biggest problem is my ISP puts a 6 hour limit on unlimited access. Go figure that one out.  I have been editing make.conf but it does the same thing I guess.
Think maybe one day this will be fixed  I did see some phone trucks the other day. We may be going to get DSL soon. Parden me while I go jump around like a idiot at even the thought of broadband. 
_________________
My rig: Gigabyte GA-970A-UD3P mobo, AMD FX-8350 Eight-Core CPU, ZALMAN CNPS10X Performa CPU cooler,
G.SKILL 32GB DDR3 PC3 12800 Memory Nvidia GTX-650 video card LG W2253 Monitor
60TBs of hard drive space using LVM
Cooler Master HAF-932 Case |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Sun May 13, 2007 12:48 am Post subject: repcachman beta 4.0 |
|
|
I've waded through the depths of portage, and returned with a new version of repcacheman.
I'm continuing to develop and add new features, but now is a good time to test and get some feedback from users. Since I have new features still in development, I chose an older revision to test, but it should work much better than the previous version and have all the old features.
My tests show this beta 4.0 uses only 10% of the resident memory of the previous version and runs 4 times faster!!
It can be run from any dir as root or dropped in place of the old /usr/bin/repcacheman.py ( not /usr/bin/repcacheman )
The code can be downloded from this temp location:
http://home.earthlink.net/~poplawtm/rep4.py.tar.gz
Last edited by flybynite on Tue May 15, 2007 12:39 am; edited 2 times in total |
|
| Back to top |
|
|
mkzelda
n00b
Joined: 22 Aug 2004
Posts: 32
|
| Posted: Mon May 14, 2007 5:06 pm Post subject: |
|
|
On the first run, it fails and the directory it creates is not where I specified in /etc/conf.d/http-replicator.
Begin Http-Replicator Setup....
created /var/cache/http-replicator/
Traceback (most recent call last):
File "/usr/bin/repcacheman", line 73, in ?
print "\tchange owner " + dir + " to " + user + " failed:"
NameError: name 'user' is not defined
When I run repcacheman again, it works properly other than ignoring my desired directory in the conf
!!! Digest verification failed:
!!! /usr/portage/distfiles/gem_plugin-0.2.2.gem
!!! Reason: Failed on RMD160 verification
!!! Got: 9715b571202ebe33d72bfd6384305b555da6f2b6
!!! Expected: 4759f2ccb75081ebe46ffffc3ad5c7ba2e20c3bc
!!! Digest verification failed:
!!! /usr/portage/distfiles/file-4.20.tar.gz
!!! Reason: Filesize does not match recorded size
!!! Got: 548412
!!! Expected: 548393
!!! Digest verification failed:
!!! /usr/portage/distfiles/mongrel-1.0.1.gem
!!! Reason: Filesize does not match recorded size
!!! Got: 159232
!!! Expected: 160256
SUMMARY:
Found 0 duplicate file(s).
Deleted 0 dupe(s).
Found 1103 new file(s).
Added 1040 of those file(s) to the cache.
Rejected 60 File(s) not in Portage.
Oops, I did that on the wrong machine.
The second results are the same. The first attempt fails, rep4.py ignores my desired cache dir, and it picked up a few more bad files. Now, I'd be happy if it'd just use my ftp pub dir instead of /var/cache/http-replicator.
--update
I also have to set the conf back to /var/cache/http-replicator for the time being or http-replicator fails. |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Tue May 15, 2007 1:00 am Post subject: |
|
|
| mkzelda wrote: | On the first run, it fails and the directory it creates is not where I specified in /etc/conf.d/http-replicator.
|
Thanks for the bug report mkzelda, I've fixed the 'user' is not defined problem.
Now did you run repcacheman from a dir or did you copy over your old /usr/bin/repcacheman? |
|
| Back to top |
|
|
mkzelda
n00b
Joined: 22 Aug 2004
Posts: 32
|
| Posted: Tue May 15, 2007 1:38 am Post subject: |
|
|
| i replaced /usr/bin/repcacheman |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Tue May 15, 2007 2:16 am Post subject: |
|
|
| mkzelda wrote: | | i replaced /usr/bin/repcacheman |
Ooops, I'm sorry. I've corrected my post above but I meant to say replace /usr/bin/repcacheman.py
/usr/bin/repcacheman just calls /usr/bin/repcacheman with the correct options which is why you had the other problems.
Either re-emerge http-replicator which won't disturb your config or edit /usr/bin/repcacheman to look like this
| Code: |
#! /bin/bash
source /etc/conf.d/http-replicator
/usr/bin/repcacheman.py $GENERAL_OPTS
|
and replace /usr/bin/repcacheman.py with the beta script.
Again, sorry for the inconvenience.
I've uploaded beta 4.1 with two typo's fixed. There is still something going on with the core code. Right now I think it is a filename collision in portage itself. I didn't change the download link but you will see rep41.py inside. |
|
| Back to top |
|
|
mkzelda
n00b
Joined: 22 Aug 2004
Posts: 32
|
| Posted: Tue May 15, 2007 5:00 pm Post subject: |
|
|
Okay, that worked, with verbose output of the portage tree. Is there a trigger to avoid the verbosity? My server performs slower when outputting scrolling text.
I'm wondering how files in the cache are treated. Are they assumed to be good, and thus unchecked? For example, if I have an overlay on another machine that my server does not, the files it grabs are stored in the cache as they are fetched, and they remain there indefinitely? So, can I put any files in the cache that my client machines might fetch, such as livecd .iso's, and so long as the client used wget with http_proxy specified it can fetch that locally? |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Tue May 15, 2007 6:41 pm Post subject: |
|
|
| mkzelda wrote: | Okay, that worked, with verbose output of the portage tree. Is there a trigger to avoid the verbosity? My server performs slower when outputting scrolling text.
|
The verbose output is just my debugging going on, it won't be in the final version. I've uploaded beta revision 4.3 that removes the scrolling and fixes the problem in the core code I mentioned earlier.
| mkzelda wrote: |
I'm wondering how files in the cache are treated. Are they assumed to be good, and thus unchecked? For example, if I have an overlay on another machine that my server does not, the files it grabs are stored in the cache as they are fetched, and they remain there indefinitely? So, can I put any files in the cache that my client machines might fetch, such as livecd .iso's, and so long as the client used wget with http_proxy specified it can fetch that locally? |
Thanks for asking! I've been trying to decide some possible options to add and who might need them. I also want the greatest possible options for users.
replicator is a general purpose proxy at heart, It will serve and cache anything that goes through it, even web browsing. There is an "alias" option to serve files from a dir of your choice in addition to the cache. It defaults to serving BINARY packages from gentoo's default location but you can add to or replace that default.
/etc/conf.d/http-replicator
| Code: |
## Local dir to serve clients. Great for serving binary packages
## See PKDIR and PORTAGE_BINHOST settings in 'man make.conf'
## --alias /path/to/serve:location will make /path/to/serve
## browsable at http://http-replicator.com:port/location
DAEMON_OPTS="$DAEMON_OPTS --alias /var/tmp/packages/All:All"
|
So if you want to serve random files you can keep them in a separate dir for easy management by fetching them with the alias url or keep them in the cache and fetch them with the http_proxy setting. Multiple alias options are allowed. Http-replicator was designed to be a secure, high performance web server with a cache.
replicator doesn't check its own cache for this reason. It won't touch anything in it's cache because it may contain user files.
The question is should replicator check it's cache?
I say no right now because it can be done better by other means. But adding that feature would be convenient for many users?
1. If replicator is a gentoo only cache, there are other distfile checking scripts that will delete files based on many tests such as not in portage, not the most current version, older than a certain date, exceed a maximum cache size, not accessed in the last 3 months, etc etc.
2. If replicator is used for other files I can't even guess how to prune the cache.
What I do is this. It could be a cron script but I do it manually by choice.
| Code: |
mv /var/cache/http-replicator/* /var/tmp/distfiles/
repcacheman
rm -rf /var/tmp/distifles/*
|
This moves the cache files to the distfile dir. This is fast because it only renames the files, it doesn't move anything on disk.
repcacheman runs which moves all good files back to the cache.
then I delete all the remaining files which are not in portage or corrupt/incomplete.
You could also move the files, run the distfile cleaning script to prune based on your desires, then run repcacheman!
There was a time when distfile cleaning scripts were hard to find, now eclean is part of gentoolkit.
I know that was probably more than you wanted to know but I hope it helped you and some lurkers |
|
| Back to top |
|
|
dahoste
Tux's lil' helper
Joined: 01 Dec 2005
Posts: 138
Location: Maryland, USA
|
| Posted: Tue Jun 19, 2007 11:27 pm Post subject: |
|
|
| flybynite: does your new beta version address the MD5 problem when computing checksums? |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Wed Jun 20, 2007 5:52 am Post subject: |
|
|
| dahoste wrote: | | flybynite: does your new beta version address the MD5 problem when computing checksums? |
Yes, the new version is fully portage manifest2 compliant and is much faster than the previous version. |
|
| Back to top |
|
|
golding
Apprentice
Joined: 07 Jun 2005
Posts: 232
Location: Adelaide / South Australia
|
| Posted: Sat Jul 07, 2007 4:53 am Post subject: |
|
|
flybynite
Some time ago (early '06 I think) I posted here that http-replicator would be started in the rc init scripts, but when I went to emerge anything I had to restart it. This behaviour has remained until yesterday.
Before then I was using a login manager of varying types from gdm to xdm and even the Enlightenment greeter, but yesterday I decided I had had enough and wanted to properly secure my lan by using proper console login procedures.
Surprise, surprise! Suddenly http-replicator did not have to be re-started after login, now it works without that annoying restart before I emerge anything.
I do not know if this is a bug, however, I thought you might like to know.
_________________
Regards, Robert
..... Some people can tell what time it is by looking at the sun, but I have never been able to make out the numbers. |
|
| Back to top |
|
|
flybynite
l33t
Joined: 06 Dec 2002
Posts: 620
|
| Posted: Mon Jul 09, 2007 3:47 am Post subject: |
|
|
| golding wrote: | flybynite
I do not know if this is a bug, however, I thought you might like to know. |
Yes, I remember
A bug was filed similar to yours (I don't think you filed it) , but I could never reproduce it. Please check if you can help maurice here
https://bugs.gentoo.org/show_bug.cgi?id=177428 |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
