| View previous topic :: View next topic |
| Author |
Message |
ct85711
Veteran
Joined: 27 Sep 2005
Posts: 1791
|
 Posted: Thu Aug 20, 2015 11:06 pm Post subject: |
 |
|
hasufell, quick question that you may have an idea (or an improvement), in regards to the hard-reset.
Last time, I did hard-reset, it also wipes the distfiles directory (and it's contents) too (assuming default configuration and locations).
Is a .gitignore file going to be passed so the distfiles directory, so git does not wipe those files unnecessarily? |
|
| Back to top |
|
 |
hasufell
Retired Dev
Joined: 29 Oct 2011
Posts: 429
|
| Posted: Thu Aug 20, 2015 11:14 pm Post subject: |
|
|
| ct85711 wrote: | hasufell, quick question that you may have an idea (or an improvement), in regards to the hard-reset.
Last time, I did hard-reset, it also wipes the distfiles directory (and it's contents) too (assuming default configuration and locations).
Is a .gitignore file going to be passed so the distfiles directory, so git does not wipe those files unnecessarily? |
I don't think "git reset --hard" removes files that are in .gitignore. At least not here. The .gitignore file in the gentoo tree currently contains "/distfiles/", so a hard reset should be safe. |
|
| Back to top |
|
|
ct85711
Veteran
Joined: 27 Sep 2005
Posts: 1791
|
| Posted: Thu Aug 20, 2015 11:33 pm Post subject: |
|
|
Yes, git will not touch anything if it was mentioned in .gitignore; it was more of I was unaware if .gitignore was being passed down to us (so we don't wipe our distfiles directory when we do --reset).
I don't know git well enough, so I may very well be wrong, but I didn't think git would pull .gitignore by default (again, you may have changed that already, hence my lack of understanding). |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Fri Aug 21, 2015 8:19 am Post subject: |
|
|
| ct85711 wrote: | Yes, git will not touch anything if it was mentioned in .gitignore; it was more of I was unaware if .gitignore was being passed down to us (so we don't wipe our distfiles directory when we do --reset).
I don't know git well enough, so I may very well be wrong, but I didn't think git would pull .gitignore by default (again, you may have changed that already, hence my lack of understanding). |
If it's been checked in (committed and pushed), it will be distributed.
Have to say I'm a bit perturbed that everyone needs to come up with scriptarounds to get what was done via rsync before; and that rsync seems a lot slower. Apparently it's refreshing every metadata.xml irrespective of whether it's changed.
It's not the specific problem that bothers me; more that there seems to have been zero testing against an offline rsync-git gateway, before moving to distribute. WTF? |
|
| Back to top |
|
|
Yamakuzure
Advocate
Joined: 21 Jun 2006
Posts: 2296
Location: Adendorf, Germany
|
| Posted: Fri Aug 21, 2015 9:04 am Post subject: |
|
|
If you modify anything in your portage repo, git must fail, as it can not know whether you want to keep your changes or not.
That is completely in order.
What puzzled me is, that I did *not* change anything! So why did the file get its modes changed ?
@steveL: For the regular user nothing changed. Only we, who want to play around with pulling directly via git do have to work around a few pieces. The default way is *not* to pull directly from git as it was stated in the news article linked above.
_________________
Edited 220,176 times by Yamakuzure |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Fri Aug 21, 2015 7:06 pm Post subject: |
|
|
OK, Yamakuzure, I'm happy to take your word for it. (I did read the news before, and again just now ofc.)
It's not what I was told on IRC, but that was in confidence, so I won't repeat it word-for-word.
The specific problem was no testing of whether manifests are generated, after the git migration, apparently leading to a retransmit of every manifest, every --sync; but that may well just be the git side.
It was mentioned because I was syncing for the first time since the switchover, and idly waiting for it to get to metadata/.
The specific problem (the header-id thing also came up) is not the concern; we expect teething issues.
The concern is that there was apparently no testing (do a dry run on a spare server, check if you can emerge -pv kde; if that works you should have consistent ebuilds and manifests) which seems to be becoming a norm, rather than an exception, eg with a base-system utility like openrc, as compared to binutils. (That development seems to be second-guessing "trends" is also worrying.)
Anyhow I won't go on about it further; I really don't want to rain on anyone's parade, and like everyone else I'm relieved the switchover has finally happened.
And again: well done to all involved. |
|
| Back to top |
|
|
freke
Veteran
Joined: 23 Jan 2003
Posts: 1029
Location: Somewhere in Denmark
|
| Posted: Fri Aug 21, 2015 8:06 pm Post subject: |
|
|
News has moved?
Just started getting: fatal: repository 'https://anongit.gentoo.org/git/proj/gentoo-news.git/' not found.... |
|
| Back to top |
|
|
hasufell
Retired Dev
Joined: 29 Oct 2011
Posts: 429
|
|
| Back to top |
|
|
freke
Veteran
Joined: 23 Jan 2003
Posts: 1029
Location: Somewhere in Denmark
|
| Posted: Fri Aug 21, 2015 8:33 pm Post subject: |
|
|
Thanks - cleaned up a bit  |
|
| Back to top |
|
|
Yamakuzure
Advocate
Joined: 21 Jun 2006
Posts: 2296
Location: Adendorf, Germany
|
| Posted: Mon Aug 24, 2015 12:18 pm Post subject: |
|
|
And another weird thing: | Code: | * Starte emerge --sync
>>> Syncing repository 'gentoo' into '/usr/portage'...
/usr/bin/git pull
remote: Counting objects: 39818, done.
remote: Compressing objects: 100% (14835/14835), done.
remote: Total 39818 (delta 27305), reused 36293 (delta 24979)
Receiving objects: 100% (39818/39818), 7.17 MiB | 2.15 MiB/s, done.
Resolving deltas: 100% (27305/27305), completed with 18092 local objects.
From git://anongit.gentoo.org/repo/gentoo
1187ddb..e4c425a master -> origin/master
Updating 1187ddb..e4c425a
error: unable to unlink old 'dev-util/idea-ultimate/metadata.xml' (Permission denied)
error: unable to unlink old 'profiles/default/bsd/fbsd/amd64/10.2/package.use.force' (Permission denied)
error: unable to unlink old 'sci-physics/looptools/metadata.xml' (Permission denied)
error: unable to unlink old 'sci-physics/yoda/metadata.xml' (Permission denied)
!!! git pull error in /usr/portage |
This is not okay. The folder "dev-util/idea-ultimate" now has permissions root:root, and as portage drops its privileges, the "permission denied" must occur of course.
If this is meant to work, the git contents must be owned by portage, or portage must sync with root privileges.
This left my /usr/portage in an ugly state with hundreds of files marked as being changed and a few dozen as being untracked. I had to do a 'git reset --hard' before I could pull again (doing an 'chown -R portage:portage' beforehand).
It looks like there is much left to be done...
_________________
Edited 220,176 times by Yamakuzure |
|
| Back to top |
|
|
hasufell
Retired Dev
Joined: 29 Oct 2011
Posts: 429
|
| Posted: Mon Aug 24, 2015 12:32 pm Post subject: |
|
|
| Yamakuzure wrote: | And another weird thing: | Code: | * Starte emerge --sync
>>> Syncing repository 'gentoo' into '/usr/portage'...
/usr/bin/git pull
remote: Counting objects: 39818, done.
remote: Compressing objects: 100% (14835/14835), done.
remote: Total 39818 (delta 27305), reused 36293 (delta 24979)
Receiving objects: 100% (39818/39818), 7.17 MiB | 2.15 MiB/s, done.
Resolving deltas: 100% (27305/27305), completed with 18092 local objects.
From git://anongit.gentoo.org/repo/gentoo
1187ddb..e4c425a master -> origin/master
Updating 1187ddb..e4c425a
error: unable to unlink old 'dev-util/idea-ultimate/metadata.xml' (Permission denied)
error: unable to unlink old 'profiles/default/bsd/fbsd/amd64/10.2/package.use.force' (Permission denied)
error: unable to unlink old 'sci-physics/looptools/metadata.xml' (Permission denied)
error: unable to unlink old 'sci-physics/yoda/metadata.xml' (Permission denied)
!!! git pull error in /usr/portage |
This is not okay. The folder "dev-util/idea-ultimate" now has permissions root:root, and as portage drops its privileges, the "permission denied" must occur of course.
If this is meant to work, the git contents must be owned by portage, or portage must sync with root privileges.
This left my /usr/portage in an ugly state with hundreds of files marked as being changed and a few dozen as being untracked. I had to do a 'git reset --hard' before I could pull again (doing an 'chown -R portage:portage' beforehand).
It looks like there is much left to be done... |
| Quote: |
usersync
Drop privileges to the owner of ${repository_location}
for emerge(1) --sync operations. Note that this feature
assumes that all subdirectories of ${repository_loca‐
tion} have the same ownership as ${repository_location}
itself. It is the user's responsibility to ensure cor‐
rect ownership, since otherwise Portage would have to
waste time validating ownership for each and every sync
operation.
|
I'd go so far to say that messing with ownerships is definitely outside of the scope of such scripts and up to the user to configure. Applying ownerships on stuff (especially recursively) on an automated basis is very dangerous, since I have no idea what else might be hidden in such directories. |
|
| Back to top |
|
|
Yamakuzure
Advocate
Joined: 21 Jun 2006
Posts: 2296
Location: Adendorf, Germany
|
| Posted: Mon Aug 24, 2015 12:34 pm Post subject: |
|
|
| hasufell wrote: | | I'd go so far to say that messing with ownerships is definitely outside of the scope of such scripts and up to the user to configure. Applying ownerships on stuff (especially recursively) on an automated basis is very dangerous, since I have no idea what else might be hidden in such directories. |
Yes. You are, of course, right. I have to investigate how this happened. I just realized, that if portage calls "git pull", then it will be the owner of new files.
But as far as I know, nothing acts on my tree apart from portage, but there must be something... I do not believe that the above issue is git's fault.
Oh, and I have "userpriv" and "userfetch", but *not* "usersync" activated.
_________________
Edited 220,176 times by Yamakuzure |
|
| Back to top |
|
|
miket
Guru
Joined: 28 Apr 2007
Posts: 497
Location: Gainesville, FL, USA
|
| Posted: Sun Aug 30, 2015 4:39 am Post subject: |
|
|
I've got a problem quite different than the one under most discussion here: it's not git-versus-rsync, but git import from CVS. One of the big promises in using git for the Portage tree is that now it will be possible to trace back into the history of the repository to see when and why changes got introduced. This is one of the big reasons we use version-control systems, you know. The other super-compelling reason to have git is that it would be easy to restore old ebuilds that might still be useful in a given system.
Here's my particular issue. When running revdep-rebuild on a new system, it kept finding media-video/dirac-1.0.2-r1 to be broken. Time for closer scrutiny. The package had a missing dependency on libcppunit-1.13.so.0. On an older machine I found that the .so comes from dev-util/cppunit. Hmm. Why isn't dev-util/cppunit a dependency of dirac? It ought to be. I searched the bugzilla and found bug 312827, which complained that media-video/dirac-1.0.2 was pulling in a unit-test framework and not using it. I can surmise that the resolution was to remove the cppunit dependency from the ebuild. To be sure, that makes it so that a library would not be merged for a portion of the code which would not be used, but there was nothing in the ebuild to keep from building the binary that requires the libary.
So how would I find the exact change to the ebuild, the time it happened and who did it except by looking at the history of the Portage repository. Now here's the rub: while I suppose that some parts of the Portage history might have been pulled into git, that certainly did not happen with dirac. The dirac-1.0.2.ebuild has a commit date of August 9 and has only one revision. There have to have been earlier revisions, and moreover, the ebuild was actually first committed to Portage in 2009.
Is the Portage git repository not supposed to show history or even just to show history from the beginning of the conversion to git? That's not good enough! I know there's a way to migrate CVS history to other version contol systems such as git. At my company, they did two such conversions over the years: one from CVS to Subversion and then from Subversion to Mercurial. We kept our history. It is important to have!
From time to time I used the old cvsweb browser to find old ebuilds. Is that still available? Since the website redesign (ugh!!), I haven't be able to find it. Can someone point it out to me?
I can't file an intelligent bug report for media-video/dirac until I can see that history. |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Sun Aug 30, 2015 10:24 am Post subject: |
|
|
miket: if you're just syncing via git, then you get a shallow-clone (no history) by default, afaik.
History was why it took so long to get the conversion done; Patrick got it to the stage of reasonably compact, but it needed the email addresses rewriting (a pretty standard git job, from what I've seen.) |
|
| Back to top |
|
|
hasufell
Retired Dev
Joined: 29 Oct 2011
Posts: 429
|
| Posted: Sun Aug 30, 2015 10:28 am Post subject: |
|
|
| steveL wrote: | | miket: if you're just syncing via git, then you get a shallow-clone (no history) by default, afaik. |
You can set in repos.conf for the gentoo repository to get unlimited history.
| steveL wrote: | | History was why it took so long to get the conversion done; |
Haha, thanks for the laugh. |
|
| Back to top |
|
|
schorsch_76
Guru
Joined: 19 Jun 2012
Posts: 452
|
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Sun Aug 30, 2015 3:43 pm Post subject: |
|
|
| hasufell wrote: | | Haha, thanks for the laugh. |
YW. Thanks for the info, but no thanks for the bitchiness.
HF now. |
|
| Back to top |
|
|
miket
Guru
Joined: 28 Apr 2007
Posts: 497
Location: Gainesville, FL, USA
|
| Posted: Sun Aug 30, 2015 7:38 pm Post subject: |
|
|
| steveL wrote: | miket: if you're just syncing via git, then you get a shallow-clone (no history) by default, afaik.
History was why it took so long to get the conversion done; Patrick got it to the stage of reasonably compact, but it needed the email addresses rewriting (a pretty standard git job, from what I've seen.) |
No.
I didn't get the git checkout by a shallow copy in Portage: I won't switch to that way of syncing until things get sorted out. My access to the repository yesterday was via the gitweb interface, which ought to reflect the whole repository, not just a shallow copy. Just to be sure, this afternoon I cloned the repository from anongit.gentoo.org in case there was some question about the action of gitweb. Again, the oldest commit to media-video/dirac/dirac-1.0.2.ebuild was August 8. This is clearly incorrect. There's history behind this file I still can't see.
Really now, I want to know what would take so long in importing history because it sure looks like none was imported. That commit for dirac-1.0.2.ebuild is the oldest commit in the whole repository. Might there be other information lurking somewhere? I don't find it. The git branch command reveals that there is no branch except the master branch.
What is telling is the commit message: | Code: | proj/gentoo: Initial commit
This commit represents a new era for Gentoo:
Storing the gentoo-x86 tree in Git, as converted from CVS.
This commit is the start of the NEW history.
Any historical data is intended to be grafted onto this point. |
What happened to everything before this? What I'm expecting to see is the history of all the commits ab initio--with all the proper commit dates. What I can see instead is reflective of what Linux did in switching from BitKeeper to Git: the whole history started anew at the conversion to git. That *may* be acceptable in the kernel, which after all, is a single project with strong central controls on what gets commited, but is both inappropriate and unnecessary in the Gentoo case. Inappropriate: there are lots of separate committers to lots of separate files that pretty generally work independently of most of the repository as a whole. Whereas the kernel comes to us in a single big tarball, all these separate packages--currently installed on our systems--get updated on wildly divergent schedules. Unnecessary: unlike the Linux situation in which BitMover put down the hammer and closed off access to the older history (and, even if there were access, there were no ready tools for the conversion), in Gentoo there is no lack of access to the older repository nor tools for the conversion.
Could it be that I'm pulling from the wrong Git repository? Is there another one with the full commit history?
| schorsch_76 wrote: | | Maybe my git repo of portage might help you . See my signature. |
I navigated through the tree view of your viewer to find dirac. Again, sadly, nothing old enough is there.
By the way, until this thing gets settled out, can anyone tell me how to use the web interface to see the CVS tree? |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Mon Aug 31, 2015 12:10 pm Post subject: |
|
|
| miket wrote: | | I didn't get the git checkout by a shallow copy in Portage: I won't switch to that way of syncing until things get sorted out. My access to the repository yesterday was via the gitweb interface, which ought to reflect the whole repository, not just a shallow copy. Just to be sure, this afternoon I cloned the repository from anongit.gentoo.org in case there was some question about the action of gitweb. Again, the oldest commit to media-video/dirac/dirac-1.0.2.ebuild was August 8. This is clearly incorrect. There's history behind this file I still can't see. |
Ouch. Yeah that's wrong, then.
| Quote: | | What I'm expecting to see is the history of all the commits ab initio--with all the proper commit dates. What I can see instead is reflective of what Linux did in switching from BitKeeper to Git: the whole history started anew at the conversion to git. That *may* be acceptable in the kernel, which after all, is a single project with strong central controls on what gets commited, but is both inappropriate and unnecessary in the Gentoo case. |
Well just to say I agree, so you don't need to argue this case with me, though others may well have a different viewpoint.
| Quote: | | Could it be that I'm pulling from the wrong Git repository? Is there another one with the full commit history? |
That does sound plausible doesn't it? Somehow I doubt you have the wrong repo.
As for web, the Gentoo site is nowhere near as nice as it used to be, ime, so can't help sorry.
AFAIK the whole history was supposed to be imported; that's what Patrick posted to his blog and the developer list about.
If it's not, (if it isn't just that you have the wrong repo) then I agree 100%: the conversion is still not done yet. |
|
| Back to top |
|
|
miket
Guru
Joined: 28 Apr 2007
Posts: 497
Location: Gainesville, FL, USA
|
| Posted: Mon Aug 31, 2015 1:32 pm Post subject: |
|
|
| steveL wrote: | | Well just to say I agree, so you don't need to argue this case with me, though others may well have a different viewpoint. |
Sorry about causing you distress. I quoted your posting because it gave the most concise summary of what (ought to have) happened. I certainly wasn't fussing at you. After all, given what you posted about your understanding of the process, I would think you'd be the least likely of the commenters in this thread to think that the Portage git repository should simply lop off old history!
At my workplace, the commit history in our VCS goes back to 2005. There have been plenty of times over the years when I've had to look back that far.
EDIT: Ah, here's that link I was trying to find: https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/. Thanks to the Wayback Machine! |
|
| Back to top |
|
|
steveL
Watchman
Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery
|
| Posted: Mon Aug 31, 2015 8:23 pm Post subject: |
|
|
| Heh nw, miket; I wasn't distressed ;) |
|
| Back to top |
|
|
Yamakuzure
Advocate
Joined: 21 Jun 2006
Posts: 2296
Location: Adendorf, Germany
|
| Posted: Tue Sep 01, 2015 5:31 am Post subject: |
|
|
Just a quick question:
Is it normal that all ChangeLog files are gone? "find /usr/portage/ -iname 'ChangeLog'" only turns up "/usr/portage/profiles/features/prefix/Changelog"...
_________________
Edited 220,176 times by Yamakuzure |
|
| Back to top |
|
|
miket
Guru
Joined: 28 Apr 2007
Posts: 497
Location: Gainesville, FL, USA
|
| Posted: Tue Sep 01, 2015 1:13 pm Post subject: |
|
|
| Yamakuzure wrote: | Just a quick question:
Is it normal that all ChangeLog files are gone? "find /usr/portage/ -iname 'ChangeLog'" only turns up "/usr/portage/profiles/features/prefix/Changelog"... |
From the bit of browsing I did, it looks like they have thrown out the change logs. There is a *huge* misconception that just because the VCS has commit messages that it is also all right to omit ancillary documentation such as change logs. In the current situation the problem is compounded: if the thought is that change logs can be reconstituted from the chain of commit messages (and lots of luck sorting it out if the committer commited several things at once), but given the current omission of history before August 8, there would be no chance to see those old commit messages either.
From what I can see, the process of the Git conversion is *definitely* not done! It strikes me that there will need to be a big do-over. |
|
| Back to top |
|
|
hasufell
Retired Dev
Joined: 29 Oct 2011
Posts: 429
|
| Posted: Tue Sep 01, 2015 2:15 pm Post subject: |
|
|
| miket wrote: | | (and lots of luck sorting it out if the committer commited several things at once) |
There is no luck involved. Git can show you _any_ commit message that has touched a given file (or directory), no matter if it has been part of a bigger commit. It is far superior to any ChangeLogs we had.
Of course, you have to learn a little bit git.
| Code: | cd /usr/portage
git log -- <category>/<package-name>
|
| miket wrote: | | From what I can see, the process of the Git conversion is *definitely* not done! It strikes me that there will need to be a big do-over. |
Not really. The old history will not be re-added to the current development repository. But a historical git-converted repo will be set up (there are a few drafts already) which you can then merge into one giant local repo via 'git replace --graft' or something, see https://git-scm.com/blog/2010/03/17/replace.html and https://github.com/gentoo/gentoo-gitmig-20150809-draft |
|
| Back to top |
|
|
Perfect Gentleman
Veteran
Joined: 18 May 2014
Posts: 1255
|
| Posted: Sat Sep 05, 2015 3:57 am Post subject: |
|
|
---
Last edited by Perfect Gentleman on Sat Sep 05, 2015 4:01 pm; edited 1 time in total |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Gentoo Chat
