(yet another) Gentoo search engine

Zougloub · n00b Joined: 14 Dec 2007 Posts: 30

I've worked a lot on the site yesterday and there were times it was unusable.
It might not be very visible since I was enhancing the structure, but I added an overlay information page, and modified the ebuild parsing to get more information from them.

If you own/know an unofficial overlay and want it referenced, you can send me the layman xml for it (using jabber, IRC, gzo's wiki or here, but here would be noisy).

Zougloub · n00b Joined: 14 Dec 2007 Posts: 30

I'm rewriting some indexer stuff to benefit from Paludis' features (EAPI stuff, cleaner tools, speed), and make the site more efficient.

think4urs11 · Posted: Mon Jul 21, 2008 10:32 pm Post subject:

a RSS-Feed would be fantastic
_________________
Nothing is secure / Security is always a trade-off with usability / Do not assume anything / Trust no-one, nothing / Paranoia is your friend / Think for yourself

Zougloub · n00b Joined: 14 Dec 2007 Posts: 30

It's g.z.o's anniversary

I did some small updates, nothing transcendant and the site is still as ugly as always... but the service is here, so I don't care.
Since I am pulling all the repos, I figured that I could make the data public, so for those who don't want to have all the scms to sync, they can use rsync to sync on g.z.o (see /repos FYI).

Think4UrS11, I'll look into that some day... currently I'm not convinced of the utility of the feature...
I'm testing something quick and dirty with diff-eix right now (there is a public /repos/eix.cache file and now diff-eix output will go to /diffs/).

alex.blackbit · Advocate Joined: 26 Jul 2005 Posts: 2397

Zougloub,

please update the text on the website, eix-sync runs layman -S for quite some time now.

Zougloub · n00b Joined: 14 Dec 2007 Posts: 30

I don't have a lot of time to work on gzo but yesterday night I took a pair of hours to perform some updates.

I updated the category listing, now it's generated from the contents of the portage tree.
If translations are sufficiently added, the site might get multilingual.
At the moment, I see no need to adapt it since only some categories are translated and usage of metadata.xml in packages is marginal.

When I logged on my server yesterday, I saw that I had some days of cron'ed indexing tasks performing at the same time, and the server was under heavy load.
FYI, there are currently 927MB of data on the repositories, and 187MB of caches.
I have not found the source of the lagging yet, I guess some bad ebuilds and bash bugs might cause some trouble...

Noticed that I forgot to have a robots.txt file, leading to unnecessary hits by Google and Cuil search engines.

MageSlayer · Posted: Sun Jul 19, 2009 2:04 pm Post subject:

pastebin on site works, but no utf-8

My Russian letters turn to hex.

Thanks for effort

Zougloub · n00b Joined: 14 Dec 2007 Posts: 30

k-nopaste seems not to support UTF-8 out of the box, I quickly patched it to have it working (author notified)

++

MageSlayer · Posted: Sun Jul 19, 2009 7:16 pm Post subject:

Still it does not work as expected.

Tested with Firefox and Opera

Zougloub · n00b Joined: 14 Dec 2007 Posts: 30

Works for me ; I posted a few pages in chinese and saw your "test" in russian.

MageSlayer · Posted: Mon Jul 20, 2009 4:43 am Post subject:

Well my Russian after posting is similar to Chineese, I would say

That's how it looks before posting
http://img249.imageshack.us/img249/7883/zaptoo.gif

That's how it looks after posting
http://img12.imageshack.us/img12/17/zaptoo2.gif

MageSlayer · Posted: Mon Jul 20, 2009 9:17 am Post subject:

Now it works ok. Thanks

Zougloub · n00b Joined: 14 Dec 2007 Posts: 30

More than 2 years that the site is online now.
g.z.o was visited by 704 distinct IPs in december (excluding the search engine crawlers), that's not a lot but that's enough to consider that the site is helping people.

Although I don't really want to release the sources because I'm not very proud of that mess, I think I can share a few facts about the site.
Every day,
- all the overlays are sync'ed using the good old layman (landing in http://gentoo.zapto.org/repos/)
- the overlays list is transformed to build the http://gentoo.zapto.org/overlays/ page
- overlays metadata is generated using Instruo (a Paludis tool)
- the modified metadata are parsed, filling a SQLite3 database
- an eix cache is built, and the 2 latest caches are diffed to produce the http://gentoo.zapto.org/diffs/ contents

The pages are served by mod_python; the code is quite simple, there is a database interface, something to produce the wanted HTTP, and something to parse the requested URIs.

I've made a C++ rewrite of the search backend, but since there is absolutely no overloading of the 1.2GHz CPU the site is running on with the current Python code, it's still in the drawers.

Alanw · n00b Joined: 07 Dec 2009 Posts: 15

This is a very nice site!

The only thing I'd like to see is different color anchor tags <a />. Trying to view orange over purple is very difficult. I have to select the text to read it. Maybe a bold white, or deep purple would help?

-Alan
_________________
www.alan-warren.com
gentoo 2.6.39-gentoo-r3
[1]Dual Xeon E5645 @ 3.7Ghz
24 GB RAM
Nvidia Quadro 4000 Fermi
[2]Dual Xeon E5520 @ 2.26Ghz
24 GB RAM
Nvidia GeForce GTX 285

Zougloub · n00b Joined: 14 Dec 2007 Posts: 30

The links styles were not specified.
Maybe your style makes them orange.
I specified them.

Alanw · n00b Joined: 07 Dec 2009 Posts: 15

Zougloub · n00b Joined: 14 Dec 2007 Posts: 30

Dear g.z.o users, please use gentoo.zougloub.eu instead of gentoo.zapto.org.
I'm tired of renewing my dyndns/no-ip memberships...

See ya !

Zougloub · n00b Joined: 14 Dec 2007 Posts: 30

An horrible mistake had made its way during an update made some times ago to enhance the site performance (> 1 month...), rendering the site quite unusable.
Found about it some days ago but there was not much time to fix it... well, now it's fixed.