from_name: Guardian Tor

subject: [Wikipedia-l] Bugs? or Features?

body:
Lars Aronsson <lars at aronsson.se> wrote

> The technical discussion is taking place on wikitech-l at nupedia.com.
> This is a short summary and unofficial status report.
>
> In the last few weeks, the Wikipedia webserver has been very slow,
> depending on new functions being added without enough concern about
> their performance under high load.
>
> At 5 pm on Thursday May 9, Jimmy disabled some of the functions that
> consumed too much time, and there was an instant improvement in
> response time.  The removed functionality include the "talk" links.

Why didn't anyone tell us this? (If this was discussed somewhere, my appologies; I missed it). Several people have pointed out the missing Talk links, but this is the first time I've seen any explanation. I'm also concerned that this action was taken without general discussion. I personally consider the Talk pages to be vital in the everyday operation of the project, as they are used to work out disputes one what should be done with various articles.

BTW, the Talk links are still present in the Collogne Blue skin, and they can still be used by typing in the URL manually.

> The implementation is studied right now, and an improved version will
> probably be installed in the next few days.  At this point, I don't
> think more programmers are needed, but if you want to stay informed,
> you can join the wikitech-l to follow the reports and discussions.

I'm a worried that that software dicisions are becoming isolated from the main Wikipedia community. Before major changes and new features are implemented on the live Wikipedia server, there should be some kind of general discussion where everyone can come to an agreement. Wikitech-l is only of interest to the programmers; I would like such discussions to take place on the wiki itself.

> It is good to see that so many are concerned to have Wikipedia back in
> full operation, with all the necessary functionality.  However, it
> does surprise me that so few were complaining about the absurdly slow
> response times in the week before.

There were many complaints about the speed for the past couple weeks.

I don't mean any of this as a personal attack against the people working on the software and running the server. I really appreciate all the hard work you're putting in. I'm simply concerned that major software changes are being made without input from us non-programming Wikipedians.

-- Stephen Gilbert
--

Powered by Outblaze


list: wikipedia-l

source: pipermail-mbox

from_name: Larry Sanger

subject: [Wikipedia-l] Re: [Wikitech-l] What happened to...?

body:
Sure enough, when I pressed "edit text of this page," the text of the
current (correct, non-cached) page came up.  So at least nobody will be
editing old versions of pages, which is a relief.

I hope the cache will be updated soon...and then automatically create a
new cached version whenever anyone saves the page?  That's how it should
work, no?

If this makes any sense, I'm sure somebody is already on it.  :-)

Hey Magnus :-) I'm around but only when something goes wrong...a
foul-weather friend, you can call me.

Larry

On Sun, 12 May 2002, Jason Richey wrote:

> I assume this is because we recently turned on caching for
> performance's sake...
>
> Jason
>
> Larry Sanger wrote:
>
> > I know wikitech-l is probably working on this, but I want to express my
> > dismay at the fact that the most recently saved version of my user page
> > (and lots of other page) is something from back in February.  That's kind
> > of scary.
> >
> > Larry
> >
> > _______________________________________________
> > Wikitech-l mailing list
> > Wikitech-l at ross.bomis.com
> > http://ross.bomis.com/mailman/listinfo/wikitech-l
>
>


list: wikitech-l

subject: [Wikitech-l] What happened to...?


from_name: Lars Aronsson

subject: [Wikipedia-l] Bugs? or Features?

body:
Guardian Tor  wrote:
> I'm a worried that that software dicisions are becoming isolated
> from the main Wikipedia community.

In my mind, Wikipedia is about volunteering to make the world better.
If you are concerned about the divide between wikipedia-l and
wikitech-l, then *you* can bridge the gap by joining both lists and
post summaries to wikipedia-l of the technical issues currently under
discussion.  That's what I did.  I'm not paid for this, and I cannot
promise to do it on a regular basis.  However, I see no point in
moving all of the technical discussion to wikipedia-l.

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


from_name: Brion L. VIBBER

subject: [Wikitech-l] Several things

body:
On dim, 2002-05-12 at 12:09, Magnus Manske wrote:
> Jimbo: The scipt that rebuilds the links is in the CVS. You probably checked
> it out already. I think it is called "rebuildLinks.php" or something
> similar...
> This will run several minutes! *Maybe* you should take the site offline for
> that, so the database doesn't get cunfused between user edits and that
> script.

Note also that it probably will fail, since there seems to be some
actual trouble accessing the "unlinked" table in the database, so the
script will just be pumping links into a black hole. :(

Unfortunately, I don't know what's really wrong there or how to fix it;
the most descriptive error message I've managed to get claims "cannot
open file: 'unlinked.MYD'"; Jason says the file does in fact exist,
which implies that it is likely corrupted. Error logs on the server may
be more helpful in diagnosing the problemo; also try "myisamchk" or
other tools for possible recovery.

However, the data in the "linked" and "unlinked" tables can, if need be,
be tossed away and regenerated via the script if the tables are
recreated successfully. (See the table setup commands in wikipedia.sql.)

-- brion vibber (brion @ pobox.com)


from_name: Magnus Manske

subject: [Wikitech-l] Speedup

body:
I finished working on the speedup for now. Some of my test data (Athlon
1800):

ab -n100 -c8   (for the Biology article):
* uncached: ca. 11 requests/sec.
* cached: ca. 20 requests/sec.

This is about 20% faster compared to what I started with this morning.

The real time killer is the {{NUMBEROFARTICLES}} variable on the main page.
Its use increases the time per page by a factor of about 10! Only a change
of the database can help here.

Another thing that takes a lot of time is the headed or the Recent Changes
page. I guess since all the links there have to be checked every time the
page is displayed, it increases the load on the database quite a bit. On my
(short) local Recent Changes page, the header text alone is responsible for
about 20% of the total rendering time. We should probably force the Recent
Changes page *not* to look for existing articles in the header text.

Everyone, please test the changes I made (especially the login procedure)
for bugy, so Jimbo can use the current version ASAP.

Magnus


subject: [Wikitech-l] "The chaching bug" - a mystery?

body:
On dim, 2002-05-12 at 17:38, Lars Aronsson wrote:
> Less than two hours ago, I found the page
> http://www.wikipedia.com/wiki/Sweden blank again.  So I clicked Edit,
> then Save, to restore its full contents.  This has happened before.
> See the page's History.
>
> Is this "the caching bug"?

Not as far as I know...

>  How does it work?  Wouldn't the obsolete
> cached copy me destroyed the first time I did this?

Sure should have been. Here's how the caching works:

* When a page is loaded, the contents of the cur_cache field are
checked. If it is not empty (!= ""), the cached HTML is dropped into the
output and various parsing/rendering steps are bypassed. Otherwise, the
page is rendered anew, shown, and stored in cur_cache for the next time
the page is shown.

* When a page is saved, the cur_cache field is cleared (set to ""), so
that the next time it's loaded (which should be immediately!) it will be
re-rendered.

* When a new page is created, the "unlinked" table is checked to see if
any other pages were linking to the previously-nonexistant page. If so,
those pages have their cur_cache cleared (set to ""), so that when next
viewed, they can show the link as being to an existing page. (This won't
happen currently, as the "unlinked" table is broken.)

"The caching bug" is due to the fact that, if caching is disabled, the
cur_cache field isn't touched at all, not even to clear caches when old
pages are saved: the software has no way of knowing the cache is old
(because there's no cache timestamp), so the old version is displayed as
if it were the current version of the page. Basically, it assumes that
caching will either be always off, or start off and turn on forever.
Turning it on, then turning it off for a while, then back on without
manually clearing the cache leaves bad cached pages.

This can easily be solved in the future by performing the cache-clears
whether caching is enabled or not. (I assume this was not done in the
first place so that the code would still work until the cur_cache field
was added to the database!)

The solution for our current problem is simply to clear all the cache
fields:
  UPDATE cur SET cur_cache="",cur_timestamp=cur_timestamp

which can be done by someone with is_developer access (that would be
nobody, currently!) or by someone with direct access to the db server
(messieurs Wales and Richey). That will get rid of all the old cached
pages, and the _correct_ caches can start to fill up again. (Except that
unlinked links may stick around for newly created pages due to the
separate "unlinked" table problem.)

>  Why does the blank page come back?

I have no idea! I haven't actually seen this problem yet, I've only seen
a couple notes that said roughly "page was blank, so I saved it again,
fine now."

Does the page _stay_ blank after reloading; ie does it consistently come
up blank until you resave it? Or does it come up blank once, but is okay
if you reload? Is it completely blank, or does it contain spaces or
something? Are there any error messages? Can you see anything suspicious
in the HTML source of the page?

If anyone with is_sysop status notices a page doing this, try doing a
direct SQL query before doing anything to the article:

  SELECT * FROM cur WHERE cur_title="Title_of_the_page"

What's in the cur_cache field? Does anything else look out of place?
(Hmm, looks like special_asksql.php needs to slip in a couple of
<nowiki></nowiki> tags.)

> Is caching really necessary for performance now?

I wouldn't doubt that it's _helping_.

>  When caching was
> activated, I thought several other functions were disabled at the same
> time.  Do we know the impact on response times of not using caching?

Alas, yes and no in that order. Since the main limiting factor is still
the multiple/complex/slow database queries, not CPU time in PHP, caching
may be a smaller help than anything else we're doing; but I'd be quite
surprised if it's not helping at all. And what's the sense in throwing
away a performance gain if we've got one?

(Note that caching does cut down a bit on database queries, since links
don't need to be checked for existence.)

-- brion vibber (brion @ pobox.com)


subject: [Wikipedia-l] What happened to...?

body:
Jimbo turned on the cache again after some time, without resetting it first.
This is only temporary, don't worry.

Magnus

PS: Glad you're still around ;)

> -----Original Message-----
> From: wikipedia-l-admin at nupedia.com
> [mailto:wikipedia-l-admin at nupedia.com]On Behalf Of Larry Sanger
> Sent: Sunday, May 12, 2002 8:05 PM
> To: wikipedia-l at nupedia.com; wikitech-l at nupedia.com
> Subject: [Wikipedia-l] What happened to...?
>
>
> I know wikitech-l is probably working on this, but I want to express my
> dismay at the fact that the most recently saved version of my user page
> (and lots of other page) is something from back in February.  That's kind
> of scary.
>
> Larry
>
> [Wikipedia-l]
> To manage your subscription to this list, please go here:
> http://www.nupedia.com/mailman/listinfo/wikipedia-l


subject: [Wikitech-l] Several things

body:
Jason: The SQL for updating the cache is

	 UPDATE cur SET cur_cache="",cur_timestamp=cur_timestamp

as Brion mailed earlier. Please run that one.

Jimbo: The scipt that rebuilds the links is in the CVS. You probably checked
it out already. I think it is called "rebuildLinks.php" or something
similar...
This will run several minutes! *Maybe* you should take the site offline for
that, so the database doesn't get cunfused between user edits and that
script.

What I said baout the Recent Changes is about the header ("Track the most
recent changes..."), *not* the list of recent changes! It is a brief text,
but it is loaded quite often, and each time, the database is checked for all
those links in that text.

That's it for now,
Magnus


body:
Less than two hours ago, I found the page
http://www.wikipedia.com/wiki/Sweden blank again.  So I clicked Edit,
then Save, to restore its full contents.  This has happened before.
See the page's History.

Is this "the caching bug"?  How does it work?  Wouldn't the obsolete
cached copy me destroyed the first time I did this?  Why does the
blank page come back?

Is caching really necessary for performance now?  When caching was
activated, I thought several other functions were disabled at the same
time.  Do we know the impact on response times of not using caching?

--
  Lars Aronsson
  <lars at aronsson.se>
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


body:
On lun, 2002-05-13 at 00:36, Lars Aronsson wrote:
> 20020511 Sat 13:20:00 769.98 200 http://www.wikipedia.com/wiki/Sweden 13360
> 20020511 Sat 13:30:00 2549.62 http://www.wikipedia.com/wiki/Sweden 0
> 20020511 Sat 13:40:00 2288.51 200 http://www.wikipedia.com/wiki/Sweden 4577
[...]
> 20020512 Sun 11:00:00 327.15 200 http://www.wikipedia.com/wiki/Sweden 12807
> 20020512 Sun 11:10:00 2549.62  http://www.wikipedia.com/wiki/Sweden 0
> 20020512 Sun 11:20:00 2549.71  http://www.wikipedia.com/wiki/Sweden 0
> 20020512 Sun 11:30:00 2549.82  http://www.wikipedia.com/wiki/Sweden 0
> 20020512 Sun 12:10:00 481.58 200 http://www.wikipedia.com/wiki/Sweden 4576

Woo-hoo! Thanks for the detailed stats, I think I've got it figured out
now. The timeouts just before the blanking look VERY suspicious...

Here's what I think is going on:

* The main database query times out on a page load.

* wikiPage::load() doesn't check the return code from mysql_query()
(oops!) and ends up with nothing for, among other things, $this->cache
and $this->text.

* Since the cache is empty (not != ""), the text is re-rendered. Blank
text actually ends up rendering to "<p><p>\n<p>\n". (Is that what you
see if you check the HTML code of the blank pages?)

* The new rendered blankness gets saved in the cache. (For some reason,
the PHP process for the page hasn't been killed for being around too
long by this point.)

* Subsequent page views get shown with the pseudo-blank cache, since
it's not actually a null string (hence, != "")

So, it's not THE caching bug, but it's A caching bug.

Fix: always check return codes! I've added a quick check to
wikiPage::load() for the main query, so if it can't execute the query it
should spit out an error message and _not_ cache the result. All the
other queries really should be checked too, though.

-- brion vibber (brion @ pobox.com)


subject: [Wikipedia-l] Difficulties with Wikipedia

body:
In light of the current difficulties, discussion on the wiki would have
been difficult at best, and impossible at worst!

However, a page devoted to technical issues (eg: scheduled downtimes,
current difficulties, fixes implemented) with a brief non-technical
explanation of the problems would probably be of use...

 ... if there is one, it's not used enough!

In terms of the status of the system, even having just the base
functionality is a Good Thing: if people can read, type and edit, then
the Wikipedia will continue. Even if that's not possible for some
reason, just letting people read (and letting them know they can't
write, today) is a Good Thing, too.

And guys: don't give up the hard work! :D

Dragon Dave.


from_name: Jimmy Wales

subject: [Wikitech-l] Editing recent changes header

body:
People are justifiably annoyed sometimes when we make technical
changes that affect them, without sufficient notice.  Our defense is
valid in the current case, i.e. there was a crisis going on, and we
had to do *something*.  But still, people are right to ask to be
notified.

It would be helpful if the Recent Changes text were editable.  In the
old software, the text at the top of the Recent Changes page was just
another wiki text.  We could (and did) edit it from time to time to
keep people updated on technical issues.  This is a great place for
it, because that text is viewed by all serious contributors many times
a day.

--Jimbo


body:
Brion L. VIBBER wrote:
> Does the page _stay_ blank after reloading; ie does it consistently come
> up blank until you resave it?

Yes it stays blank.  I can tell you how I discovered this.  I
suspected that my response time monitoring script was seeing timeouts
instead of real pages when response times where in the 500+ seconds
range.  So I added HTTP status code and document length to my log
file.  I can tell you the Main Page is 18K in normal cases and 4K when
only the page header and footer are returned.

With times in GMT, the normal sequence is like this:

date         time     resp status    url                 bytes
20020510 Fri 11:40:00 9.02 200 http://www.wikipedia.com/ 18459
20020510 Fri 11:50:00 1.13 200 http://www.wikipedia.com/ 18459

As you can see, the status code is 200 and response time very
reasonable (1.13 - 9.02 seconds).  But then something happens at
12:00 GMT (5:20 am PDT) Friday:

20020510 Fri 12:00:00 0.90 200 http://www.wikipedia.com/ 3999
20020510 Fri 12:10:00 1.09 200 http://www.wikipedia.com/ 3999

It stays this way until restored at 14:10 GMT (7:10 am PDT) Friday:

20020510 Fri 13:50:00 7.80 200 http://www.wikipedia.com/ 3999
20020510 Fri 14:00:00 8.71 200 http://www.wikipedia.com/ 3999
20020510 Fri 14:10:00 1.27 200 http://www.wikipedia.com/ 18444
20020510 Fri 14:20:00 1.18 200 http://www.wikipedia.com/ 18444

The same thing happens at 13:20 GMT (6:20 am PDT) Saturday:

20020511 Sat 13:00:00 1.65 200 http://www.wikipedia.com/ 18443
20020511 Sat 13:10:00 1.12 200 http://www.wikipedia.com/ 18443
20020511 Sat 13:20:00 174.32 200 http://www.wikipedia.com/ 3998
20020511 Sat 13:30:00 3.31 200 http://www.wikipedia.com/ 3998

And is restored at 15:40 GMT (8:40 am PDT) Saturday:

20020511 Sat 15:20:00 0.59 200 http://www.wikipedia.com/ 3998
20020511 Sat 15:30:00 0.59 200 http://www.wikipedia.com/ 3998
20020511 Sat 15:40:00 7.68 200 http://www.wikipedia.com/ 18445
20020511 Sat 15:50:00 1.18 200 http://www.wikipedia.com/ 18445

And 15:30-16:30 GMT (8:30 am - 9:30 am PDT) Sunday:

20020512 Sun 15:00:00 13.99 200 http://www.wikipedia.com/ 18443
20020512 Sun 15:20:00 1.88 200 http://www.wikipedia.com/ 18443
20020512 Sun 15:30:00 0.59 200 http://www.wikipedia.com/ 3998
20020512 Sun 15:40:00 0.68 200 http://www.wikipedia.com/ 3998
[...]
20020512 Sun 16:10:00 0.61 200 http://www.wikipedia.com/ 3998
20020512 Sun 16:20:00 0.62 200 http://www.wikipedia.com/ 3998
20020512 Sun 16:30:00 4.79 200 http://www.wikipedia.com/ 18397
20020512 Sun 16:40:00 1.22 200 http://www.wikipedia.com/ 18289

Obviously, I didn't restore the Main Page since I don't have the
privileges to do this.  And I have no idea who or what did this.

For the [[Sweden]] page some response time observations are so long or
timed out (zero length, no status code) that log entries appear in the
wrong order.  The two restore times (15:50 GMT = 8:50 am PDT Saturday,
and 22:50 GMT = 3:50 pm PDT Sunday) correspond to the revision History
page (8:48 and 15:48), when I restored the page:

20020511 Sat 13:10:00 45.22 200 http://www.wikipedia.com/wiki/Sweden 13360
20020511 Sat 13:20:00 769.98 200 http://www.wikipedia.com/wiki/Sweden 13360
20020511 Sat 13:30:00 2549.62 http://www.wikipedia.com/wiki/Sweden 0
20020511 Sat 13:40:00 2288.51 200 http://www.wikipedia.com/wiki/Sweden 4577
20020511 Sat 13:50:00 940.10 200 http://www.wikipedia.com/wiki/Sweden 4577
[...]
20020511 Sat 15:30:00 0.80 200 http://www.wikipedia.com/wiki/Sweden 4577
20020511 Sat 15:40:00 0.81 200 http://www.wikipedia.com/wiki/Sweden 4577
20020511 Sat 15:50:00 1.22 200 http://www.wikipedia.com/wiki/Sweden 12807
20020511 Sat 16:00:00 1.09 200 http://www.wikipedia.com/wiki/Sweden 12807
[...]
20020512 Sun 10:40:00 384.72 200 http://www.wikipedia.com/wiki/Sweden 12807
20020512 Sun 11:00:00 327.15 200 http://www.wikipedia.com/wiki/Sweden 12807
20020512 Sun 11:10:00 2549.62  http://www.wikipedia.com/wiki/Sweden 0
20020512 Sun 11:20:00 2549.71  http://www.wikipedia.com/wiki/Sweden 0
20020512 Sun 11:30:00 2549.82  http://www.wikipedia.com/wiki/Sweden 0
20020512 Sun 12:10:00 481.58 200 http://www.wikipedia.com/wiki/Sweden 4576
20020512 Sun 11:40:00 2286.75 200 http://www.wikipedia.com/wiki/Sweden 4576
[...]
20020512 Sun 22:30:00 1.90 200 http://www.wikipedia.com/wiki/Sweden 4576
20020512 Sun 22:40:00 0.80 200 http://www.wikipedia.com/wiki/Sweden 4576
20020512 Sun 22:50:00 1.10 200 http://www.wikipedia.com/wiki/Sweden 12805
20020512 Sun 23:10:00 1.10 200 http://www.wikipedia.com/wiki/Sweden 12805

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


from_name: Jason Richey

subject: [Wikitech-l] Several things

in_reply_to: <MLEPJFEEOFMFHGIHGJPICEOACAAA.magnus.manske@epost.de>; from magnus.manske@epost.de on Sun, May 12, 2002 at 09:09:28PM +0200

body:
Done.

mysql> UPDATE cur SET cur_cache="",cur_timestamp=cur_timestamp;
Query OK, 50142 rows affected (44.27 sec)
Rows matched: 50759  Changed: 50142  Warnings: 0

Jason

Magnus Manske wrote:

> Jason: The SQL for updating the cache is
>
> 	 UPDATE cur SET cur_cache="",cur_timestamp=cur_timestamp

--
"Jason C. Richey" <jasonr at bomis.com>


from_name: Karen AKA Kajikit

body:
"d.n.mckee" wrote:
>
> In light of the current difficulties, discussion on the wiki would have
> been difficult at best, and impossible at worst!
>
> However, a page devoted to technical issues (eg: scheduled downtimes,
> current difficulties, fixes implemented) with a brief non-technical
> explanation of the problems would probably be of use...
>
>  ... if there is one, it's not used enough!
>
> In terms of the status of the system, even having just the base
> functionality is a Good Thing: if people can read, type and edit, then
> the Wikipedia will continue. Even if that's not possible for some
> reason, just letting people read (and letting them know they can't
> write, today) is a Good Thing, too.
>
> And guys: don't give up the hard work! :D

I agree. Basic functionality has been restored and we can 'do stuff'
even if it's not fancy. That's what's really important. But it might be
a Good Thing to add another page to the special pages section for
technical updates/'what's in the pipeline'/'what we're working on' so
that people can stay informed more easily.

--

Karen AKA Kajikit

Come and visit my part of the web:
Kajikit's Corner: http://Kajikit.netfirms.com/
Aussie Support Mailing List: http://groups.yahoo.com/group/AussieSupport
Allergyfree Eating Recipe Swap:
http://groups.yahoo.com/group/Allergyfree_Eating

Love and huggles to all!


subject: [Wikitech-l] Several things

body:
On lun, 2002-05-13 at 10:00, Jason Richey wrote:
> Done.
>
> mysql> UPDATE cur SET cur_cache="",cur_timestamp=cur_timestamp;
> Query OK, 50142 rows affected (44.27 sec)
> Rows matched: 50759  Changed: 50142  Warnings: 0

Huge thanks, Jason!

If at some point we can get the software upgraded to the CVS version, we
can also fix these problems:

(Major problems fixed in CVS)
* old page versions overwrite the cache when viewed
* no Talk page links
* special:Watchlist grinds the database to unusable levels
* (possibly) pages spontaneously appearing empty

(Less severe, but darned annoying problems fixed in CVS)
* some inter-language links broken
* recent changes only bolds certain watchlist items
* wrong link colors and other style issues
* buggy experimental auto-wikification feature too "in-your-face"
others ?

(Assuming the code is otherwise in a usable state. It hasn't exploded on
me recently.)

-- brion vibber (brion @ pobox.com)


subject: [Wikipedia-l] Main page broken

body:
On ven, 2002-05-10 at 07:38, Robert Graham Merkel wrote:
> I don't know how, I don't know why, but the main page is
> giving me "describe the new page here" at the moment.
>
> Doesn't appear to be in the Recent Changes, so I assume
> there's some other breakage gone on.

This and several other cache-related bugs have been located and fixed in
the software repository. It should be updated soon, and this nightmare
will soon be over!

In short, we've been having three problems with the page caching:

* Old versions of pages were left in the cache from a couple months ago.
These have now been cleared out; thanks Jason!

* Old versions can still get into the cache again when you load them up
from the history list. A fix is ready in CVS.

* Pages sometimes spontaneously turn into "Describe the new page here."
This appears to be caused by database timeouts that result in a blank
page being "loaded" and saved over the cache; a workaround that should
stop this is ready in CVS.

Thanks, everbody, for not lynching the programming team while we work on
fixing this stuff. :)

-- brion vibber (brion @ pobox.com)


subject: [Wikipedia-l] Bugs? or Features?

body:
On lun, 2002-05-13 at 18:01, Guardian Tor wrote:
> The first is if a Bomis sysadmin has to take drastic steps to keep
> Wikipedia running (i.e. temporarily removing major functionality),
> could some sort of announcement  made soon afterwards? That way, the
> community doesn't have to guess what's going on.

That would be nice, yes. :)

There's been talk of making the text at the top of Recent Changes
editable again (as on the old software), which would enable this kind of
major news to be placed both on the main page and recent changes, the
two main entry points to the site. A courtesy message to wikipedia-l
would be helpful, too.

> The second is my concern about what the Cunctator calls "featuritis".
> I think that any *major* new software features should be discussed in
> the general Wikipedia assembly. For example, what is the
> "Autowikification" button (no offence intended to whoever implemented
> it!)? What does it do? Why is it useful? How do I use it? Most
> importantly, do we need it? It sounds like it automatically links
> words to existing articles. If so, I would argue against the utility
> of such a creature.

See the thread "Wikification function proposal" in the Wikipedia-l
archives circa February 27. There was discussion; consensus was not
really achieved; Magnus later decided to add it as an experimental
feature (as mentioned on Wikitech-l, 7 April), and it ended up on the
main Wikipedia mainly because no one remembered to take it out before
the software suddenly got upgraded last weekend! In the latest revision
(not yet installed) it's been made optional, the button hidden by
default, until such time as it is working, documented, and desired.

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] Several things

in_reply_to: <1021339581.3487.2.camel@rdaneel.moisty.org>; from brion@pobox.com on Mon, May 13, 2002 at 06:26:20PM -0700

body:
Done.  Yikes.

Fatal error: Cannot instantiate non-existent class: skinstandard in /home/wiki-newest/work-http/wikiPage.php on line 1131

Site down for the moment, probably I can fix it in moments.

Brion L. VIBBER wrote:

> On lun, 2002-05-13 at 10:00, Jason Richey wrote:
> > Done.
> >
> > mysql> UPDATE cur SET cur_cache="",cur_timestamp=cur_timestamp;
> > Query OK, 50142 rows affected (44.27 sec)
> > Rows matched: 50759  Changed: 50142  Warnings: 0
>
> Huge thanks, Jason!
>
> If at some point we can get the software upgraded to the CVS version, we
> can also fix these problems:
>
> (Major problems fixed in CVS)
> * old page versions overwrite the cache when viewed
> * no Talk page links
> * special:Watchlist grinds the database to unusable levels
> * (possibly) pages spontaneously appearing empty
>
> (Less severe, but darned annoying problems fixed in CVS)
> * some inter-language links broken
> * recent changes only bolds certain watchlist items
> * wrong link colors and other style issues
> * buggy experimental auto-wikification feature too "in-your-face"
> others ?
>
> (Assuming the code is otherwise in a usable state. It hasn't exploded on
> me recently.)
>
> -- brion vibber (brion @ pobox.com)
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at ross.bomis.com
> http://ross.bomis.com/mailman/listinfo/wikitech-l


body:
"d.n.mckee" <d.n.mckee at durham.ac.uk> wrote:

> In light of the current difficulties, discussion on the wiki would have
> been difficult at best, and impossible at worst!
>
> However, a page devoted to technical issues (eg: scheduled downtimes,
> current difficulties, fixes implemented) with a brief non-technical
> explanation of the problems would probably be of use...
>
>  ... if there is one, it's not used enough!

Yes, that's exactly what I'm looking for. Jimmy, would it be possible to have a page for notices of planned downtime, emergency situations and the like? Maybe a static web page that isn't on the wiki would be best.

> And guys: don't give up the hard work! :D

Indeed!

-- Stephen Gilbert

________________________________
Wikipedia: The Free Encyclopedia
   http://www.wikipedia.com

--

Powered by Outblaze


subject: [Wikipedia-l] Main page broken

in_reply_to: <1021341043.3428.4.camel@rdaneel.moisty.org>; from brion@pobox.com on Mon, May 13, 2002 at 06:50:43PM -0700

body:
The fixes Brion outlines here have just now been installed on the server.  There still
appear to be some minor problems, but at least the talk: links are back!

Brion L. VIBBER wrote:

> On ven, 2002-05-10 at 07:38, Robert Graham Merkel wrote:
> > I don't know how, I don't know why, but the main page is
> > giving me "describe the new page here" at the moment.
> >
> > Doesn't appear to be in the Recent Changes, so I assume
> > there's some other breakage gone on.
>
> This and several other cache-related bugs have been located and fixed in
> the software repository. It should be updated soon, and this nightmare
> will soon be over!
>
> In short, we've been having three problems with the page caching:
>
> * Old versions of pages were left in the cache from a couple months ago.
> These have now been cleared out; thanks Jason!
>
> * Old versions can still get into the cache again when you load them up
> from the history list. A fix is ready in CVS.
>
> * Pages sometimes spontaneously turn into "Describe the new page here."
> This appears to be caused by database timeouts that result in a blank
> page being "loaded" and saved over the cache; a workaround that should
> stop this is ready in CVS.
>
> Thanks, everbody, for not lynching the programming team while we work on
> fixing this stuff. :)
>
> -- brion vibber (brion @ pobox.com)
>
> [Wikipedia-l]
> To manage your subscription to this list, please go here:
> http://www.nupedia.com/mailman/listinfo/wikipedia-l


subject: [Wikipedia-l] Bugs? or Features?

body:
lcrocker at nupedia.com wrote:

> Yes, new _features_ or major changes in features should be
> discussed on the list at large.  Again, this one situation was
> an emergency and I'm glad that Jim was able to take unilateral
> action to rescue the project when it was needed, and I wouldn't
> want his ability to do that overly burdened with discussion.

I agree. I don't oppose unilateral action by Jimmy in dire situations, and Wikipedia was grinding to a halt looked pretty dire to me. My only request is that when such action is taken, some sort of general notice is given, either via Wikipedia-L or Wikipedia Announcements.

> I also think programmers ought to have some discretion on minor
> features and changes.  But generally I agree that any changes
> in the interface should be discussed on the general list.

Absolutely. It would be terrible if programmers felt they were being micromanaged and had to run every code change by a mass of non-programmers.

-- Stephen Gilbert

________________________________
Wikipedia: The Free Encyclopedia
   http://www.wikipedia.com

--

Powered by Outblaze


subject: [Wikipedia-l] Main page broken

in_reply_to: <r5iadr2oebk.fsf@gill.maths.keele.ac.uk>; from wiki@gwowen.freeserve.co.uk on Tue, May 14, 2002 at 05:59:43PM +0100

body:
Oh, dear.  I'm on it.

Gareth Owen wrote:

> Jimmy Wales <jwales at bomis.com> writes:
>
> > The fixes Brion outlines here have just now been installed on the server.
> > There still
> > appear to be some minor problems, but at least the talk: links are back!
>
> http://www.wikipedia.com/wiki/Miles_Davis&action=edit
> Fatal error: Call to undefined function: doedit() in
> /home/wiki-newest/work-http/wiki.phtml on line 95
>
> ... that doesn't look minor
> --
> Gareth Owen
> "Wikipedia does rock.  By the count on the "brilliant prose" page, there
>  are 14 not-bad articles so far" -- Larry Sanger (12 Jan 2001)
>
> [Wikipedia-l]
> To manage your subscription to this list, please go here:
> http://www.nupedia.com/mailman/listinfo/wikipedia-l


subject: [Wikipedia-l] Bugs? or Features?

body:
----- Original Message -----
Lars Aronsson wrote

> In my mind, Wikipedia is about volunteering to make the world better.
> If you are concerned about the divide between wikipedia-l and
> wikitech-l, then *you* can bridge the gap by joining both lists and
> post summaries to wikipedia-l of the technical issues currently under
> discussion.

I hope I don't sound like I'm giving orders; I know we're all volunteers here, including Mr. Wales, who has volunteered his servers and talents to keep the whole show running. I'm not asking for summaries of all technical discussions, nor do I think they should be moved to Wikipedia-L (they were just moved *off* of this list not that long ago). When I say I'm worried about the programmers becoming isolated (note that I don't think the programmering subcommunity *is* isolated yet), I'm mainly talking about two things.

The first is if a Bomis sysadmin has to take drastic steps to keep Wikipedia running (i.e. temporarily removing major functionality), could some sort of announcement  made soon afterwards? That way, the community doesn't have to guess what's going on.

The second is my concern about what the Cunctator calls "featuritis". I think that any *major* new software features should be discussed in the general Wikipedia assembly. For example, what is the "Autowikification" button (no offence intended to whoever implemented it!)? What does it do? Why is it useful? How do I use it? Most importantly, do we need it? It sounds like it automatically links words to existing articles. If so, I would argue against the utility of such a creature.

However, I also think that the programmers should not have to be nitpicked by a bunch of people who don't code. ;-) It's only major stuff that I'm talking about here.

So, it's not that I want someone to sum up "This Week on Wikitech-L" all the time. Anyone interested in that can join the list. I'm just looking for ways to keep the rest of the community in the loop concerning major changes to everyday Wikipedia work.

-- Stephen Gilbert

________________________________
Wikipedia: The Free Encyclopedia
   http://www.wikipedia.com
--

Powered by Outblaze


subject: [Wikitech-l] Re: Link update script

in_reply_to: <MLEPJFEEOFMFHGIHGJPIMEOFCAAA.magnus.manske@epost.de>; from magnus.manske@epost.de on Tue, May 14, 2002 at 07:23:06PM +0200

body:
I've switched to the CVS version and corrected a few bugs.  But the
site is sluggish again, and load is creeping upward.  I wonder if
getOtherNamespaces is the issue again?

Magnus Manske wrote:

> When you've fixed the current version, please run the link update script. I
> just fixed the database so it will work properly (I hope;)
>
> The script is in the CVS, run it with
> 	php updateLinks.php
> (or whatever it is actually called)
>
> Attention: This will take several minutes to complete! You might consider
> taking the site offline for the time being.
>
> Magnus
>


subject: [Wikitech-l] Re: Link update script

in_reply_to: <MLEPJFEEOFMFHGIHGJPIEEOGCAAA.magnus.manske@epost.de>; from magnus.manske@epost.de on Tue, May 14, 2002 at 08:56:41PM +0200

body:
Things seem more or less o.k. now. I'm running rebuildLinks.php -- it will be running
for several more hours, it seems.

Magnus Manske wrote:

> > -----Original Message-----
> > From: wikitech-l-admin at nupedia.com
> > [mailto:wikitech-l-admin at nupedia.com]On Behalf Of Jimmy Wales
> > Sent: Tuesday, May 14, 2002 7:23 PM
> > To: wikitech-l at nupedia.com
> > Subject: [Wikitech-l] Re: Link update script
> >
> >
> > I've switched to the CVS version and corrected a few bugs.  But the
> > site is sluggish again, and load is creeping upward.  I wonder if
> > getOtherNamespaces is the issue again?
>
> I don't think so - it should be fixed so it only checks for a single article
> (talk or the original topic) now. In fact, I cut the time for page rendering
> down by 20% compared to the patched version (without the talk links) of
> yours.
>
> Anyway, currently the 'pedia seems to run fast and stable. What does the top
> command say? Is the "really big process" bug back??
>
> Magnus
>


from_name: Gareth Owen

subject: [Wikipedia-l] Main page broken

in_reply_to: Jimmy Wales's message of "Tue, 14 May 2002 09:27:40 -0700"

body:
Jimmy Wales <jwales at bomis.com> writes:

> The fixes Brion outlines here have just now been installed on the server.
> There still
> appear to be some minor problems, but at least the talk: links are back!

http://www.wikipedia.com/wiki/Miles_Davis&action=edit
Fatal error: Call to undefined function: doedit() in
/home/wiki-newest/work-http/wiki.phtml on line 95

... that doesn't look minor
--
Gareth Owen
"Wikipedia does rock.  By the count on the "brilliant prose" page, there
 are 14 not-bad articles so far" -- Larry Sanger (12 Jan 2001)


subject: [Wikipedia-l] Bugs? or Features?

body:
Guardian Tor  wrote:

> The second is my concern about what the Cunctator calls
> "featuritis". I think that any *major* new software features should
> be discussed in the general Wikipedia assembly.

I agree with this.  But there is a problem to determine in advance
what is a "major" and what is a minor feature.  Obviously, a feature
or implementation detail that makes the system grind to a halt is a
major issue.  This time, the "talk" links turned out to be one of the
major issues because of the inefficient way they were implemented,
and temporarily removing the talk links made the website run faster.
(Let's hope the improved implementation is up and running soon.)

I think the autowikification button is a most interesting feature, but
perhaps it should be disabled by default, with each user having the
option to enable it from the personal preferences page.  This way, it
wouldn't confuse the beginners.  It could also be turned on
automatically for all logged in users who have pressed "save" more
than 50 times. This would be close to magic (like an adventure game,
where you gain "experience points" from slaying dragons), but still
require very little explanation.

My granddad's old TV set had a knob at the back for horizontal sync.
That was a major feature to him, because the picture would just roll
around until he adjusted the sync.  However, instead of teaching
everybody about this feature, the TV manufacturers were able to hide
the function in a self-regulating circuit inside all new TV sets.  In
the same way, if the wikitech-l gang is able to keep the website
running nice and smooth, the wikipedia-l crowd can focus on issues
about how to write articles, and nobody will ask for tech updates.

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


from_name: Brion Vibber

subject: [Wikitech-l] Re: Link update script

body:
Tisdag, 14 Maj 2002, Jimmy Wales skribte:
> I've switched to the CVS version and corrected a few bugs.  But the
> site is sluggish again, and load is creeping upward.  I wonder if
> getOtherNamespaces is the issue again?

It could be the temporary workaround I had put in place for the
certain-common-words-cause-searches-to-fail-completely bug, which falls
back to a slow but reliable LIKE search when the MATCH fails.

I've disabled the fallback in CVS; update special_dosearch.php and see if
performance (if not search results) improves.

The search system needs some major work, alas...

-- brion vibber (brion @ pobox.com)


subject: [Wikipedia-l] Bugs? or Features?

body:
> > The second is my concern about what the Cunctator calls
> > "featuritis". I think that any *major* new software features should
> > be discussed in the general Wikipedia assembly.
>
> I agree with this.  But there is a problem to determine in advance
> what is a "major" and what is a minor feature.  Obviously, a feature
> or implementation detail that makes the system grind to a halt is a
> major issue.  This time, the "talk" links turned out to be one of the
> major issues because of the inefficient way they were implemented,
> and temporarily removing the talk links made the website run faster.
> (Let's hope the improved implementation is up and running soon.)

It is, and working great (except it seems to be broken at the moment...)

> I think the autowikification button is a most interesting feature, but
> perhaps it should be disabled by default, with each user having the
> option to enable it from the personal preferences page.  This way, it
> wouldn't confuse the beginners.  It could also be turned on
> automatically for all logged in users who have pressed "save" more
> than 50 times. This would be close to magic (like an adventure game,
> where you gain "experience points" from slaying dragons), but still
> require very little explanation.

Already done in the brand-new version (the option method, not the
may-the-force-be-with-you thing;)

> My granddad's old TV set had a knob at the back for horizontal sync.
> That was a major feature to him, because the picture would just roll
> around until he adjusted the sync.  However, instead of teaching
> everybody about this feature, the TV manufacturers were able to hide
> the function in a self-regulating circuit inside all new TV sets.  In
> the same way, if the wikitech-l gang is able to keep the website
> running nice and smooth, the wikipedia-l crowd can focus on issues
> about how to write articles, and nobody will ask for tech updates.

There's definitely room for optimizing the update process (and we need
constant updates, if only to make the 'pedia fast again, or install
bugfixes).

Magnus


subject: [Wikitech-l] Fixes to watchlist

body:
I've just checked in a rewrite of the watchlist function; instead of
doing a zillion separate database queries, it now does just two: one to
get the watchlist from the user table, the other to grab the info needed
to fill out the table/list.

Please, check over it and make sure it looks okay before we let yet
another horribly misshappen watchlist version wander over the land!

As a hack to get non-blank, non-talk namespaces listed until we have a
separate field that contains the title but *not* the namespace, all
valid namespaces get prepended to the titles-to-be-watched in the list
given to the database to check against.

I've also fixed the bug whereby clicking "add to my watchlist" on a page
with a non-blank namespace results in a redirect to the equivalent title
in the blank namespace.

(It would also be possible, if preferred, to limit the namespace-laxity
in the watchlist to just "X" and "X talk" pairs. Personally, I've seen
enough pages get moved from blank to "wikipedia:" that I'd prefer to
keep complete namespace agnosticism in my watchlist: the less I have to
add and subtract things to keep my watchlist relevent, the better.)

-- brion vibber (brion @ pobox.com)


from_name: Axel Boldt

subject: [Wikitech-l] "Really big process bug"?

body:
> Anyway, currently the 'pedia seems to run fast and stable.

I have been seeing several fast accesses and every once in a while an
extremely slow one. It could be that one of the special functions (or
search?) behaves badly.

> Is the "really big process" bug back??

What is the "really big process" bug?

I have been wondering before whether it's possible to have memory
leaks in php scripts: is everything automatically released once the
page is served? Somebody somewhere could be eating memory, and that
would slow everything. If memory is indeed the bottleneck, then
caching might not be a good idea, since we in effect transport twice
as much data between database and php script, and I would expect that
the data is buffered in memory on both ends.

Some top/ps outputs would really help.

Axel


subject: [Wikitech-l] "Really big process bug"?

in_reply_to: <200205150438.GAA05584@linde.uni-paderborn.de>; from axel@uni-paderborn.de on Wed, May 15, 2002 at 06:38:51AM +0200

body:
A review of the uptime results suggests that we aren't quite as fast as we were in
the previous version (when I essentially disabled getOtherNamespaces), but we're
doing o.k. now.  I think we still have some work to do.

Axel Boldt wrote:
> I have been wondering before whether it's possible to have memory
> leaks in php scripts: is everything automatically released once the
> page is served?

We are running apc php caching:
http://apc.communityconnect.com/

In practice this means that all the .php scripts get precompiled and
saved to disk as .php_apc.  They are precompiled and cached so that
"the overhead of parsing and compiling can almost completely be
eliminated".  This might also introduce a new place for memory leaks
to happen.

However, the machine does not appear to be swapping most of the time.
I suspect that when the load really spikes up (review
http://www.wikipedia.com/uptime.txt for a log of recent uptime
output), we are swapping.

> Somebody somewhere could be eating memory, and that
> would slow everything. If memory is indeed the bottleneck, then
> caching might not be a good idea, since we in effect transport twice
> as much data between database and php script, and I would expect that
> the data is buffered in memory on both ends.

Possibly.

> Some top/ps outputs would really help.

This is a typical looking 'top'.

  8:54am  up 42 days, 23:24,  1 user,  load average: 6.05, 4.93, 5.15
172 processes: 163 sleeping, 8 running, 1 zombie, 0 stopped
CPU states: 73.3% user, 26.2% system,  0.0% nice,  0.3% idle
Mem:  1029824K av,  934008K used,   95816K free,       0K shrd,   49900K buff
Swap:  506008K av,   64200K used,  441808K free                  517092K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT  LIB %CPU %MEM   TIME COMMAND
 7887 root      20   0 51632  45M 28972 R       0 27.5  4.4   0:04 mysqld
12467 nobody    19   0  8516 7856  5076 R       0  4.3  0.7   0:03 httpd
 8154 nobody    20   0  8868 8104  5640 R       0  3.8  0.7   0:04 httpd
12468 root      12   0 51632  45M 28972 S       0  3.8  4.4   0:01 mysqld
17146 nobody    19   0  7724 7068  4128 R       0  3.8  0.6   0:00 httpd
17141 jwales    19   0   936  936   672 R       0  3.3  0.0   0:00 top
16989 nobody    14   0  8280 7712  4900 R       0  3.0  0.7   0:01 httpd
17181 nobody    14   0  3424 3424   956 R       0  3.0  0.3   0:00 wiki.cgi
17136 root      10   0 51632  45M 28972 S       0  2.3  4.4   0:00 mysqld
 7879 nobody     9   0  8772 8116  4728 S       0  2.1  0.7   0:03 httpd
17182 nobody    14   0  1896 1896   896 R       0  2.0  0.1   0:00 q.cgi
 7723 nobody    17   0  7820 7096  4676 S       0  1.8  0.6   0:03 httpd
17148 nobody    11   0  7884 7256  4452 S       0  1.8  0.7   0:00 httpd
17170 nobody     9   0  7108 6480  4096 S       0  1.6  0.6   0:00 httpd
17158 nobody     9   0  7780 7152  4452 S       0  1.5  0.6   0:00 httpd
17165 nobody    10   0  7000 6372  4064 S       0  1.5  0.6   0:00 httpd
 9949 nobody    11   0     0    0     0 Z       0  1.3  0.0   0:00 httpd <defunct>
17144 nobody    10   0  7876 7248  4452 S       0  1.3  0.7   0:00 httpd
17134 nobody    10   0  7560 6936  4260 S       0  1.1  0.6   0:00 httpd
17058 nobody    10   0  7660 7048  4384 S       0  1.0  0.6   0:00 httpd
17067 nobody    10   0  8032 7420  4288 S       0  1.0  0.7   0:00 httpd
17143 nobody    12   0  7020 6432  4120 S       0  1.0  0.6   0:00 httpd
17159 nobody    11   0  7080 6452  4132 S       0  1.0  0.6   0:00 httpd
17166 nobody     9   0  7068 6440  4128 S       0  1.0  0.6   0:00 httpd
17169 nobody     9   0  7696 7084  4468 S       0  1.0  0.6   0:00 httpd
16958 nobody    10   0  8024 7444  4632 S       0  0.8  0.7   0:03 httpd
16978 nobody    11   0  7976 7352  4696 S       0  0.8  0.7   0:01 httpd
16983 nobody    10   0  7960 7348  4696 S       0  0.8  0.7   0:02 httpd
17084 root       9   0 51632  45M 28972 S       0  0.8  4.4   0:05 mysqld
17167 nobody     9   0  7112 6484  4148 S       0  0.8  0.6   0:00 httpd
17168 nobody     9   0  6988 6360  4068 S       0  0.8  0.6   0:00 httpd
17083 nobody    10   0  7336 6724  4228 S       0  0.6  0.6   0:00 httpd
 7730 nobody    10   0  8632 7884  5444 S       0  0.5  0.7   0:04 httpd
 7850 nobody    10   0  8556 7836  5444 S       0  0.5  0.7   0:03 httpd
16985 nobody    10   0  7236 6648  4240 S       0  0.5  0.6   0:00 httpd
17052 nobody     9   0  8144 7572  4668 S       0  0.5  0.7   0:00 httpd
17063 root       9   0 51632  45M 28972 S       0  0.5  4.4   0:05 mysqld
17145 nobody     9   0  7028 6404  4072 S       0  0.5  0.6   0:00 httpd
 7851 root      10   0 51632  45M 28972 S       0  0.3  4.4   0:00 mysqld


subject: [Wikitech-l] Re: Link update script

body:
On Tue, May 14, 2002 at 07:06:26PM -0700, Brion Vibber wrote:
> Tisdag, 14 Maj 2002, Jimmy Wales skribte:
> > I've switched to the CVS version and corrected a few bugs.  But the
> > site is sluggish again, and load is creeping upward.  I wonder if
> > getOtherNamespaces is the issue again?
>
> It could be the temporary workaround I had put in place for the
> certain-common-words-cause-searches-to-fail-completely bug, which falls
> back to a slow but reliable LIKE search when the MATCH fails.

Hi, remember me? :-) Has it been considered to recompile MySQL? That way you
could also remove the annoying 4-character limit for search words. See:

http://www.mysql.com/doc/F/u/Fulltext_Fine-tuning.html

Even better would be a switch to MySQL4.0, in which case the extra code for
boolean searches can be omitted.

-- Jan Hidders


from_name: Pierre Abbat

subject: [Wikitech-l] response times

body:
On Wednesday 15 May 2002 09:50, Lars Aronsson wrote:
> Response times above some limit (say, 30, 60 or 120 seconds) can be
> defined as absurdly long, because the user will have left for other
> websites and is no longer waiting for the response.  Instead of
> spending more system resources (CPU cycles and allocated memory) on
> these requests, it would be better to set a hard time out (in PHP or
> Apache) and return an error message that says "sorry for the delay".
> This would free up system resources that can be better used to serve
> other requests.
>
> In <http://www.php.net/manual/en/function.set-time-limit.php>, the PHP
> function set_time_limit() is said to have a default of 30 seconds,
> unless the configuration file has defined max_execution_time.  Will
> calling this function set the time limit for the current request only,
> or set a permanent value for the server?  What happens when PHP
> execution times out?  Is the connection to the client abruptly closed?
> Or is an error message returned?  Does an error message appear in the
> log file?  I haven't seen any timeouts of this kind.

PHP aborts the script, outputs an error message saying it timed out, and
closes the connection.

phma


subject: [Wikitech-l] response times

body:
I'm looking again at my log files, comparing three days:
 - Thursday May 9,  before Jimmy's change (old talk links)
 - Friday   May 10, after Jimmy's change (no talk links)
 - Tuesday  May 14, with the new version (new talk links)

The overall performance was best on Friday.  It is now getting worse
again, albeit not as bad as Thursday.  The new version is a real
improvement over the old talk links, but we aren't quite done yet.

The number of OK responses (HTTP status code 200) which take absurdly
long (longer than 60 seconds) is still very high (3-30 %), with the
Main Page of the English Wikipedia being the main exception (0 %).

Response times above some limit (say, 30, 60 or 120 seconds) can be
defined as absurdly long, because the user will have left for other
websites and is no longer waiting for the response.  Instead of
spending more system resources (CPU cycles and allocated memory) on
these requests, it would be better to set a hard time out (in PHP or
Apache) and return an error message that says "sorry for the delay".
This would free up system resources that can be better used to serve
other requests.

In <http://www.php.net/manual/en/function.set-time-limit.php>, the PHP
function set_time_limit() is said to have a default of 30 seconds,
unless the configuration file has defined max_execution_time.  Will
calling this function set the time limit for the current request only,
or set a permanent value for the server?  What happens when PHP
execution times out?  Is the connection to the client abruptly closed?
Or is an error message returned?  Does an error message appear in the
log file?  I haven't seen any timeouts of this kind.

However, even if a PHP execution timeout is set, the limit will not
include time that the request spent waiting to start execution.  This
wait could happen in the UNIX socket listen backlog, waiting for the
connection to be accepted, or inside Apache, waiting for a child
process to become available.  Increasing the value of a parameter like
ListenBacklog (in Apache httpd.conf) is not necessarily a solution,
because this will only keep more requests in queue, increasing overall
response time.  Instead, the problem should be fixed at the exit end
of the queue.  The key to better performance is keeping the server
fast and queues short, getting things done.

Here are some pages on Apache performance issues:

 - Hints on Running a High-Performance Web Server,
    http://httpd.apache.org/docs/misc/perf.html
 - Apache Performance Notes,
    http://httpd.apache.org/docs/misc/perf-tuning.html
 - Professional Apache, chapter 8,
    http://www.devshed.com/Talk/Books/ProApache/
 - Tuning Your Apache Web Server,
    http://dcb.sun.com/practices/howtos/tuning_apache.jsp
 - Linux HTTP Benchmarking HOWTO,
    http://www.xenoclast.org/doc/benchmark/HTTP-benchmarking-HOWTO/

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikitech-l] the bug bug

body:
On mer, 2002-05-15 at 16:58, Lars Aronsson wrote:
>
> >From http://meta.wikipedia.com/wiki.phtml, the link for reporting bugs
> goes to a page gets hooked up in some redirect loop,
> http://wikipedia.sourceforge.net/fpw/wiki.phtml?title=wikipedia%3ABugs

I've just replaced these broken links with links to the bug report and
feature request pages on www.wikipedia.com.

> The displayed text says:
>
>   You are being redirected to wikipedia.com
>   Warning: Cannot add header information - headers already sent by
>   (output started at
>   /home/groups/w/wi/wikipedia/htdocs/fpw/wiki.phtml:5) in
>   /home/groups/w/wi/wikipedia/htdocs/fpw/wiki.phtml on line 81

Same without specifying the title. Somebody put this at the top of the
file:

<META HTTP-EQUIV=Refresh CONTENT=\"0; URL=http://wikipedia.com">
You are being redirected to wikipedia.com

<? ... the PHP code for the wiki....

(The backslash is probably breaking it such that instead of refreshing
to the main wikipedia site, it refreshes to itself!) I'm guessing
somebody was Magnus, as the file is marked as owned by "magnus_m". I
don't have write access to the file or the directory, so I can't fix it.

-- brion vibber (brion @ pobox.com)


from_name: Daniel Mayer

subject: [Wikipedia-l] What is the "Edit user settings" feature?

body:
What is the "Edit user settings" feature? I hadn't heard anything about it
before, but apparently now any sysop can edit the privileges of any other
user -- including other sysops. A privilege formerly only granted to Jimbo.

This could be a useful feature if the sysops knew what each level does and
there was some type of framework for sysops to follow (thinking of checks and
balances here).

Also, what does it mean to be a:

 System operator
(I assume this means a sysop since this is checked for the registered sysops
-- does this also mean that any sysop can promote any user to sysop status,
or demote any sysop to a common user? If so, then a feature like this
probably should have been discussed here -- on the policy mailing list --
before it was implemented.)

 System developer
(Does this refer to one of the wikiware gods (Like Brion or Magnus)? Probably
should have been discussed before implementation too)

 Trusted hand
(What privileges does this grant a user beyond the defaults?)

-- maveric


body:
On mer, 2002-05-15 at 22:28, Daniel Mayer wrote:
> What is the "Edit user settings" feature? I hadn't heard anything about it
> before, but apparently now any sysop can edit the privileges of any other
> user -- including other sysops. A privilege formerly only granted to Jimbo.
>
> This could be a useful feature if the sysops knew what each level does and
> there was some type of framework for sysops to follow (thinking of checks and
> balances here).

Bah! Always asking for documentation, you people! :)

> Also, what does it mean to be a:
>
>  System operator
> (I assume this means a sysop since this is checked for the registered sysops

Yes.

> -- does this also mean that any sysop can promote any user to sysop status,
> or demote any sysop to a common user? If so, then a feature like this
> probably should have been discussed here -- on the policy mailing list --
> before it was implemented.)

It does seem a potentially controversial point... I think Magnus stuck
it in specifically so he could grant himself developer priviledges and
fix a problem with the database that was causing trouble (see below).
I'm not sure whether this is intended as a permanent feature or if it's
a entirely good idea.

In any case, having a convenient interface page for setting user
priveleges is probably not a bad thing; if it's preferred that only the
guy who owns the server gets to wield it, that's easily fixed.

>  System developer
> (Does this refer to one of the wikiware gods (Like Brion or Magnus)? Probably
> should have been discussed before implementation too)

This grants the ability to run SQL queries directly on the database that
could affect the data.

This is occasionally useful to fix something: for instance, Magnus was
recently able to recreate the "unlinked" link table which had been
corrupted and was no longer accessible -- thus breaking 'Most Wanted'
and several other things -- which could then be fully rebuilt by a
script run by Jimmy. But it's easy to, say, delete the entire database
if you don't know what you're doing; hence the restriction to known
developers.

I'm quite sure that was discussed on wikipedia-l a month or so ago...
Yeah, here it is:
http://www.nupedia.com/pipermail/wikipedia-l/2002-April/001707.html

>  Trusted hand
> (What privileges does this grant a user beyond the defaults?)

As far as I know there is nothing in the code that currently checks for
this, and I'm not sure what it's intended to mean. Magnus?

-- brion vibber (brion @ pobox.com)


from_name: Jens Frank

subject: [Wikitech-l] Link to non-existing pages

body:
Hello,

I just clicked a red link (indicating that there currently is no article yet).
The link triggered had a &action=edit -suffix and thus invoked the new page
in edit-mode. The only problem was that in the meantime, just after the
page was put to the pagecache, the article has been written. So I ended with
the edit-mode of a quite fine article. While this is probably OK for a
wikipedia contributor I think this might be confusing for a "user".

I would like to ask whether it might make sense to change the URL invoked
by red links to not end in "&action=edit". Then the automatic edit-mode for
non-existent articles is triggered in case of a non-existing article and
for articles written in the meantime the link would just show the article
expected.

        Best Regards,

		jens frank
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 232 bytes
Desc: not available
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20020516/af6660ad/attachment.pgp>


subject: [Wikitech-l] the bug bug


subject: [Wikipedia-l] AW: Re: [Wikipedia-l] What is the "Edit user settings" feature?

body:
>>  Trusted hand
>> (What privileges does this grant a user beyond the defaults?)
>
>As far as I know there is nothing in the code that currently checks for
>this, and I'm not sure what it's intended to mean. Magnus?

Originally, this was intended to be something between "normal" user and
sysop, someone who can do things like deleting pages, but without, say,
SQL access.
As this was solved by making eveyone who wants to sysop and restricting
sysop SQL access, it is currently not used at all.
If (one day, in a galaxy far, far away;) we can decide on implementing a
"stable" namespace for keeping good articles as a stable copy, the "trusted"
status might come in handy as the "key" for copying articles to that namespace
(if we let everybody do that, it won't be a stable namespace anymore...).

Magnus

________________________________________
Zeitschriftenabos online bestellen - jetzt neu im Infoboten! http://www.epost.de


in_reply_to: <1021529641.3311.0.camel@rdaneel.moisty.org>; from brion@pobox.com on Wed, May 15, 2002 at 11:13:59PM -0700

body:
Brion L. VIBBER wrote:
> It does seem a potentially controversial point... I think Magnus stuck
> it in specifically so he could grant himself developer priviledges and
> fix a problem with the database that was causing trouble (see below).
> I'm not sure whether this is intended as a permanent feature or if it's
> a entirely good idea.

Wikipedia is based on trust.  If we can't trust sysops, we're in big
trouble anyway. :-)

I was opposed to the general ability to let people (even sysops)
execute arbitrary SQL commands via the web, not because of trust per
se, but because of the high risk of an accident.  It makes me
personally very nervous when I have to type in raw SQL, and so I
imagine it's not a good thing for anyone to be doing, except for
developers and then only for specific and emergency-ish reasons.

Sysop status should be available to just about anyone who asks, and
the privileges of the sysop should be so minimal as to justify this
policy.

The main thing we want to prevent is the notion of an "elite" who
directs the wikipedia and gets to say who participates in driving the
content *through the use of special powers*.  The rare exceptions
should be agonized over, and should be restricted to people who just
refuse to even try to get along with us.

I think that all sysops and especially developers should work really really
hard to make sure to NEVER use any special powers in a content argument.  It's
just not fair, and not in the spirit of wikipedia consensus.

For example, I'd be pretty pissed at a developer who made a raw SQL command
to delete another user, or to delete an article that was in contention.  Yuck.

--Jimbo


subject: [Wikitech-l] New server

body:
Good news!  I'm buying Wikipedia a new and much more powerful server.

Bomis.com needs a new server in our web cluster.  As it turns out,
ross.bomis.com (the current wikipedia server) is physically identical
to our existing servers in our web cluster (because I bought ross at
the same time), and is therefore ideal to include in the web cluster
without complicating my load balancing over there.

This means that the newly purchased server, which should be here in a
couple of weeks, will be set up exclusively for wikipedia.  I will be
moving personal websites of mine (like jimmywales.com and
kirawales.com) and experimental websites of mine OFF this machine, but
not to the new machine.

The new machine will be setup with different root password, and no
real connection to the rest of my network.  This means that I can give
developers Unix shell accounts on the machine, and even the root
password in some cases.

The new machine will be a dual Athlon with 2 gig of ram and a much
bigger SCSI harddrive than what we have now.  The performance of the
machine should be a lot better than the existing single PIII with 1
gig of ram.

This will all come to pass in the next couple of weeks.  This will remove
me as a key bottleneck in the development/release cycle.

--Jimbo


subject: [Wikitech-l] New server specs

body:
Tyan S2462 motherboard with
Dual AMD Athlon MP 1800+/266FSB
2 GB of ECC Reg. DDR-266/PC2100 ram (4 - 512 Mb pieces)
IBM 36 GB Ultra 160 / 10 K RPM
Dual onboard 3Com 10/100 adapters

For the install, we'll go with MySQL 4.0 unless someone gives us a
good reason not to do it -- this will enable us to cut the minimum
keyword length in the search engine down to less than 4.

We'll probably install the latest stable Kernel at the time we get the machine,
2.4.18 right now.

I'm tempted to go with Apache 2.0.36, but I've haven't used Apache 2.x
in production, so maybe we won't start with that.  Opinions welcome.

This should be a pretty sweet setup.

The new server will host Wikipedia and Nupedia ONLY.

--Jimbo


subject: [Wikipedia-l] speed at last

body:
Since 21:00 GMT (2 pm PDT) Wednesday, Wikipedia is really fast:

 - Static images load in less than one second.
 - Wiki pages load in 5 seconds (often less than 2).
 - Recent Changes loads in 5 seconds in 86% of my samples.
 - No responses ever take longer than 15 seconds.

I don't know what caused this, but I love it.
Now I can start to promote Wikipedia more actively.

Note that I'm in Sweden and the server is in San Diego, and the single
roundtrip time (ping) alone accounts for 0.25 seconds.  Theoretically
(speed of light, size of Earth), this could be reduced to 0.10
seconds.  Starting "wget", opening a socket (one roundtrip), sending a
request, and receiving a response (second roundtrip) containing the
Wikipedia logotype, all in 0.70 seconds is pretty amazing.  It never
took longer than 0.91 seconds in the last 14 hours.

--
  Lars Aronsson <lars at aronsson.se>
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikitech-l] speed at last


subject: [Wikipedia-l] speed at last

in_reply_to: Karen AKA Kajikit's message of "Thu, 16 May 2002 21:44:16 +1000"

body:
Karen AKA Kajikit <kaji at labyrinth.net.au> writes:

> Lars Aronsson wrote:
>> I don't know what caused this, but I love it.
>> Now I can start to promote Wikipedia more actively.
>
> It's good isn't it!

I'm glad someone's getting joy.  I'm timing out 4 times out of 5...
--
Gareth Owen
"Wikipedia does rock.  By the count on the "brilliant prose" page, there
 are 14 not-bad articles so far" -- Larry Sanger (12 Jan 2001)


subject: [Wikipedia-l] Re: What is the "Edit user settings" feature?

body:
It seems plain to me that Jimbo and Magus should have developer status
and that the "Edit user settings" feature should be restricted to
developers.

Axel


subject: [Wikitech-l] AW: Re: (from [Wikipedia-l]) speed at last

body:
-- Original Nachricht --

>Gareth Owen wrote:
>>
>> Karen AKA Kajikit <kaji at labyrinth.net.au> writes:
>>
>> > Lars Aronsson wrote:
>> >> I don't know what caused this, but I love it.
>> >> Now I can start to promote Wikipedia more actively.
>> >
>> > It's good isn't it!
>>
>> I'm glad someone's getting joy.  I'm timing out 4 times out of 5...
>
>I said it WAS good... alas it seems to have ground to a halt again. I
>went from rapid access to all pages to no access at all. :(

I had similar experiences today - either really fast, or not at all. Strangely,
I had a fast page, a link from there (in a new window) timed out, and another
one (started some seconds later) was fast again.
The only explanation I can come up with is that some requests go to a slow
thread/process, which would match with the "really big process" bug we encountered
some time ago. Jimbos top-dump showed a zombie process, which was probably
it.

Is there a way we can find out if the bad process starts only on certain
pages (e.g., the Most Wanted), or if it just happens at random? That would
help enormously in error search.

Magnus

________________________________________
Zeitschriftenabos online bestellen - jetzt neu im Infoboten! http://www.epost.de


subject: [Wikitech-l] New server

body:
> Good news!  I'm buying Wikipedia a new and much more powerful server.

See, Brion, it finally worked out - we just had to keep the script slow
enough ;)

> The new machine will be setup with different root password, and no
> real connection to the rest of my network.  This means that I can give
> developers Unix shell accounts on the machine, and even the root
> password in some cases.

Cool!

> The new machine will be a dual Athlon with 2 gig of ram and a much
> bigger SCSI harddrive than what we have now.  The performance of the
> machine should be a lot better than the existing single PIII with 1
> gig of ram.

[[drooling]]

Magnus


subject: [Wikipedia-l] speed at last

body:
Lars Aronsson wrote:
>
> Since 21:00 GMT (2 pm PDT) Wednesday, Wikipedia is really fast:
>
>  - Static images load in less than one second.
>  - Wiki pages load in 5 seconds (often less than 2).
>  - Recent Changes loads in 5 seconds in 86% of my samples.
>  - No responses ever take longer than 15 seconds.
>
> I don't know what caused this, but I love it.
> Now I can start to promote Wikipedia more actively.

It's good isn't it! That said, my wikipedia screen has just timed out
twice in a row... I'll try restarting Netscape and see if that helps.
Sometimes it's Netscape going screwy and not the site I'm trying to
access.

--

Karen AKA Kajikit

Come and visit my part of the web:
Kajikit's Corner: http://Kajikit.netfirms.com/
Aussie Support Mailing List: http://groups.yahoo.com/group/AussieSupport
Allergyfree Eating Recipe Swap:
http://groups.yahoo.com/group/Allergyfree_Eating

Love and huggles to all!


subject: [Wikipedia-l] speed at last

body:
Gareth Owen wrote:
>
> Karen AKA Kajikit <kaji at labyrinth.net.au> writes:
>
> > Lars Aronsson wrote:
> >> I don't know what caused this, but I love it.
> >> Now I can start to promote Wikipedia more actively.
> >
> > It's good isn't it!
>
> I'm glad someone's getting joy.  I'm timing out 4 times out of 5...

I said it WAS good... alas it seems to have ground to a halt again. I
went from rapid access to all pages to no access at all. :(

--

Karen AKA Kajikit

Come and visit my part of the web:
Kajikit's Corner: http://Kajikit.netfirms.com/
Aussie Support Mailing List: http://groups.yahoo.com/group/AussieSupport
Allergyfree Eating Recipe Swap:
http://groups.yahoo.com/group/Allergyfree_Eating

Love and huggles to all!


subject: [Wikitech-l] Cause for slowdowns: spiders?

body:
Right now, I'm seeing nice and fast responses, except every
once in a while everything slows to a halt. If that's due to our
script, then there must be some really bad, really rare special
function somewhere. I doubt that.

Maybe the slowdowns are due to spiders that hit our site and request
several pages at once, in parallel, like many of these multithreaded
programs do. I read somewhere that everything2.com for this very
reason has disallowed spiders completely and doesn't even allow Google
to index their site anymore.

Maybe we should search the server logs for several rapid requests from
the same IP, and try to correlate those to load averages?

Axel


body:
On Thursday 16 May 2002 19:43, Axel Boldt wrote:
> Right now, I'm seeing nice and fast responses, except every
> once in a while everything slows to a halt. If that's due to our
> script, then there must be some really bad, really rare special
> function somewhere. I doubt that.

How long do slowdowns last? I've been trying for at least half an hour to
submit a change to a page. I get nothing but timeouts. Sometimes I can't even
get a connect.

phma


subject: [Wikipedia-l] Redirects broken?

body:
On ven, 2002-05-17 at 09:55, Gareth Owen wrote:
> http://www.wikipedia.com/wiki/R.E.M.
> and others.  You can't even edit these, as they're marked "Protected"?
>
> WTHeck?

You may have noticed that the title of the article redirected to was not
"REM" but "#REDIRECT [[REM]]". This being a highly invalid article
title, you're not allowed to edit it!

A fix for the problem (which I think is triggered by a funky newline
character) is already in CVS waiting to be put online to see if it
helps. (in wikiPage.php)

In the meantime; if someone else created the redirect, try going back to
edit it (via the "redirected from X" link). Delete the whole thing and
type it in again, that should fix it. (Unless your browser also puts in
the funky newlines.)

Also make sure there is always a space between "#REDIRECT" and "[[bla
bla bla]]" (this causes a similar problem; also fixed in CVS).

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] response times

body:
On ven, 2002-05-17 at 08:35, Lars Aronsson wrote:
> (Is the Esperanto Wikipedia running Perl again?  Didn't it already use
> the new PHP software?)

Not "again" but "still". There's a test server (test-eo.wikipedia.com),
but something didn't go right in the database conversion (so we can't
seriously test anything on it!) and nobody at Bomis has touched it
since. :(

Note that I have not been able to reproduce the conversion error (the
character set conversion which should have changed "cx" etc into proper
UTF-8 characters didn't happen); it all works fine and dandy on my
computer. We'd be ever so grateful if Jason or Jimmy could try it again
sometime...

-- brion vibber (brion @ pobox.com)


body:
On Friday 17 May 2002 08:25, Pierre Abbat wrote:
> How long do slowdowns last? I've been trying for at least half an hour to
> submit a change to a page. I get nothing but timeouts. Sometimes I can't
> even get a connect.

Okay, I got a time on this. It happened between 4:16 and 5:48 server time.
There were no unsuccessful searches between that time.

phma


in_reply_to: Axel Boldt's message of "Fri, 17 May 2002 01:43:39 +0200 (MET DST)"

body:
Axel Boldt <axel at uni-paderborn.de> writes:

> Right now, I'm seeing nice and fast responses, except every
> once in a while everything slows to a halt. If that's due to our
> script, then there must be some really bad, really rare special
> function somewhere. I doubt that.

Its a punt, and I've no real evidence ... but ...
when the 'pedia comes back after some slowtime, there often seems to be an

Wikipedia:Unsuccessful searches (date); [Unsuccessful search for foobar]

at the top of RecentChanges

I might be imagining it, I certainly can't reproduce it at will.
--
Gareth Owen


subject: [Wikitech-l] response times

in_reply_to: <1021655224.4456.2.camel@rdaneel.moisty.org>; from brion@pobox.com on Fri, May 17, 2002 at 10:07:03AM -0700

body:
I'll look at it today...

Jason

Brion L. VIBBER wrote:

> We'd be ever so grateful if Jason or Jimmy could try it again
> sometime...

--
"Jason C. Richey" <jasonr at bomis.com>


subject: [Wikitech-l] Testbed for wiki software

body:
On ven, 2002-05-17 at 11:21, Lee Daniel Crocker wrote:
> I contacted Magnus and Brion earlier this week to let them know
> that I had set up accounts for them on my server which they can
> use to test wiki software changes.

Uhh, you did? Hmm, was that Tuesday? Fetchmail burped and sent a couple
messages into the ether that day, could have been one of them.

-- brion vibber (brion @ pobox.com)


subject: [Wikipedia-l] Creative Commons

body:
http://creativecommons.org/

Laurence Lessig's Creative Commons project could be important to the
Wikipedia and Nupedia projects.  Have a look.

Larry


from_name: Lee Daniel Crocker

subject: [Wikitech-l] Testbed for wiki software

body:
I contacted Magnus and Brion earlier this week to let them know
that I had set up accounts for them on my server which they can
use to test wiki software changes. It has a fairly large database
dump installed, and plenty of space to experiment, and all the
softare needed to develop and test.

Brion hasn't contacted me, but Magnus tells me the setup is
basically OK, so I'll make the same offer to other developers:
my server is available as a testbed; please take advantage of
it.  It would be GREAT if some enterprising developer who didn't
want to muck with the wikipedia software itself would take on
writing some testing scripts--just some code that pounds on a
running wiki for a while and exercises its functions.  We could
then run that against the running wiki on my site before
installing any software change. It's no big deal if my site
goes down now and then.

I also gave Magnus and Brion access to the new reorganized
software base I've been working on, and I may open that up to
a wider audience soon.  It's doing the basic functions of
displaying pages with links and all the wiki formatting, and
handles user login and prefs.  I still need to move the code
for page editing/history/etc., many special pages, and some
more internationalization over.  It's going remarkably fast,
but it will still take another week or two.

The new software should not only be easier to understand,
faster, and more scaleable, but I've included plenty of hooks
for debugging and logging features that will make it easier
to diagnose problems.  Just keeping everyone up to date.

--
Lee Daniel Crocker <lee at piclab.com> <http://www.piclab.com/lee/>
"All inventions or works of authorship original to me, herein and past,
are placed irrevocably in the public domain, and may be used or modified
for any purpose, without permission, attribution, or notification."--LDC


subject: [Wikipedia-l] robots and spiders

body:
On ven, 2002-05-17 at 13:04, General Wesc (LKBM) wrote:
> I have to wonder though...if a spider goes to Recent Changes and then to
> "Last 5000 changes" (and last 90 days, and last 30 days, and last 2500
> changes, and last 1000 changes, and every such combination) it seems to
> me the server load could get pretty high.

Does anyone _really_ ever want to look at the last 5000 changes?

As for the higher day values, they'll be useful for the less active
other-language wikipedias once they're converted, though it might be
good to have "intelligent" scaling on that bar. 90 days worth of changes
on the English 'pedia would go well over ever the 5000 changes limit...
(At the moment, 5000 only gets us back to about April 27.)

> Perhaps talk pages should be
> spidered, but not recent changes or the history (diff/changes).

A robots.txt could easily be set up to disallow
/wiki/special%3ARecentChanges (and various case variations). That only
stops _nice_ spiders, of course.

History links would need to be changed to be sufficiently
distinguishable, for instance using /wiki.phtml?title=Foo&action=history
etc; then ban /wiki.phtml.

-- brion vibber (brion @ pobox.com)


from_name: wojtek pobratyn

subject: [Wikipedia-l] Re: [Wikitech-l] Cause for slowdowns: spiders?

body:
----- Original Message -----
From: "Axel Boldt" <axel at uni-paderborn.de>
To: <wikitech-l at nupedia.com>
Sent: Friday, May 17, 2002 1:43 AM
Subject: [Wikitech-l] Cause for slowdowns: spiders?

| Right now, I'm seeing nice and fast responses, except every
| once in a while everything slows to a halt. If that's due to our
| script, then there must be some really bad, really rare special
| function somewhere. I doubt that.
|
| Maybe the slowdowns are due to spiders that hit our site and request
| several pages at once, in parallel, like many of these multithreaded
| programs do. I read somewhere that everything2.com for this very
| reason has disallowed spiders completely and doesn't even allow Google
| to index their site anymore.

Robots can be useful as probably Google contributes a lot
to bringing new visitors  to the site. Being a community this surely is
a usefull thing. An issue worth considering is banning robots from
special pages, Recent Changes, Talk, User, etc. by an appropriate META tag.
IMHO,  robots and spiders should only be allowed to go over the articles.

<snip>

regards,
[[user:WojPob]]


from_name: Chuck Smith

subject: [Wikipedia-l] robots and spiders

body:
> Robots can be useful as probably Google contributes
> a lot
> to bringing new visitors  to the site. Being a
> community this surely is
> a usefull thing. An issue worth considering is
> banning robots from
> special pages, Recent Changes, Talk, User, etc. by
> an appropriate META tag.
> IMHO,  robots and spiders should only be allowed to
> go over the articles.
>
> <snip>
>
> regards,
> [[user:WojPob]]

I COMPLETELY disagree with this.  Let the robots crawl
everything.  It's better that someone finds one of our
Talk or User pages and cruises on over to our main
site than to simply find a completely website!

Chuck

=====
Come to my homepage!  Venu al mia hejmpagxo!
http://amuzulo.babil.komputilo.org/
====
Venu al la senpaga, libera enciklopedio
esperanta reta! http://eo.wikipedia.com/

_________________________________________________________
Do You Yahoo!?
Informaci?n de Estados Unidos y Am?rica Latina, en Yahoo! Noticias.
Vis?tanos en http://noticias.espanol.yahoo.com


subject: [Wikitech-l] response times

body:
On ven, 2002-05-17 at 11:06, Jason Richey wrote:
> I'll look at it today...

Thanks!

-- brion vibber (brion @ pobox.com)

> Jason
>
> Brion L. VIBBER wrote:
>
> > We'd be ever so grateful if Jason or Jimmy could try it again
> > sometime...
>
> --
> "Jason C. Richey" <jasonr at bomis.com>


from_name: The Cunctator

subject: [Wikipedia-l] robots and spiders

body:
On 5/17/02 3:26 PM, "Chuck Smith" <msochuck at yahoo.com> wrote:

>> Robots can be useful as probably Google contributes
>> a lot
>> to bringing new visitors  to the site. Being a
>> community this surely is
>> a usefull thing. An issue worth considering is
>> banning robots from
>> special pages, Recent Changes, Talk, User, etc. by
>> an appropriate META tag.
>> IMHO,  robots and spiders should only be allowed to
>> go over the articles.
>>
>> <snip>
>>
>> regards,
>> [[user:WojPob]]
>
> I COMPLETELY disagree with this.  Let the robots crawl
> everything.  It's better that someone finds one of our
> Talk or User pages and cruises on over to our main
> site than to simply find a completely website!
>
> Chuck
>
I agree with Chuck, strongly.


from_name: lcrocker at nupedia.com

subject: [Wikipedia-l] Robots and special pages

body:
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://lists.wikimedia.org/pipermail/wikipedia-l/attachments/20020517/6b2446a2/attachment.diff


subject: [Wikitech-l] response times

in_reply_to: <1021671045.5102.2.camel@rdaneel.moisty.org>; from brion@pobox.com on Fri, May 17, 2002 at 02:30:43PM -0700

body:
I have re-imported the data from the usemod-eo wiki to the php-eo
wiki.  Let me know if it looks better...

Jason

Brion L. VIBBER wrote:

> On ven, 2002-05-17 at 11:06, Jason Richey wrote:
> > I'll look at it today...
>
> Thanks!
>
> -- brion vibber (brion @ pobox.com)
>
> > Jason
> >
> > Brion L. VIBBER wrote:
> >
> > > We'd be ever so grateful if Jason or Jimmy could try it again
> > > sometime...
> >
> > --
> > "Jason C. Richey" <jasonr at bomis.com>
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at ross.bomis.com
> http://ross.bomis.com/mailman/listinfo/wikitech-l

--
"Jason C. Richey" <jasonr at bomis.com>


body:
On Fri, May 17, 2002 at 08:25:19AM -0400, Pierre Abbat wrote:
> On Thursday 16 May 2002 19:43, Axel Boldt wrote:
> > Right now, I'm seeing nice and fast responses, except every
> > once in a while everything slows to a halt. If that's due to our
> > script, then there must be some really bad, really rare special
> > function somewhere. I doubt that.

I'm not following this closely so maybe this is a stupid remark but I know
that Brion disabled his code that did LIKE queries when MATCH gave empty
results. Has this change already become effective for the actual Wikipedia?

-- Jan Hidders