from_name: Jimmy Wales

subject: [Wikitech-l] whitespace in search

in_reply_to: <200203091736.g29HakP01755@levi.aronsson.se>; from lars@aronsson.se on Sat, Mar 09, 2002 at 06:36:46PM +0100

body:
Lars Aronsson wrote:
> This is a software problem that should not be exposed to the user.
> The four letter limit is stupid, and should be raised to allow three
> letter words.  If my search queary contains a whitespace, each
> separate word could still be sent to the MySQL search engine, and the
> resulting hit lists can be joined so that pages containing both words
> are listed ahead of pages that contain only one of the words.

I concur.

In my own personal experience writing the search engine for Bomis and
the old fastcgi Wikipedia search engine, it is not particularly costly
to have on-the-fly scoring logic that's suitable for the problem at
hand.  Even some very ad hoc measures (scoring titles higher than the
body is very powerful for Wikipedia, for example... at Bomis, I have a
measure of "uselessness" for words that does a nice job of helping
relevancy over a raw keyword search) can be hugely beneficial in
having the search results cause joy in the searcher.

I personally wonder about the raw performance of MySQL as compared to
btree dbm files.  I could, in theory, write a perl script to go
through and index the current wikipedia database once per night, using
some of my "tricks of the trade", and make the search engine a lot
better than it is now.

However, I wonder if that's the right approach.  The downside to doing
it my old-fashioned way is that the search engine only updates as
often as I set the cron to update the search engine index.  If the
data is always "live" in the mysql database, and if it's just as fast,
then that's obviously the better way to do it.

I don't have any actual knowledge, I just know that fascgi perl + DB_File btree
stuff is very fast.  (Bomis handles about 100x the traffic on 3x the servers!)

--Jimbo


list: wikitech-l

source: pipermail-mbox

from_name: Tomasz Wegrzanowski

subject: [Wikitech-l] Transition of Polish Wikipeda to PHP

body:
I tried this script and I have a few things to say.

First, I seriously propose using UTF-8 instead of Latin-2 for Polish Wikipedia.
Browsers shouldn't have any problems with it. At least Mozilla don't have any.
And it will be more "standard", so for example it will be possible to cut&paste kanjis
into it (I tried and it works, that's main reason for this proposal).

On the other hand I don't know how would people without right
fonts react to kanjis on text box.

I think I see the right way.

Simpler way:
markup input  ::= what user sends after clicking 'submit'
markup output ::= what user get after clicking 'edit this page'

On wiki markup output:
	if (user_option[unicode_fonts] != "yes_i_have_all_fonts_installed")
		s/(unicode_char)/numeric[$1]/;
On wiki markup input (options-independent, some people may have only fonts but no input installed):
	s/(numeric)/unicode_char[$1]/;

Even better way:

On wiki markup output:
	if (user_option[unicode_fonts] != "yes_i_have_all_fonts_installed")
		s/(unicode_char)/special_entity[$1]/;
On wiki markup input (options-independent, some people may have only fonts but no input installed):
	s/(special_entity)/unicode_char[$1]/;

It would also make sense to generate kanjis pngs instead of fonts if
(user_option[unicode_fonts] != "yes_i_have_all_fonts_installed"),
but that's a issue for future.

It might also make some sense to split user_option[unicode_fonts] into more than one option,
like japanese, chinese, arabic, cyrilic, hebrew, etc.

What do you think about this ?

----

Second, someone should create wikiTextPl.php ;)
There is a bit of code there, but most of translations can be probably taken
from Rozeta UseMod script.


from_name: Brion L. VIBBER

subject: [Wikitech-l] whitespace in search

body:
On sab, 2002-03-09 at 09:36, Lars Aronsson wrote:
> Jan Hidders wrote:
> > No. The problem is that because of how the MySQL indexing works you
> > cannot do that anymore. For the same reason you also cannot search
> > for anything that is not a letter or words with less than 4 letters.
>
> Well, I *am* a programmer, but the minute that Julie Kemp finds out
> about this, you're gonna be in big trouble.  :-)
>
> This is a software problem that should not be exposed to the user.
> The four letter limit is stupid, and should be raised to allow three
> letter words.

Just three? What if I'm looking up the ancient city of "Ur" or the
fictional land of "Oz"? Or "Wen Ho Li"?

I'd rather there were no length limitation. (This can be adjusted by
recompiling mysql and re-indexing the database. Alternatively, as Jan
has suggested the index field can be munged such that two-character
words would be counted as 4 characters in the indexing.)

Rather, if we're going to eliminate "useless" search terms, we should
have a (per-language) list of such words.

>  If my search queary contains a whitespace, each
> separate word could still be sent to the MySQL search engine, and the
> resulting hit lists can be joined so that pages containing both words
> are listed ahead of pages that contain only one of the words.

I concur.

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] International upgrades

body:
On sab, 2002-03-09 at 14:21, Lars Aronsson wrote:
> Jan Hidders wrote:
> > We cannot index UTF-8.
>
> We shouldn't.  We should strip down to 7bit U.S. ASCII before
> indexing.  Searching for o should find any occurance of ?, ? or ?.
> This works great for English, Swedish, Norwegian, Danish, Finnish, and
> German.  I have successfully tried this on other websites before, but
> I cannot speak for other languages.  Of course, the search expression
> must be stripped in the same way before the search is performed.

That's only relevant for accented Latin characters, obviously. Hebrew,
Arabic, Cyrillic, Greek, Chinese and Japanese characters still need to
be retained and searchable. (However we can similarly fold together
cases and accents for Greek, perhaps final/medial forms for Greek,
Hebrew, and Arabic, and possibly katakana/hiragana for Japanese.)

So yes, we need to index UTF-8 if we're using it.

> Also, in the stripping down, any E following a wovel could be removed,
> to avoid the confusion between spellings like Gottingen, Goettingen,
> and G?ttingen, and that Danish poet Oehlenschl?ger.
>
> This sort of search will yield a few hits too many, which is good.
> I'm not advocating soundex matching here, but soundex could be
> implemented in the same way.

I have no objection to the above. Would match potato/potatoe, too. :)

-- brion vibber (brion @ pobox.com)


from_name: Lars Aronsson

subject: [Wikitech-l] whitespace in search

body:
Jan Hidders wrote:
> No. The problem is that because of how the MySQL indexing works you
> cannot do that anymore. For the same reason you also cannot search
> for anything that is not a letter or words with less than 4 letters.

Well, I *am* a programmer, but the minute that Julie Kemp finds out
about this, you're gonna be in big trouble.  :-)

This is a software problem that should not be exposed to the user.
The four letter limit is stupid, and should be raised to allow three
letter words.  If my search queary contains a whitespace, each
separate word could still be sent to the MySQL search engine, and the
resulting hit lists can be joined so that pages containing both words
are listed ahead of pages that contain only one of the words.

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikitech-l] International upgrades

body:
On sab, 2002-03-09 at 18:50, Lars Aronsson wrote:
> Brion L. VIBBER wrote:
> > That's only relevant for accented Latin characters, obviously. Hebrew,
> > Arabic, Cyrillic, Greek, Chinese and Japanese characters still need to
> > be retained and searchable.
>
> Are we talking about Greek/Hebrew characters in the English/German
> Wikipedia now?  I think users of the English/German Wikipedia won't
> have Greek/Hebrew keyboards,

Excepting Greeks and Israelis, obviously. ;)

> so ASCII searching would do just fine.

But why bother creating a special separate ASCII-only search, when the
non-Latin code is necessary for other languages and we're using a
unified character set?

Why *shouldn't* I be able to search for the occasional Greek, Hebrew, or
Japanese word in the original spelling on the English wikipedia, if we
allow people to put them in in the first place?

> I have no idea how to implement search in the Greek/Hebrew Wikipedia.

As stated above: do whatever accent/case/other equivalent conversion is
necessary (exactly as you propose for Latin characters), and perform
some conversion so that MySQL doesn't reject the UTF-8 non-ascii
characters as word separators (in an ideal world, we'd just configure
MySQL to understand UTF-8; otherwise, replacing raw bytes with hex codes
should work fine).

-- brion vibber (brion @ pobox.com)


list: wikipedia-l

from_name: Ian

subject: [Wikipedia-l] Everything in the last digest

body:
Um, when you get down to it, the PHP does produce HTML. Couldn't you just
have the mirrors display static html pages?

It would be a new feature in the PHP code - it would have to produce pages
with only relative links, except for the edit features, which would link
directly to wikipedia.com. And some way of communicating to mirrors of
what to download when.

Ian

> The existing PHP code will be useless in generating a static site
> suitable for mirroring.
>
> Well, I do have the wt2txt utility, which will convert articles into
> DocBook XML, which is easily publishable in many formats including
> PDF, PostScript, text, and good old html.
>
> Sime I'm already working on very much the same problem, there's no
> need for you to duplicate work. My code is all under the GPL. So if
> any programmers want to help with this, talk to me.
>
> --
> David C. Merrill                         http://www.lupercalia.net
> Linux Documentation Project                   david at lupercalia.net
> Collection Editor & Coordinator            http://www.linuxdoc.org
>
> I wonder if in part why so many people are angry at Microsoft is not just
> because their products frustrate them so much, but also because this
> frustration is ignored. The computer makes people feel like they are
> dummies, when in fact it is the computer that is stupid.
> 	--MIT Associate Professor Rosalind Picard
> [Wikipedia-l]
> To manage your subscription to this list, please go here:
> http://www.nupedia.com/mailman/listinfo/wikipedia-l
>


from_name: kband at www.llamacom.com

subject: [Wikipedia-l] Short word searches

body:
As someone has previously pointed out, short word searches (such as
Ur and Oz) are important. There shouldn't be a letter minimum, if
at all possible.
We should probably use google as a good guide for usability standards.
Google lets you search for single letters, for example.

I understand there may be technical difficulties in reducing the minimum,
but we should consider it a flaw if we have a four-letter minimum,
or even a three-letter or two-letter minimum.

I've cc'ed wikipedia-l because this is a policy discussion as much as a
technical discussion.

--tc


subject: [Wikitech-l] Short word searches


subject: [Wikitech-l] International upgrades

body:
Jan Hidders wrote:
> We cannot index UTF-8.

We shouldn't.  We should strip down to 7bit U.S. ASCII before
indexing.  Searching for o should find any occurance of ?, ? or ?.
This works great for English, Swedish, Norwegian, Danish, Finnish, and
German.  I have successfully tried this on other websites before, but
I cannot speak for other languages.  Of course, the search expression
must be stripped in the same way before the search is performed.

Also, in the stripping down, any E following a wovel could be removed,
to avoid the confusion between spellings like Gottingen, Goettingen,
and G?ttingen, and that Danish poet Oehlenschl?ger.

This sort of search will yield a few hits too many, which is good.
I'm not advocating soundex matching here, but soundex could be
implemented in the same way.

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikitech-l] International upgrades

body:
Brion L. VIBBER wrote:
> That's only relevant for accented Latin characters, obviously. Hebrew,
> Arabic, Cyrillic, Greek, Chinese and Japanese characters still need to
> be retained and searchable.

Are we talking about Greek/Hebrew characters in the English/German
Wikipedia now?  I think users of the English/German Wikipedia won't
have Greek/Hebrew keyboards, so ASCII searching would do just fine.

I have no idea how to implement search in the Greek/Hebrew Wikipedia.

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


from_name: Axel Boldt

subject: [Wikitech-l] Re: whitespace in search

body:
Jimbo, the tricks that you mention (weighing titles more than text,
weighing rare words more than common ones, stopwords etc.) are already
used by mysql's fulltext index; I don't think it would make sense to
reinvent the wheel, especially since we now have real-time searching
and boolean searches, both of which are kind of nice. I am also pretty
sure that the mysql index is reasonably fast, being written in C.
They explain a bit about it at
http://www.mysql.com/doc/F/u/Fulltext_Search.html

Regarding the three letter limit: since we right now already parse
the search string for AND, OR and NOT anyway, it should be pretty easy
to remove short words from the search string, start the query without them,
and then later report the results with a warning like
  The short word "the" was ignored.

Axel


subject: [Wikitech-l] New diff engine from phpwiki

body:
I just checked in a new diff engine from phpwiki 1.3.3 (GPL). It can
display diffs line by line and highlight the changed words. I also
added a diff display to the edit conflict screen.

I'm very impressed with the phpwiki project. They are also working on
internationalization issues; I think we could do far worse than
check out what they have come up with.

Axel


subject:
[Wikitech-l] Re: [Intlwiki-l] More translation issues, preliminary version of
 wikiTextPl.php

body:
On dim, 2002-03-10 at 06:34, Tomasz Wegrzanowski wrote:
> Everyone on these lists who can speak Polish - please check this translation
> and whether internationalization works in general. Not all strings have been translated yet.

Great! I don't speak but a few words of Polish myself, but I'll take a
general look over it this evening.

> It uses UTF-8.
> I think that's the right way.
> If you think otherwise please say why.

I agree.

> 1) Print footer ("This article is from Wikipedia (URL), the free online
> encyclopedia. You can find this article at URL") is in wikiPage.php
> and it should be in wikiText*.php
> That is it should be internationalizable.

Huh. Well, that'll need to be fixed...

> 2) The same word is used for names of languages and names of non-English
> Wikipedias. That's wrong for Polish because "language" is masculine
> but "Wikipedia" is feminine, so the word should be different
> ("polski" and "polska").

Where exactly does this come up? The only place in the interface that
the languages are displayed (that I can think of) is in the link bar
with links to other languages (ex http://wikipedia.com/wiki/Esperanto).
As it is, these are going to need to be shortened, probably to
2/3-letter language codes, because the line gets waaay too long with
many languages.

> 3) There are problems with interwiki links. If Polish article title contain
> non-ascii characters, I can't easily link it using [[pl:Name]].
> The "right" solution is of course using UTF-8 for all wikipedias.
> We will have to do it some day, as not doing it is causing
> more problems than it saves. But that's not an urgent issue.

Yes, that's what we're planning.

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] Feature request: Kana entities

body:
On dim, 2002-03-10 at 04:34, Lars Aronsson wrote:
> Brion L. Vibber wrote:
> > On ven, 2002-03-08 at 09:20, Jimmy Wales wrote:
> > > Really?  There are kanji in articles about Japan?
> > Yeeessss.... You have such a difficult time accepting this. :)
>
> For the matter of implementing the search engine, Latin search and
> Kanji search could be two different functions.  Just like image search
> (Google style) is a third function and mathematic equations search
> could be a fourth kind of search, once LaTeX support is integrated in
> Wikipedia.  To me, the kanji is just like images and I have no
> keyboard to input that in the search window anyway.
>
> The English Wikipedia might implement all three searches, but the
> Norwegian and German ones might only need the Latin search (until a
> significant number of German Wikipedia pages have kanji, images or
> equations in them).
>
> Perhaps this separation of implementations can help us get forward?
> I still have no advice for the non-Latin Wikipediae.

Well, my point is that there is no need whatsoever to make these
separate functions. They work 100% THE SAME WAY. You put in text, it
munges it a bit to make non-ASCII characters behave, and searches for
it. No images involved.

No great hardship to the person who never types a kanji. (Or an
o-with-umlaut!)

No reason to make them separate.

> > The English wikipedia isn't just for English monolinguals, is it?
>
> Is this the new politically correct term for Americans?  :-)

:)

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] whitespace in search

in_reply_to: <1015726974.7876.0.camel@rdaneel.moisty.org>; from brion@pobox.com on Sat, Mar 09, 2002 at 06:22:53PM -0800

body:
Brion L. VIBBER wrote:
> Rather, if we're going to eliminate "useless" search terms, we should
> have a (per-language) list of such words.

A useful and simple (though not perfect) measure of uselessness is how
many pages are returned for a given word.  In English, 'a', 'an' and
'the' will appear in nearly every article.  In Japanese, 'wa' and
other similar marker words will appear in nearly every article.

The more articles that are returned for a given search term, the less
informative it is.

--Jimbo


subject: [Wikitech-l] Re: whitespace in search

in_reply_to: <200203100253.DAA10500@linde.uni-paderborn.de>; from axel@uni-paderborn.de on Sun, Mar 10, 2002 at 03:53:32AM +0100

body:
Axel Boldt wrote:
> Jimbo, the tricks that you mention (weighing titles more than text,
> weighing rare words more than common ones, stopwords etc.) are already
> used by mysql's fulltext index;

I was not aware of that!  So, very good then!

> Regarding the three letter limit: since we right now already parse
> the search string for AND, OR and NOT anyway, it should be pretty easy
> to remove short words from the search string, start the query without them,
> and then later report the results with a warning like
>   The short word "the" was ignored.

I would that this is what we should do, at a bare minimum, from the
principle of least astonishment.  Still, what if there are legitimate
2 letter searches, like for "Ur" as someone else pointed out?

--Jimbo


in_reply_to: <200203100612.HAA11370@linde.uni-paderborn.de>; from axel@uni-paderborn.de on Sun, Mar 10, 2002 at 07:12:12AM +0100

body:
We should contact the leaders of that project and see what kind of synergy
there might be between the two.

One primary difference is that they are working on general wiki code,
while all of our coding is really subordinate to the _encyclopedia_
project.  There are features that we will want that they will not, and
features that they will want which we will not.  Probably we'll end up
with very different end results 5 years from now.

However, it seems likely that there's so much in common that...

--Jimbo


subject: [Wikitech-l] PHP script internationalization issues

body:
I have found some problems with this script.

1) Print footer ("This article is from Wikipedia (URL), the free online
encyclopedia. You can find this article at URL") is in wikiPage.php
and it should be in wikiText*.php
That is it should be internationalizable.

2) The same word is used for names of languages and names of non-English
Wikipedias. That's wrong for Polish because "language" is masculine
but "Wikipedia" is feminine, so the word should be different
("polski" and "polska").

3) There are problems with interwiki links. If Polish article title contain
non-ascii characters, I can't easily link it using [[pl:Name]].
The "right" solution is of course using UTF-8 for all wikipedias.
We will have to do it some day, as not doing it is causing
more problems than it saves. But that's not an urgent issue.


body:
Brion L. Vibber wrote:
> On ven, 2002-03-08 at 09:20, Jimmy Wales wrote:
> > Really?  There are kanji in articles about Japan?
> Yeeessss.... You have such a difficult time accepting this. :)

For the matter of implementing the search engine, Latin search and
Kanji search could be two different functions.  Just like image search
(Google style) is a third function and mathematic equations search
could be a fourth kind of search, once LaTeX support is integrated in
Wikipedia.  To me, the kanji is just like images and I have no
keyboard to input that in the search window anyway.

The English Wikipedia might implement all three searches, but the
Norwegian and German ones might only need the Latin search (until a
significant number of German Wikipedia pages have kanji, images or
equations in them).

Perhaps this separation of implementations can help us get forward?
I still have no advice for the non-Latin Wikipediae.

> The English wikipedia isn't just for English monolinguals, is it?

Is this the new politically correct term for Americans?  :-)

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikitech-l] More translation issues, preliminary version of wikiTextPl.php

body:
Everyone on these lists who can speak Polish - please check this translation
and whether internationalization works in general. Not all strings have been translated yet.

It uses UTF-8.
I think that's the right way.
If you think otherwise please say why.

You can "easily" setup PHP local wikipedia by:

$ cvs -d:pserver:anonymous at cvs.wikipedia.sourceforge.net:/cvsroot/wikipedia login
<click enter for password>
$ mkdir wikipedia
$ cd wikipedia
$ cvs -d:pserver:anonymous at cvs.wikipedia.sourceforge.net:/cvsroot/wikipedia -z3 co .
$ cd phpwiki/fpw
* edit wikiLocalSettings.php to something like:
<?
$wikiLanguage = "pl";
$wikiThisDBserver = "localhost";   # the IP address of the computer running mysqld
$wikiSQLServer = "wikipedia";      # the mysql database used for Wikipedia
$wikiThisDBuser = "taw";           # the mysql user
$wikiThisDBpassword = "xxx";       # mysql password
$wikiCurrentServer = "http://localhost";
$THESCRIPT = "/~taw/wikipedia/wiki.phtml";
$wikiArticleSource = "$wikiCurrentServer$THESCRIPT?title=$1";
?>
* copy and gunzip wikiTextPl.php from my mail
$ ls -s ~/src/wikipedia/phpwiki/fpw/ ~/public_html/wiki
* set up mod_php4 in apache if not already done, by Webmin or otherwise
* set up mysql permisions if not already done, by Webmin or otherwise
* follow instruction from README file to set up mysql database
* now try and enjoy local instalation of wikipedia

Issues:
* importing Polish Rozeta/UseMod tree.
* RewriteRules
* translate all yet not traslated strings
* fix any wrong translations

I would like to see Polish wikipedia moved to new software as soon as possible.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: wikiTextPl.php.gz
Type: application/octet-stream
Size: 9108 bytes
Desc: not available
URL: <http://lists.wikimedia.org/pipermail/wikitech-l/attachments/20020310/2181198a/attachment.obj>


subject: [Wikitech-l] Re: whitespace in search

in_reply_to: <200203101915.UAA14725@linde.uni-paderborn.de>; from axel@uni-paderborn.de on Sun, Mar 10, 2002 at 08:15:07PM +0100

body:
Axel Boldt wrote:
> The only way to make mysql index shorter words is by recompiling and
> setting MIN_WORD_LEN to a different number. They claim that changing
> this variable from the current value of 4 to 0 will enlarge the index
> by a factor of 20. So maybe we should go with 2. After that, the
> indexes have to be rebuilt.

Wow, factor of 20!  That's pretty severe.  2 seems good enough for now.

I'll try this out in a few days time.


subject: [Wikitech-l] Re: [Intlwiki-l] More translation issues, preliminary version of wikiTextPl.php

body:
On Sun, Mar 10, 2002 at 08:40:34AM -0800, Brion L. VIBBER wrote:
> > 2) The same word is used for names of languages and names of non-English
> > Wikipedias. That's wrong for Polish because "language" is masculine
> > but "Wikipedia" is feminine, so the word should be different
> > ("polski" and "polska").
>
> Where exactly does this come up? The only place in the interface that
> the languages are displayed (that I can think of) is in the link bar
> with links to other languages (ex http://wikipedia.com/wiki/Esperanto).
> As it is, these are going to need to be shortened, probably to
> 2/3-letter language codes, because the line gets waaay too long with
> many languages.

My mistake. $wikiLanguageNames is used only for links to other
language versions (with [[pl:Foo]]). So it should just use one form.

The second place where it appears - links on main pages - are not automatic.
So this isn't any problem after all.


subject: [Wikitech-l] whitespace in search

body:
On Sun, Mar 10, 2002 at 09:28:46AM -0800, Jimmy Wales wrote:
> Brion L. VIBBER wrote:
> > Rather, if we're going to eliminate "useless" search terms, we should
> > have a (per-language) list of such words.
>
> A useful and simple (though not perfect) measure of uselessness is how
> many pages are returned for a given word.  In English, 'a', 'an' and
> 'the' will appear in nearly every article.  In Japanese, 'wa' and
> other similar marker words will appear in nearly every article.

I'm wondering how is search going to work in Japanese.
Not only some articles are in romaji and other kanjis,
but kanji words usually aren't separated by whitesace,
so it might be a bit difficult.

> The more articles that are returned for a given search term, the less
> informative it is.

Only if they are not sorted.
We could just give less priority to more frequent word and more priority
to less frequent or something like that. So for example search for "the foo"
would rate +1 point for every the and +10 for every foo (which is 10x less
frequent than "the"). And then sort the results according to this score.


from_name: Chuck Smith

subject: [Wikitech-l] English monolinguals

body:
> > The English wikipedia isn't just for English
> monolinguals, is it?
>
> Is this the new politically correct term for
> Americans?  :-)

S?, tienes raz?n.

Ja, du bist richtig.

Jes, vi pravas.

Chuck  :)

=====
Come to my homepage!  Venu al mia hejmpagxo!
http://amuzulo.babil.komputilo.org/
====
Venu al la senpaga, libera enciklopedio
esperanta reta! http://eo.wikipedia.com/

_________________________________________________________
Do You Yahoo!?
Informaci?n de Estados Unidos y Am?rica Latina, en Yahoo! Noticias.
Vis?tanos en http://noticias.espanol.yahoo.com


subject: [Wikitech-l] Re: [Intlwiki-l] PHP script internationalization issues

body:
On Sun, Mar 10, 2002 at 03:11:02PM +0000, Joao M?rio Miranda wrote:
> Tomasz Wegrzanowski wrote:
> > 3) There are problems with interwiki links. If Polish article title contain
> > non-ascii characters, I can't easily link it using [[pl:Name]].
>
> I have the same problem with the portuguese wkipedia. I can't
> link to [[pt:portugu?s]].

You can link by %-escapes (which you can copy from URL).
This method sucks but is the only way until we all adopt UTF-8.

I strongly recommend setting all non-English Wikipedias to UTF-8
on transition to PHP script, regardless of whatever encoding is
currently most popular for your languages.

I also think that English Wikipedia should move to UTF-8.


subject: [Wikitech-l] Re: whitespace in search

body:
>Still, what if there are legitimate
>2 letter searches, like for "Ur" as someone else pointed out?

The only way to make mysql index shorter words is by recompiling and
setting MIN_WORD_LEN to a different number. They claim that changing
this variable from the current value of 4 to 0 will enlarge the index
by a factor of 20. So maybe we should go with 2. After that, the
indexes have to be rebuilt.

Axel


subject: [Wikipedia-l] Short word searches

body:
At the very least, there should be a convient way to search with Google.

The problem with Google is that it does not update very much. What's
currently in the cache is from January 26.

I noticed this problem as well, when I tried to search for "Ian". (-:

Ian Monroe
http://mlug.missouri.edu/~eean/

On Sat, 9 Mar 2002, kband at www.llamacom.com XXXXXXXXXXXXXXXXXXXXX wrote:

> As someone has previously pointed out, short word searches (such as
> Ur and Oz) are important. There shouldn't be a letter minimum, if
> at all possible.
> We should probably use google as a good guide for usability standards.
> Google lets you search for single letters, for example.
>
> I understand there may be technical difficulties in reducing the minimum,
> but we should consider it a flaw if we have a four-letter minimum,
> or even a three-letter or two-letter minimum.
>
> I've cc'ed wikipedia-l because this is a policy discussion as much as a
> technical discussion.
>
> --tc
> [Wikipedia-l]
> To manage your subscription to this list, please go here:
> http://www.nupedia.com/mailman/listinfo/wikipedia-l
>


body:
On dim, 2002-03-10 at 09:54, Jimmy Wales wrote:
> pl.wikipedia.com
>
> How about I set this up on Monday?  Do I need a new conversion script?

I'll need to tweak the conversion script to convert the Latin-2 database
to UTF-8, and drop in a quick fix to make searching work on non-ASCII
words. Shouldn't be hard, I'll drop a note when it's checked in.

The full set of encoding-related changes to the code still need to be
determined and made, but if we start it off using UTF-8, the database
should be ready.

(Also, we may need to rebuild the search index later, as that part of
the code may still change. But then this is the case if we change the
mysql indexing anyway.)

> I believe that this is best:
>
> 1.  I set up pl2.wikipedia.com as the "beta test" Polish wikipedia,
> and import the existing data from pl.wikipedia.com using a conversion
> script.
>
> 2.  We run this for one week, inviting as many Polish speakers as we
> can find to test it, incorporating bug fixes all week as quickly as
> they come in.
>
> 3.  The following Monday, I make the conversion "for real".

Sounds good to me. Can we commit to such a schedule for Esperanto as
well? We've only put it off so far because we were going to change the
database internals, but since we seem to have decided on UTF-8 it's not
a problem to convert ASAP and make the changes later.

> I'd like to follow up with Spanish and Esperanto as quickly as possible.

FYI, I currently have a beta running the converted Esperanto wiki at
http://leukas.dyndns.org/wiki/ . A more officialish server wouldn't hurt
though, as this is limited to the reliability of my DSL line.

-- brion vibber (brion @ pobox.com)


body:
On dim, 2002-03-10 at 08:40, Brion L. VIBBER wrote:
> On dim, 2002-03-10 at 06:34, Tomasz Wegrzanowski wrote:
> > Everyone on these lists who can speak Polish - please check this translation
> > and whether internationalization works in general. Not all strings have been translated yet.
>
> Great! I don't speak but a few words of Polish myself, but I'll take a
> general look over it this evening.

Okay, it looks pretty good to me! I can't speak for the correctness of
the translations, of course, but most obvious functions seem to work and
I didn't see gratuitous English text in the main places. I've checked it
into the CVS server. (Yay!) I'm not sure if all the log names are right
(eg, Log:Uploady), some of those may be hardcoded English internally.
(But, "log" shouldn't be relied on anyway.)

Also, test that all the talk/dyskusja pages work correctly; currently
there's some magic in places that expects all talk pages (including user
talk, wikipedia talk, log talk, etc) to have the same word as just plain
talk pages in them. Right now you don't have that, and it doesn't seem
to like it. ie, we have a user page in the "U?ytkownik:" namespace,
which goes to "U?ytkownik Talk:" (which it won't let me create) rather
than something like "U?ytkownika Dyskusja:" (let's pretend that was a
correct genitive case or something). However, this whole part of things
is rather rough now, and needs work. (My mind is going, can you tell?
It's too late to be working on software...)

I also added a quick Latin2-to-UTF-8 convertor to convertWiki2SQL.php;
uncomment the Polish section at the top and it'll enable it (as well as
the alternate field separator and "Dyskusja" as the default /Talk link).

A couple of warnings; the conversion script currently doesn't fill in
link information, since the entries in the cur table were removed.
Support should be added there for filling the new linked and unlinked
tables, but for now you can just run rebuildLinks.php after creating the
new database.

Also, searching with non-ASCII characters is still broken until the much
discussed, not yet implemented second index field is added. (Or we
figure out how to make MySQL behave without one.)

-- brion vibber (brion @ pobox.com)


from_name: Ward Cunningham

subject: [Wikitech-l] Re: WikiHistory

body:
Lars Aronsson wrote:

> However, the Internet Archive's Wayback machine shows no sign of any Wiki in November 1996. So where was the Wiki hiding during 1994, 1995, and 1996?

I was careful to keep wiki quiet early on.

I've dug up some more dated information and added it to the end of the WikiHistory page. It looks like wiki went public in March of 1995.

    http://c2.com/cgi/wiki?WikiHistory

--
Ward Cunningham
v 503-245-5633   mailto:ward at c2.com
f 503-246-5587   http://c2.com/


body:
On sab, 2002-03-09 at 22:12, Axel Boldt wrote:
> I just checked in a new diff engine from phpwiki 1.3.3 (GPL). It can
> display diffs line by line and highlight the changed words. I also
> added a diff display to the edit conflict screen.

Nice!

Do you think it would be better to show the diff between the two edit
boxes rather than below both? I suspect that users are likely not to see
it way down there.

-- brion vibber (brion @ pobox.com)


subject:
[Wikitech-l] Re: [Intlwiki-l] More translation issues,
	preliminary version of wikiTextPl.php

body:
On dim, 2002-03-10 at 09:53, Tomasz Wegrzanowski wrote:
> My mistake. $wikiLanguageNames is used only for links to other
> language versions (with [[pl:Foo]]). So it should just use one form.
>
> The second place where it appears - links on main pages - are not automatic.
> So this isn't any problem after all.

Ah, good.

I have fixed the print footer -- set the non-English message in
$wikiPrintFooter.

Also, the conversion script now builds the link tables, so running
rebuildLinks.php is no longer necessary. Next step: bringing in the old
edit history!

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] WikiHistory

body:
To Ward and to Wikipedia's tech list,

On http://c2.com/cgi/wiki?WikiHistory it is claimed that this (c2.com)
is the first Wiki website, founded in 1994, as a supplement to the
Portland Pattern Repository.  The next dates on the page say that
RecentVisitors and PeopleIndex were added in 1994, NotSoRecentChanges
in 1995, ThreadMode, ThreadModeConsideredHarmful, EditCopy and
WikiCategories in 1996.

However, the Internet Archive's Wayback machine shows no sign of any
Wiki in November 1996, when the c2.com website was first archived.
The Portland Pattern Repository is there, but it consists of plain
HTML files.  The Wiki Wiki Web does show on the second archiving in
December 1997.  See http://web.archive.org/web/*/http://c2.com

So where was the Wiki hiding during 1994, 1995, and 1996?

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikitech-l] Wiki conversion script updates

body:
The conversion script now includes old revisions of articles. I've tried
it on the Esperanto and Polish databases which I had handy, so far it
seems to work. Please test on other languages if you dare.

It shouldn't be too hard to modify this code to extract just the old
versions from the old English wikipedia and drop them into the current
database there, as well.

Notes:
* Since user accounts are not transferred, there is no numerical user ID
to put in the old_user field. Currently this results in the wiki
thinking the user name in old_user_text was an IP address and trying to
mask the last 3 digits, and not making links to user pages in the
history lists. The digit masking is definitely wrong, however not making
the links is arguably correct behavior.

* The most recent revision still has its user, comment, and timestamp
wiped and replaced with "conversion script", "automatic conversion", and
time of conversion. Would it not be nicer to keep the previous user,
comment, and timestamp, as is done with the older revisions?

* We might, however, still want to add a note that conversion took
place, so it's an obvious cutoff in the history list.

* Do we want to run fixLinks() on the old page versions? (This changes
/subpage links into Page:subpage links.) Right now I do so to preserve
link functionality, but this may not be appropriate, as it changes the
content of previous versions slightly. The purpose of keeping old
versions is to see what changed, so we might prefer to have the
unchanged (and no longer working) /subpage links. Comments?

-- brion vibber (brion @ pobox.com)


subject: [Intlwiki-l] [SOS][Wikitech-l]  killer bug

body:
On mar, 2002-03-12 at 04:02, 421621 at ingta.unizar.es wrote:
> S.O.S.
>
> If you get
>  http://es.wikipedia.com/
>
> liks have this format:
>  http://es.wikipedia.com/?stuff
>
>  and.. not work!
>
> the good thing is this:
>  http://es.wikipedia.com/wiki.cgi?stuff

It looks okay to me. Either Jimbo fixed it *really* fast, or the bug
only shows up in certain circumstances.

Can you check again, and if it's still broken, give more information to
help us reproduce the problem? (What browser, what preferences you have
set, etc)

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] Importing Polish Wikipedia

body:
If you're reading this, I hit "send" by mistake.

On mar, 2002-03-12 at 03:29, Tomasz Wegrzanowski wrote:
> Generally it worked, but there were some problems
>
> 1)
> Conversion script tried to convert .db.orig files:
>
> Now there are articles like "Propozycje Tematow.db.o"
> with contents "There was an error converting this file : file not found."

Should be harmless...

> 2)
> Article q{"Casablanca"} (WITH quotes, i don't know how to write
> it down, let's say that q{} are stronger quotes ;)), currently
> a redirect to q{Casablanca (film)} was converted to q{\"Casablanca"}.
> I wouldn't mind if it was just removed from database.

!! Okay, that's just wrong. Hmm.

Actually, on my machine I don't even see this article converted. I'm not
sure what's going on there, I seem to have a lot of articles missing.
Possibly some weird PHP problem, as near as I can tell it's simply not
reading all the files in every directory.

> 3)
> Some articles are weird, like:
> Polimery
> 12c12 < [?rednia masa cz?steczkowa polimer?w]? --- > Masa cz?steczkowa polimer?w
>
> That looks like a diff which was imported as article.
> I have no idea what could cause it.

The usemod db format doesn't seem to be quite as consistent as the
conversion script wants... I'll look it over again in the morning.

> Major problem: z-with-dot letter seems to be screwed in links.
> Links from http://local_copy_of_wikipedia/wiki.phtml?title=Wy%C5%BCsze_uczelnie_w_Polsce
> to subpages are broken.
> They work on Polish UseMod Wikipedia, so it must be problem with conversion script.
> Maybe there are some mistakes in latin2 -> unt8 table ?

Ah, looks like parts of the links are accidentally converted from latin2
to utf-8 twice. (Thanks for catching that! It doesn't show up on the
Esperanto database I usually test with, which is already mostly UTF-8
and doesn't change on a second conversion.) I've moved the conversion to
before the link extraction, it seems better now.

-- brion vibber (brion @ pobox.com)


in_reply_to: <1015935522.6550.0.camel@rdaneel.moisty.org>; from brion@pobox.com on Tue, Mar 12, 2002 at 04:18:40AM -0800

body:
I didn't fix it.

Brion L. VIBBER wrote:

> On mar, 2002-03-12 at 04:02, 421621 at ingta.unizar.es wrote:
> > S.O.S.
> >
> > If you get
> >  http://es.wikipedia.com/
> >
> > liks have this format:
> >  http://es.wikipedia.com/?stuff
> >
> >  and.. not work!
> >
> > the good thing is this:
> >  http://es.wikipedia.com/wiki.cgi?stuff
>
> It looks okay to me. Either Jimbo fixed it *really* fast, or the bug
> only shows up in certain circumstances.
>
> Can you check again, and if it's still broken, give more information to
> help us reproduce the problem? (What browser, what preferences you have
> set, etc)
>
> -- brion vibber (brion @ pobox.com)
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at ross.bomis.com
> http://ross.bomis.com/mailman/listinfo/wikitech-l


subject: [Wikitech-l] Re: WikiHistory

body:
Ward Cunningham wrote:
> I've dug up some more dated information and added it to the end of
> the WikiHistory page. It looks like wiki went public in March of
> 1995.
>     http://c2.com/cgi/wiki?WikiHistory

Cool!  The annual 100 % growth rate tells us this is an Internet
phenomenon.  Don't you keep webserver access logs?

(My own Scandinavian literature gopher-then-website had its
first access at 8:10 pm on December 13, 1992, to be exact.  :-)
http://www.lysator.liu.se/runeberg/admin/timeline.html)

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


from_name: 421621 at ingta.unizar.es

subject: [SOS][Wikitech-l]  killer bug

body:
S.O.S.

If you get
 http://es.wikipedia.com/

liks have this format:
 http://es.wikipedia.com/?stuff

 and.. not work!

the good thing is this:
 http://es.wikipedia.com/wiki.cgi?stuff

---------------------------------------------
This message was sent using Endymion MailMan.
http://www.endymion.com/products/mailman/


subject: [Wikitech-l] Importing Polish Wikipedia

body:
Generally it worked, but there were some problems

1)
Conversion script tried to convert .db.orig files:

Now there are articles like "Propozycje Tematow.db.o"
with contents "There was an error converting this file : file not found."

2)
Article q{"Casablanca"} (WITH quotes, i don't know how to write
it down, let's say that q{} are stronger quotes ;)), currently
a redirect to q{Casablanca (film)} was converted to q{\"Casablanca"}.
I wouldn't mind if it was just removed from database.

3)
Some articles are weird, like:
Polimery
12c12 < [?rednia masa cz?steczkowa polimer?w]? --- > Masa cz?steczkowa polimer?w

That looks like a diff which was imported as article.
I have no idea what could cause it.


subject: [Wikitech-l] Importing Polish Wikipedia, part 2

body:
Major problem: z-with-dot letter seems to be screwed in links.

Links from http://local_copy_of_wikipedia/wiki.phtml?title=Wy%C5%BCsze_uczelnie_w_Polsce
to subpages are broken.
They work on Polish UseMod Wikipedia, so it must be problem with conversion script.
Maybe there are some mistakes in latin2 -> unt8 table ?


subject: [SOS][Wikitech-l]  killer bug

body:
On Tue, Mar 12, 2002 at 12:02:35PM +0000, 421621 at ingta.unizar.es wrote:
> S.O.S.
>
> If you get
>  http://es.wikipedia.com/
>
> liks have this format:
>  http://es.wikipedia.com/?stuff
>
>  and.. not work!
>
> the good thing is this:
>  http://es.wikipedia.com/wiki.cgi?stuff

I tried and links have format like:
	http://es.wikipedia.com/wiki.cgi?Medicina
and work.


subject: [Wikitech-l] Our search engine syntax/semantics

body:
On mar, 2002-03-12 at 12:49, Lars Aronsson wrote:
> Is there a way to create a link directly to a Wikipedia search using
> the new software? In the old system, I used to link to
>
>   http://www.wikipedia.com/search.fcgi?request=Oregon
>
> but that doesn't work anymore (and it shouldn't, since the old
> system isn't used anymore).

Sure:

  http://www.wikipedia.com/wiki.phtml?search=Oregon
or
  http://www.wikipedia.com/wiki/&search=Oregon

(Is there any reason we do searches with POST rather than GET? You can
always construct the URL manually, but it's nicer to be able to
cut-n-paste.)

-- brion vibber (brion @ pobox.com)


body:
On mar, 2002-03-12 at 12:56, Magnus Manske wrote:
> Congratulations! You succeeded where I failed miserably.

Now, if I can only figure out how to read those occasional garbled
_current_ page files...

> Suggestion: Keep all old versions as they are, and generate an "artificial"
> edit, signed with "conversion script", where the sctual "subpage
> translation" takes place. So all old versions are stored correctly, and the
> conversion itself is more transparent.

Sounds reasonable. I'll try hacking that into place later today.

> Really, great work! :)

Thanks!

-- brion vibber (brion @ pobox.com)


body:
By parsing the search string, we get the benefit of full boolean
search, which is definitely cool. But our current regime has one
severe downside: any search which has a single short word or stopword
or non-existent word in it will automatically fail since we assume an
implicit "AND" between all search terms. The unadorned mysql "MATCH"
operator does not have this downside: if you search for "the chinese
wall" for example, "the" will be silently ignored and you get the
expected hit.

I am wondering if we can combine the best of those worlds. This should
decrease the number of complaints about short search terms
dramatically, maybe even to the point that we can keep the current
index size.

How about this: every subsequence of the query string which doesn't
contain any +/- boolean operators (see below) is passed to the MATCH
operator as is, which assumes an implicit "give me the best matches
you can find for these terms, the more matching terms the better".
Then we could have two additional operators: + and -. If a word is
preced by +, it *must* be presents, if a word is preceded by - it
*cannot* be present. That allows to express any complicated query we
can right now, but should result in much fewer failed searches. Does
that seem feasible?

Axel


from_name: 'Dragon' Dave McKee

subject: [Wikipedia-l] Four letter words...

body:
> > I understand there may be technical difficulties in reducing the minimum,
> > but we should consider it a flaw if we have a four-letter minimum,
> > or even a three-letter or two-letter minimum.
> >

I'd also consider the error message given when searching for short words
a Bad Idea.

  [!! SYNTAX ERROR: the word 'the' is too short, the index requires at
least 4 characters]
   AND "cunctuator"

I think it would be better if words shorter than four letters (or stop
words) were ignored, and a more polite error given if it can't search.

Dragon Dave.


body:
Is there a way to create a link directly to a Wikipedia search using
the new software? In the old system, I used to link to

  http://www.wikipedia.com/search.fcgi?request=Oregon

but that doesn't work anymore (and it shouldn't, since the old
system isn't used anymore).

Linking to searches is very useful in some cases.  Google allows this
(http://google.com/search?q=Oregon) and most Wikis do.

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


body:
Brion L. VIBBER wrote:
>   http://www.wikipedia.com/wiki.phtml?search=Oregon

Thanks, this works great!

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikitech-l] Importing Polish Wikipedia

body:
On mar, 2002-03-12 at 06:03, Brion L. VIBBER wrote:
> If you're reading this, I hit "send" by mistake.

Oops. :)

> On mar, 2002-03-12 at 03:29, Tomasz Wegrzanowski wrote:
> > 3)
> > Some articles are weird, like:
> > Polimery
> > 12c12 < [?rednia masa cz?steczkowa polimer?w]? --- > Masa cz?steczkowa polimer?w
> >
> > That looks like a diff which was imported as article.
> > I have no idea what could cause it.
>
> The usemod db format doesn't seem to be quite as consistent as the
> conversion script wants... I'll look it over again in the morning.

Fixed now.

It should now also import old versions unchanged (without converting
subpage links) up to and including the current version, then add the
conversion.

Also, I added localisable strings $wikiSeeAlso, $wikiAutomatedConversion
and $wikiConversionScript.

-- brion vibber (brion @ pobox.com)


subject: [Wikitech-l] Examining history of redirect pages

body:
I just checked in a fix for an issue that was reported on Bug Reports:
the history of pages which contain a #REDIRECT is invisible; if you
click on "Redirected from" after being redirected, you immediately get
an edit screen.

I changed that behavior: clicking on "Redirected from ..." will now
give you a regular page which shows the redirect, lets you click on
History, Edit this page, Pages that link here etc. as ususal.

I introduced a new global $redirect. If the request
&action=view&redirect=no comes in, then we operate as usual except we
don't follow redirects.

Axel


subject: [Wikipedia-l] 1911 britannica online

in_reply_to: <3C902D08.829935E5@explicacoes.com> from "João Mário Miranda" at Mar 14, 2002 04:54:32 AM

body:
>
> Someone published the 1911 britannica online:
>
> http://1911encyclopedia.org/
>
All right, now we're talking!

Conversion scripts to the ready....


from_name: Bryan Derksen

subject: [Wikipedia-l] 1911 britannica online

body:
At 04:40 PM 3/14/02 +1100, Robert Graham Merkel wrote:
>Whilst this is very good news, on my limited sampling
>it's not something I'd be sucking into the Wikipedia
>using a conversion script.
>
>The first article I examined was one about W.G. Grace
>(19th century English cricketer who revolutionised the
>game).  Only the first paragraph of what appears to be
>a much longer article is there, and it is laden with
>typos - some probably from the scanning process, others
>from the original.

It looks like the rest of the article on W.G. Grace is at the beginning of
the next page; http://97.1911encyclopedia.org/G/GR/GRACES_THE.htm

I noticed something similar with the mount circeo article which I pasted
across, too; they did a very poor job of chopping the source text up, but I
think it's all here.

>Still, it's an outstanding resource and I look forward to
>incorporating a lot of material from it.

Me too. It's just a little more labor-intensive than simple copy-paste or
scripting can manage. :)

--
"Let there be light." - Last words of Bomb #20, "Dark Star"


subject: [Wikipedia-l] 1911 britannica online

in_reply_to: <20020314164030.A24813@trell.vic.bigpond.net.au> from "Robert Graham Merkel" at Mar 14, 2002 04:40:30 PM

body:
> On Thu, 14 Mar 2002 16:20:41 kband at www.llamacom.com wrote:
> > >
> > > Someone published the 1911 britannica online:
> > >
> > > http://1911encyclopedia.org/
> > >
> > All right, now we're talking!
>
> Whilst this is very good news, on my limited sampling
> it's not something I'd be sucking into the Wikipedia
> using a conversion script.
>
> The first article I examined was one about W.G. Grace
> (19th century English cricketer who revolutionised the
> game).  Only the first paragraph of what appears to be
> a much longer article is there, and it is laden with
> typos - some probably from the scanning process, others
> from the original.
>
What was the process for adding the A's?
Were the A's all complete, without typos?

--tc


from_name: João Mário Miranda

subject: [Wikipedia-l] 1911 britannica online

body:
Someone published the 1911 britannica online:

http://1911encyclopedia.org/

--
jo?o
http://www.nonio.com


from_name: koyaanisqatsi at nupedia.com

subject: [Wikipedia-l] 1911 britannica online

body:
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://lists.wikimedia.org/pipermail/wikipedia-l/attachments/20020314/96b0712b/attachment.diff


from_name: bderksen

subject: [Wikipedia-l] 1911 britannica online

body:
I just got word back from the editor of the encyclopedia site, he told me that
in a few days they'd be revising the site ith more abilities. One of them will
be a complete download of the text, so no need for wget just yet.


subject: [Wikipedia-l] 1911 encyclopedia

body:
It is my understanding, from the edition of the 'A' volume of this
encyclopedia published by the fine folks over at Project Gutenburg,
that it is "uncool" to use the name "Britannica" in association with
this book.  The reason is that although the _copyright_ has indeed
properly expired, the _trademark_ is still that of an active company.
So, while the text is fair game, the name is not.

I'm cc:'ing this message to editor at 1911encyclopedia.org, just so that
person will know about this.  But my opinion is not very
authoritative.  I'd just suggest getting the word 'Britannica' off the
site _in the logo_, while feeling free to use the word in a text
explanation of where this information comes from.  (My understanding
is that it's o.k. to say a trademark in just about any context, but
not o.k. to use it in a fashion that might mislead people into
thinking that you are the organization in question.)

The site, as it stands, doesn't really make clear that the people
behind it are not the official Britannica people.

What a wonderful thing to have online.

--Jimbo


from_name: Tim Chambers

subject: [Wikipedia-l] 1911 britannica online

body:
I'm even more encouraged than ever that the 'pedia can earn a
reputation as an authoratative source after having skimmed through
http://1911encyclopedia.org/. Not only is this particular Web site's
content a junk pile (scanning errors and sloppy cataloging), but it
reminded me that some of the 1911's entries are surprisingly thin.
Surely we can do better! (I think we already have --
http://www.wikipedia.com/wiki/Brilliant_prose.)

<>< Tim

__________________________________________________
Do You Yahoo!?
Yahoo! Sports - live college hoops coverage
http://sports.yahoo.com/


from_name: lcrocker at nupedia.com

subject: [Wikipedia-l] 1911 britannica online

body:
>> Whilst this is very good news, on my limited sampling
>> it's not something I'd be sucking into the Wikipedia
>> using a conversion script.
>
>What was the process for adding the A's?
>Were the A's all complete, without typos?

The "A" articles were compiled and edited by hand by
the folks at Project Gutenberg.  Many of them were then
added to Wikipedia more or less automatically, after
which many of them were further hand-edited where
appropriate (for example, I updated "Almond" to reflect
more recent biological classification).

The same process will be needed for these, so that
should keep me and others busy for a long time.  I'm
also curious whether they plan to scan the illustrations.
Those, too, are now in the public domain, and some of
the articles make little sense without them (look at
"Abbey", for example).


from_name: Jimmy Wales

subject: [Wikipedia-l] 1911 encyclopedia

in_reply_to: <3C90E644.A112753D@explicacoes.com>; from jmiranda@explicacoes.com on Thu, Mar 14, 2002 at 06:04:52PM +0000

body:
Oh, very good, and I'm very glad to hear that about the Britannica people!
Very reasonable and gracious of them!

Jo?o M?rio Miranda wrote:

> They have legal notice at the site:
>
> Legal Notice:
>
> The information on this site is protected against unauthorized use
> by others by  a variety of laws and statutes. For more information
> on what you can and cannot do with the information on this site,
> please contact
>
> editor at 1911encyclopedia.com
>
> Thanks
>
> and they are using the name with permition:
>
> Special thanks goes out to the people at Encyclopaedia Britannica
> who have graciously granted permission to us to use this material
> and their name on this site. Without their kind assistance, this
> site could not be.
>
> We urge anyone interested in a modern, up-to-date encyclopedia to
> visit their site at http://www.eb.com.
>
> Joao
> http://www.nonio.com
>
> Jimmy Wales wrote:
> >
> > It is my understanding, from the edition of the 'A' volume of this
> > encyclopedia published by the fine folks over at Project Gutenburg,
> > that it is "uncool" to use the name "Britannica" in association with
> > this book.  The reason is that although the _copyright_ has indeed
> > properly expired, the _trademark_ is still that of an active company.
> > So, while the text is fair game, the name is not.
> >
> > I'm cc:'ing this message to editor at 1911encyclopedia.org, just so that
> > person will know about this.  But my opinion is not very
> > authoritative.  I'd just suggest getting the word 'Britannica' off the
> > site _in the logo_, while feeling free to use the word in a text
> > explanation of where this information comes from.  (My understanding
> > is that it's o.k. to say a trademark in just about any context, but
> > not o.k. to use it in a fashion that might mislead people into
> > thinking that you are the organization in question.)
> >
> > The site, as it stands, doesn't really make clear that the people
> > behind it are not the official Britannica people.
> >
> > What a wonderful thing to have online.
> >
> > --Jimbo
> >
> > [Wikipedia-l]
> > To manage your subscription to this list, please go here:
> > http://www.nupedia.com/mailman/listinfo/wikipedia-l
>
> --
> jo?o
> http://www.nonio.com
> http://www.explicacoes.com
> [Wikipedia-l]
> To manage your subscription to this list, please go here:
> http://www.nupedia.com/mailman/listinfo/wikipedia-l


from_name: Hannes Hirzel

subject: [Wikipedia-l] 1911 britannica online

body:
On Thu, 14 Mar 2002 kband at www.llamacom.com wrote:

> > On Thu, 14 Mar 2002 16:20:41 kband at www.llamacom.com wrote:
> > > >
> > > > Someone published the 1911 britannica online:
> > > >
> > > > http://1911encyclopedia.org/
> > > >
> > > All right, now we're talking!
> >
> > Whilst this is very good news, on my limited sampling
> > it's not something I'd be sucking into the Wikipedia
> > using a conversion script.
> >
> > The first article I examined was one about W.G. Grace
> > (19th century English cricketer who revolutionised the
> > game).  Only the first paragraph of what appears to be
> > a much longer article is there, and it is laden with
> > typos - some probably from the scanning process, others
> > from the original.

It seems that the scanning was just done column-wise and that there
were not enough efforts to separate the entries. But it may be
useful as a source anyway.

HJH


subject: [Wikipedia-l] 1911 encyclopedia

in_reply_to: <3C90F054.DC275F80@explicacoes.com>; from jmiranda@explicacoes.com on Thu, Mar 14, 2002 at 06:47:48PM +0000

body:
Jo?o M?rio Miranda wrote:
> Pagewise.com is a for profit organization:
>
> http://www.pagewise.com/
>
> that owns the http://www.WriteForCash.com/ site.
>
> Probably, they will claim some kind of copyrights over the
> digitalisation of the 1911 Encyclopedia.

I doubt that very much.  If they were going to do that, presumably the Britannica
people would not have been so gracious.

Strange as it may seem, sometimes businesses do things that don't
directly contribute to profits.  Maybe the owner just _believes in the
idea_, and is willing to support it.

--Jimbo


from_name: Robert Graham Merkel

subject: [Wikipedia-l] 1911 britannica online

in_reply_to: <20020314052041.31067.qmail@www.llamacom.com>; from kband@www.llamacom.com on Thu, Mar 14, 2002 at 16:20:41 +1100

body:
On Thu, 14 Mar 2002 16:20:41 kband at www.llamacom.com wrote:
> >
> > Someone published the 1911 britannica online:
> >
> > http://1911encyclopedia.org/
> >
> All right, now we're talking!

Whilst this is very good news, on my limited sampling
it's not something I'd be sucking into the Wikipedia
using a conversion script.

The first article I examined was one about W.G. Grace
(19th century English cricketer who revolutionised the
game).  Only the first paragraph of what appears to be
a much longer article is there, and it is laden with
typos - some probably from the scanning process, others
from the original.

Still, it's an outstanding resource and I look forward to
incorporating a lot of material from it.

--
------------------------------------------------------------
Robert Merkel	                           rgmerk at mira.net

Go You Big Red Fire Engine
-- Unknown Audience Member at Adam Hills standup gig
------------------------------------------------------------


subject: [Wikipedia-l] 1911 encyclopedia

body:
They have legal notice at the site:

Legal Notice:

The information on this site is protected against unauthorized use
by others by  a variety of laws and statutes. For more information
on what you can and cannot do with the information on this site,
please contact

editor at 1911encyclopedia.com

Thanks

and they are using the name with permition:

Special thanks goes out to the people at Encyclopaedia Britannica
who have graciously granted permission to us to use this material
and their name on this site. Without their kind assistance, this
site could not be.

We urge anyone interested in a modern, up-to-date encyclopedia to
visit their site at http://www.eb.com.

Joao
http://www.nonio.com

Jimmy Wales wrote:
>
> It is my understanding, from the edition of the 'A' volume of this
> encyclopedia published by the fine folks over at Project Gutenburg,
> that it is "uncool" to use the name "Britannica" in association with
> this book.  The reason is that although the _copyright_ has indeed
> properly expired, the _trademark_ is still that of an active company.
> So, while the text is fair game, the name is not.
>
> I'm cc:'ing this message to editor at 1911encyclopedia.org, just so that
> person will know about this.  But my opinion is not very
> authoritative.  I'd just suggest getting the word 'Britannica' off the
> site _in the logo_, while feeling free to use the word in a text
> explanation of where this information comes from.  (My understanding
> is that it's o.k. to say a trademark in just about any context, but
> not o.k. to use it in a fashion that might mislead people into
> thinking that you are the organization in question.)
>
> The site, as it stands, doesn't really make clear that the people
> behind it are not the official Britannica people.
>
> What a wonderful thing to have online.
>
> --Jimbo
>
> [Wikipedia-l]
> To manage your subscription to this list, please go here:
> http://www.nupedia.com/mailman/listinfo/wikipedia-l

--
jo?o
http://www.nonio.com
http://www.explicacoes.com


subject: [Wikipedia-l] 1911 britannica online

body:
On Thu, 14 Mar 2002, Tim Chambers wrote:

> I'm even more encouraged than ever that the 'pedia can earn a
> reputation as an authoratative source after having skimmed through
> http://1911encyclopedia.org/. Not only is this particular Web site's
> content a junk pile (scanning errors and sloppy cataloging), but it
> reminded me that some of the 1911's entries are surprisingly thin.
Well these are a bit harsh words. At least it is a valuable source of open
content we can use. And to use it means to integrate edit them and
integrate the content which is still valuable. For various issues the 1911
edition is valuable. It will help to come up with let's say 10'000
additional entries more easily.

> Surely we can do better! (I think we already have --
> http://www.wikipedia.com/wiki/Brilliant_prose.)

_yes_

HJH


subject: [Wikipedia-l] 1911 encyclopedia

body:
Jimmy Wales wrote:
>
> Oh, very good, and I'm very glad to hear that about the Britannica people!
> Very reasonable and gracious of them!

The 1911encyclopedia.org domain was registered by

Pagewise, inc
byron reese
815-A Brazos St. #PMB 534
austin, TX 78701
US
Phone: 512-478-4566
Fax..: 512-478-4726
Email: byron at pagewise.com

Pagewise.com is a for profit organization:

http://www.pagewise.com/

that owns the http://www.WriteForCash.com/ site.

Probably, they will claim some kind of copyrights over the
digitalisation of the 1911 Encyclopedia.

--
jo?o
http://www.nonio.com


subject: [Wikipedia-l] 1911 encyclopedia

body:
On Thu, 14 Mar 2002, Jo?o M?rio Miranda wrote:

[snip]

> The 1911encyclopedia.org domain was registered by
>
> Pagewise, inc
> byron reese
> 815-A Brazos St. #PMB 534
> austin, TX 78701
> US
> Phone: 512-478-4566
> Fax..: 512-478-4726
> Email: byron at pagewise.com
>
> Pagewise.com is a for profit organization:
>
> http://www.pagewise.com/
>
> that owns the http://www.WriteForCash.com/ site.
>
> Probably, they will claim some kind of copyrights over the
> digitalisation of the 1911 Encyclopedia.
>
That's not possible from the point of view of the copyright law as
scanning is not a creative expression and the copyright of the original
material has expired. The only thing which is possible is to claim
copyright on additional markup, layout, presentation and additional
graphics.

HJH


from_name: Lars Aronsson

subject: [Wikipedia-l] 1911 encyclopedia

body:
Jo?o M?rio Miranda wrote:
> Pagewise.com is a for profit organization:
> Probably, they will claim some kind of copyrights over the
> digitalisation of the 1911 Encyclopedia.

Does anybody know if this text is the same as the one from Project
Gutenberg, or a new scan?

If two independent scans exist, it might make sense to download both
and run "diff" as a first step to create an improved version.

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikipedia-l] 1911 encyclopedia

body:
>> Probably, they will claim some kind of copyrights over the
>> digitalisation of the 1911 Encyclopedia.

>That's not possible from the point of view of the copyright law as
>scanning is not a creative expression and the copyright of the
>original material has expired. The only thing which is possible is to
>claim copyright on additional markup, layout, presentation and
>additional graphics.

I think they fully deserve the legal copyright on their markup,
layout, presentation and additional graphics :-)

More seriously: even if the quality of the scanned text will be
improved, I do hope that nobody will whip together a script to suck
the 1911 contents into Wikipedia wholesale. Every article needs to
be read, checked for OCR mistakes, accuracy and outdated language,
wikified and merged with whatever material we already have on the
topic. Many technical articles will be useless because the terminology
has changed. In mathematics for instance, it is next to impossible to
read a paper that was written in the 19th century.

Axel


subject: [Wikipedia-l] 1911 encyclopedia

body:
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://lists.wikimedia.org/pipermail/wikipedia-l/attachments/20020315/9490691b/attachment.diff


from_name: Brion L. VIBBER

subject: [Wikitech-l] Polish Wikipedia and PHP

body:
On ven, 2002-03-15 at 12:23, Tomasz Wegrzanowski wrote:
> I have two issues with import script:
>
> 1)
> It still tries to import db.orig files from UseMod wikipedia.
> Patch attached. These warnings say when there is something wrong.

Committed.

> 2)
> "Most wanted" page doesn't seem to be up-to-date after import.
> Many of listed pages actually exist.
> Even editing them or pages that link to them doesn't help.
>
> Affected pages are:
> 4th place)  53 artykulow chce Stany Zjednoczone/Podzia&lstroke; terytorialny
> 7th place)  40 artykulow chce Zwi&aogonek;zek chemiczny
> 11th place) 37 artykulow chce Cz&aogonek;steczka
>
> And others.
>
> What they have in common is that they contain Polish diactrics.

Aha, was missing some $recodeCharset in getAllTopics(). Fixed.

-- brion vibber (brion @ pobox.com)


from_name: Vicki Rosenzweig

subject: [Wikipedia-l] 1911 encyclopedia

body:
At 07:39 PM 3/15/02 +0100, you wrote:
> >> Probably, they will claim some kind of copyrights over the
> >> digitalisation of the 1911 Encyclopedia.
>
>Well, how could they tell where we got it? We could have typed the stuff in
>by hand, after all...
>
> > More seriously: even if the quality of the scanned text will be improved,
>I do hope that nobody will whip together a script to suck the 1911 contents
>into Wikipedia wholesale.

Indeed. I had to google to update the one on the Pribilof islands--the 1911
version ended with the months during which seal hunting "is" legal, meaning
when it was legal in 1911. This has no connection to current US law.

I assume there will be similar changes in many geographical entries.
--
Vicki Rosenzweig
vr at redbird.org
http://www.redbird.org


subject: [Wikipedia-l] 1911 encyclopedia

body:
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://lists.wikimedia.org/pipermail/wikipedia-l/attachments/20020315/3484bd6a/attachment.diff


subject: [Wikipedia-l] 1911 encyclopedia

body:
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://lists.wikimedia.org/pipermail/wikipedia-l/attachments/20020315/371052ff/attachment.diff


from_name: Magnus Manske

subject: [Wikipedia-l] 1911 encyclopedia

body:
>> Probably, they will claim some kind of copyrights over the
>> digitalisation of the 1911 Encyclopedia.

Well, how could they tell where we got it? We could have typed the stuff in
by hand, after all...

> More seriously: even if the quality of the scanned text will be improved,
I do hope that nobody will whip together a script to suck the 1911 contents
into Wikipedia wholesale.

*If* there's going to be a script, I suggest the following:
- Only create entrys that are empty on the wiki; keep the others for manual
"prescan" (on existing topics, we might not need the old text at all)
- Add some text (in red?) as first and last line of the text, stating where
it is from, handle with care etc.
- Let the script create a list of these articles as [[links]], then add that
list as "wikipedia:1911 conversion" (or something) so we can mark off the
entries that were proofread by real living humans.

ALternatively, we could create a read-only "1911" namespace, dump it all
there and do the conversion by hand.

Magnus


from_name: Tomasz Wegrzanowski

subject: [Wikitech-l] Polish Wikipedia and PHP

body:
I have two issues with import script:

1)
It still tries to import db.orig files from UseMod wikipedia.
Patch attached. These warnings say when there is something wrong.

2)
"Most wanted" page doesn't seem to be up-to-date after import.
Many of listed pages actually exist.
Even editing them or pages that link to them doesn't help.

Affected pages are:
4th place)  53 artykulow chce Stany Zjednoczone/Podzia&lstroke; terytorialny
7th place)  40 artykulow chce Zwi&aogonek;zek chemiczny
11th place) 37 artykulow chce Cz&aogonek;steczka

And others.

What they have in common is that they contain Polish diactrics.

----

Anyway these are minor problems.
Jimbo, could you set up test site pl2.wikipedia.com with PHP
script and data imported from pl.wikipedia.com ?
Preferably after this patch is commited to CVS.
-------------- next part --------------
--- convertWiki2SQL.php.orig	Fri Mar 15 20:25:39 2002
+++ convertWiki2SQL.php	Fri Mar 15 20:37:55 2002
@@ -347,10 +347,14 @@
 				$a = getTopics ( "$dir/$entry" ) ;
 				foreach ( $a as $x ) array_push ( $ret , "$entry/$x" ) ;
 			} else {
-				$x = substr ( $entry , 0 , strlen ( $entry ) - 3 ) ;
-				array_push ( $ret , $x ) ;
+				if (substr ($entry, strlen ( $entry ) - 3, 3) == '.db') {
+				    $x = substr ( $entry , 0 , strlen ( $entry ) - 3 ) ;
+				    array_push ( $ret , $x ) ;
+				} else {
+				    print "Warning: File \"$entry\" doesn't seem to contain an article
\n" ;
 				}
 			}
+		}
 	}
 	closedir($mydir);


from_name: Ian

subject: [Wikipedia-l] 1911 encyclopedia

body:
On Thu, 14 Mar 2002, Hannes Hirzel hirzel at spw.unizh.ch XXXXXXXXXXXXXXXXXXXXX wrote:
> That's not possible from the point of view of the copyright law as
> scanning is not a creative expression and the copyright of the original
> material has expired. The only thing which is possible is to claim
> copyright on additional markup, layout, presentation and additional
> graphics.
>
> HJH

Then how come in Project Gutenberg, as I understand it, you have to use an
actual old copy of the book? If you want to digitize Mark Twain, you can't
pick up a copy at Wal-Mart, you have to go to the used book store.

Perhaps I'm mistaken.

Ian Monroe
http://mlug.missouri.edu/~eean/


subject: [Wikipedia-l] 1911 encyclopedia

body:
On Sat, 16 Mar 2002, Ian wrote:

> On Thu, 14 Mar 2002, Hannes Hirzel hirzel at spw.unizh.ch XXXXXXXXXXXXXXXXXXXXX wrote:
> > That's not possible from the point of view of the copyright law as
> > scanning is not a creative expression and the copyright of the original
> > material has expired. The only thing which is possible is to claim
> > copyright on additional markup, layout, presentation and additional
> > graphics.
> >
> > HJH
>
> Then how come in Project Gutenberg, as I understand it, you have to use an
> actual old copy of the book? If you want to digitize Mark Twain, you can't
> pick up a copy at Wal-Mart, you have to go to the used book store.
>
> Perhaps I'm mistaken.
>
> Ian Monroe
> http://mlug.missouri.edu/~eean/
>

Because it is possible that the Wal-Mart copy is edited, this means has
modernized spelling and perhaps additional footsnotes or remarks. The
editing process makes it a new work of which again somebody holds the
copyright.

So to be sure you are on the safe side you better use the 70 year old
edition.
If you are sure that the Wal-Mart edition is character by character
identical to the 70 year old version you could use it.

HJH


subject: [Wikipedia-l] 1911 encyclopedia

body:
Hannes Hirzel wrote:
>
> On Sat, 16 Mar 2002, Ian wrote:
>
> > On Thu, 14 Mar 2002, Hannes Hirzel hirzel at spw.unizh.ch XXXXXXXXXXXXXXXXXXXXX wrote:
> > > That's not possible from the point of view of the copyright law as
> > > scanning is not a creative expression and the copyright of the original
> > > material has expired. The only thing which is possible is to claim
> > > copyright on additional markup, layout, presentation and additional
> > > graphics.
> > >
> > > HJH
> >
> > Then how come in Project Gutenberg, as I understand it, you have to use an
> > actual old copy of the book? If you want to digitize Mark Twain, you can't
> > pick up a copy at Wal-Mart, you have to go to the used book store.
> >
> > Perhaps I'm mistaken.
> >
> > Ian Monroe
> > http://mlug.missouri.edu/~eean/
> >
>
> Because it is possible that the Wal-Mart copy is edited, this means has
> modernized spelling and perhaps additional footsnotes or remarks. The
> editing process makes it a new work of which again somebody holds the
> copyright.
>
> So to be sure you are on the safe side you better use the 70 year old
> edition.
> If you are sure that the Wal-Mart edition is character by character
> identical to the 70 year old version you could use it.

The problem, I think, is that publishers make all kinds of
unsuported claims. For example, from the Bibliomania.com FAQ:

----
What is the copyright status of texts on Bibliomania.com?

Most texts on our site are in the public domain. However
Bibliomania.com Ltd has copyright in the HTML versions we have
created for our web site. You are free to download these texts
for personal use, but they may not be used for any commercial
purpose, or republished in any form (including on the internet)
without our prior email permission. Bibliomania.com Ltd has and
will take legal action worldwide to protect its rights.
----

--
jo?o
http://www.nonio.com


subject: [Wikitech-l] Polish PHP site ?

in_reply_to: <20020318172019.GA6469@tavaiah>; from taw@users.sourceforge.net on Mon, Mar 18, 2002 at 06:20:19PM +0100

body:
Yes, I am sick today and staying home, which means that I will
probably get MORE work done, ha ha.

--Jimbo

Tomasz Wegrzanowski wrote:

> Jimbo:
> When will you be able to setup a test site for Polish Wikipedia
> on PHP script ? If you don't have time, we may set it up somewhere
> else (just for testing, not any fork).
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l at ross.bomis.com
> http://ross.bomis.com/mailman/listinfo/wikitech-l


subject: [Wikipedia-l] Fork within a language

in_reply_to: <200203182157.g2ILvKg23906@levi.aronsson.se>; from lars@aronsson.se on Mon, Mar 18, 2002 at 10:57:20PM +0100

body:
Lars Aronsson wrote:
> Forgive me if this gets philosophical, and not very practical.

No, I think it is very interesting!

> There are, of course, already other Wikis.  http://c2.com/ (the
> original Wiki, founded in 1994) has the Portland Pattern Repository,
> devoted to object-oriented software development and eXtreme
> Programming.  Some of the information there might overlap with
> Wikipedia, but most of it would be considered too non-encyclopedic if
> it was suddenly copied to Wikipedia.

Also, I don't think their content is released under an open license.

> Could a Wiki devoted to history have a place outside of Wikipedia?
> When describing London, it would focus on the city's historic
> features, not on the facets of today's London.  Then again,
> Wikipedia's entries on many things are focused on history.  It is
> almost as if Wikipedia is that history Wiki.  History, after all, is
> so much more in line with the contents of an encyclopedia than is
> object-oriented software development.

Right.  But imagine a travel-guide wiki.  The information would not necessarily
be NPOV as in an encyclopedia -- it could be and probably should be quirky and
opinionated.  Such a guide might use wikipedia content as a foundation, but we
probably wouldn't (due to our NPOV interests) be able to use much of what they
generated.  No problem.

> Could a leftist-point-of-view Wiki exist side by side with Wikipedia?
> It would carefully point out any misuse of power, and list activist
> and political groups.  Its logotype could be a hammer-and-sickle or
> simply a red star on white background.  (There is already a leftist
> encyclopedia (non-Wiki) in Danish on http://www.leksikon.org/)

Boy, I personally find it very funny to imagine a
leftist-point-of-view wiki pointing out "any misuse of power".  :-)
Perhaps that should say "any misuse of power not sanctioned by
leftists".

> A youth culture Wiki might list all the hot dance clubs in London, but
> forget the British Museum.  (Does Wikipedia list any clubs at all?)
> The entire Wiki could be white text on black background.
>
> I think target groups, focus, design, logotypes would be different
> for each one of these Wikis.  Just like websites are different today.
> Some titles (like "London") would exist in several of them, but with a
> different slant.  Some titles (like "British Museum") might only exist
> in Wikipedia.
>
> If all of these Wikis existed side by side, how would Wikipedia best
> take advantage of this expanded network?  Should Wikipedia be its
> backbone, or try to be self-sufficient, ignoring the outside world?
>
> Just to be clear: I'm not suggesting a fork of Wikipedia.

Right.

I think that we have to stick to our central mission: the encyclopedia.  The idea
of "encyclopedia" is one that we all understand, and I think it has been miraculous
in terms of letting people from possibly wildly different viewpoints work together.
It gives us a simple test for whether or not something belongs here.

But by no means do I think this exhausts the possibilities for vibrant wiki communities.

--Jimbo


subject: [Wikipedia-l] Fork within a language

body:
Lars wrote:
> If all of these Wikis existed side by side, how would Wikipedia best
> take advantage of this expanded network?  Should Wikipedia be its
> backbone, or try to be self-sufficient, ignoring the outside world?

There already has been discussion about how to "connect" wikis. UseMod
is a leading advocate for an InterWiki protocol. I updated
http://www.wikipedia.com/wiki/InterWiki with a pointer to more
information. Start there.

<>< Tim

__________________________________________________
Do You Yahoo!?
Yahoo! Sports - live college hoops coverage
http://sports.yahoo.com/


subject: [Wikipedia-l] Spinoff project

in_reply_to: <002a01c1cedd$963a9340$6401a8c0@april>; from aprilrosanina@charter.net on Mon, Mar 18, 2002 at 07:32:44PM -0500

body:
Rosa Williams wrote:

> Also, I'd like to ask something of an etiquette question - to Jimbo
>Wales in particular, and to the Wikipedians in general. How do people
>feel about Wikipedia material being ported over and adapted for use in
>the textbook project?  I know what the "public domain" rules are, but
>I don't want to do anything that would create bad feeling or set up
>for trouble between the sites. I do, of course, have Wikipedia linked
>back from a couple of places. ;)

I think it's wonderful!

I think that the Right Thing To Do is to link back to us on any pages that have
our stuff as a source.  Something as simple as "This page partly based on
http://www.wikipedia.com/wiki/Whatever" would be great.

I don't intend to be a real stickler about this sort of thing, meaning
the linkback.  The one thing that I don't want, is for Microsoft or
AOL or Yahoo or someone like that to take our content without
attribution.  I think that's going to be a huge opportunity for the
free content to grow like crazy, and so letting people know we exist
is great.

And we should have a prominent page somewhere for "fellow travelers", i.e. other
people who are doing neat things with wiki and a free license, like you, and like
the Sevilla group, etc.

--Jimbo


subject: [Wikipedia-l] 1911 encyclopedia

body:
At 04:45 AM 3/18/02 +0000, Joao wrote:

>The problem, I think, is that publishers make all kinds of
>unsuported claims. For example, from the Bibliomania.com FAQ:
>
>----
>What is the copyright status of texts on Bibliomania.com?
>
>Most texts on our site are in the public domain. However
>Bibliomania.com Ltd has copyright in the HTML versions we have
>created for our web site. You are free to download these texts
>for personal use, but they may not be used for any commercial
>purpose, or republished in any form (including on the internet)
>without our prior email permission. Bibliomania.com Ltd has and
>will take legal action worldwide to protect its rights.

I'm not a lawyer (standard disclaimer), but it's possible that they do
have a claim on the HTML code. Computer code is considered copyrightable,
though algorithms aren't.

On the other hand, if you were to grab one of those files and strip the HTML
off, you'd probably be in the clear. If I understand the matter, which I
may not.
--
Vicki Rosenzweig
vr at redbird.org
http://www.redbird.org


subject: [Wikipedia-l] Fork within a language

body:
On Mon, 18 Mar 2002, Lars Aronsson lars at aronsson.se XXXXXXXXXXXXXXXXXXXXX wrote:
[snip]
> Could a Wiki devoted to history have a place outside of Wikipedia?
> When describing London, it would focus on the city's historic
> features, not on the facets of today's London.  Then again,
> Wikipedia's entries on many things are focused on history.  It is
> almost as if Wikipedia is that history Wiki.  History, after all, is
> so much more in line with the contents of an encyclopedia than is
> object-oriented software development.

I think the history of London certainly has a place in an encyclopedia.
One problem I think Wikipedia is facing, but is sorting out, is how to
deal with large amounts of text about a subject. I don't think we should
force people to decide between linking to page about current London and
its history. The current solution seems to be to have an opening page
[[London]] and then [[London (History)]] or [[History of London]].

> Could a leftist-point-of-view Wiki exist side by side with Wikipedia?
> It would carefully point out any misuse of power, and list activist
> and political groups.  Its logotype could be a hammer-and-sickle or
> simply a red star on white background.  (There is already a leftist
> encyclopedia (non-Wiki) in Danish on http://www.leksikon.org/)

Um, no. If it did, it would be sit next to in the same way Nupedia sits
next to Wikipedia.

> A youth culture Wiki might list all the hot dance clubs in London, but
> forget the British Museum.  (Does Wikipedia list any clubs at all?)
> The entire Wiki could be white text on black background.
>
> I think target groups, focus, design, logotypes would be different
> for each one of these Wikis.  Just like websites are different today.
> Some titles (like "London") would exist in several of them, but with a
> different slant.  Some titles (like "British Museum") might only exist
> in Wikipedia.
>
> If all of these Wikis existed side by side, how would Wikipedia best
> take advantage of this expanded network?  Should Wikipedia be its
> backbone, or try to be self-sufficient, ignoring the outside world?
>
> Just to be clear: I'm not suggesting a fork of Wikipedia.
>

If such places are created and released their stuff under a usable
license or public domain, we could list them on the Public Domain
resources page with the rest of them. None of them are really within the
mission of Wikipedia.

Ian Monroe
http://mlug.missouri.edu/~eean/


subject: [Wikitech-l] Polish PHP site ?

body:
Jimbo:
When will you be able to setup a test site for Polish Wikipedia
on PHP script ? If you don't have time, we may set it up somewhere
else (just for testing, not any fork).


from_name: Rosa Williams

subject: [Wikipedia-l] Spinoff project

body:
As those of you who read the meta pages will know, I'm applying the Wiki idea to a slightly different project, with the object of creating a "science hypertextbook".  Like Wikipedia, this will all be under GNU Free Documentation License.

I note that the Wikipedians muster a good deal of technical expertise among themselves, and I'd like to invite anyone interested in a more teaching-oriented approach (i.e. building from basic subjects to advanced ones)  to participate.  It's at http://www.renaissoft.com/april/cgi-bin/wiki.pl for those who are interested.

Also, I'd like to ask something of an etiquette question - to Jimbo Wales in particular, and to the Wikipedians in general. How do people feel about Wikipedia material being ported over and adapted for use in the textbook project?  I know what the "public domain" rules are, but I don't want to do anything that would create bad feeling or set up for trouble between the sites. I do, of course, have Wikipedia linked back from a couple of places. ;)

Thanks for any constructive input,
-- April
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.wikimedia.org/pipermail/wikipedia-l/attachments/20020318/849b4357/attachment.htm


subject: [Wikipedia-l] Spinoff project

body:
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://lists.wikimedia.org/pipermail/wikipedia-l/attachments/20020318/a95c96b8/attachment.diff


subject: [Wikipedia-l] Spinoff project

body:
Thanks; you also need to add a clear copyright notice to
the site itself.  In the absence of clear notice, the content
of the site is by default fully copyrighted, and using GFDL
content there would be a violation.  I know this legal stuff
is a real nuisance--I'd just as soon it was all repealed.
But while it's still the law of the land, you do have to
pay attention to the little details.

>----- Original Message -----
>From: <lcrocker at nupedia.com>
>To: <wikipedia-l at nupedia.com>
>Sent: Monday, March 18, 2002 10:57 PM
>Subject: Re: [Wikipedia-l] Spinoff project
>
>> Rosa's project specifically says that any materials to be
>> posted on her site must be in the public domain, so unless she
>> changes that she won't be able to use anything from Wikipedia
>> (except for stuff she can get independently from authors like
>> me who release their work to the public domain as well as
>> putting it here).
>
>Hm - checking back, I did say that. I was, of course, thinking of sources
>other than FDL-released sources, I'll amend it to say "public domain or
>otherwise legally available for GNU FDL release". Thanks for pointing
that
>wrinkle out.
>
>--April
>
>[Wikipedia-l]
>To manage your subscription to this list, please go here:
>http://www.nupedia.com/mailman/listinfo/wikipedia-l


subject: [Wikipedia-l] Fork within a language

body:
I'm trying to understand where the boundaries for Wikipedia are.
Forgive me if this gets philosophical, and not very practical.

There has been a lot of talk about languages that fork away from the
Wikipedia project.  But what about forks within the language?  Will
the English Wikipedia always be the single source, or when will it be
reasonable to set up another Wiki?

There are, of course, already other Wikis.  http://c2.com/ (the
original Wiki, founded in 1994) has the Portland Pattern Repository,
devoted to object-oriented software development and eXtreme
Programming.  Some of the information there might overlap with
Wikipedia, but most of it would be considered too non-encyclopedic if
it was suddenly copied to Wikipedia.

Could a Wiki devoted to history have a place outside of Wikipedia?
When describing London, it would focus on the city's historic
features, not on the facets of today's London.  Then again,
Wikipedia's entries on many things are focused on history.  It is
almost as if Wikipedia is that history Wiki.  History, after all, is
so much more in line with the contents of an encyclopedia than is
object-oriented software development.

Could a leftist-point-of-view Wiki exist side by side with Wikipedia?
It would carefully point out any misuse of power, and list activist
and political groups.  Its logotype could be a hammer-and-sickle or
simply a red star on white background.  (There is already a leftist
encyclopedia (non-Wiki) in Danish on http://www.leksikon.org/)

A youth culture Wiki might list all the hot dance clubs in London, but
forget the British Museum.  (Does Wikipedia list any clubs at all?)
The entire Wiki could be white text on black background.

I think target groups, focus, design, logotypes would be different
for each one of these Wikis.  Just like websites are different today.
Some titles (like "London") would exist in several of them, but with a
different slant.  Some titles (like "British Museum") might only exist
in Wikipedia.

If all of these Wikis existed side by side, how would Wikipedia best
take advantage of this expanded network?  Should Wikipedia be its
backbone, or try to be self-sufficient, ignoring the outside world?

Just to be clear: I'm not suggesting a fork of Wikipedia.

--
  Lars Aronsson (lars at aronsson.se)
  Aronsson Datateknik
  Teknikringen 1e, SE-583 30 Linuxk?ping, Sweden
  tel +46-70-7891609
  http://aronsson.se/ http://elektrosmog.nu/ http://susning.nu/


subject: [Wikipedia-l] Spinoff project

body:
----- Original Message -----
From: <lcrocker at nupedia.com>
To: <wikipedia-l at nupedia.com>
Sent: Monday, March 18, 2002 10:57 PM
Subject: Re: [Wikipedia-l] Spinoff project

> Rosa's project specifically says that any materials to be
> posted on her site must be in the public domain, so unless she
> changes that she won't be able to use anything from Wikipedia
> (except for stuff she can get independently from authors like
> me who release their work to the public domain as well as
> putting it here).

Hm - checking back, I did say that. I was, of course, thinking of sources
other than FDL-released sources, I'll amend it to say "public domain or
otherwise legally available for GNU FDL release". Thanks for pointing that
wrinkle out.

--April


from_name: Axel Boldt

subject: [Wikitech-l] sample database dump with new schema

body:
Hi,

I took the old database test dump
http://wikipedia.sourceforge.net/fpw/newiki.sql.gz and applied the
database scheme changes from updSchema.sql and updLinks.php to create
a new dump which conforms with our current database scheme, so that
new hackers have something to play with. Where can I upload it?

Axel


subject: [Wikitech-l] [gblanchard@arcsy.co.jp: French & Japanese wiki !!]

body:
I will write and ask him to subscribe here.  Let's give him a big welcome.

----- Forwarded message from Guillaume Blanchard <gblanchard at arcsy.co.jp> -----

From: "Guillaume Blanchard" <gblanchard at arcsy.co.jp>
Date: Tue, 19 Mar 2002 16:21:51 +0900
To: <jwales at bomis.com>
Subject: French & Japanese wiki !!

Hello James,
I'm a french progrmmer living in Japan (Yokohama).
I joined the french wikipedia project last week.
It's so exciting !!
I already wrote some article about video games & egyptology.
Speaking about Wiki around me, I founded a japanese friend who wnat to help
and grow up Wiki-japan.
The probleme is the CGI actualy in the homepage is an english version.
I talk with Brion Vibber and Yussefsan about this, they say a PHP version is
under construction. I understood the way of how to help programming in
sourceforge, but how to update the server version of fr.wikipedia or
ja.wikipedia ?
Physicaly, the database is on sourceforge ?
Can I access the data base from my own homepage ?
Cheers,

Guillaume (alias Aoineko)


subject: [Wikipedia-l] Spinoff project

body:
----- Original Message -----
From: <lcrocker at nupedia.com>
To: <wikipedia-l at nupedia.com>; "Rosa Williams" <aprilrosanina at charter.net>
Sent: Tuesday, March 19, 2002 12:18 AM
Subject: Re: Re: [Wikipedia-l] Spinoff project

> Thanks; you also need to add a clear copyright notice to
> the site itself.  In the absence of clear notice, the content
> of the site is by default fully copyrighted, and using GFDL
> content there would be a violation.  I know this legal stuff
> is a real nuisance--I'd just as soon it was all repealed.
> But while it's still the law of the land, you do have to
> pay attention to the little details.

That one I *did* remember - it's at the bottom, marked "Copyleft", with all
the details on that link. I may add some print to the main page to make that
even clearer, though.  Again, the advice is appreciated - anything that
helps fortify the site against future problems is a good thing!
-- April


subject: [Wikitech-l] Minor conversion script problem

body:
On mar, 2002-03-19 at 09:33, Tomasz Wegrzanowski wrote:
> Conversion script tries to convert q{/foo/} to q{[[Articlename/foo|/foo]]/}
> But q{/foo/} are not subpage links in UseMod wiki.
> They aren't links at all.

But q{/Foo/} are links. I've added a quick fix to the conversion script
to only do automatic link conversion when they start with an uppercase
letter; this should help with many (but not all) cases.

> q{/foo/} are actually used to write down pronunciacion in a few
> articles. But maybe that should be used that way.

Also, the conversion script shouldn't touch these inside <nowiki> or
<pre> tags, but I don't know a nice clean way to do that with regexps.

-- brion vibber (brion @ pobox.com)


subject: [Wikipedia-l] Fork within a language

body:
At 06:12 PM 3/19/02 +0200, you wrote:
>On Monday 18 March 2002 23:57, Lars Aronsson wrote:
>
> > I'm trying to understand where the boundaries for Wikipedia are.
> > Forgive me if this gets philosophical, and not very practical.
> >
> > Could a Wiki devoted to history have a place outside of Wikipedia?
> > When describing London, it would focus on the city's historic
> > features, not on the facets of today's London.  Then again,
> > Wikipedia's entries on many things are focused on history.  It is
> > almost as if Wikipedia is that history Wiki.  History, after all, is
> > so much more in line with the contents of an encyclopedia than is
> > object-oriented software development.
>
>Assuming FDL for all instances, perhaps cut'n'pasting from a history wiki
>into wikipedia might be easier than the other way round.
>
> > Could a leftist-point-of-view Wiki exist side by side with Wikipedia?
>
>Only on the sinister side ... <g>
><<OK, how many of you figured that one out?>>
>
> > It would carefully point out any misuse of power
>
>You mean as in [[Stalin]], [[Pol Pot]], [[Mao]] . . . ?

As in Emma Goldman pointing out the flaws of socialist Russia around 1920.
Or were you thinking of the way the U.S. allied with [[Stalin]]? (This stuff
really isn't one-sided, and I'm getting tired of reading left-bashing on a
completely unrelated thread.)

> >, and list activist
> > and political groups.
>
>Expect lots and lots of vandalism.

A valid and important point. On the other hand, the wiki model
can be used in less open-to-everyone modes than the wikipedia.

> > A youth culture Wiki might list all the hot dance clubs in London, but
>
>Feasible, but youth "culture" changes so quickly that you might as well
>start from scratch every two years or so.

What you'd do, I think, is change as it went: add new clubs, edit/update
descriptions, remove those that had closed (or tag them as "closed, here
are some historic notes").
--
Vicki Rosenzweig
vr at redbird.org
http://www.redbird.org