UnaMesa : saq - all changes

Po2JS

SaqImtiaz — Sun, 29 Aug 2010 13:47:00 GMT

...

spidermonkey (apt-get install spidermonkey-bin)
gettext (apt-get install gettext)
Python modules (included in Python 2.6+) : simplejson and gettext
Notes
I've commented the script so you can follow what it is doing. Please note that:
The script does not currently have much in the way of error handling and we would need to add some before using this in production.
the script is currently hardcoded for the French translation, but could easily be modified to accept an argument defining the language to work on
it currently downloads the PO file from Pootle but this would be removed before we used it in production. Instead, this script would be fired by a SVN post-commit hook every time an admin "pushed" changes to SVN from Pootle and would work on the copy of the PO file in SVN.

Po2JS

SaqImtiaz — Sun, 29 Aug 2010 13:37:35 GMT

...

This will download the latest French PO file from Pootle on EC2 and convert it into a french translation plugin: fr-translation.js
You can then cook this into a TiddlyWiki (which includes Martins changes - patch:http://dl.dropbox.com/u/385674/gettext.diff).

...

the gettext ~~line.~~line
jslib: jquery.gettext-1.0.4.js
jslib: fr-translation.js
jquery: jQuery.encoding.digests.sha1.js
jquery: jQuery.twStylesheet.js
Dependencies
spidermonkey (apt-get install spidermonkey-bin)

Po2JS

SaqImtiaz — Sun, 29 Aug 2010 13:36:34 GMT

The scripts and associated files can be downloaded here: http://dl.dropbox.com/u/385674/po2js.zip
Pootle link: http://ec2-75-101-153-235.compute-1.amazonaws.com:8080/
Instructions
Extract and run po2js.sh.

Po2JS

SaqImtiaz — Sun, 29 Aug 2010 13:36:02 GMT

The scripts and associated files can be downloaded here: http://dl.dropbox.com/u/385674/po2js.zip
Instructions
Extract and run po2js.sh.
This will download the latest French PO file from Pootle on EC2 and convert it into a french translation plugin: fr-translation.js
You can then cook this into a TiddlyWiki (which includes Martins changes - patch:http://dl.dropbox.com/u/385674/gettext.diff).
Because of previously discussed load order issues it must be included in the source of the file immediately after the jquery.gettext plugin. I've done this by modifying my copy of http://svn.tiddlywiki.org/Trunk/core/jquery/plugins/split.recipe with the line "jslib: fr-translation.js" immediately after the gettext line.
Dependencies
spidermonkey (apt-get install spidermonkey-bin)
gettext (apt-get install gettext)
Notes
I've commented the script so you can follow what it is doing. Please note that:
the script is currently hardcoded for the French translation, but could easily be modified to accept an argument defining the language to work on
it currently downloads the PO file from Pootle but this would be removed before we used it in production. Instead, this script would be fired by a SVN post-commit hook every time an admin "pushed" changes to SVN from Pootle and would work on the copy of the PO file in SVN.
The current approach for turning the JSON obtained from the PO file into JavaScript that would work with the jquery.gettext plugin is very simplistic and concatenates a plugin-header, the JSON and a plugin-footer to create the plugin. (Files in templates folder). Once we are sure how we intend to include the plugin in TiddlyWiki and what metadata should be included, a cleaner approach using templates can be devised.

Supporting localized TiddlyWiki distributions

122.49.210.54 — Fri, 29 Jan 2010 03:48:46 GMT

...

With regards to linking to resources in other languages, I recommend we avoid using country specific domains. Not only are we actually providing language specific distributions and not geographically specific, but the use of country specific domains may lead to issues with having to obtain and maintain domains for a growing number of countries.
Instead it seems appropriate to adopt the widely used practice of suffixing the url with the language code. Eg: tiddlywiki.com/en and tiddlywiki.com/es

...

a simple ~~site~~document that provides
Ideally we would want to update these sites every time there is a new TiddlyWiki release not only updating the TiddlyWiki version used but updating a "What's New" tiddler.
If we start providing officially supported localized distributions of TiddlyWiki the upgrade mechanism should ideally be updated as well. Currently it only updates the TiddlyWiki core but could be adapted to also update the appropriate translation plugin.

pblog notes

SaqImtiaz — Wed, 23 Dec 2009 20:17:41 GMT

...

templates: http://cdent.tumblr.com/search/templates
handlers: http://github.com/tiddlyweb/tiddlyweb-plugins/blob/master/cachinghoster/cachinghoster.py#L128
filters: http://tiddlyweb.peermore.com/wiki/recipes/editor/tiddlers/filter and http://github.com/tiddlyweb/tiddlyweb/blob/master/test/test_bag_filtering.py#L21
$ sudo apt-get install python-pip
$ pip install -U pip==dev

pblog notes

SaqImtiaz — Wed, 23 Dec 2009 18:45:52 GMT

...

RSS
if sorting by titles, 1 vs 001 ?
deployment and URI structure.
/ standard apache handling
/project1/ tweb
/project2/ tweb
TiddlyPhotog
* individual entries as static HTML pages

pblog notes

SaqImtiaz — Wed, 23 Dec 2009 18:37:35 GMT

~~Must~~
Must be resiliant

...

n-1 link (provide next and previous photo context in templates?)
RSS
if sorting by titles, 1 vs 001 ?
TiddlyPhotog
* individual entries as static HTML pages

pblog notes

SaqImtiaz — Wed, 23 Dec 2009 17:57:29 GMT

Must be resiliant to deleting photos/pages. Therefore previous and next links cannot be simple n+1 and n-1 link
TiddlyPhotog
* individual entries as static HTML pages

pblog notes

SaqImtiaz — Wed, 23 Dec 2009 17:30:57 GMT

TiddlyPhotog

...

HTML pages
** title
** sequence number (ID)
** single photo
** navigation links for previous/next
** expose data

...

free with tiddlers]*tiddlers]
* URI structure:
** / is latest entry
** /photo/$ID /photos/$page*
** /photos/$page
* photos stored

...

not binary tiddlers)*tiddlers)
* thumbnails for collection pages*pages
* admin interface
** add new entry
** upload photo
** scheduling (hides $pubdate > $today)*$today)
* hosted on GAE (optional)
~~Resources**templates: http://cdent.tumblr.com/search/templateshandlers:~~
Resources
templates: http://cdent.tumblr.com/search/templates
handlers: http://github.com/tiddlyweb/tiddlyweb-plugins/blob/master/cachinghoster/cachinghoster.py#L128
$

...

apt-get install ~~python-pip$~~python-pip
$ pip install -U ~~pip==dev$~~pip==dev
$ pip install -U ~~tiddlywebwiki$~~tiddlywebwiki
$ pip install -U ~~tiddlywebplugins.utils$~~tiddlywebplugins.utils
$ twinstance ~~/tmp/myInstance$~~/tmp/myInstance
$ cd ~~/tmp/myInstance$~~/tmp/myInstance
$ cat > ~~sample_plugin.pydef~~sample_plugin.py
def init(config): ~~print~~
print "hello ~~world"^D$~~world"
^D
$ vi tiddlywebconfig.py
http://gist.github.com/262592
Sample ~~app:http://gist.github.com/262592~~app:
http://gist.github.com/262592

pblog notes

SaqImtiaz — Wed, 23 Dec 2009 17:30:31 GMT

TiddlyPhotog
* individual entries as static HTML pages title sequence number (ID) single photo navigation links for previous/next expose data for each entry as JSON [comes for free with tiddlers]* URI structure: / is latest entry /photo/$ID /photos/$page* photos stored in file system (i.e. not binary tiddlers)* thumbnails for collection pages* admin interface add new entry upload photo scheduling (hides $pubdate > $today)* hosted on GAE (optional)
Resources**templates: http://cdent.tumblr.com/search/templateshandlers: http://github.com/tiddlyweb/tiddlyweb-plugins/blob/master/cachinghoster/cachinghoster.py#L128
$ sudo apt-get install python-pip$ pip install -U pip==dev$ pip install -U tiddlywebwiki$ pip install -U tiddlywebplugins.utils$ twinstance /tmp/myInstance$ cd /tmp/myInstance$ cat > sample_plugin.pydef init(config): print "hello world"^D$ vi tiddlywebconfig.py # add 'sample_plugin' to system_plugins
http://gist.github.com/262592
Sample app:http://gist.github.com/262592

Supporting localized TiddlyWiki distributions : re: country specific domain names

martin — Mon, 14 Dec 2009 15:33:19 GMT

If we are to have language specific start pages, then I favour using subdomains rather than different paths.

On the issue of accreditation, I think this is problematic. I think it is one thing having a list of resources in different languages, but if we accredit these then we are giving some kind of guarantee. How can we do this if we don't understand the text in the language file?

I agree with Greg - it's much more important to provide more support for the translation process than it is to do compliance checking or accreditation.

Supporting localized TiddlyWiki distributions : Translation process

martin — Mon, 14 Dec 2009 14:33:25 GMT

There are two main ways we get TiddlyWiki translations:

1) someone decides to adopt a locale, and translates and supports a locale (for example Bram Chen for Chinese or Ton van Rooijen for Dutch). This person has a commitment to TiddlyWiki and maintaining the translation. They also need to be reasonably technically savy: they need to understand javascript and subversion.

2) someone offers to do a one-off translation (perhaps because they need it, or they want to "give something back"). This person does not necessarily want to continue to support the translation. They may need help in parsing javascript strings, and they probably will need someone to submit the translation to subversion.

I think it is unrealistic to expect there to a dedicated support person for each locale, or even for the majority of locales. This means we need to provide a means for itinerant translators to contribute to a translation. In particular need need to:

i) be able to extract (either manually or automatically) the untranslated strings from a translation
ii) put those strings in an accessible place, in a form that a tiddlywiki user (as opposed to a tiddlywiki developer or plugin writer) can easily edit. A wiki is a possible place.
iii) have a means of notifying users in other languages that we have new strings to translate and asking them to do the translation
iv) put the newly translated strings back into the subversion locale

I think (i) and (iv) are reasonably easy to address - this could be done with some differencing scripts, and actually the workload is small enough to have a reasonably sizable manual component.

Items (ii) and (iii) are harder to solve. Any views on these?

Supporting localized TiddlyWiki distributions : re: country specific domain names

SaqImtiaz — Fri, 11 Dec 2009 06:45:45 GMT

"I agree with the strategy of having language specific "start pages" and then maybe having an additional pages of language specific resources. (Maybe subdomains, e.g. en.tiddlywiki.org, is easier than different paths but same basic idea.)"

This sounds like a good approach. One caveat with using subdomains is that they are already in use for things such as plugins.tiddlywiki.org and svn.tiddlywiki.org and this may lead to some confusion.

With regards to country specific domains if we are only talking about a handful of popular domains then the purchase of them by UnaMesa might be feasible. We could redirect them to the page listing resources for that language. In the event of third parties owning the domains we could request the same.

Should they choose to host a localized version of TiddlyWiki in their own language instead, it may indeed be possible to somewhat automate the verification of the language pack by comparison to the hash listed at TiddlyHub. This assumes however that the language packs are created and hosted at TiddlyWiki.org. i.e. a Chinese site would need to use our endorsed translation pack to be accredited and listed.

I agree that this is not an ideal approach as it may become onerous over time. However our options might be limited if we wish to start accrediting sites and distributions created by third parties.

Will invite input from Bram and Martin and then hopefully we can proceed with a more concrete proposal for providing support for the translation process.

Supporting localized TiddlyWiki distributions

SaqImtiaz — Fri, 11 Dec 2009 06:31:34 GMT

...

However, any change towards facilitating the translation process will not be possible without co-ordinating with the release team.
One caveat is that we would have to ensure that the translation process did not delay the release schedule.
The different language versions could be distributed from TiddlyWiki.org. Not only is this an opportunity for UnaMesa to contribute to the community and play a more active role in distributing TiddlyWiki but would also minimize any extra work for the maintainers of TiddlyWiki.com.
Each translated version of TiddlyWiki could have its own TiddlyWiki site with basic information. This is discussed in the section below.
Support

...

Ideally we would want to update these sites every time there is a new TiddlyWiki release not only updating the TiddlyWiki version used but updating a "What's New" tiddler.
If we start providing officially supported localized distributions of TiddlyWiki the upgrade mechanism should ideally be updated as well. Currently it only updates the TiddlyWiki core but could be adapted to also update the appropriate translation plugin.
~~The different language versions could be distributed from TiddlyWiki.org. Not only~~If the creation of such information sites is ~~this an opportunity for UnaMesa to contribute to~~not immediately feasible we should still proceed with the ~~community and play a more active role in~~idea of distributing empty TiddlyWiki ~~but would also minimize any extra work for the maintainers of TiddlyWiki.com.~~documents in different languages.

Supporting localized TiddlyWiki distributions

SaqImtiaz — Fri, 11 Dec 2009 06:29:12 GMT

...

There may be other support communities such as http://groups.google.com/group/tiddlywiki-zh but these are not monitored by UnaMesa associates and the wider TiddlyWiki community due to the language barrier.
Possibilities for UnaMesa to better support localized distributions
~~One key issue to consider is that UnaMesa should avoid to be seen as officially endorsing any TiddlyWiki resources that we are not able to adequately monitor and maintain due to language barriers.~~
Distribution
Facilitating the availability of TiddlyWiki distributions in other languages in a timely manner is probably the area in which UnaMesa can make the most significant contribution.

...

Ideally we would want to update these sites every time there is a new TiddlyWiki release not only updating the TiddlyWiki version used but updating a "What's New" tiddler.
If we start providing officially supported localized distributions of TiddlyWiki the upgrade mechanism should ideally be updated as well. Currently it only updates the TiddlyWiki core but could be adapted to also update the appropriate translation plugin.
The different language versions could be distributed from TiddlyWiki.org. Not only is this an opportunity for UnaMesa to contribute to the community and play a more active role in distributing TiddlyWiki but would also minimize any extra work for the maintainers of TiddlyWiki.com.

Supporting localized TiddlyWiki distributions : country specific domain names

greg — Mon, 07 Dec 2009 17:50:40 GMT

Saq,

You make a good point about not wanting to maintain many different domain names.

I agree with the strategy of having language specific "start pages" and then maybe having an additional pages of language specific resources. (Maybe subdomains, e.g. en.tiddlywiki.org, is easier than different paths but same basic idea.)

However I also think it is a good idea to have some solution for country specific domains. Otherwise we run the risk of spammers obtaining the country specific names in areas where the country domain is widely used. e.g. TiddlyWiki.fr, TiddlyWiki.cn, ... Perhaps we can have a standard recommendation that individuals in those countries can follow to "be in compliance" with our recommendation and then we can list them. For example we maybe identify the "official" translations as those things which are identical to the english version except for the language pack -- and then make sure that the language pack does not include malicious javascript.

I'm not totally happy with the above, just suggesting an idea of an approach. I think we should move ahead with trying to provide more support for the translation process and at least maintain a list of resources for the supported languages.

-Greg

Supporting localized TiddlyWiki distributions

SaqImtiaz — Mon, 07 Dec 2009 14:23:31 GMT

...

There have been previous discussions in the community about facilitating the translation process by providing translators with a diff of the changes in the locale.en.js file. The 'TiddlyWiki release team' has alluded to this being possible but not further action has been forthcoming in this direction.
Furthermore, to provide timely access to translated distributions of TiddlyWiki it would be desirable to provide community translators with said diff in good time prior to each new TiddlyWiki release. Whilst this will require greater co-ordination between the release team and the community, the relative infrequency of new TiddlyWiki releases makes this a viable option.

...

only are ~~their~~there tools available

...

community to ~~contribute.~~contribute as well.
UnaMesa could play the role of liaison between the release team and the community translators, thereby avoiding the burden of facilitating translations from falling on the release team. Furthermore UnaMesa could develop the current TiddlyWiki build tools to facilitate the use of localization tools. (would require agreement from the development/release team)
However, any change towards facilitating the translation process will not be possible without co-ordinating with the release team.

...

in English ~~before~~first - it
Ideally we would want to update these sites every time there is a new TiddlyWiki release not only updating the TiddlyWiki version used but updating a "What's New" tiddler.
If we start providing officially supported localized distributions of TiddlyWiki the upgrade mechanism should ideally be updated as well. Currently it only updates the TiddlyWiki core but could be adapted to also update the appropriate translation plugin.

Supporting localized TiddlyWiki distributions

SaqImtiaz — Mon, 07 Dec 2009 12:42:09 GMT

...

Providing organized support to say the Mandarin speaking community - as we do the TiddlyWiki Google Groups - would lead to expectations that we will do the same for other language groups with a sizeable number of users, eg: Spanish.
What might be more sensible for UnaMesa to do instead is to highlight existing support mechanisms in other languages by means of easy-to-find links from TiddlyWiki.com and TiddlyWiki.org
With regards to linking to resources in other languages, I recommend we avoid using country specific domains. Not only are we actually providing language specific distributions and not geographically specific, but the use of country specific domains may lead to issues with having to obtain and maintain domains for a growing number of countries.
Instead it seems appropriate to adopt the widely used practice of suffixing the url with the language code. Eg: tiddlywiki.com/en and tiddlywiki.com/es
With regards to creating simplified sites for information and distribution in other languages, one challenge that we will face is to ensure that the information provided is accurate and current. I suggest that the best way to go about this is by creating a simple site that provides a basic introduction and answers the most common questions, whilst shying away from too many details or things that are likely to change in future versions. If we were to create such a site in English before - it could be a subsection of the tiddlers that currently make up TiddlyWiki.com - we could then use the above mentioned translation tools to translate it into different languages. The TiddlyWiki build tools could then be used to create the localized sites. Using a subsection of the TiddlyWiki.com content would not only make the initial translations less time consuming but would also make maintenance easier as changes would not be necessitated as often.
Ideally we would want to update these sites every time there is a new TiddlyWiki release not only updating the TiddlyWiki version used but updating a "What's New" tiddler.

Supporting localized TiddlyWiki distributions

SaqImtiaz — Mon, 07 Dec 2009 12:39:23 GMT

...

What might be more sensible for UnaMesa to do instead is to highlight existing support mechanisms in other languages by means of easy-to-find links from TiddlyWiki.com and TiddlyWiki.org
With regards to creating simplified sites for information and distribution in other languages, one challenge that we will face is to ensure that the information provided is accurate and current. I suggest that the best way to go about this is by creating a simple site that provides a basic introduction and answers the most common questions, whilst shying away from too many details or things that are likely to change in future versions. If we were to create such a site in English before - it could be a subsection of the tiddlers that currently make up TiddlyWiki.com - we could then use the above mentioned translation tools to translate it into different languages. The TiddlyWiki build tools could then be used to create the localized sites. Using a subsection of the TiddlyWiki.com content would not only make the initial translations less time consuming but would also make maintenance easier as changes would not be necessitated as often.
Ideally we would want to update these sites every time there is a new TiddlyWiki release not only updating the TiddlyWiki version used but updating a "What's New" tiddler.
If we start providing officially supported localized distributions of TiddlyWiki the upgrade mechanism should ideally be updated as well. Currently it only updates the TiddlyWiki core but could be adapted to also update the appropriate translation plugin.