• Talks
• Web Collaboration And Python
• Talk

Start slideshow

Internet-based Collaboration with MoinMoin and Other Python Web Applications

Covered by this talk:

• Experiences from public project/activity Web sites

• Using MoinMoin and Wiki solutions for collaboration

• Working with other useful collaborative tools
• Some ideas for future activities

Presenter: Paul Boddie (<paul AT boddie DOT org DOT uk>)

Some of the material in this talk is presented for future reference, so that you can revisit it and follow some of the links. I may not show any code, but you can follow the links and get to real software projects.

Background

• 1995: Started using Python
• 1996-1998: First tried to introduce Python in my place of work

• 2002: My involvement with the PythonInfo Wiki begins

• (2004: I start writing my own Python Web frameworks)

• 2006: Python returns to CERN with EuroPython 2006; I learn some things about MoinMoin

• 2007-2010: Helping EuroPython's organisers work together

Here, the emphasis will be more on practical collaboration than pure technology. Note how the desire to use something, to improve it, and to promote it leads to collaborative efforts to further its use.

Getting Involved

Most people "get involved" with a community naturally via...

• Mailing lists, newsgroups, discussion forums
• IRC, chat, blogging, microblogging, social networks
• Bug reports, contributions
• Web site development/maintenance, documentation
• Infrastructure development/maintenance

The first kinds of involvement can be fairly casual; the latter kinds usually involve having the project's "house keys". Remember that for things like conferences, there are other things like registration and even venue-related activities, like looking after equipment, which are somewhat outside the scope of this talk.

Getting Involved

Anyone can get involved eventually:

1. Maintain fairly useful document: Python Web Modules Overview

2. Maintain fairly useful replacement: Python Web Frameworks Overview

3. Have content incorporated into the PythonInfo Wiki

4. Maintain more and more Wiki content, fight spam, ...
5. Eventually get administrator privileges
6. Learn, write and share more, perhaps even influence others

This is how people can get involved in a community without really doing much coding. But that lets others get on with the coding.

The What and Why of Wiki

• A Wiki is just a Web site:

  • Wikipedia is the famous one, but there are two on python.org (for Python and Jython)

• Its pages can be edited through the Web
• Formatting can be done using plain text
• Anyone (subject to policy) can add, edit/improve and manage pages

• Very simple "edit, preview, save" processes, not strict or workflow-intensive

• A sensible choice for community-edited content

This summary is just for anyone who isn't really aware of what a Wiki actually is.

The Python Wiki

• The Python (or PythonInfo) Wiki is the sum of the community's knowledge

• If you know an area of expertise well, try and contribute!

• If you want to widen Python's global reach, help out with the Languages page!

• Why not find other Python-related Wikis and help maintain them?

  • WSGI, Mercurial, MoinMoin, PyQt ...

Wiki contributions are a way of getting familiar with collaboration tools while helping others. I think that the Python Wiki could be the cornerstone of python.org in future. On the next slides, some ways of learning how to collaborate via a Wiki medium will be given.

Maintaining Summary Pages

• WebFrameworks (was WebProgramming)

• PythonImplementations

• PythonEditors

• GuiProgramming, ParallelProcessing

• DatabaseInterfaces (uses templates)

Here are some summary pages that might be worth looking at. Some are more chaotic than others. DatabaseInterfaces uses templates which people don't always want to edit.

Maintaining Summary Pages

• Self-promotion usually rears its ugly head
  • People want their Web framework to be more visible than the rest
• Editorial policies and controls keep the playing field level
• Sometimes, separate pages keep the summaries focused

  • WebProgramming became WebProgramming (an overview) plus WebFrameworks, ContentManagementSystems, WebClientProgramming

• Large summaries should not "scare people" - in fact, a wide range of solutions is a healthy sign

Remote collaboration always involves misunderstandings as well as misbehaviour. Here, we want to find out how to make things work. Social factors are often more important than having completely optimised technology.

Maintaining Documentation Pages

• Tutorials/instructions:

  • PublishingRepositories on the Mercurial Wiki

  • PythonSpeed/PerformanceTips

  • Python2orPython3

• Guides/overviews: PythonWarts

• Proposals and position papers:

  • CodingProjectIdeas/StandardLibrary/RestructuredStandardLibrary

  • MarketingPython

There's a lot of useful content just waiting to be discovered! Why not contribute your own documentation?

Maintaining Documentation Pages

• Tutorials/instructions often just accumulate content
  • People just add tips, tricks, recipes, experiences ("this worked for me!")
  • People tend to tread lightly around others' contributions
• At some point, someone has to introduce structure and consistency

  • Improvement also involves fixing/expanding the document if people don't understand something

• The aim is to eliminate repetition and provide fast access to necessary information

Again, finding out how to make things work is crucial. That way, the benefit can be gained from everybody's contributions. The feedback cycle is critical, too: with EuroPython, it hasn't been clear whether anyone is really reading our "documentation" or communications.

The Rewards of Wiki Maintenance

• You learn a lot about what is out there ("community wisdom")

  • Try to help others to benefit (example: PythonSpeed/Profiling)

• Publicising things like software projects and solutions helps them and potential users (like you)
  • People might not reinvent existing solutions but try doing new things instead
• You are combining contributions to make a community resource
  • This is quite different from the "throwaway remark" mentality of social networking tools

Knowing which solutions exist can be very helpful since it makes you aware of tools that you can use, which then makes some domains easier to get into, and it can even help you think of new kinds of projects.

Applying a Wiki to a New Project

"I think one of the hardest things is getting people to submit content, and what we do should make that process a lot easier. Another hard thing was the bottleneck where things that needed changing on the website had to go through official people who had official rights to change things. Moving to a wiki should change that." - Laura Creighton

So far, a study of Wiki-based collaboration has been studied as a way of contributing to a community, but what about taking this knowledge and applying it elsewhere?

Some Common Myths

• "Wikis are messy!"
• "Wikis mean that anyone on the Internet can edit it!"
• "Put it somewhere on the Wiki! ... It's a mess! I can't find anything!"
• "I don't want to learn the syntax! ... It doesn't look any good!"
• "Wikis don't help me structure my content!"
  • Meanwhile, in a bug tracker: "Make a new bug for it! ... I can't find the bug!"

• "Let them have their Wiki, we'll put the real content on the proper site! ... Can someone update the Web site? <days pass by>"

Editorial Maintenance

• Deal with unstructured editing (encourage good but structured edits)
• An editorial policy sets the ground rules
• Impose consistency and house styles (formatting, page names)
• Use categories and subpages

Sometimes people just need encouragement to make big but respectful changes. You can always revert bad edits, but people need to know what is good and bad.

Preventing Vandalism and Spam

• MoinMoin and other Wiki solutions can overcome spam

• Use the TextCHA support in MoinMoin

• Or use ACLs to control write access

• MoinMoin 1.6.x and later are capable of dealing with all this

Here, we just want to emphasise that Wikis are "safe" choices. It doesn't have to be like the Wild West!

Wiki Weaknesses

• Web browser editing is tiresome for large texts

• MoinMoin's GUI editor needs improvement

• Tables are often tricky to edit (also in most mark-up languages)

• Some editing and maintenance tasks could be automated or better supported

• "Full" version control features are often absent (annotate/blame)

• Example page: PythonEditors

Weaknesses are also areas for improvement, of course. It is possible to try and improve things in order to make contribution easier, either by writing extensions or by helping projects like MoinMoin.

Making It Easier to Contribute

• Sprints at EuroPython

• The main page is called Sprints and gives an overview of sprinting

• Sprint pages are subpages of Sprints and are listed using the PageList macro:

  <<PageList(Sprints/)>>

• Use the NewPage macro with a template to provide an easy way to make a new page:

  <<NewPage(SprintTemplate,Add a sprint!,Sprints)>>

• When a contributor fills out the form field and presses the Add a sprint! button, they edit the template

• New sprint pages appear automatically

Sometimes, people need support and reassurance that they are doing the right thing when editing. Similar to the above, attachments can be listed, and this was done when providing talk materials summaries for previous conferences.

Common Mistakes of New Projects

Some lessons from EuroPython infrastructure projects:

• Overambition: a long list of desirable features
• A pristine foundation: get rid of working but "inelegant" solutions
• Monolithic technology: it all has to be done within one framework
• The result: something which can be elegant but is often unfinished or inadequate

There's always a discussion about how to make the ultimate open source conference solution, but I'm going to argue for choosing a Wiki, sticking with it, and augmenting it with other things if appropriate.

A Case Study: EuroPython 2006-2010

The Advantage of a Wiki for New Projects

• Getting something up and running is fast

• Content is easy to add, and user-generated content is well supported

  • For EuroPython, content is king!

• Wiki solutions like MoinMoin are extensible - they're also frameworks!

• The Wiki environment is good for prototyping, and prototyping is often what new projects need

Getting things up and running can still be tricky. I propose some nice scripts for common configuration tasks.

Adopting a Visual Style

It's important to adopt a visual style consistent with other services and distinct from any defaults.

• Why bother? To avoid a number of negative perceptions:
  • "They don't really know what they're doing" - someone has set up a Wiki but doesn't know what to do next
  • "They aren't really serious" - it doesn't look like the rest of the site
  • "It's just a Wiki" - people are just going to use it for scribbling private notes
  • "They'll take it down in a bit" - it was easy to put up as an "experiment", so it'll go away when they want to try something else
  • "Hey, I'm editing Wikipedia!"

• Examples: Apache Lucene, The Balloon Project

A Comparison of Theming

The CSS usually requires a lot of time for any kind of theming.

Assuming that you're not only going to be deploying a Wiki, it's worth considering the effort required to change the visual style of various solutions.

Making a Wiki the "Main Site"

• We did this for EuroPython 2008

• If anything, current versions of MoinMoin are even more suitable

• Wiki plus theme (including sponsor banner), macros (feed reader, login hint, site updates, vote recorder)

• Why not have a "normal" main site and a Wiki for informal stuff?
  • Because it fragments the content, leads to duplication, makes the Wiki seem like a playpen
  • And the "normal" site still won't be as easy to maintain

The important thing is that it doesn't have to look like a traditional Wiki. Some people might not like the look, which was based on the previous site (but less rigid), but the site was usable. We could have been slightly more liberal with write access, and newer antispam measures would have helped, too.

The Top Priorities of New Projects

• Something which communicates the goals of the project well
• It's worth spending time on the "public face" of the project
  • A blog as a project's Web site often suggests unreadiness

  • A generic "project page" on SourceForge or Google Code can turn people away

• Don't forget licensing considerations!

When one sees a blog as a project's Web site, it gives the impression of something peripheral - a hobby, perhaps - and not something that is being seriously pursued. Generic project pages often leave the impression of a code drop - somewhere to upload files and nothing more - which puts up a barrier to collaboration.

Where's the Python?

"We need a Wiki/tracker/repository browser!"

Fortunately, the Python offerings are all first-class solutions.

If you can help it, it's best to choose something in Python (if you're a Python programmer). For those convinced that this is the kind of thing they should be doing themselves, or for people who are going to be asked to deploy tools at work, it's in your interest to pursue Python-based solutions (within reason).

Examples of Collaboration

The various services depend on the kind of endeavour (source browsers, package indexes for software, registrations and submissions for conferences). But almost everything needs a coherent Web presence. The Web lets us blend different code bases almost seamlessly, and only stuff like authentication can be tricky.

A Case Study: FSFE Fellowship Wiki

• The FSFE Fellowship has a collaboration platform (site, blogs, Wiki, mailing lists)
• A consistent visual style had been made (and implemented for the site and blogs)

• They had chosen MoinMoin for the Wiki but needed a theme

• Principle objective: glue the existing CSS onto an appropriate page structure
• Solved problems: special page title layout, multiple page translations

As noted earlier, most of the effort was spent in making sure that the CSS behaved itself, occasionally tweaking the structure to force the CSS to do the right thing.

A Case Study: FSFE Fellowship Wiki

• The need for a calendar solution:

  • Preferably an existing application or an extension to a deployed application (Free Software, of course)
  • Integrate with other services
• Why not run a calendar out of the Wiki?

  • PythonEvents versus Ubuntu's Fridge Calendar

    • Google Calendar not "free as in freedom" or necessarily convenient

• FSFE Fellowship Events uses EventAggregator

  • Make it less error-prone to enter events and maintain summaries in a Wiki

EventAggregator was a learning experience and an attempt to improve the state of Wiki calendar offerings. It doesn't support recurring events at the moment, but would probably be suitable for Ubuntu's needs if it did.

What about "Single Platform" Solutions?

• "How messy! Surely a Django-based Wiki/blogging/tracking/calendaring/... solution is preferable!"

• Aren't existing applications good enough and proven?

  • Usually, the "main site" and/or Wiki aspects of such solutions are clearly weaker than the competition

• Is it really worth your time reinventing existing solutions?

• Is it really so hard to co-deploy them using Apache and to theme them?

• The strength of the Web is that you can mix completely different technologies and they can still look the same

The usual argument is that an existing project is hard to understand or inelegant. But even inelegant solutions impart knowledge about a domain. Moreover, existing solutions are widely used and knowing how to integrate them with other solutions is arguably a form of expertise that is sought after.

Sustainable Models of Collaboration

• How well do such sites work?
• Easy to navigate? Do you know how to find all the features?
• Easy to join? Can you get access and customise it?
• Easy to contribute to? Can you add or change content?

• Easy to emulate? Could you set it up for your own projects?

Whether you write a monolithic solution or glue together existing ones, you still have to evaluate the resulting experience. Being able to repeat the exercise is important because it could save other people a lot of time trying to set such things up for themselves.

Some Ideas for Community Projects

In activities like EuroPython, it feels like every year is a repeat of the previous one, some of the same solved issues recur to be solved again. We should be trying to grow the area of experience and expertise, not struggle with the same things over and over again.

Some Objectives

• Prevent work being redone over and over
• Share best practices with others
• Improve the existing services rather than trying to surpass them with something new
• Retain existing volunteers instead of alienating them

There's just no point in writing lots of new code when existing solutions are good enough (or thereabouts).

The Main Messages

• You can get involved with projects like Python and activities like EuroPython

• There are good solutions for collaboration
• These solutions can "work together", too!
• You get to learn about and help Python-related projects
• Everybody wins: you gain experience, the projects gain from usage, the community gains from the result

I want to encourage people to investigate the projects that are out there and to consider helping them, even if only in a small way. I've helped random software projects in minor ways, but it can make a difference and help people concentrate on what they are doing, leaving other things to random volunteers like myself.

Thanks to...

The wider Python community and everyone responsible for making Python and Python solutions as useful and interesting as they are!

See http://wiki2010.europython.eu/Talks/Web Collaboration And Python for more information and resources.

Bonus Material

Maintaining Summary Pages

• Structured templates describing each solution

  PostgreSQL

  URL

  http://www.postgresql.org/

  licence

  BSD

  platforms

  Unix, win32

• Templates don't always encourage contribution

  • One exception: DatabaseInterfaces

• It's typically easier to have list of links with brief descriptions

Maintaining Documentation Pages

• One comprehensive page or several specific pages?
  • You may end up having to collect and merge many existing pages
  • Eventually, some content will need moving to specialised pages

• Follow mailing lists and discussion groups and attempt to address problems experienced by the (potential) readership

• Ultimately, tutorials/instructions can be rendered obsolete by automation

Editorial Maintenance

Keep the page count down:

• Stop people creating new pages when they could edit existing ones

• PythonProjects links to MostPopularPythonProjects

• Applications links to WellKnownPythonPrograms

• The more places the same stuff gets added, the more places can be wrong or become out of date

Editorial Maintenance

Deal with unstructured editing:

• Get people to write coherently in the right places
• An editorial policy makes expectations clear:

  Editorial Notes

  The above lists should be arranged in ascending alphabetical order - please respect this when adding new entries. If, as the developer of a listed application, you disagree with the classification of the work, please move it into the appropriate category or create a new category, respecting the ascending alphabetical order of the categories. When specifying release dates please use the format YYYY-MM-DD.

• Policies may deal with self-promotion, agenda-pushing, style/grammar mavens

Editorial Maintenance

• Impose consistency: always format things in the same way; keep examples (such as filenames) consistent

• Impose house styles: although some prefer WikiNames, it can obscure or distort content (example: RoundUp)

• Use categories and subpages: they make lists of related pages easier to maintain

Preventing Vandalism

• Use ACLs and groups to lock pages down:

  • As an admin user, edit affected pages adding...

    #acl All:read

  • For more widespread problems, a full lock-down involves editing the Wiki configuration:

    acl_rights_before = 'All:read'
    

Protecting Resources

• Keep some pages under strict control using a trusted editors group

• Make a TrustedEditorsGroup page

  1. Set an ACL on the page to stop anyone editing it:

     #acl All:read

  2. Put a bullet-list of usernames on the page

Protecting Resources

• Either add an ACL to protected pages:

  #acl TrustedEditorsGroup:read,write,delete,revert All:read

• Or potentially change the configuration:

  acl_rights_before = 'TrustedEditorsGroup:read,write,delete,revert'
  

• Then just add a shorter ACL to protected pages:

  #acl All:read

Spamming

• Spam prevention used to be a huge problem for MoinMoin 1.5.x and earlier

• There was a BadContent and LocalBadContent pattern matching system which disallowed edits using "bad patterns"

  • This may have been useful for learning certain Chinese words, but caused a lot of work
  • A recent example on a Wiki which doesn't seem to use TextCHA:

    [a-z]{30,}

    This stopped repeated page creation with names like GisfsyuPjusfbsd and content like hfpifhenvphfnvpspdahneauhnveifhsoiha.

Spamming

• Banning users using groups works to a limited extent, but still has its uses:

  • It's the opposite of TrustedEditorsGroup: define a BannedUsersGroup page and populate it with usernames

  • Change the configuration to give such users no access whatsoever:

    • acl_rights_before = 'BannedUsersGroup:'
      

Spamming

• In MoinMoin 1.6 and later, the TextCHA antispam feature has transformed the situation

  • Define "editing questions" that challenge people to prove familiarity with the site or its purpose
  • "Which programming language is this site about?"

Making Summaries Easier to Maintain

• Just like PageList lists pages, AttachList lists attachments

• On the Talk Materials page:

  <<AttachList>>

• A list of attached talks is shown
• Contributors just have to click a link to see the upload form

  [[TalkMaterials|Click here to upload a file|&action=AttachFile]]

Your Potential Users

• Even existing projects are "new" to potential users
• Existing users may need persuading

  • "I prefer my text editor and Subversion!"

• Arguments from positions of luxury don't address participation issues
• Top-down or "from the inside (of the project) out" communication isn't good collaboration

There's an ongoing argument about this kind of thing around python.org. I want a link to the Wiki added/restored on the front page, but arguments about "screen real-estate" mean that it may just never happen.

What about a Content Management System?

• You need just enough automation to make a system that is better than the one that preceded it

• It's tempting to design or adopt a rigid system for various reasons:
  • "This is structured information: we need a database!"
  • "Getting at the data will be slow: let's use a database!"
• Always ask whether the architectural changes will deliver enough benefits:
  • Can you handle unstructured information in a structured fashion?

  • Are you accessing millions of records millions of times per hour, or do you just think it will be slow?

  • What do your potential users have to say?

Writing Themes

• The CSS part of most themes takes a lot of time
• For individual systems, the theme code or templates requires a varying amount of time
• Wiki solutions may have just the theme "boilerplate" and what appears in pages...
  • Sometimes, there are special things that appear (forms, special tables)
  • Making sure the special stuff works can be time-consuming
• Other solutions may have different kinds of pages, each of which needs special attention

If you deploy Wiki and other solutions, that visual style requirement is going to come up. Although you can spend a lot of time on it, it's arguably worthwhile, so this gives a few warnings about the task at hand.

Writing Extensions

Some ideas:

• Permit nicer formatting of search results

  • Make a category search for CategoryFellowshipGroup return Berlin, not groups/Berlin

  • Put an EventAggregator calendar view in front of search results

• Tools to promote trusted editors, edit TextCHA questions through the Web
• Support RSS more widely
• Have editing conveniences such as "quick edit" buttons
• Integrate other kinds of data (version control, bug tracking)

This is where your Python expertise comes in. It's preferable to having to write PHP to do something similar.

A Case Study: FSFE Fellowship Wiki

• Problem: page titles had a special place in the layout

  • This is a bit like MediaWiki, but we still wanted page names to look good as titles

• Solution #1: try and make page names prettier

  • FrontPage becomes "Front Page"

• Solution #2: drop initial headings on pages
  • Tell everyone not to have an initial level 1 heading

A Case Study: FSFE Fellowship Wiki

• Problem: support multiple translations of documents (Advocacy faq (English))

  • Although MoinMoin supports languages, a Wikipedia-style list of languages feature wasn't available

• Solution #1: obtain known translations of system pages using the translation dictionary

  • FrontPage (English), StartSide (Norwegian), ...

• Solution #2: have a naming convention for other pages (PageName_xx where xx is the language code)

  • Advocacy_faq_en (English), Advocacy_faq_fr (French), ...

This just illustrates that MoinMoin's underlying page store is somewhat accessible and that there are ways of extending the solution to meet the needs of users.