Maintaining an OpenBlock Site

Upgrades

Make Backups!

As with any software, it is prudent to make backups of your database and your code before attempting any upgrades, so that you can roll back in the event of problems.

The most important things to back up are:

• Your database - see http://www.postgresql.org/docs/8.3/static/backup.html and http://blog.cleverelephant.ca/2010/09/postgis-back-up-restore.html
• Your settings file
• Any custom templates or code you wrote - use a version control system!
• Any uploaded content in the directory that settings.MEDIA_ROOT points to

Check the Release Notes

See OpenBlock 1.2 (Released 5/23/2012) for a list of changes since the last release. It's probably much more detailed than you need, but it's good to know what you're getting, and this file will alert you of backward incompatibilities that may impact any custom code you may have written.

Update the Code

To install updated openblock code (from source) and all dependencies:

pip install -r ebpub/requirements.txt
pip install -e ebpub
pip install -r ebdata/requirements.txt
pip install -e ebdata
pip install -r obadmin/requirements.txt
pip install -e obadmin
pip install -r obdemo/requirements.txt
pip install -e obdemo

To upgrade from stable packages, the procedure is the same as described at Installing Stable Packages.

Database Migrations

Whenever upgrading your copy of the OpenBlock code, there may be updates to Model code which require corresponding changes to your existing database.

You can do this with one command (it's prudent to make a database backup first):

django-admin.py syncdb --migrate

To check what migrations exist and which ones you've already run, you can do:

django-admin.py migrate --list

Under the hood, this uses South to automate these database migrations. If you really want to see what the migrations do, or need to write your own, you'll want to read the South documentation to understand how they work. Then look in the migrations/ subdirectories located under the various app directories, notably ebpub/ebpub/db/migrations/ to see what the existing migration scripts look like.

If a migration gets stuck...

If you run django-admin.py migrate and it seems to hang -- just sitting there indefinitely -- this typically means some other code is trying to write to the database, but the migration needs an exlusive lock to alter some tables, so it waits "forever" for those other scripts to go away. (See http://south.aeracode.org/wiki/FAQ#ImusingPostgreSQLandmigrationsjusthangindefinitely ). Typically these will be scraper scripts. To fix it, either restart the database, or kill all the other processes that are writing to the database. The migration should then finish with no trouble.

Downtime

When your code is installed but your database migrations haven't yet finished, the live site may give errors. It may make sense to temporarily replace your site with eg. a static "system is down for maintenance" page. Doing so is beyond the scope of this documentation.

The OpenBlock Admin UI

The admin UI is built on the standard Django admin interface. Here is an overview of some things you can do with it that are specific to OpenBlock.

Start by pointing your browser at /admin/

Moderating User-Submitted Content

If you have any Schemas with allow_flagging=True, eg. if you have enabled the Neighbornews package then logged-in users can flag those NewsItems as possible spam or inappropriate content.

To moderate these flagged items, go to the admin UI and, under the "Moderation" heading, click on "News Item Flags".

The list of flags is sorted with the most recent new, unmoderated flags at top.

There are two ways you can moderate items: in bulk, or one at a time.

Moderating One at a Time

If you click on a News Item Flag in the list, you'll see details about the flagged News Item, who flagged it, when, and why.

There is also a link to the public view of the NewsItem, if you want to examine it more fully in context.

You'll see two buttons at top: "Reject and Delete it" and "Approve it". Click one of those and you're done.

Moderating in Bulk

From the main list of News Item Flags, you can check all items you want to approve or reject, and then from the "Action" pull-down menu, you can select Approve, Reject, or Delete.

What effect does each action have?

Whether singly or in bulk, here is the meaning of the actions you can take:

• Approve - This will mark all flags on this NewsItem as "approved". The flags are not deleted (though we might revisit that decision), but the NewsItem is no longer shown as flagged on the public site, and those Flags will be moved off the top of the list of new flags.
• Delete - This will permanently delete the NewsItem and all Flags on it. You cannot undo this action.
• Delete flag - The selected Flag(s) is/are deleted. This has no effect on the associated NewsItem(s) and any Flags not specifically selected.

Upload NewsItems from Spreadsheets

This is a good way to get news into your site if you have spreadsheets that are updated rarely (or never).

Browse to /admin/db/newsitem/ and click "Import from spreadsheet".

(Currently only CSV files are supported.)

Alternative: Importing spreadsheets via a scraper

If you have a spreadsheet that updates frequently, it may be worthwhile to use the spreadsheet scraper which provides the same functionality.

What Goes Where

OpenBlock needs to know how to store the data from each column of your spreadsheet.

Each column might correspond to:

• a core field of the NewsItem model. Only 'title' and some form of location are strictly required; the rest have reasonable defaults, but you typically want to provide 'description' and 'item_date' and maybe 'url' too. See below for more about how locations are recognized.
• Or, a custom attribute - if you have any - defined by the relevant Schema.

There are three ways you can tell OpenBlock where to store the data from each cell of the spreadsheet. Which one you choose depends on what your spreadsheet looks like and how often you'll be loading similar sheets:

1. Modify the original spreadsheet:

Modify or create your spreadsheet so the first row contains NewsItem field names, and Attribute names relevant to your Schema. Do not provide a second spreadsheet. This is fine for a one-time deal; Not recommended if you're going to be loading similar spreadsheets frequently, because you'd have to modify the first row every time.

Example items sheet:

title	item_date	description	location_name	reason
Bob	12/31/2011	group therapy	123 Main St	feeling depressed
Carol	2012-01-01	film premiere	456 Broadway	got free tickets

In this example, we assume the Schema has a text SchemaField named "reason". If it doesn't, the fourth column would just be ignored. All the other columns correspond to core NewsItem fields.)

2. Spreadsheets with a header with column names:

1. Before the first time you want to load a spreadsheet, save a second copy of the spreadsheet.
2. In the copy, delete all rows but that first header row.
3. In the copy, in each cell of the second row, enter a NewsItem field name (or Attribute name relevant to your Schema).
4. In the second row, leave un-needed cells blank.
5. Use the copy as the second (optional) spreadsheet.

OpenBlock will use this second spreadsheet to "map" the original column names to NewsItem fields and attributes. You can keep this "mapping" spreadsheet handy for the next time you want to load a similar spreadsheet.

Example items sheet:

Who	When	What	Where	Why
Bob	12/31/2011	group therapy	123 Main St	feeling depressed
Carol	2012-01-01	film premiere	456 Broadway	got free tickets

Example mapping sheet:

Who	When	What	Where	Why
title	item_date	description	location_name	reason

3. Spreadsheets without a header:

1. Before the first time you want to load a spreadsheet, save a second copy of the spreadsheet.
2. In the copy, delete all rows but the first row.
3. In the copy, replace each cell in the first row with a NewsItem field name (or Attribute name relevant to your Schema).
4. In the first row, leave un-needed cells blank.
5. Provide the copy as the second (optional) spreadsheet.

In this case, OpenBlock will use this second spreadsheet to "map" original column numbers to NewsItem fields and attributes. This "mapping" spreadsheet can be re-used next time you load a similar spreadsheet, too.

Example items sheet:

Bob	12/31/2011	group therapy	123 Main St	feeling depressed
Carol	2012-01-01	film premiere	456 Broadway	got free tickets

Example mapping sheet:

title

item_date

description

location_name

reason

What if some rows are inconsistent?

OpenBlock will just ignore any rows that it can't successfully save.

Locations

OpenBlock will attempt to determine each NewsItem's location according to this procedure:

• If there is a "location" field, try to split it into a latitude,longitude pair; if that's not within bounds, try treating it as a longitude,latitude pair. If it doesn't look like a pair of numbers, ignore it.
• If there are fields named "latitude" and "longitude", or "lat" and "lon" or "long" or "lng", use those.
• If there is a "location_name" field, try geocoding that.
• Otherwise, combine all text fields and try to extract addresses using ebdata.nlp and geocode them.
• If all of the above fails, just save the item with no location.
• If a point is found by any of the above means, but there is no "location_name", one will be derived by reverse-geocoding the point.

Avoiding Duplicates

By default, OpenBlock assumes that any change in any field except item_date or pub_date indicates a new, unique NewsItem.

This can result in duplicates if eg. a minor correction is made in a description or title. To avoid this, you would need to figure out which fields really identify a unique row, and provide them as the unique fields option.

(Note you can't currently use SchemaField names here; only core fields of NewsItem.)