db Package

db Package

admin Module

Hooks up ebpub.db.models to the django.contrib.admin UI.

breadcrumbs Module

Breadcrumbs helpers. These functions return lists of (label, url) pairs.

constants Module

Various numbers that appear throughout the code but don't change often enough to warrant going in settings.py.

context_processors Module

ebpub-specific data needed by various templates.

ebpub.db.context_processors.map_context(request)

Context variables needed on pages that use maps.

feeds Module

class ebpub.db.feeds.AbstractLocationFeed

Bases: ebpub.db.feeds.EbpubFeed

Abstract base class for ebpub.db.models.Location-aware RSS feeds.

newsitems_for_obj(obj, qs, block_radius)

Get the relevant NewsItems as a queryset.

fields Module

Custom Fields for OpenBlock models.

class ebpub.db.fields.OpenblockImageField(*args, **kwargs)

Bases: easy_thumbnails.fields.ThumbnailerImageField

A model Field based on ThumbnailerImageField that makes sure we save a correct relative filename.

Uses OpenblockImageFormField as its formfield.

class ebpub.db.fields.OpenblockImageFormField(*args, **kwargs)

Bases: django.forms.fields.ImageField

A version of ImageField that fixes EXIF rotation on upload, so eg. mobile phone photos will have the correct orientation.

to_python(value)

Fixes rotation if needed.

forms Module

Custom Forms for ebpub.db.models.

models Module

Overview: NewsItems and Schemas

The ebpub system is capable of handling many disparate types of news -- e.g., crime, photos and restaurant inspections. Each type of news is referred to as a Schema, and an individual piece of news is a NewsItem.

Core Fields of NewsItems

The NewsItem model in itself is generic -- a lowest-common denominator of each piece of news on the site. It has:

• title (required)
• url (optional)
• description (optional)
• location_name (optional but highly desirable; can be reverse-geocoded from location if you have one)
• location (optional but highly desirable; can be geocoded from location_name if you have one)
• item_date (default: today)
• pub_date (default: current date and time)

Flexible data: SchemaFields and Attributes

If you'd like to extend your NewsItems to include more information, you can use SchemaFields.

Each piece of news is described by:

• One NewsItem instance, with just the core fields like title and description.
• One corresponding Attribute instance, which is a dictionary-like object containing extra data, and is available as newsitem.attributes.
• One Schema that identifies the "type" of NewsItem; and
• A set of SchemaFields, each of which describes: a valid key for the attributes dictionary; the type of its allowed values; and some configuration metadata about how to display and use that attribute.

This is intended to be flexible enough for real-world news data, while still allowing for fast database queries. For more background, you might be interested in the video Behind the scenes of EveryBlock.com.

Why not NoSQL?

ebpub was originally written around the time of the rise in popularity of schemaless document storage systems commonly lumped together as nosql, which would have made this one aspect of ebpub rather trivial. However, at the time, none of them had much in the way of geospatial capabilities; even today, none of them are as full-featured as PostGIS.

Examples might make this clearer. To assign the whole attributes dictionary:

ni = NewsItem.objects.get(...)
ni.attributes = {'sale_price': 19}
# There is no need to call ni.save() or ni.attributes.save();
# the assignment operation does that behind the scenes.

To assign a single value:

ni.attributes['sale_price'] = 19
# Again there is no need to save() anything explicilty.

To get a value:

print ni.attributes['sale_price']

Or, from a database perspective: The "db_attribute" table stores arbitrary attributes for each NewsItem, and the "db_schemafield" table is the key for those attributes. A SchemaField says, for example, that the "int01" column in the db_attribute table for the "real estate sales" Schema corresponds to the "sale price".

We'll walk through this example in detail below.

Detailed Example

Imagine you have a "real estate sales" Schema, with an id of 5. Say, for each sale, you want to store the following information:

• address
• sale date
• sale price

The first two fields should go in NewsItem.location_name and NewsItem.item_date, respectively -- there's no reason to put them in the Attribute table, because the NewsItem table has a slot for them.

Sale price is a number (we'll assume it's an integer), so create a SchemaField defining it, with these values:

field = SchemaField(schema_id = 5,
    name = 'sale_price',
    real_name = 'int01',
    pretty_name = 'sale price',
    pretty_name_plural = 'sale prices',
    display = True,
    display_order = 1,
    is_searchable = False,
   )

Line 1. schema_id is the id of our "real estate sales" schema.

Line 2. name is the alphanumeric-and-underscores-only name for this field. (Used in URLs, and as the key for newsitem.attributes, and for the NewsItemQuerySet.by_attribute() method.) This value must be unique with respect to the schema_id.

Line 3. real_name is the column to use in the db_attribute model. Choices are: int01-07, text01, bool01-05, datetime01-04, date01-05, time01-02, varchar01-05. This value must be unique with respect to the schema_id.

Lines 4-5. pretty_name and pretty_name_plural are the human-readable name for this attribute.

Line 6. display controls whether to display the value on the site.

Line 7: sort_order` - An integer representing what order it should be displayed on newsitem_detail pages.

Line 8: is_searchable - Whether you can do text searches on this field. Only works with text or varchar fields.

Once you've created this SchemaField, the value of "int01" for any db_attribute row with schema_id=5 will be the sale price.

Having done all that, using the field is as easy as:

from ebpub.db.models import NewsItem
ni = NewsItem(schema__id=5, title='the title', description='the description', ...)
ni.save()
ni.attributes['sale_price'] = 59

Searching by Attributes

There is a simple API for searching NewsItems by attribute values:

sale_price = SchemaField.objects.get(schema__id=5, name='sale_price') NewsItem.objects.filter(schema__id=5).by_attribute(sale_price, 59)

For details see NewsItemQuerySet.by_attribute().

Attributes: Under the hood

The dictionary-like API is provided thanks to the combination of the AttributesDescriptor, AttributeDict, and Attribute classes; see the source code for details.

Images

NewsItems have a newsitemimage_set reverse relationship with the NewsItemImage model, allowing any number of images to be associated with one NewsItem. All the images for a NewsItem can be retrieved via item.newsitemimage_set.all().

Dates

The distinction between NewsItem.pub_date and NewsItem.item_date is intended for data sets where there's a lag in publishing or where the data is updated infrequently or irregularly.

For example, on EveryBlock.com, Chicago crime data is published a week after it is reported, so a crime's item_date is the day of the crime report, whereas the pub_date is the day the data was published to EveryBlock.com.

Location, location, location

NewsItems have several distinct notions of location:

• NewsItem.location_name is a human-readable version of the location; it can be anything, but typically it describes an address, block, geographic area, or landmark.

• NewsItem.location is used frequently; typically a point, and typically set when scraping data, by geocoding if not provided in the source data. This is used in many views for finding relevant NewsItems (indirectly; actually see below about NewsItemLocations).

• NewsItem.location_set is a convenient way to get all Locations that overlap this item's location. It's a many-to-many relationship (via the NewsItemLocation model). The NewsItemLocations are created by a sql trigger whenever self.location changes; not set by any python code. Used in various views for filtering.

• NewsItem.location_object is a single Location reference; theoretically to be explicitly assigned by a scraper script when there's no known address or geographic point for this NewsItem but we know the name of the general area it's within.

  For example, many stories might mention a town or city name, or a police report might tell you the precinct. In practice, this field is usually Null; more importantly it's only used currently (2011-12-06) by self.location_url(), for linking back to a location view from a newsitem view. (Example of where everyblock.com uses this: NYC crime aggregates, eg. http://nyc.everyblock.com/crime/by-date/2010/8/23/3364632/ )

  See also this ticket http://developer.openblockproject.org/ticket/93 about possibly making more use of self.location_object.

Aggregates

Several parts of ebpub display aggregate totals of NewsItems for a particular Schema; for example, charts of how many were added each day.

Because these calculations can be expensive, there's an infrastructure for caching the aggregate numbers regularly in separate tables (db_aggregate*).

To do this, just run the update_aggregates script on the command line.

You'll want to do this on a regular basis, depending on how often you update your data. Some parts of the site (such as charts) will not be visible until you populate the aggregates.

Event-like News Types

In order for OpenBlock to treat a news type as being about (potentially) future events, rather than news from the (recent) past, there is a simple convention that you should follow:

1. Set the schema's is_event=True.
2. Add a SchemaField with name='start_time'. It should be a Time field, i.e. real_name should be one of time01, time02, etc. Leave is_filter, is_lookoup, is_searchable, and is_charted set to False. The pretty_name can be whatever you like of course.
3. Optionally add another SchemaField with name='end_time', if your data source will include this information.
4. When adding NewsItems of this type, the NewsItem's item_date field should be set to the date on which the event will (or already did) take place, and attributes['start_time'] should be set to the (local) time it will start, and attributes['end_time'] (if needed) should be set to the (local) end time.

All-day events can be represented by leaving start_time empty.

There is no special support for repeating events or other advanced calendar features.

Lookups

Lookups are a way to reduce duplication and support fast searching for similar NewsItems.

Consider the "real estate" schema we talked about in earlier examples. We want to add a field representing "property type" for each real estate sale NewsItem.

We could store it as a varchar field (in which case we'd set real_name='varchar01') -- but that would cause a lot of duplication and redundancy, because there are only a couple of property types -- the set ['single-family', 'condo', 'land', 'multi-family']. To represent this set, we can use a Lookup -- a way to normalize the data.

To do this, set SchemaField.is_lookup=True and make sure to use an 'int' column for SchemaField.real_name. For example:

field = SchemaField(schema_id = 5,
    name = 'property_type',
    real_name = 'int02',
    is_lookup=True,
    pretty_name = 'property type',
    pretty_name_plural = 'property types',
    display = True,
    display_order = 2,
   )

Then, for each record, get or create a Lookup object that represents the data, and use the Lookup's id in the appropriate attribute field. For example:

condo = Lookup.objects.get_or_create_lookup(
    schema_field=field, name='condo')
newsitem['property_type'] = condo

Note the convenience function Lookup.objects.get_or_create_lookup().

Many-to-many Lookups

Sometimes a NewsItem has multiple values for a single attribute. For example, a restaurant inspection can have multiple violations. In this case, you can use a many-to-many Lookup. To do this, just set SchemaField.is_lookup=True as before, but use a varchar field for the SchemaField.real_name. Then, in the db_attribute column, set the value to a string of comma-separated integers of the Lookup IDs:

field = SchemaField.objects.get(schema_id=5, name='property_type')
field.real_name = 'varchar01'
field.save()

newsitem.attributes['property_type'] = '1,2,3'

"Featured" Lookups

A Lookup instance can have featured=True, which you can use to mark some Lookup values as "special" for eg. navigation purposes. One example use case would be special tags or keywords that mark any relevant NewsItems for inclusion in a special part of your homepage.

To work with Lookups that are marked with featured=True, there are several useful tools:

• Lookup.objects.featured_lookups_for(newsitem, name) will, given a NewsItem instance and an attribute name, find all featured Lookups for that attribute which are relevant to that NewsItem.
• The same functionality is available in templates via the featured_lookups_for_item template tag.
• get_featured_lookups_by_schema is a tag that finds all currently featured Lookups, and URLs to find relevant NewsItems.

Charting and filtering lookups

Set SchemaField.is_filter=True on a lookup SchemaField, and the detail page for the NewsItem (newsitem_detail) will automatically link that field to a page that lists all of the other NewsItems in that Schema with that particular Lookup value.

Set SchemaField.is_charted=True on a lookup SchemaField, and the detail page for the Schema (schema_detail) will include a chart of the top 10 lookup values in the last 30 days' worth of data. Similar charts are on the place detail overview page. (This assumes aggregates are populated; see the Aggregates section below.)

module contents

class ebpub.db.models.AggregateAll(*args, **kwargs)

Bases: ebpub.db.models.AggregateBaseClass

Total items in the schema.

Parameters:

• id (AutoField) -- Id
• schema (ForeignKey, required) -- Schema
• total (IntegerField, required) -- Total

class ebpub.db.models.AggregateBaseClass(*args, **kwargs)

Bases: django.db.models.base.Model

Aggregates provide for quick lookups of NewsItems by various buckets, eg. number of NewsItems added on one day.

Parameters:

• schema (ForeignKey, required) -- Schema
• total (IntegerField, required) -- Total

class ebpub.db.models.AggregateDay(*args, **kwargs)

Bases: ebpub.db.models.AggregateBaseClass

Total items in the schema with item_date on the given day

Parameters:

• id (AutoField) -- Id
• schema (ForeignKey, required) -- Schema
• total (IntegerField, required) -- Total
• date_part (DateField, required) -- Date part

class ebpub.db.models.AggregateFieldLookup(*args, **kwargs)

Bases: ebpub.db.models.AggregateBaseClass

Total items in the schema with schema_field's value = lookup

Parameters:

• id (AutoField) -- Id
• schema (ForeignKey, required) -- Schema
• total (IntegerField, required) -- Total
• schema_field (ForeignKey, required) -- Schema field
• lookup (ForeignKey, required) -- Lookup

class ebpub.db.models.AggregateLocation(*args, **kwargs)

Bases: ebpub.db.models.AggregateBaseClass

Total items in the schema in location, summed over that last 30 days

Parameters:

• id (AutoField) -- Id
• schema (ForeignKey, required) -- Schema
• total (IntegerField, required) -- Total
• location_type (ForeignKey, required) -- Location type
• location (ForeignKey, required) -- Location

class ebpub.db.models.AggregateLocationDay(*args, **kwargs)

Bases: ebpub.db.models.AggregateBaseClass

Total items in the schema in location with item_date on the given day

Parameters:

• id (AutoField) -- Id
• schema (ForeignKey, required) -- Schema
• total (IntegerField, required) -- Total
• location_type (ForeignKey, required) -- Location type
• location (ForeignKey, required) -- Location
• date_part (DateField, required) -- Date part

class ebpub.db.models.Attribute(*args, **kwargs)

Bases: django.db.models.base.Model

Extended metadata for NewsItems.

Each row contains all the extra metadata for one NewsItem instance. The field names are generic, so in order to know what they mean, you must look at the SchemaFields for the Schema for that NewsItem.

You don't normally access an Attribute instance directly. You usually go through the dictionary-like API provided by Flexible data: SchemaFields and Attributes.

Parameters:

• news_item (OneToOneField, required) -- News item
• schema (ForeignKey, required) -- Schema
• varchar01 (CharField) -- Varchar01
• varchar02 (CharField) -- Varchar02
• varchar03 (CharField) -- Varchar03
• varchar04 (CharField) -- Varchar04
• varchar05 (CharField) -- Varchar05
• date01 (DateField) -- Date01
• date02 (DateField) -- Date02
• date03 (DateField) -- Date03
• date04 (DateField) -- Date04
• date05 (DateField) -- Date05
• time01 (TimeField) -- Time01
• time02 (TimeField) -- Time02
• datetime01 (DateTimeField) -- Datetime01
• datetime02 (DateTimeField) -- Datetime02
• datetime03 (DateTimeField) -- Datetime03
• datetime04 (DateTimeField) -- Datetime04
• bool01 (NullBooleanField) -- Bool01
• bool02 (NullBooleanField) -- Bool02
• bool03 (NullBooleanField) -- Bool03
• bool04 (NullBooleanField) -- Bool04
• bool05 (NullBooleanField) -- Bool05
• int01 (IntegerField) -- Int01
• int02 (IntegerField) -- Int02
• int03 (IntegerField) -- Int03
• int04 (IntegerField) -- Int04
• int05 (IntegerField) -- Int05
• int06 (IntegerField) -- Int06
• int07 (IntegerField) -- Int07
• text01 (TextField) -- Text01
• text02 (TextField) -- Text02

class ebpub.db.models.DataUpdate(*args, **kwargs)

Bases: django.db.models.base.Model

Scraper scripts can use this to keep track of each time we populate NewsItems of a given Schema.

Parameters:

• id (AutoField) -- Id
• schema (ForeignKey, required) -- Schema
• update_start (DateTimeField, required) -- When the scraper/importer started running.
• update_finish (DateTimeField, required) -- When the scraper/importer finished.
• num_added (IntegerField, required) -- Num added
• num_changed (IntegerField, required) -- Num changed
• num_deleted (IntegerField, required) -- Num deleted
• num_skipped (IntegerField, required) -- Num skipped
• got_error (BooleanField) -- Got error

class ebpub.db.models.Location(*args, **kwargs)

Bases: django.db.models.base.Model

A polygon that represents a geographic area, such as a specific neighborhood, ZIP code boundary or political boundary. Each Location has an associated LocationType (e.g., "neighborhood"). To add a Location to the system, follow these steps:

1. Create a LocationType.

2. Get the Location's geographic representation (a set of longitude/latitude points that determine the border of the polygon). You might want to draw this on your own using desktop GIS tools or online tools, or you can try to get the data from a company or government agency. (You can even draw simple shapes in the OpenBlock admin UI.)

3. With the geographic representation, create a row in the "db_location" table that describes the Location. See below for what the various fields mean.

   You can create Locations in various ways: use the admin UI; use the script add_location to create one by specifying its geometry in well-known text (WKT) format; use the script import_locations to import them from shapefiles; or use the Django model API; or do a manual SQL INSERT statement.

Parameters:

• id (AutoField) -- Id
• name (CharField, required) -- e.g., "35th Ward"
• normalized_name (CharField, required) -- Normalized name
• slug (SlugField, required) -- Slug
• location_type (ForeignKey, required) -- Location type
• location (GeometryField, required) -- Location
• display_order (SmallIntegerField, required) -- Display order
• city (CharField, required) -- City
• source (CharField, required) -- Source
• area (FloatField) -- In square meters. This is populated automatically.
• population (IntegerField) -- Optional. If used, typicall found in census data.
• user_id (IntegerField) -- Used for 'custom' Locations created by end users.
• is_public (BooleanField) -- Whether this is publically visible, or requires the staff cookie
• description (TextField) -- Description
• creation_date (DateTimeField) -- Creation date
• last_mod_date (DateTimeField) -- Last mod date

class ebpub.db.models.LocationSynonym(*args, **kwargs)

Bases: django.db.models.base.Model

Represents an alternate name for a Location.

Parameters:

• id (AutoField) -- Id
• pretty_name (CharField, required) -- Pretty name
• normalized_name (CharField, required) -- Normalized name
• location (ForeignKey, required) -- Location this is a synonym for.

class ebpub.db.models.LocationType(*args, **kwargs)

Bases: django.db.models.base.Model

Used for classifying and grouping Location.

You'll want to create at least one LocationType with the slug set to the same value as settings.DEFAULT_LOCTYPE_SLUG, because that's used in various default URLs. By default this is set to "neighborhoods".

Parameters:

• id (AutoField) -- Id
• name (CharField, required) -- for example, "Ward" or "Congressional District"
• plural_name (CharField, required) -- Plural name
• scope (CharField, required) -- e.g., "Chicago" or "U.S.A.". For display only; has no effect.
• slug (SlugField, required) -- Slug
• is_browsable (BooleanField) -- Whether this is displayed on location_type_list.
• is_significant (BooleanField) -- Whether this can be used to filter NewsItems, shows up in 'nearby locations', etc.

class ebpub.db.models.Lookup(*args, **kwargs)

Bases: django.db.models.base.Model

Lookups are a normalized way to store Attribute fields that have only a few possible values.

For more context, see Lookups.

Parameters:

• id (AutoField) -- Id
• schema_field (ForeignKey, required) -- This must be a SchemaField whose real_name is an int or varchar column.
• name (CharField, required) -- Human-readable name of this lookup value.
• code (CharField) -- Value used for queries. May differ from name if name is modified from the original data source, eg. to make name prettier. code should not be modified from the original source data.
• slug (SlugField, required) -- URL-safe identifier
• description (TextField) -- Description
• featured (BooleanField) -- Whether this lookup value is 'special' eg. for use in navigation.

class ebpub.db.models.NewsItem(*args, **kwargs)

Bases: django.db.models.base.Model

A NewsItem is broadly defined as "something with a date and a location." For example, it could be a local news article, a building permit, a crime report, or a photo.

For the big picture, see Overview: NewsItems and Schemas

Parameters:

• id (AutoField) -- Id
• schema (ForeignKey, required) -- What kind of news is this and what extra fields does it have?
• title (CharField, required) -- the "headline"
• description (TextField) -- Description
• url (TextField) -- link to original source for this news
• pub_date (DateTimeField) -- Date/time this Item was added to the OpenBlock site; default now.
• item_date (DateField) -- Date (without time) this Item occurred, or failing that, the date of publication on the original source site; default today.
• last_modification (DateTimeField) -- Last modification
• location (GeometryField) -- Coordinates where this news occurred.
• location_name (CharField, required) -- Human-readable address or name of place where this news item occurred.
• location_object (ForeignKey) -- Optional reference to a Location where this item occurred, for use when we know the general area but not specific coordinates.

attributes_for_template()

Return a list of AttributeForTemplate objects for this NewsItem. The objects are ordered by SchemaField.display_order.

class ebpub.db.models.NewsItemImage(*args, **kwargs)

Bases: django.db.models.base.Model

NewsItems can optionally be associated with any number of images.

Parameters:

• id (AutoField) -- Id
• news_item (ForeignKey, required) -- News item
• image (OpenblockImageField, required) -- Upload an image

class ebpub.db.models.NewsItemLocation(*args, **kwargs)

Bases: django.db.models.base.Model

Many-to-many mapping of NewsItem to Location where the geometries intersect.

This is both an optimization - so we don't have to do spatial searches very much - and a useful abstraction in that a NewsItem may be relevant in any number of places. You can get all associated Locations from a NewsItem by doing newsitem.location_set.all(), and all associated NewsItems from a Location by doing location.newsitem_set.all().

Normally you don't have to worry about creating NewsItemLocations: there are database triggers that update this table whenever a NewsItem's location is set or updated.

Parameters:

• id (AutoField) -- Id
• news_item (ForeignKey, required) -- News item
• location (ForeignKey, required) -- Location

class ebpub.db.models.NewsItemManager

Bases: django.contrib.gis.db.models.manager.GeoManager

Available as NewsItem.objects

by_attribute(*args, **kwargs)

See NewsItemQuerySet.by_attribute()

by_request(request)

See NewsItemQuerySet.by_request()

date_counts(*args, **kwargs)

See NewsItemQuerySet.date_counts()

get_query_set()

Returns a NewsItemQuerySet

text_search(*args, **kwargs)

See NewsItemQuerySet.text_search()

top_lookups(*args, **kwargs)

See NewsItemQuerySet.top_lookups()

class ebpub.db.models.NewsItemQuerySet(model=None, query=None, using=None)

Bases: django.contrib.gis.db.models.query.GeoQuerySet

Adds special methods for searching NewsItem.

by_attribute(schema_field, att_value, is_lookup=False)

Returns a QuerySet of NewsItems whose attribute value for the given SchemaField is att_value.

For example:

items = NewsItem.objects.filter(schema_id=1)
sf = SchemaField.objects.get(name='violation', schema_id=1)
items.by_attribute(sf, 'unsanitary work surface')

If att_value is a list, this will do the equivalent of an "OR" search, returning all NewsItems that have an attribute value in the att_value list.

Handles many-to-many lookups correctly behind the scenes.

If is_lookup is True, then each att_value must be either a Lookup instance, or the 'code' field of a Lookup instance, or an id of a Lookup instance. (If is_lookup is False, then only ids will work.)

Does not support comparisons other than simple equality testing.

by_request(request)

Returns a QuerySet that does additional request-specific filtering; currently this just uses get_schema_manager(request) to limit the schemas that are visible during this request.

date_counts()

Returns a dictionary mapping {item_date: count}, i.e. the number of NewsItem created each day.

text_search(schema_field, query)

Returns a QuerySet of NewsItems whose attribute for a given schema field matches a text search query.

top_lookups(schema_field, count)

Returns a list of {lookup, count} dictionaries representing the top Lookups for this QuerySet.

class ebpub.db.models.Schema(*args, **kwargs)

Bases: django.db.models.base.Model

Describes a type of NewsItem. A NewsItem has exactly one Schema, which describes its Attributes, via associated SchemaFields.

nb. to get all NewsItem instances for a Schema, you can do the usual as per http://docs.djangoproject.com/en/dev/topics/db/queries/#backwards-related-objects: schema.newsitem_set.all()

nb. Some Schemas may not be visible to some users, if eg. is_public=False. To abstract this, use the ebpub.utils.view_utils.get_schema_manager() function, rather than directly using Schema.objects or Schema.public_objects.

(To filter NewsItems appropriately, do NewsItem.objects.by_request(request) which will take care of using the right Schema manager.)

Parameters:

• id (AutoField) -- Id
• name (CharField, required) -- Name
• plural_name (CharField, required) -- Plural name
• indefinite_article (CharField, required) -- eg.'a' or 'an'
• slug (SlugField, required) -- Slug
• min_date (DateField, required) -- The earliest available pub_date for this Schema
• last_updated (DateField, required) -- Last date any NewsItems were loaded for this Schema.
• date_name (CharField, required) -- Human-readable name for the item_date field
• date_name_plural (CharField, required) -- Date name plural
• importance (SmallIntegerField, required) -- Bigger number is more important; used for sorting in some places.
• is_public (BooleanField) -- Set False if you want only people with the staff cookie to be able to see it.
• is_special_report (BooleanField) -- Whether to use the schema_detail_special_report view for these items, eg. for displaying items that have a known general Location but not a specific point.
• is_event (BooleanField) -- Whether these items are (potentially) future events rather than news in the past.
• can_collapse (BooleanField) -- Whether RSS feed should collapse many of these into one.
• has_newsitem_detail (BooleanField) -- Whether to show a detail page for NewsItems of this schema, or redirect to the NewsItem's source URL instead.
• allow_comments (BooleanField) -- Whether to allow users to add comments to NewsItems of the schema. Only applies to items with detail page.
• allow_flagging (BooleanField) -- Whether to allow users to flag NewsItems of this schema as spam or inappropriate.
• allow_charting (BooleanField) -- Whether aggregate charts are displayed on the home page of this Schema
• uses_attributes_in_list (BooleanField) -- Whether attributes should be preloaded for NewsItems of this Schema, in the list view
• number_in_overview (SmallIntegerField, required) -- Number of records to show on place_overview
• map_icon_url (TextField) -- Set this to a URL to a small image icon and it will be displayed on maps. Should be roughly 40x40 pixels. Optional.
• map_color (CharField) -- CSS color code used on maps to display this type of news. eg #FF0000. Only used if map_icon_url is not set. Optional.
• edit_window (FloatField) -- How long, in hours, the creator of an item is allowed to edit it. Set to 0 to disallow edits by non-Admin users. Set to -1 to allow editing forever.
• short_description (TextField) -- Short description
• summary (TextField) -- Summary
• source (TextField) -- Where this information came from, as one or more URLs, one per line.
• short_source (CharField) -- Short source
• update_frequency (CharField) -- Update frequency

class ebpub.db.models.SchemaField(*args, **kwargs)

Bases: django.db.models.base.Model

Describes the meaning of one Attribute field for one Schema type.

Parameters:

• id (AutoField) -- Id
• schema (ForeignKey, required) -- Schema
• pretty_name (CharField, required) -- Human-readable name of this field, for display.
• pretty_name_plural (CharField, required) -- Plural human-readable name
• name (SlugField, required) -- Name
• real_name (CharField, required) -- Column name in the Attribute model. 'varchar01', 'varchar02', etc.
• display (BooleanField) -- Whether to display value on the public site.
• is_lookup (BooleanField) -- Whether the value is a foreign key to Lookup.
• is_filter (BooleanField) -- Whether to link to list of items with the same value in this field. Assumes is_lookup=True.
• is_charted (BooleanField) -- Whether the schema detail view displays a chart for this field; also see "trends" tabs on place overview page. Assumes is_lookup=True.
• display_order (SmallIntegerField, required) -- Display order
• is_searchable (BooleanField) -- Whether the value is searchable by content. Doesn't make sense if is_lookup=True.

browse_by_title()

Returns FOO in 'Browse by FOO', for this SchemaField.

is_many_to_many_lookup()

Returns True if this SchemaField is a many-to-many lookup.

is_type(*data_types)

Returns True if this SchemaField is of any of the given data types.

Allowed values are 'varchar', 'date', 'time', 'datetime', 'bool', 'int'.

smart_pretty_name()

Returns the pretty name for this SchemaField, taking into account many-to-many fields.

class ebpub.db.models.SearchSpecialCase(*args, **kwargs)

Bases: django.db.models.base.Model

Used as a fallback for location searches that don't match any Location, Intersection, etc.

Parameters:

• id (AutoField) -- Id
• query (CharField, required) -- Normalized form of search queries that match this special case.
• redirect_to (CharField) -- Optional absolute URL to redirect to on searches that match the query.
• title (CharField) -- Title to display on the results page if we don't redirect.
• body (TextField) -- Body to display on the result page if we don't redirect. HTML is OK.

ebpub.db.models.field_mapping(schema_id_list)

Given a list of schema IDs, returns a dictionary of dictionaries, mapping schema_ids to dictionaries mapping the fields' name->real_name. Example return value:

{1: {u'crime_type': 'varchar01', u'crime_date', 'date01'},
 2: {u'permit_number': 'int01', 'to_date': 'date01'},
}

ebpub.db.models.get_city_locations()

If we have configured multiple_cities, find all Locations of the city_location_type. Otherwise, empty query set.

ebpub.db.models.get_valid_real_names()

Field names of Attribute, suitable for use as SchemaField.real_name.

schemafilters Module

API that abstracts filtering NewsItems by various criteria. Features:

• validation of filter parameters.
• detecting when more user input is needed, so views can provide a disambiguation UI.
• consistent normalized URLs and breadcrumbs.
• consistent order of filter application

TODO:

• generate URLs for the REST API too?
• profile optimal order of filters for sort()? Currently guessing it should be something like: schema, date, non-lookup attrs, block/location, lookup attrs, text search. This will need profiling with lots of test data.

class ebpub.db.schemafilters.AttributeFilter(request, context, queryset, *args, **kwargs)

Bases: ebpub.db.schemafilters.NewsitemFilter

Base class for more specific types of attribute filters (LookupFilter, TextSearchFilter, etc).

class ebpub.db.schemafilters.BlockFilter(request, context, queryset, *args, **kwargs)

Bases: ebpub.db.schemafilters.NewsitemFilter

Filters on intersecting ebpub.streets.models.Block.

apply()

filtering by Block.

class ebpub.db.schemafilters.BoolFilter(request, context, queryset, *args, **kwargs)

Bases: ebpub.db.schemafilters.AttributeFilter

Filters on boolean attributes.

class ebpub.db.schemafilters.DateFilter(request, context, queryset, *args, **kwargs)

Bases: ebpub.db.schemafilters.NewsitemFilter

Filters on NewsItem.item_date. The start_date and end_date args are inclusive. They can be the same; missing end_date implies start == end.

apply()

filtering by Date

exception ebpub.db.schemafilters.DuplicateFilterError(msg, url=None)

Bases: ebpub.db.schemafilters.FilterError

Raised if we try to add conflicting filters to a FilterChain.

class ebpub.db.schemafilters.FilterChain(data=None, request=None, context=None, queryset=None, schema=None)

Bases: django.utils.datastructures.SortedDict

A set of NewsitemFilters, to be applied in a predictable order.

The update_from_request() method can be used to configure one based on the request URL.

Also handles URL generation.

add(key, *values, **kwargs)

Given a key that is a string or a SchemaField, construct an appropriate NewsitemFilter with the values as arguments, and save it as self[key], where the new key is either the string key or derived automatically from the SchemaField.

This does no parsing of values. The filter added may be determined by the type of the values passed. Eg. if the value is a Block, Location, or LocationType, a LocationFilter or BlockFilter will be added under the key 'location'. If the value is a datetime, a DateFilter or PubDateFilter will be added depending on the key.

Yes, this smells too complicated.

For convenience, this returns self.

add_by_place_id(pid)

pid is a place id string as used by parse_pid and make_pid, identifying a location or block (and if a block, a radius).

add_by_schemafield(schemafield, *values, **kwargs)

Given a SchemaField, construct an appropriate NewsitemFilter with the values as arguments, and save it as self[schemafield.name].

For convenience, returns self.

apply(queryset=None)

Applies each filter in the chain.

filters_for_display()

If a filter has no label, that means don't show it in various places in the UI. This is a convenience to get only the values that should be shown.

make_breadcrumbs(additions=(), removals=(), stop_at=None, base_url=None)

Returns a list of (label, URL) pairs suitable for making breadcrumbs for the schema_filter view.

If base_url is passed, URLs generated will be include that that; otherwise fall back to self.base_url, which falls back to self.schema.url().

If stop_at is passed, the key specified will be the last one used for the breadcrumb list.

If removals is passed, the specified filter keys will be excluded from the breadcrumb list.

If additions is passed, the specified (key, NewsitemFilter) pairs will be added to the end of the breadcrumb list.

(In some cases, you can pass (key, [args]) and it will figure out what kind of NewsitemFilter to create. TODO: document this!!)

Also, if self.other_query_params is a dictionary, its items will be added as query parameters to all the URLs. This can be used to add or preserve query parameters that aren't relevant to the FilterChain.

In all URLs, query parameters will be sorted alphabetically by name.

make_url(additions=(), removals=(), stop_at=None, base_url=None)

Makes one URL representing all the filters of this filter chain, for the schema_filter view.

replace(key, *values)

Same as self.add(), but instead of raising DuplicateFilterError on existing keys, replaces them.

sort()

Put keys in optimal order.

update_from_request(filter_sf_dict)

Update the list of filters based on the request params.

After this is called, it's recommended to redirect to a normalized form of the URL, which you can get via self.sort(); self.make_url()

filter_sf_dict is a mapping of name -> SchemaField which have either is_filter or is_searchable True. We remove SchemaFields that we create filters for. (This is so that templates can display input widgets for the ones we're not already filtering by.)

TODO: This should not bail out on the first error, it should do as much as possible and signal multiple errors. (Use the forms framework?)

validate()

Check whether any of the filters were requested without a required value. If so, return info about what's needed, as a dict. Stops on the first one that returns anything.

Can raise FilterError.

exception ebpub.db.schemafilters.FilterError(msg, url=None)

Bases: exceptions.Exception

All-purpose exception for invalid filter parameters and the like.

If self.url is set, it may be used for redirection.

class ebpub.db.schemafilters.IdFilter(request, context, queryset, *args, **kwargs)

Bases: ebpub.db.schemafilters.NewsitemFilter

Filters by NewsItem ids, which may be a list.

apply()

Filtering by ID.

class ebpub.db.schemafilters.LocationFilter(request, context, queryset, *args, **kwargs)

Bases: ebpub.db.schemafilters.NewsitemFilter

Filters on intersecting ebpub.db.models.Location.

apply()

filtering by Location

class ebpub.db.schemafilters.LookupFilter(request, context, queryset, *args, **kwargs)

Bases: ebpub.db.schemafilters.AttributeFilter

Filters on Lookup attributes (see ebpub.db.models for more info).

Note the args are expected to be either Lookup instances or Lookup.slug, but not Lookup.code, because the slugs are safe for use in URLs and the codes may not be.

class ebpub.db.schemafilters.NewsitemFilter(request, context, queryset=None, *args, **kw)

Bases: object

Base class for filtering NewsItems.

apply()

mutate and return the queryset, and modify any other state that needs sharing with others.

get_query_params()

Suitable for composing query strings out of dictionaries.

validate()

If we didn't get enough info from the args, eg. it's a Location filter but no location was specified, then return a dict of stuff for putting in a template context.

If we have enough information, returns an empty dict.

The dict keys are::

'filter_key': The filter slug. 'param_name': Query parameter for this filter, for forms or URLs. 'param_label': Human-readable name of this filter. 'option_list': A list of possible argument values for this filter. Each is a dict with keys 'value' (usable for input values) and 'name' (human-readable). 'select_multiple': boolean, whether you can filter on more than one value.

... or maybe this should be something more generic across both REST and UI views

class ebpub.db.schemafilters.PubDateFilter(request, context, queryset, *args, **kwargs)

Bases: ebpub.db.schemafilters.DateFilter

Filters on NewsItem.pub_date.

class ebpub.db.schemafilters.SchemaFilter(request, context, queryset, *args, **kwargs)

Bases: ebpub.db.schemafilters.NewsitemFilter

Filters by NewsItem.schema, which may be a list.

Schemas that the current user is not allowed to see will be filtered out.

class ebpub.db.schemafilters.TextSearchFilter(request, context, queryset, *args, **kwargs)

Bases: ebpub.db.schemafilters.AttributeFilter

Does a text search on values of the given attribute.

ebpub.db.schemafilters.radius_from_slug(slug)

Extract radius from a string like 8-blocks, 1-block, ...

ebpub.db.schemafilters.radius_slug(radius)

Return radius string like 8-blocks, 1-block ...

urlresolvers Module

ebpub.db.urlresolvers.filter_reverse(slug, args)

Generate a reverse schema_filter URL.

utils Module

ebpub.db.utils.block_radius_value(request)

Get block radius from either query string or cookie, or default.

ebpub.db.utils.get_place_info_for_request(request, *args, **kwargs)

A utility function that abstracts getting some commonly used location-related information: a place (Location or Block), its type, a bbox, a list of nearby locations, etc.

ebpub.db.utils.make_search_buffer(geom, block_radius)

Returns a polygon of a buffer around a block's centroid.

geom is the centroid of the block, and block_radius is the number of blocks.

ebpub.db.utils.populate_attributes_if_needed(newsitem_list, schema_list, get_lookups=True)

Optimization helper function that takes a list of NewsItems and ensures the ni.attributes pseudo-dictionary is populated, for all NewsItems whose schemas have uses_attributes_in_list=True. This is accomplished with a minimal amount of database queries.

The values in the NewsItem.attributes pseudo-dictionary are Lookup instances in the case of Lookup fields. Otherwise, they're the direct values from the Attribute table.

(Note this is different than accessing NewsItem.attributes without having called this function, in which case Lookups are not dereferenced automatically. Client code such as AttributesForTemplate should handle both cases - or really .attributes should be fixed to be consistent.)

schema_list should be a list of all Schemas that are referenced in newsitem_list.

Note that the list is edited in place; there is no return value.

views Module

ebpub.db.views.ajax_place_date_chart(request)

Returns HTML fragment containing a chart of how many news items were added for each day over a short period (length defined by constants.DAYS_SHORT_AGGREGATE_TIMEDELTA).

Expects request.GET['pid'] and request.GET['s'] (a Schema ID).

ebpub.db.views.ajax_place_lookup_chart(request)

Returns HTML fragment -- expects request.GET['pid'] and request.GET['sf'] (a SchemaField ID).

ebpub.db.views.block_bbox(block, radius)

Given a ebpub.streets.models.Block, and an integer radius, returns a geometry representing a bounding box around the block.

Assumes block` has wkt attribute.

ebpub.db.views.get_date_chart(schemas, start_date, end_date, counts)

Returns a list that's used to display a date chart for the given schemas. Note that start_date and end_date should be datetime.date objects, NOT datetime.datetime objects, and they are inclusive, i.e. the resulting chart will include both start_date and end_date.

counts should be a nested dictionary: {schema_id: {date: count}}

The list will be in order given by the schemas parameter.

ebpub.db.views.get_date_chart_agg_model(schemas, start_date, end_date, agg_model, kwargs=None)

start_date and end_date are inclusive.

ebpub.db.views.homepage(request)

Front page of the default OpenBlock theme.

ebpub.db.views.location_type_list(request)

Default view of /locations; just redirect to the default loc type.

ebpub.db.views.newsitem_detail(request, *args, **kwargs)

Page displaying a single NewsItem.

ebpub.db.views.newsitems_geojson(request)

Get a list of newsitems, optionally filtered for one place ID and/or one schema slug.

Response is a geojson string.

ebpub.db.views.place_detail_overview(request, *args, **kwargs)

Recent news AND upcoming events for a Location or Block, grouped by Schema.

ebpub.db.views.place_detail_timeline(request, *args, **kwargs)

Recent news OR upcoming events for the given Location or Block.

ebpub.db.views.schema_detail_special_report(request, schema)

For display of schemas where is_special_report=True.

ebpub.db.views.schema_filter(request, slug)

List NewsItems for one schema, filtered by various criteria in the query params (eg. date, location, or values of SchemaFields).

ebpub.db.views.search(request, schema_slug='')

Performs a location search and redirects to the address/xy page.

Subpackages

• bin Package
  • bin Package
  • activate_schema Module
  • add_location Module
  • alphabetize_locations Module
  • delete_newsitems Module
  • export_schema Module
  • geocode_newsitems Module
  • import_hoods Module
  • import_locations Module
  • import_zips Module
  • update_aggregates Module
• templatetags Package
  • templatetags Package
  • dateutils Module
  • eb Module
  • eb_filter Module
  • eb_json Module
  • full_links Module
  • mapping Module
  • raw Module