User guide

This user guide covers using CKAN’s web interface to organize, publish and find data. CKAN also has a powerful API (machine interface), which makes it easy to develop extensions and links with other information systems.

Some web UI features relating to site administration are available only to users with sysadmin status.

What is CKAN?

CKAN is a tool for making open data websites. (Think of a content management system like WordPress - but for data, instead of pages and blog posts.) It helps you manage and publish collections of data. It is used by national and local governments, research institutions, and other organizations who collect a lot of data.

Once your data is published, users can use its faceted search features to browse and find the data they need, and preview it using maps, graphs and tables - whether they are developers, journalists, researchers, NGOs, citizens, or even your own staff.

Datasets and resources

For CKAN purposes, data is published in units called “datasets”. A dataset is a parcel of data - for example, it could be the crime statistics for a region, the spending figures for a government department, or temperature readings from various weather stations. When users search for data, the search results they see will be individual datasets.

A dataset contains two things:

  • Information or “metadata” about the data. For example, the title and publisher, date, what formats it is available in, what license it is released under, etc.
  • A number of “resources”, which hold the data itself. CKAN does not mind what format the data is in. A resource can be a CSV or Excel spreadsheet, XML file, PDF document, image file, linked data in RDF format, etc. CKAN can store the resource internally, or store it simply as a link, the resource itself being elsewhere on the web. A dataset can contain any number of resources. For example, different resources might contain the data for different years, or they might contain the same data in different formats.

Users, organizations and authorization

CKAN users can register user accounts and log in. Normally (depending on the site setup), login is not needed to search for and find data, but is needed for all publishing functions: datasets can be created, edited, etc by users with the appropriate permissions.

Normally, each dataset is owned by an “organization”. A CKAN instance can have any number of organizations. For example, if CKAN is being used as a data portal by a national government, the organizations might be different government departments, each of which publishes data. Each organization can have its own workflow and authorizations, allowing it to manage its own publishing process.

An organization’s administrators can add individual users to it, with different roles depending on the level of authorization needed. A user in an organization can create a dataset owned by that organization. In the default setup, this dataset is initially private, and visible only to other users in the same organization. When it is ready for publication, it can be published at the press of a button. This may require a higher authorization level within the organization.

Datasets cannot normally be created except within organizations. It is possible, however, to set up CKAN to allow datasets not owned by any organization. Such datasets can be edited by any logged-in user, creating the possibility of a wiki-like datahub.

 

Using CKAN

Registering and logging in

Note

Registration is needed for most publishing features and for personalization features, such as “following” datasets. It is not needed to search for and download data.

To create a user ID, use the “Login” link at the top of any page. CKAN will forward you to SOI login page. You will be prompted to select either Public or Employee account. Select and enter the information:

  • Username – This should be either illinois.gov or public.external.illinois.gov account. For example, “illinois\testuser” or “central\testuser (if you are part of trusted domain)”.
  • Password – enter the same password in both boxes

Illinois.gov logon screen

If there are problems with any of the fields, service will tell you the problem and enable you to correct it. When the fields are filled in correctly, CKAN will create your user account and/or automatically log you in. If you do not have an account please select to create a new one and follow the prompts.

Note

Please note you will need get in touch with State Data Practice @ “DoIT.SDP.Support@illinois.gov” to request organization administration and data publishing roles. Please submit ESR to DoIT helpdesk together with below information:

  • Username
  • Organization Name
  • Access Type
  • Justification

 

Features for publishers

Adding a new dataset

Note

You will need to be a member of an organization in order to add and edit datsets. On http://demo.ckan.org, you can add a dataset without being in an organization, but dataset features relating to authorization and organizations will not be available.

Step 1. You can access CKAN’s “Create dataset” screen in two ways.

  1. Select the “Datasets” link at the top of any page. From this, above the search box, select the “Add Dataset” button.
  2. Alternatively, select the “organizations” link at the top of a page. Now select the page for the organization that should own your new dataset. Provided that you are a member of this organization, you can now select the “Add Dataset” button above the search box.

Step 2. CKAN will ask for the following information about your data. (The actual data will be added in step 4.)

  • Title – this title will be unique across CKAN, so make it brief but specific. E.g. “UK population density by region” is better than “Population figures”.
  • Description – You can add a longer description of the dataset here, including information such as where the data is from and any information that people will need to know when using the data.
  • Tags – here you may add tags that will help people find the data and link it with other related data. Examples could be “population”, “crime”, “East Anglia”. Hit the <return> key between tags. If you enter a tag wrongly, you can use its delete button to remove it before saving the dataset.
  • License – it is important to include license information so that people know how they can use the data. This field should be a drop-down box. If you need to use a license not on the list, contact your site administrator.
  • Organization - If you are a member of any organizations, this drop-down will enable you to choose which one should own the dataset. Ensure the default chosen is the correct one before you proceed. (Probably most users will be in only one organization. If this is you, CKAN will have chosen your organization by default and you need not do anything.)

Create dataset 1st screen

Note

By default, the only required field on this page is the title. However, it is good practice to include, at the minimum, a short description and, if possible, the license information. You should ensure that you choose the correct organization for the dataset, since at present, this cannot be changed later. You can edit or add to the other fields later.

Step 3. When you have filled in the information on this page, select the “Next: Add Data” button. (Alternatively select “Cancel” to discard the information filled in.)

Step 4. CKAN will display the “Add data” screen.

This is where you will add one or more “resources” which contain the data for this dataset. Choose a file or link for your data resource and select the appropriate choice at the top of the screen:

  • If you are giving CKAN a link to the data, like http://example.com/mydata.csv, then select “Link to a file” or “Link to an API”. (If you don’t know what an API is, you don’t need to worry about this option - select “Link to a file”.)
  • If the data to be added to CKAN is in a file on your computer, select “Upload a file”. CKAN will give you a file browser to select it.

Step 5. Add the other information on the page. CKAN does not require this information, but it is good practice to add it:

  • Name – a name for this resource, e.g. “Population density 2011, CSV”. Different resources in the dataset should have different names.
  • Description – a short description of the resource.
  • Format – the file format of the resource, e.g. CSV (comma-separated values), XLS, JSON, PDF, etc.

Step 6. If you have more resources (files or links) to add to the dataset, select the “Save & add another” button. When you have finished adding resources, select “Next: Additional Info”.

Step 7. CKAN displays the “Additional data” screen.

  • Visibility – a Public dataset is public and can be seen by any user of the site. A Private dataset can only be seen by members of the organization owning the dataset and will not show up in searches by other users.
  • Author – The name of the person or organization responsible for producing the data.
  • Author e-mail – an e-mail address for the author, to which queries about the data should be sent.
  • Maintainer / maintainer e-mail – If necessary, details for a second person responsible for the data.
  • Custom fields – If you want the dataset to have another field, you can add the field name and value here. E.g. “Year of publication”. Note that if there is an extra field that is needed for a large number of datasets, you should talk to your site administrator about changing the default schema and dataset forms.

create dataset screen 2

Note

Everything on this screen is optional, but you should ensure the “Visibility” is set correctly. It is also good practice to ensure an Author is named.

Changed in version 2.2: Previous versions of CKAN used to allow adding the dataset to existing groups in this step. This was changed. To add a dataset to an existing group now, go to the “Group” tab in the Dataset’s page.

Step 8. Select the ‘Finish’ button. CKAN creates the dataset and shows you the result. You have finished!

You should be able to find your dataset by typing the title, or some relevant words from the description, into the search box on any page in your CKAN instance.

Editing a dataset

You can edit the dataset you have created, or any dataset owned by an organization that you are a member of. (If a dataset is not owned by any organization, then any registered user can edit it.)

  1. Go to the dataset’s page. You can find it by entering the title in the search box on any page.
  2. Select the “Edit” button, which you should see above the dataset title.
  3. CKAN displays the “Edit dataset” screen. You can edit any of the fields (Title, Description, Dataset, etc), change the visibility (Private/Public), and add or delete tags or custom fields.
  4. When you have finished, select the “Update dataset” button to save your changes.

edit metadata screen 1

Adding, deleting and editing resources

  1. Go to the dataset’s “Edit dataset” page (steps 1-2 above).
  2. In the left sidebar, there are options for editing resources. You can select an existing resource (to edit or delete it), or select “Add new resource”.
  3. You can edit the information about the resource or change the linked or uploaded file. For details, see steps 4-5 of “Adding a new resource”, above.
  4. When you have finished editing, select the button marked “Update resource” (or “Add”, for a new resource) to save your changes. Alternatively, to delete the resource, select the “Delete resource” button.

Deleting a dataset

  1. Go to the dataset’s “Edit dataset” page (see “Editing a dataset”, above).
  2. Select the “Delete” button.
  3. CKAN displays a confirmation dialog box. To complete deletion of the dataset, select “Confirm”.

Note

The “Deleted” dataset is not completely deleted. It is hidden, so it does not show up in any searches, etc. However, by visiting the URL for the dataset’s page, it can still be seen (by users with appropriate authorization), and “undeleted” if necessary. If it is important to completely delete the dataset, contact your site administrator.

Managing an organization

When you create an organization, CKAN automatically makes you its “Admin”. From the organization’s page you should see an “Admin” button above the search box. When you select this, CKAN displays the organization admin page. This page has two tabs:

  • Info – Here you can edit the information supplied when the organization was created (title, description and image).
  • Members – Here you can add, remove and change access roles for different users in the organization. Note: you will need to know their username on CKAN.

/base/images/manage_organization.png

By default CKAN allows members of organizations with three roles:

  • Member – can see the organization’s private datasets
  • Editor – can edit and publish datasets
  • Admin – can add, remove and change roles for organization members

Finding data

Searching the site

To find datasets in CKAN, type any combination of search words (e.g. “health”, “transport”, etc) in the search box on any page. CKAN displays the first page of results for your search. You can:

  • View more pages of results
  • Repeat the search, altering some terms
  • Restrict the search to datasets with particular tags, data formats, etc using the filters in the left-hand column

If there are a large number of results, the filters can be very helpful, since you can combine filters, selectively adding and removing them, and modify and repeat the search with existing filters still in place.

If datasets are tagged by geographical area, it is also possible to run CKAN with an extension which allows searching and filtering of datasets by selecting an area on a map.

/base/images/search_the_site.png

Searching within an organization

If you want to look for data owned by a particular organization, you can search within that organization from its home page in CKAN.

  1. Select the “Organizations” link at the top of any page.
  2. Select the organization you are interested in. CKAN will display your organization’s home page.
  3. Type your search query in the main search box on the page.

CKAN will return search results as normal, but restricted to datasets from the organization.

If the organization is of interest, you can opt to be notified of changes to it (such as new datasets and modifications to datasets) by using the “Follow” button on the organization page. You must have a user account and be logged in to use this feature.

Exploring datasets

When you have found a dataset you are interested and selected it, CKAN will display the dataset page. This includes

  • The name, description, and other information about the dataset
  • Links to and brief descriptions of each of the resources

/base/images/exploring_datasets.png

The resource descriptions link to a dedicated page for each resource. This resource page includes information about the resource, and enables it to be downloaded. Many types of resource can also be previewed directly on the resource page. .CSV and .XLS spreadsheets are previewed in a grid view, with map and graph views also available if the data is suitable. The resource page will also preview resources if they are common image types, PDF, or HTML.

The dataset page also has two other tabs:

  • Activity stream – see the history of recent changes to the dataset
  • Related items – see any links to web pages related to this dataset, or add your own links.

If the dataset is of interest, you can opt to be notified of changes to it by using the “Follow” button on the dataset page. You must have a user account and be logged in to use this feature.

API guide

This section documents CKAN’s API, for developers who want to write code that interacts with CKAN sites and their data.

CKAN’s Action API is a powerful, RPC-style API that exposes all of CKAN’s core features to API clients. All of a CKAN website’s core functionality (everything you can do with the web interface and more) can be used by external code that calls the CKAN API. For example, using the CKAN API your app can:

The DataStore API

The CKAN DataStore offers an API for reading, searching and filtering data without the need to download the entire file first. The DataStore is an ad hoc database which means that it is a collection of tables with unknown relationships. This allows you to search in one DataStore resource (a table in the database) as well as queries across DataStore resources.

Data can be written incrementally to the DataStore through the API. New data can be inserted, existing data can be updated or deleted. You can also add a new column to an existing table even if the DataStore resource already contains some data.

Triggers may be added to enforce validation, clean data as it is loaded or even record record histories. Triggers are PL/pgSQL functions that must be created by a sysadmin.

You will notice that we tried to keep the layer between the underlying PostgreSQL database and the API as thin as possible to allow you to use the features you would expect from a powerful database management system.

A DataStore resource can not be created on its own. It is always required to have an associated CKAN resource. If data is stored in the DataStore, it will automatically be previewed by the preview extension.

Making a DataStore API request

Making a DataStore API request is the same as making an Action API request: you post a JSON dictionary in an HTTP POST request to an API URL, and the API also returns its response in a JSON dictionary.

API reference

Note

Lists can always be expressed in different ways. It is possible to use lists, comma separated strings or single items. These are valid lists: ['foo', 'bar']'foo, bar'"foo", "bar" and 'foo'. Additionally, there are several ways to define a boolean value. Trueon and 1are all vaid boolean values.

Note

The table structure of the DataStore is explained in Internal structure of the database.

ckanext.datastore.logic.action.datastore_create(contextdata_dict)

Adds a new table to the DataStore.

The datastore_create action allows you to post JSON data to be stored against a resource. This endpoint also supports altering tables, aliases and indexes and bulk insertion. This endpoint can be called multiple times to initially insert more data, add fields, change the aliases or indexes as well as the primary keys.

To create an empty datastore resource and a CKAN resource at the same time, provide resource with a valid package_id and omit theresource_id.

If you want to create a datastore resource from the content of a file, provide resource with a valid url.

See Fields and Records for details on how to lay out records.

Parameters:
  • resource_id (string) – resource id that the data is going to be stored against.
  • force (bool (optional, default: False)) – set to True to edit a read-only resource
  • resource (dictionary) – resource dictionary that is passed to resource_create(). Use instead of resource_id(optional)
  • aliases (list or comma separated string) – names for read only aliases of the resource. (optional)
  • fields (list of dictionaries) – fields/columns and their extra metadata. (optional)
  • records (list of dictionaries) – the data, eg: [{“dob”: “2005”, “some_stuff”: [“a”, “b”]}] (optional)
  • primary_key (list or comma separated string) – fields that represent a unique key (optional)
  • indexes (list or comma separated string) – indexes on table (optional)
  • triggers (list of dictionaries) – trigger functions to apply to this table on update/insert. functions may be created with datastore_function_create(). eg: [ {“function”: “trigger_clean_reference”}, {“function”: “trigger_check_codes”}]

Please note that setting the aliasesindexes or primary_key replaces the exising aliases or constraints. Setting records appends the provided records to the resource.

Results:

Returns: The newly created data object.
Return type: dictionary

See Fields and Records for details on how to lay out records.

ckanext.datastore.logic.action.datastore_run_triggers(contextdata_dict)

update each record with trigger

The datastore_run_triggers API action allows you to re-apply exisitng triggers to an existing DataStore resource.

Parameters: resource_id (string) – resource id that the data is going to be stored under.

Results:

Returns: The rowcount in the table.
Return type: int

ckanext.datastore.logic.action.datastore_upsert(contextdata_dict)

Updates or inserts into a table in the DataStore

The datastore_upsert API action allows you to add or edit records to an existing DataStore resource. In order for the upsert and updatemethods to work, a unique key has to be defined via the datastore_create action. The available methods are:

upsert

Update if record with same key already exists, otherwise insert. Requires unique key.

insert

Insert only. This method is faster that upsert, but will fail if any inserted record matches an existing one. Does not require a unique key.

update

Update only. An exception will occur if the key that should be updated does not exist. Requires unique key.

Parameters:
  • resource_id (string) – resource id that the data is going to be stored under.
  • force (bool (optional, default: False)) – set to True to edit a read-only resource
  • records (list of dictionaries) – the data, eg: [{“dob”: “2005”, “some_stuff”: [“a”,”b”]}] (optional)
  • method (string) – the method to use to put the data into the datastore. Possible options are: upsert, insert, update (optional, default: upsert)

Results:

Returns: The modified data object.
Return type: dictionary

ckanext.datastore.logic.action.datastore_info(contextdata_dict)

Returns information about the data imported, such as column names and types.

Return type: A dictionary describing the columns and their types.
Parameters: id (A UUID) – Id of the resource we want info about

ckanext.datastore.logic.action.datastore_delete(contextdata_dict)

Deletes a table or a set of records from the DataStore.

Parameters:
  • resource_id (string) – resource id that the data will be deleted from. (optional)
  • force (bool (optional, default: False)) – set to True to edit a read-only resource
  • filters (dictionary) – filters to apply before deleting (eg {“name”: “fred”}). If missing delete whole table and all dependent views. (optional)

Results:

Returns: Original filters sent.
Return type: dictionary

ckanext.datastore.logic.action.datastore_search(contextdata_dict)

Search a DataStore resource.

The datastore_search action allows you to search data in a resource. DataStore resources that belong to private CKAN resource can only be read by you if you have access to the CKAN resource and send the appropriate authorization.

Parameters:
  • resource_id (string) – id or alias of the resource to be searched against
  • filters (dictionary) – matching conditions to select, e.g {“key1”: “a”, “key2”: “b”} (optional)
  • q (string or dictionary) – full text query. If it’s a string, it’ll search on all fields on each row. If it’s a dictionary as {“key1”: “a”, “key2”: “b”}, it’ll search on each specific field (optional)
  • distinct (bool) – return only distinct rows (optional, default: false)
  • plain (bool) – treat as plain text query (optional, default: true)
  • language (string) – language of the full text query (optional, default: english)
  • limit (int) – maximum number of rows to return (optional, default: 100)
  • offset (int) – offset this number of rows (optional)
  • fields (list or comma separated string) – fields to return (optional, default: all fields in original order)
  • sort (string) – comma separated field names with ordering e.g.: “fieldname1, fieldname2 desc”
  • include_total (bool) – True to return total matching record count (optional, default: true)
  • records_format (controlled list) – the format for the records return value: ‘objects’ (default) list of {fieldname1: value1, ...} dicts, ‘lists’ list of [value1, value2, ...] lists, ‘csv’ string containing comma-separated values with no header, ‘tsv’ string containing tab-separated values with no header

A listing of all available resources can be found at the alias _table_metadata.

Results:

The result of this action is a dictionary with the following keys:

Return type:

A dictionary with the following keys

Parameters:
  • fields (list of dictionaries) – fields/columns and their extra metadata
  • offset (int) – query offset value
  • limit (int) – query limit value
  • filters (list of dictionaries) – query filters
  • total (int) – number of total matching records
  • records (depends on records_format value passed) – list of matching results

ckanext.datastore.logic.action.datastore_search_sql(contextdata_dict)

Execute SQL queries on the DataStore.

The datastore_search_sql action allows a user to search data in a resource or connect multiple resources with join expressions. The underlying SQL engine is the PostgreSQL engine. There is an enforced timeout on SQL queries to avoid an unintended DOS. DataStore resource that belong to a private CKAN resource cannot be searched with this action. Use datastore_search() instead.

Note

This action is only available when using PostgreSQL 9.X and using a read-only user on the database. It is not available in legacy mode.

Parameters: sql (string) – a single SQL select statement

Results:

The result of this action is a dictionary with the following keys:

Return type:

A dictionary with the following keys

Parameters:
  • fields (list of dictionaries) – fields/columns and their extra metadata
  • records (list of dictionaries) – list of matching results

ckanext.datastore.logic.action.datastore_make_private(contextdata_dict)

Deny access to the DataStore table through datastore_search_sql().

This action is called automatically when a CKAN dataset becomes private or a new DataStore table is created for a CKAN resource that belongs to a private dataset.

Parameters: resource_id (string) – id of resource that should become private

ckanext.datastore.logic.action.datastore_make_public(contextdata_dict)

Allow access to the DataStore table through datastore_search_sql().

This action is called automatically when a CKAN dataset becomes public.

Parameters: resource_id (string) – if of resource that should become public

ckanext.datastore.logic.action.set_datastore_active_flag(modeldata_dictflag)

Set appropriate datastore_active flag on CKAN resource.

Called after creation or deletion of DataStore table.

ckanext.datastore.logic.action.datastore_function_create(contextdata_dict)

Create a trigger function for use with datastore_create

Parameters:
  • name (string) – function name
  • or_replace (bool) – True to replace if function already exists (default: False)
  • rettype (string) – set to ‘trigger’ (only trigger functions may be created at this time)
  • definition (string) – PL/pgSQL function body for trigger function

ckanext.datastore.logic.action.datastore_function_delete(contextdata_dict)

Delete a trigger function

Parameters: name (string) – function name

Download resource

A DataStore resource can be downloaded in the CSV file format from {CKAN-URL}/datastore/dump/{RESOURCE-ID}.

For an Excel-compatible CSV file use {CKAN-URL}/datastore/dump/{RESOURCE-ID}?bom=true.

Other formats supported include tab-separated values (?format=tsv), JSON (?format=json) and XML (?format=xml). E.g. to download an Excel-compatible tab-separated file use {CKAN-URL}/datastore/dump/{RESOURCE-ID}?format=tsv&bom=true.

Fields

Fields define the column names and the type of the data in a column. A field is defined as follows:

{
     "id":    # a string which defines the column name
     "type":  # the data type for the column
}
                                                    

Field types are optional and will be guessed by the DataStore from the provided data. However, setting the types ensures that future inserts will not fail because of wrong types. See Field types for details on which types are valid.

Example:

[
    {
        "id": "foo",
        "type": "int4"
    },
    {
        "id": "bar"
        # type is optional
    }
]
                                                    

Records

A record is the data to be inserted in a DataStore resource and is defined as follows:

{
    "<id>":  # data to be set
    # .. more data
}
                                                    

Example:

[
    {
        "foo": 100,
        "bar": "Here's some text"
    },
    {
        "foo": 42
    }
]
                                                    

Field types

The DataStore supports all types supported by PostgreSQL as well as a few additions. A list of the PostgreSQL types can be found in the type section of the documentation. Below you can find a list of the most common data types. The json type has been added as a storage for nested data.

In addition to the listed types below, you can also use array types. They are defines by prepending a _ or appending [] or [n] where n denotes the length of the array. An arbitrarily long array of integers would be defined as int[].

text

Arbitrary text data, e.g. Here's some text.

json

Arbitrary nested json data, e.g {"foo": 42, "bar": [1, 2, 3]}. Please note that this type is a custom type that is wrapped by the DataStore.

date

Date without time, e.g 2012-5-25.

time

Time without date, e.g 12:42.

timestamp

Date and time, e.g 2012-10-01T02:43Z.

int

Integer numbers, e.g 427.

float

Floats, e.g. 1.61803.

bool

Boolean values, e.g. true0

You can find more information about the formatting of dates in the date/time types section of the PostgreSQL documentation.

Resource aliases

A resource in the DataStore can have multiple aliases that are easier to remember than the resource id. Aliases can be created and edited with the datastore_create() API endpoint. All aliases can be found in a special view called _table_metadata. See Internal structure of the database for full reference.

HTSQL support (not supported in current release)

The ckanext-htsql extension adds an API action that allows a user to search data in a resource using the HTSQL query expression language. Please refer to the extension documentation to know more.

Comparison of different querying methods

The DataStore supports querying with multiple API endpoints. They are similar but support different features. The following list gives an overview of the different methods.

  datastore_search() datastore_search_sql() HTSQL
Ease of use Easy Complex Medium
Flexibility Low High Medium
Query language Custom (JSON) SQL HTSQL
Join resources No Yes No

Internal structure of the database

The DataStore is a thin layer on top of a PostgreSQL database. Each DataStore resource belongs to a CKAN resource. The name of a table in the DataStore is always the resource id of the CKAN resource for the data.

As explained in Resource aliases, a resource can have mnemonic aliases which are stored as views in the database.

All aliases (views) and resources (tables respectively relations) of the DataStore can be found in a special view called _table_metadata. To access the list, open http://{YOUR-CKAN-INSTALLATION}/api/3/action/datastore_search?resource_id=_table_metadata.

_table_metadata has the following fields:

_id

Unique key of the relation in _table_metadata.

alias_of

Name of a relation that this alias point to. This field is null iff the name is not an alias.

name

Contains the name of the alias if alias_of is not null. Otherwise, this is the resource id of the CKAN resource for the DataStore resource.

oid

The PostgreSQL object ID of the table that belongs to name.

Extending DataStore

Starting from CKAN version 2.7, backend used in DataStore can be replaced with custom one. For this purpose, custom extension must implement ckanext.datastore.interfaces.IDatastoreBackend, which provides one method - register_backends. It should return dictonary with names of custom backends as keys and classes, that represent those backends as values. Each class supposed to be inherited from ckanext.datastore.backend.DatastoreBackend.

Note

Example of custom implementation can be found at ckanext.example_idatastorebackend

ckanext.datastore.backend.get_all_resources_ids_in_datastore()

Helper for getting id of all resources in datastore.

Uses get_all_ids of active datastore backend.

exception ckanext.datastore.backend.InvalidDataError

Exception that’s raised if you try to add invalid data to the datastore.

For example if you have a column with type “numeric” and then you try to add a non-numeric value like “foo” to it, this exception should be raised.

class ckanext.datastore.backend.DatastoreBackend

Base class for all datastore backends.

Very simple example of implementation based on SQLite can be found in ckanext.example_idatastorebackend. In order to use it, set datastore.write_url to ‘example-sqlite:////tmp/database-name-on-your-choice’

Prop _backend: mapping(schema, class) of all registered backends
Prop _active_backend:
  current active backend

classmethod register_backends()

Register all backend implementations inside extensions.

classmethod set_active_backend(config)

Choose most suitable backend depending on configuration

Parameters: config – configuration object
Return type: ckan.common.CKANConfig

classmethod get_active_backend()

Return currently used backend

configure(config)

Configure backend, set inner variables, make some initial setup.

Parameters: config – configuration object
Returns: config
Return type: CKANConfig

create(contextdata_dict)

Create new resourct inside datastore.

Called by datastore_create.

Parameters: data_dict – See ckanext.datastore.logic.action.datastore_create
Returns: The newly created data object
Return type: dictonary

upsert(contextdata_dict)

Update or create resource depending on data_dict param.

Called by datastore_upsert.

Parameters: data_dict – See ckanext.datastore.logic.action.datastore_upsert
Returns: The modified data object
Return type: dictonary

delete(contextdata_dict)

Remove resource from datastore.

Called by datastore_delete.

Parameters: data_dict – See ckanext.datastore.logic.action.datastore_delete
Returns: Original filters sent.
Return type: dictonary

search(contextdata_dict)

Base search.

Called by datastore_search.

Parameters:
  • data_dict – See ckanext.datastore.logic.action.datastore_search
  • fields (list of dictionaries) – fields/columns and their extra metadata
  • offset (int) – query offset value
  • limit (int) – query limit value
  • filters (list of dictionaries) – query filters
  • total (int) – number of total matching records
  • records (list of dictionaries) – list of matching results
Return type:

dictonary with following keys

search_sql(contextdata_dict)

Advanced search.

Called by datastore_search_sql. :param sql: a single seach statement :type sql: string

Return type:

dictonary

Parameters:
  • fields (list of dictionaries) – fields/columns and their extra metadata
  • records (list of dictionaries) – list of matching results

make_private(contextdata_dict)

Do not display resource in search results.

Called by datastore_make_private. :param resource_id: id of resource that should become private :type resource_id: string

make_public(contextdata_dict)

Enable serch for resource.

Called by datastore_make_public. :param resource_id: id of resource that should become public :type resource_id: string

resource_exists(id)

Define whether resource exists in datastore.

resource_fields(id)

Return dictonary with resource description.

Called by datastore_info. :returns: A dictionary describing the columns and their types.

resource_info(id)

Return DataDictonary with resource’s info - #3414

resource_id_from_alias(alias)

Convert resource’s alias to real id.

Parameters: alias (string) – resource’s alias or id
Returns: real id of resource
Return type: string

get_all_ids()

Return id of all resource registered in datastore.

Returns: all resources ids
Return type: list of strings

create_function(*args**kwargs)

Called by datastore_function_create action.

drop_function(*args**kwargs)

Called by datastore_function_delete action.