Development

Development should follow a problem-solution approach.

See also

The reader philosophy

Roadmap

The plan is to continue evolving the library to support as many “feed reader application” use cases as possible, while still following the The reader philosophy. Even if a specific feature is not a good fit for the library itself, it should be possible to find a more generic solution that makes it possible to build the feature on top.

Following is an unsorted, non-exhausive list of known areas for improvement. I am working on reader based on my current interests, in my spare time, but I will prioritize supporting contributors (discussions, reviews and so on).

See also

Open issues and Design notes.

Command-line interface

The Command-line interface is more or less stable,[*] although both the output and config loading need more polish and additional tests.

A full-blown terminal feed reader is not in scope, since I don’t need one, but I’m not opposed to the idea.

Web application

The Web application is “unsupported”, in that it’s not all that polished, and I don’t have time to do major improvments. But, I am using it daily, and it will keep working until a better one exists.

Long term, I’d like to:

  • re-design it from scratch to improve usability (see #318 for a wishlist)

  • switch to htmx instead of using a home-grown solution

  • spin it off into a separate package/project

Backwards compatibility

reader uses semantic versioning.

Breaking compatibility is done by incrementing the major version, announcing it in the Changelog, and raising deprecation warnings for at least one minor version before the new major version is released (if possible).

There may be minor exceptions to this, e.g. bug fixes and gross violation of specifications; they will be announced in the Changelog with a This is a minor compatibility break warning.

Schema migrations for the default storage must happen automatically. Migrations can be removed in new major versions, with at least 3 months provided since the last migration.

What is the public API

reader follows the PEP 8 definition of public interface.

The following are part of the public API:

  • Every interface documented in the API reference.

  • Any (documented) module, function, object, method, and attribute, defined in the reader package, that is accessible without passing through a name that starts with underscore.

  • The number and position of positional arguments.

  • The names of keyword arguments.

  • Argument types (argument types cannot become more strict).

  • Attribute types (attribute types cannot become less strict).

Undocumented type aliases (even if not private) are not part of the public API.

Other exceptions are possible; they will be marked aggresively as such.

See also

The Twisted Compatibility Policy, which served as inspiration for this.

Internal API

The Internal API is not stable, but the long term goal is for it to become so.

In order to support / encourage potential users (e.g. plugins, alternate storage implementations), changes should at least be announced in the Changelog.

Supported Python versions

The oldest Python version reader should support is:

  • the newest CPython available on the latest Ubuntu LTS (3 months after LTS release)

  • at least 1 stable PyPy version

This usually ends up being the last 3 stable CPython versions.

Dropping support for a Python version should be announced at least 1 release prior.

Releases

For convenience, reader only releases major and minor versions (bugfixes go in minor versions). Changes go only to the next release (no backports).

Making a release

Note

scripts/release.py already does most of these.

Making a release (from x to y == x + 1):

  • (release.py) bump version in src/reader/__init__.py to y

  • (release.py) update changelog with release version and date

  • (release.py) make sure tests pass / docs build

  • (release.py) clean up dist/: rm -rf dist/

  • (release.py) build tarball and wheel: python -m build

  • (release.py) push to GitHub

  • (release.py prompts) wait for GitHub Actions / Codecov / Read the Docs builds to pass

  • upload to test PyPI and check: twine upload --repository-url https://test.pypi.org/legacy/ dist/*

  • (release.py) upload to PyPI: twine upload dist/*

  • (release.py) tag current commit with <major>.<minor> and <major>.x (e.g. when releasing 1.20: 1.20 and 1.x)

  • (release.py prompts) create release in GitHub

  • build docs from latest and enable y docs version (should happen automatically after the first time)

  • (release.py) bump versions from y to (y + 1).dev0, add (y + 1) changelog section

  • (release.py prompts) trigger Read the Docs build for <major>.x (doesn’t happen automatically)

Documentation

Following are notes about what documentation should look like, especially for the stable high-level API, since that’s what most users will see.

We prefer type information in the method description, not in the signature, since the result is more readable. For the same reason, we prefer hand-written Sphinx-style field list types.

We still use autodoc-provided type hints as fallback for parameters that don’t have hand-written types, for type documentation for dataclasses, and for the unstable Internal API, where it’s too much effort to maintain hand-written types.

Known issues (October 2023, Sphinx version ~7):

  • Overloads are shown with full annotation regardless of autodoc_typehints (known, documented behavior). May get better with https://github.com/sphinx-doc/sphinx/issues/10359.

  • Type aliases that do not come from hand-written types but from the autodoc typehints are expanded in-place; this also affects the overload type annotations. The documented work-around is to add the aliases to autodoc_type_aliases.

  • Type alias names that appear in parameter types do not link to the documentation in Type aliases. May get better with https://github.com/sphinx-doc/sphinx/issues/9705

Design notes

Folowing are various design notes that aren’t captured somewhere else (either in the code, or in the issue where a feature was initially developed).

Why use SQLite and not SQLAlchemy?

tl;dr: For “historical reasons”.

In the beginning:

  • I wanted to keep things as simple as possible, so I don’t get demotivated and stop working on it. I also wanted to try out a “problem-solution” approach.

  • I think by that time I was already a great SQLite fan, and knew that because of the relatively single-user nature of the thing I won’t have to change databases because of concurrency issues.

  • The fact that I didn’t know exactly where and how I would deploy the web app (and that SQLite is in stdlib) kinda cemented that assumption.

Since then, I did come up with some of my own complexity: there’s a SQL query builder, a schema migration system, and there were some concurrency issues. SQLAlchemy would have likely helped with the first two, but not with the last one (not without dropping SQLite).

Note that it is possible to use a different storage implementation; all storage stuff happens through a DAO-style interface, and SQLAlchemy was the main real alternative I had in mind. The API is private at the moment (1.10), but if anyone wants to use it I can make it public.

It is unlikely I’ll write a SQLAlchemy storage myself, since I don’t need it (yet), and I think testing it with multiple databases would take quite some time.

Multiple storage implementations

Detailed requirements and API discussion: #168#issuecomment-642002049.

Minimal work needed to support alternate storages: #168#issuecomment-1383127564.

Storage internal API documented in version 3.10 (November 2023) in #325.

Database optimization

Some general guidance on schema/index design: #327#issuecomment-1859147186.

Speeding up get_entries(sort='recent'):

  • first attempt at adding indexes: #134

  • using a computed column (recent_sort) didn’t change things very much: #279

  • an index on recent_sort alone is not enough for pagination, the index needs to match 1:1 the WHERE clause: #330.

Speeding up get_entry_counts(feed=...):

Parser

file:// handling, feed root, per-URL-prefix parsers (later retrievers, see below):

Requests session plugins:

Retriever / parser split:

Alternative feed parsers:

Lessons learned from the twitter plugin:

  • It is useful for a retriever to pass an arbitrary resource to the parser.

    This is already codified in RetrieverType() and ParserType() being generic.

  • It is useful for a Retriever to store arbitrary caching data; the plugin (mis)used http_etag to store the (integer) id of the newest tweet in the thread.

    It would be nice to formalize this into a single “arbitrary caching data” attribute; also see this comment.

  • It is useful for a Retriever to pass arbitrary data to itself; the plugin (mis)used http_etag to pass from process_feed_for_update() to __call__():

    • the bearer token and the ids of recent entries (used to retrieve tweets)

    • the ids of entries to re-render, triggered by a one-off tag (passed along to the parser)

    This distinction was made so that process_feed_for_update() takes all the decisions upfront (possibly taking advantage of Storage.get_feeds_for_update() future optimisations to e.g. also get tags), and calling the retriever (in parallel) doesn’t do any reader operations.

    It would be nice to formalize this as well.

  • A plugin can coordinate between a custom retriever and custom parser with an unregistered RetrieveResult MIME type (e.g. application/x.twitter).

  • A plugin can keep arbitrary data as a content with an unregistered type (e.g. application/x.twitter+json).

Metrics

Some thoughts on implementing metrics: #68#issuecomment-450025175.

Per-call timings introduced in the timer experimental plugin.

Query builder

Survey of possible options: #123#issuecomment-582307504.

In 2021, I’ve written an entire series about it: https://death.andgravity.com/query-builder

Pagination for methods that return iterators

Why do it for the private implementation: #167#issuecomment-626753299 (also a comment in storage code).

Detailed requirements and API discussion for public pagination: #196#issuecomment-706038363.

reader types to Atom mapping

This whole issue: #153.

Sort by random

Some thoughts in the initial issue: #105.

Sort by tag values

It may be useful to be able to sort by tag values in order to allow sorting by cached entry counts: #306#issuecomment-1694655504.

Entry/feed “primary key” attribute naming

This whole issue: #159#issuecomment-612914956.

Change feed URL

From the initial issue:

Resource tags / metadata

Feed tags

Detailed requirements and API discussion, and a case study of how to implement categories on top of tags: #184#issuecomment-689587006.

Merging tags and metadata, and the addition of a new, generic (global, feed, entry) tag API: #266#issuecomment-1013739526.

Entry tags

#228#issuecomment-810098748 discusses three different kinds of entry user data, how they would be implemented, and why I want more use-cases before implementing them (basically, YAGNI):

  • entry searchable text fields (for notes etc.)

  • entry tags (similar to feed tags, may be used as additional bool flags)

  • entry metadata (similar to feed metadata)

    • also discusses how to build an enclosure cache/preloader (doesn’t need special reader features besides what’s available in 1.16)

#253 discusses using entry tags to implement the current entry flags (read, important); tl;dr: it’s not worth adding entry tags just for this. #327 discusses using entry tags for has_enclosures; tl;dr: it wouldn’t save a lot of code, it would be only a bit slower, and it reconfirms that read and important are integral to the data model, so we still want them as regular columns.

After closing #228 with wontfix in late 2021, in early 2022 (following the #266 tag/metadata unification) I implemented entry and global tags in #272; there’s a list of known use cases in the issue description.

Resource tags

Optimistic locking for tags: #308.

Filter tags by prefix: #309.

User-added entries

Discussion about API/typing, and things we didn’t do: #239.

Feed updates

Some thoughts about adding a map argument: #152#issuecomment-606636200.

How update_feeds() is like a pipeline: comment.

Data flow diagram for the update process, as of v1.13: #204#issuecomment-779709824.

update_feeds_iter():

Disabling updates:

Updating entries based on a hash of their content (regardless of updated):

Decision to ignore feed.updated when updating feeds: #231.

Deleting entries

Requirements, open questions, and how it interacts with entry_dedupe: #96.

A summary of why it isn’t easy to do: #301#issuecomment-1442423151.

Counts API

Detailed requirements and API discussion: #185#issuecomment-731743327.

Tracking additional statistics (e.g. read_modified): #254; how to expose said statistics: #254#issuecomment-1807064610.

Notebook with a successful attempt to determine a feed “usefulness” score based on how many entries I mark as read / important / don’t care; highlights a number of gaps in the reader API: https://gist.github.com/lemon24/93222ef4bc4a775092b56546a6e6cd0f

Using None as a special argument value

This comment: #177#issuecomment-674786498.

Batch methods

Some initial thoughts on batch get methods (including API/typing) in #191 (closed with wontfix, for now).

Why I want to postpone batch update/set methods: #187#issuecomment-700740251.

tl:dr: Performance is likely a non-issue with SQLite, convenience can be added on top as a plugin.

See the 2.12 reader._app.ResourceTags class for an idea of how to represent a bunch of tags in a reserved-name-scheme-agnostic way (useful e.g. for when get_entries() should return tags x, y, z of each entry).

Some web app measurements that show a few cases where batch methods may help: #306#issuecomment-1694655504.

Using a single Reader objects from multiple threads

Some thoughts on why it’s difficult to do: #206#issuecomment-751383418.

Requirements and use cases: #206#issuecomment-1179739301.

When/how to run pragma optimize: #206#issuecomment-1183660880.

Full support added in version 2.16 (July 2022).

Plugins

List of potential hooks (from mid-2018): #80.

Minimal plugin API (from 2021) – case study and built-in plugin naming scheme: #229#issuecomment-803870781.

We’ll add / document new (public) hooks as needed.

“Tag before, clear tag after” pattern for resilient plugins: #246#issuecomment-1596097300.

Update hook error handling:

Reserved names

Requirements, thoughts about the naming scheme and prefixes unlikely to collide with user names: #186 (multiple comments).

Wrapping underlying storage exceptions

Which exception to wrap, and which not: #21#issuecomment-365442439.

In version 3.10 (November 2023), all internal APIs were changed to use timezone-aware datetimes, with the timezone set to UTC, in preparation for support for any timezone.

Timezone handling

Aware vs. naive, and what’s needed to go fully aware: #233#issuecomment-881618002.

OPML support

Thoughts on dynamic lists of feeds: #165#issuecomment-905893420.

entry_dedupe

Using MinHash to speed up similarity checks (maybe): https://gist.github.com/lemon24/b9af5ade919713406bda9603847d32e5

REST API

Some early thoughts: #192#issuecomment-700773138 (closed with wontfix, for now).

Web application