This release marks reader’s 5th anniversary and its 2000th commit.
This is the last release to support Python 3.8; see #298 for details.
Support Python 3.11. (#289)
Postpone update-related imports until needed. Shortens time from process start to usable Reader instance by 3x (imports are 72% faster). (#297)
Refactor parser internals. (#297)
Plugins using the (unstable) session hooks should replace:
edit_history_tweet_ids(fails in tweepy 4.11, warns in tweepy >4.12).
Fix bug in
mark_as_readcausing updates to fail if an entry had no title.
In the CLI, don’t suppress the traceback of
ReaderError, since it would also suppress it for bugs.
In the CLI, stop using deprecated
readtimeplugin dependency on readtime (which has a transitive dependency on lxml, which does not always have PyPy Windows wheels on PyPI). The
readtimeextra is deprecated, but remains available to avoid breaking dependent packages. (#286)
Sort entries by added date most of the time, with the exception of those imported on the first update. Previously, entries would be sorted by added only if they were published less than 7 days ago, causing entries that appear in the feed months after their published to never appear at the top (so the user would never see them). (#279)
This release contains backwards incompatible changes.
Remove old database migrations.
mark_as_readconfig tag name migration.
If you are upgrading from reader 2.10 or newer, no action is required.
If you are upgrading to reader 3.0 from a version older than 2.10, you must open your database with reader 2.10 or newer once, to run the removed migrations:
pip install 'reader>=2.10,<3' && \ python - db.sqlite << EOF import sys from reader import make_reader from reader.plugins.mark_as_read import _migrate_pre_2_7_metadata as migrate_mark_as_read reader = make_reader(sys.argv) for feed in reader.get_feeds(): migrate_mark_as_read(reader, feed) print("OK") EOF
Remove code that issued deprecation warnings in versions 2.* (#268):
object_idproperty of data objects and related exceptions
Make some of the parameters of the following positional-only (#268):
object_idproperty of data objects in favor of new property
resource_idis the same as
object_id, except for feeds and feed-related exceptions it is of type
object_idwill be removed in version 3.0. (#266, #268)
Do not attempt too hard to run
PRAGMA optimizeif the database is busy. Prevents rare “database is locked” errors when multiple threads using the same reader terminate at the same time. (#206)
Skip with a warning entries that have no <guid> or <link> in an RSS feed; only raise
ParseErrorif all entries have a missing id. (Note that both Atom and JSON Feed entries are required to have an id by their respective specifications.) Thanks to Mirek Długosz for the issue and pull request. (#281)
In the web app, use the read time provided by the
readtimeplugin, instead of calculating it on each page load. Speeds up the rendering of the entries page by 20-30%, hopefully winning back the time lost when the read time feature was first added in 2.6. (#275)
In the web app, also show the read time for search results.
get_tags()support for the
(None,)(any feed) and
None(any resource) wildcard resource values.
This is a minor compatibility break, but is unlikely to affect existing users; the usefulness of the wildcards was limited, because it was impossible to tell to which resource a (key, value) pair belongs.
Allow passing a (feed URL,) 1-tuple anywhere a feed URL can be passed to a
global_metadataexperimental plugin (superseded by global tags).
In the web application, support editing entry and global metadata. Fix broken delete metadata button. Fix broken error flashing.
update_feeds()memory usage by ~35% (using the maxrss before the call as baseline; overall process maxrss decreases by ~20%). The improvement is not in reader code, but in feedparser; reader will temporarily vendor feedparser until the fix makes it upstream and is released on PyPI. (#265)
Deprecate feed-specific tag and metadata methods (#266):
All deprecated methods/exceptions will be removed in version 3.0.
In the web application, show maxrss when debug is enabled. (#269)
In the web application, decrease memory usage of the entries page when there are a lot of entries (e.g. for 2.5k entries, maxrss decreased from 115 MiB to 75 MiB), at the expense of making “entries for feed” slightly slower. (#269)
mark_as_readplugin now uses the
.reader.mark-as-readmetadata for configuration. Feeds using the old metadata,
.reader.mark_as_read, will be migrated automatically on update until reader 3.0.
Allow running arbitrary actions before updating feeds via
Retrieve feeds in parallel, but parse them serially; previously, feeds would be parsed in parallel. Decreases Linux memory usage by ~20% when using
workers; the macOS decrease is less notable. (#261)
Reuse the requests session when retrieving feeds; previously, each feed would get its own session.
Add support for CLI plugins.
In the web application, show entry read time.
Noneif missing in the feed (
updatedbecame optional in version 2.0). Use
updated_not_nonefor the pre-2.5 behavior. Do not swap
Entry.updatedfor RSS feeds where
updatedis missing. (#183)
Support PyPy 3.8.
Fix bug where deleting an entry and then adding it again (with the same id) would fail if search was enabled and
update_search()was not run before adding the new entry.
Enable search by default. (#252)
In the web application, show the feed subtitle. (#223)
Support Python 3.10. (#248)
Please comment in #140 / open an issue if you were relying on the old behavior.
entry_dedupebug introduced in 2.2, causing the newest read entry to be marked as unread if none of its duplicates are read (idem for important). This was an issue only when re-running the plugin for existing entries, not for new entries (since new entries are unread/unimportant).
Record when an entry is marked as read/important, and make it available through
important_modified. Allow providing a custom value using the
In the web application, allow marking an entry as don’t care (read + unimportant explicitly set by the user) with a single button. (#254)
In the web application, show the entry read modified / important modified timestamps as button tooltips. (#254)
Use an index for
get_entry_counts(feed=...)calls. Makes the /feeds?counts=yes page load 2-4x faster. (#251)
In the web application, show the feed entry count averages as a bar sparkline. (#249)
Make the minimum SQLite version and required SQLite compile options
reader._storagemodule globals, for easier monkeypatching. (#163)
This is allows supplying a user-defined
json_array_lengthfunction on platforms where SQLite doesn’t come with the JSON1 extension (e.g. on Windows with stock Python earlier than 3.9; details).
Note these globals are private, and thus not covered by the backwards compatibility policy.
This release contains backwards incompatible changes.
Remove old database migrations.
If you are upgrading from reader 1.15 or newer, no action is required.
If you are upgrading to reader 2.0 from a version older than 1.15, you must open your database with reader 1.15 or newer once, to run the removed migrations:
pip install 'reader>=1.15,<2' && \ python - db.sqlite << EOF import sys from reader import make_reader make_reader(sys.argv) print("OK") EOF
Remove code that issued deprecation warnings in versions 1.* (#183):
get_feed_metadata(feed, key, default=no value, /)form of
make_reader()now defaults to
None(don’t open local feeds) instead of
''(full filesystem access).
Drop Python 3.6 support. (#237)
Support PyPy 3.7. (#234)
Skip enclosures with no
url; previously, they would result in a parse error. (#240)
Stop using Travis CI (only use GitHub Actions). (#199)
The signature of
UpdatedFeed(url, new, updated)to
UpdatedFeed(url, new, modified).
This is a minor compatibility break, but only affects third-party code that instantiates UpdatedFeed directly with
updatedas a keyword argument.
Readerfeed metadata methods:
For backwards compatibility, the old method signatures will continue to work until version 2.0, when they will be removed. (#183)
get_feed_metadata(feed, key[, default]) -> valueform is backwards-compatible only when the arguments are positional.
This is a minor compatibility break; the following work in 1.17, but do not in 1.18:
# raises TypeError reader.get_feed_metadata(feed, key, default=None) # returns `(key, value), ...` instead of `value` reader.get_feed_metadata(feed, key=key)
get_feed_metadata_item()) is intended to have positional-only arguments, but this cannot be expressed easily until Python 3.8.
MetadataNotFoundErrorremains available, and is a superclass of
FeedMetadataNotFoundErrorfor backwards compatibility. (#228)
The signatures of the following exceptions changed:
Takes a new required
keyargument, instead of no required arguments.
Takes only one required argument,
urlargument has been removed.
This is a minor compatibility break, but only affects third-party code that instantiates these exceptions directly.
feed_url; for backwards compatibility, the old attribute will be available as a property until version 2.0, when it will be removed. (#183).
The signature of
EntryError(and its subclasses) changed from
This is a minor compatibility break, but only affects third-party code that instantiates these exceptions directly with
urlas a keyword argument.
For backwards compatibility, the old methods will continue to work until version 2.0, when they will be removed. (#183)
When serving the web application with
python -m reader serve, don’t set the
Refererheader for cross-origin requests. (#209)
Prevents spurious updates for feeds whose
updatedchanges excessively (either because the entries’ content changes excessively, or because an RSS feed does not have a
dc:dateelement, and feedparser falls back to
regex_mark_as_readexperimental plugin is now built-in. To use it with the CLI / web application, use the plugin name instead of the entry point (
The config metadata key and format changed; the config will be migrated automatically on the next feed update, during reader version 1.17 only. If you used
regex_mark_as_readand are upgrading to a version >1.17, install 1.17 (
pip install reader==1.17) and run a full feed update (
python -m reader update) before installing the newer version.
sqlite-releasesunstable extras are not available anymore. Use the
unstable-pluginsextra to install dependencies of the unstable plugins instead.
In the web application, allow updating a feed manually. (#195)
ua_fallbackplugin by default.
To use them with the CLI / web application, use the plugin name instead of the entry point:
reader._plugins.enclosure_dedupe:enclosure_dedupe -> reader.enclosure_dedupe reader._plugins.feed_entry_dedupe:feed_entry_dedupe -> reader.entry_dedupe reader._plugins.ua_fallback:init -> reader.ua_fallback
pluginsextra; plugin loading machinery does not have additional dependencies anymore.
Mention in the User guide that all reader functions/methods can raise
TypeErrorif passed invalid arguments. There is no behavior change, this is just documenting existing, previously undocumented behavior.
Limit content-only updates (not due to an
updatedchange) to 24 consecutive updates, to prevent spurious updates for entries whose content changes excessively (for example, because it includes the current time). (#225)
Previously, entries would be updated only if the entry
updatedwas newer than the stored one.
Fix bug causing entries that don’t have
updatedset in the feed to not be updated if the feed is marked as stale. Feed staleness is an internal feature used during storage migrations; this bug could only manifest when migrating from 0.22 to 1.x. (found during #179)
Minor web application improvements.
Minor CLI improvements.
make_reader()to set a timeout for retrieving HTTP(S) feeds. The default (connect timeout, read timeout) is (3.05, 60) seconds; the previous behavior was to never time out.
PRAGMA user_versioninstead of a version table. (#210)
PRAGMA application_idto identify reader databases; the id is
readin ASCII / UTF-8. (#211)
reader updatecommand to show a progress bar and update summary (with colors), instead of plain log output. (#204)
Fix broken Mypy config following 0.800 release. (#213)
JSON Feed support. (#206)
Split feed retrieval from parsing; should make it easier to add new/custom parsers. (#206)
Prevent any logging output from the
readerlogger by default. (#207)
<link rel=alternative ...>tags as a feed detection heuristic.
<a>tags as a fallback feed detection heuristic.
In the web application, fix bug causing the entries page to crash when counts are enabled.
object_idproperty that allows getting the unique identifier of a data object in a uniform way. (#196)
In the web application, add links to toggle feed/entry counts. (#185)
Allow disabling feed updates for specific feeds. (#187)
Add methods to get aggregated feed and entry counts. (#185)
In the web application: allow disabling feed updates for a feed; allow filtering feeds by whether they have updates enabled; do not show feed update errors for feeds that have updates disabled. (#187)
In the web application, show feed and entry counts when
?counts=yesis used. (#185)
In the web application, use YAML instead of JSON for the tags and metadata fields.
Reraise unexpected errors caused by parser bugs instead of replacing them with an
sqlite_releasescustom parser plugin.
Refactor the HTTP feed sub-parser to allow reuse by custom parsers.
Add a user guide, and improve other parts of the documentation. (#194)
Support Python 3.9. (#199)
Support Windows (requires Python >= 3.9). (#163)
Use GitHub Actions to do macOS and Windows CI builds. (#199)
ua_fallback. Retry any feed that gets a 403, not just those served by Cloudflare. (#181)
Fix type annotation to avoid mypy 0.790 errors. (#198)
Drop feedparser 5.x support (deprecated in 1.7); use feedparser 6.x instead. (#190)
Allow changing the URL of a feed in the web application. (#149)
Add more tag navigation links to the web application. (#184)
feed_entry_dedupeplugin, copy the important flag from the old entry to the new one. (#140)
Add new methods to support feed tags:
get_feed_tags(). Allow filtering feeds and entries by their feed tags. (#184)
feedparser 5.x support is deprecated in favor of feedparser 6.x. Using feedparser 5.x will raise a deprecation warning in version 1.7, and support will be removed the following version. (#190)
Tag-related web application features: show tags in the feed list; allow adding/removing tags; allow filtering feeds and entries by their feed tag; add a page that lists all tags. (#184)
In the web application, allow showing only feeds that failed / did not fail. (#189)
<meta>tags as a feed detection heuristic.
Add a few property-based tests. (#188)
make_reader(), which allows limiting local feed parsing to a specific directory or disabling it altogether. Using it is recommended, since by default reader will access any local feed path (in 2.0, local file parsing will be disabled by default). (#155)
Fail fast for feeds that return HTTP 4xx or 5xx status codes, instead of (likely) failing later with an ambiguous XML parsing error. The cause of the raised
ParseErroris now an instance of
cloudflare_ua_fixplugin (work around Cloudflare sometimes blocking requests). (#181)
feedparser 6.0 (beta) compatibility fixes.
Use rowid when deleting from the search index, instead of the entry id. Previously, each
update_search()call would result in a full scan, even if there was nothing to update/delete. This should reduce the amount of reads significantly (deleting 4 entries from a database with 10k entries resulted in an 1000x decrease in bytes read). (#178)
close(). This should increase the performance of all methods. As an example, in #178 it was found that
update_search()resulted in a full scan of the entries table, even if there was nothing to update; this change should prevent this from happening. (#143)
PRAGMA optimizeis a no-op in SQLite versions earlier than 3.18. In order to avoid the case described above, you should run ANALYZE regularly (e.g. every few days).
Work to reduce the likelihood of “database is locked” errors during updates (#175):
Prepare entries to be added to the search index (
update_search()) outside transactions.
Fix bug causing duplicate rows in the search index when an entry changes while updating the search index.
Update the search index only when the indexed values change (details below).
Use SQLite WAL (details below).
Update the search index only when the indexed values change. Previously, any change on a feed would result in all its entries being re-indexed, even if the feed title or the entry content didn’t change. This should reduce the
update_search()run time significantly.
Require at least click 7.0 for the
Do not fail for feeds with incorrectly-declared media types, if feedparser can parse the feed; this is similar to the current behavior for incorrectly-declared encodings. (#171)
In the web application, display a nice error message for invalid search queries instead of returning an HTTP 500 Internal Server Error.
Other minor web application improvements.
Minor CLI logging improvements.
Show details about feed update errors in the web application. (#68)
Fall back to <link> as entry id if an entry in an RSS feed has no <guid>; previously, feeds like this would fail on update. (#170)
Minor web application improvements (show feed added/updated date).
In the web application, handle previewing an invalid feed nicely instead of returning an HTTP 500 Internal Server Error. (#172)
Internal API changes to support multiple storage implementations in the future. (#168)
Minor web application improvements.
Remove unneeded additional query in methods that use pagination (for n = len(result) / page size, always do n queries instead n+1).
search_entries()are now 33–7% and 46–36% faster, respectively, for results of size 32–256. (#166)
All queries are now chunked/paginated to avoid locking the SQLite storage for too long, decreasing the chance of concurrent queries timing out; the problem was most visible during
update_search(). This should cap memory usage for methods returning an iterable that were not paginated before; previously the whole result set would be read before returning it. (#167)
Allow changing the entry sort order in the web application. (#105)
Use a query builder instead of appending strings manually for the more complicated queries in search and storage. (#123)
Make searching entries faster by filtering them before searching; e.g. if 1/5 of the entries are read, searching only read entries is now ~5x faster. (enabled by #123)
Make all private submodules explicitly private. (#156)
All direct imports from
readercontinue to work.
reader.core.*modules moved to
reader.*(most of them prefixed by
The web application WSGI entry point moved from
The entry points for plugins that ship with reader moved from
Require at least beautifulsoup4 4.5 for the
searchextra (before, the version was unspecified). (#161)
Rename the web application dependencies extra from
execute()in the SQLite storage. Makes updating feeds (excluding network calls) 5-10% faster. (#144)
In the web app, redirect to the feed’s page after adding a feed. (#119)
In the web app, show highlighted search result snippets. (#122)
Minor consistency improvements to the web app search button. (#122)
Add support for web application plugins. (#80)
The enclosure tag proxy is now a plugin, and is disabled by default. See its documentation for details. (#52)
In the web app, the “add feed” button shows a preview before adding the feed. (#145)
In the web app, if the feed to be previewed is not actually a feed, show a list of feeds linked from that URL. This is a plugin, and is disabled by default. (#150)
Fix bug in
enable_search()that caused it to fail if search was already enabled and the reader had any entries.
Require at least requests 2.18 (before, the version was unspecified).
Make the database schema more strict regarding nulls. (#138)
Tests are now run in a random order. (#142)
Improve entry page rendering for text/plain content. (#117)
Improve entry page rendering for images and code blocks. (#126)
Show enclosures on the entry page. (#128)
Show the entry author. (#129)
Fix bug causing the enclosure tag proxy to use too much memory. (#133)
Start using mypy on the core modules. (#132)
Unify plugin loading and error handling code. (#112)
Minor improvements to CLI error reporting.
Increase timeout of the button actions from 2 to 10 seconds.
enclosure_dedupeplugin (deduplicate enclosures of an entry). (#78)
servecommand now supports loading plugins. (#78)
reader.app.wsginow supports loading plugins. (#78)
Move the core modules to a separate subpackage and enforce test coverage (
make coveragenow fails if the coverage for core modules is less than 100%). (#101)
Support Python 3.8 development branch.
docsextras (to install development requirements).
Build HTML documentation when running tox.
docsmake targets (to run tox / build HTML docs).
Support Python 3.7.
Allow changing the feed sort order in the web application. (#98)
Released on 2018-12-22
regex_mark_as_readplugin (mark new entries as read based on a regex). (#79)
feed_entry_dedupeplugin (deduplicate new entries for a feed). (#79)
Plugin loading machinery dependencies are now installed via the
Add a plugins section to the documentation.
Released on 2018-11-25
Released on 2018-10-21
reader servecommand (broken in 0.1).
Fix wrong submit button being used when pressing enter in non-button fields. (#69)
requests-mockin parser tests instead of a web server (test suite run time down by ~35%). (#90)
Released on 2018-09-15
Initial release; public API stable.
Support broken Tumblr feeds via the the