This is manifestly not a feed.
EOF } make_html_head () { cat >"$1" <This is manifestly not a feed.
EOF } make_html_body () { cat >"$1" <This is manifestly not a feed.
EOF cat >>"$1" cat >>"$1" <bug fixed 2006-01-07. #
/ has a workaround in feedparser for sgmllib. add "tidyhtml false" write_desc <
" \ "1
" \ "2
/" begin "broken processing instruction" write_desc <
" workaround for non-block-level HTML; this may be useful if you have a plugin that is doing different HTML sanitisation, or if your template already forces a block-level element around article descriptions. Fix -l for feeds with non-ASCII characters in their titles. Provide human-readable __feed_id__ in items (patch from David Durschlag), and add feed-whatevername class to the default item template; this should make it somewhat easier to add per-feed styles. Handle feeds that are local files correctly, and handle file: URLs in feedparser (reported by Chris Niekel). Allow feed arguments to be given on indented lines after the "feed" or "feeddefaults" lines; this makes it possible to have spaces in feed arguments. Add a meta element to the default template to stop search engines indexing rawdog pages (patch from Rick van Rein). Add new feeds at the end of the config file rather than before the first feed line (patch from Decklin Foster). - rawdog 2.1 Fix a character encoding problem with format=text feeds. Add proxyuser and proxypassword options for feeds, so that you can use per-feed proxies requiring HTTP Basic authentication (patch from Jon Nelson). Add a manual page (written by Decklin Foster). Remove extraneous #! line from feedparser.py (reported by Decklin Foster). Update an article's modified date when a new version of it is seen (reported by Decklin Foster). Support nested ifs in templates (patch from David Durschlag), and add __else__. Make the README file list all the options that rawdog now supports (reported by David Durschlag). Make --verbose work even if it's specified after an action (reported by Dan Noe and David Durschlag). - rawdog 2.0 Update to feedparser 3.3. This meant reworking some of rawdog's internals; state files from old versions will no longer work with rawdog 2.0 (and external programs that manipulate rawdog state files will also be broken). The new feedparser provides a much nicer API, and is significantly more robust; several feeds that previously caused feedparser internal errors or Python segfaults now work fine. Add an --upgrade option to import state from rawdog 1.x state files into rawdog 2.x. To upgrade from 1.x to 2.x, you'll need to perform the following steps after installing the new rawdog: - cp -R ~/.rawdog ~/.rawdog-old - rm ~/.rawdog/state - rawdog -u - rawdog --upgrade ~/.rawdog-old ~/.rawdog (to copy the state) - rawdog -w - rm -r ~/.rawdog-old (once you're happy with the new version) Keep track of a version number in the state file, and complain if you use a state file from an incompatible version. Remove support for the old option syntax ("rawdog update write"). Remove workarounds for early 1.x state file versions. Save the state file in the binary pickle format, and use cPickle instead of pickle so it can be read and written more rapidly. Add hideduplicates and allowduplicates options to attempt to hide duplicate articles (based on patch from Grant Edwards). Fix a bug when sorting feeds with no titles (found by Joseph Reagle). Write the updated state file more safely, to reduce the chance that it'll be damaged or truncated if something goes wrong while it's being written (requested by Tim Bishop). Include feedfinder, and add a -a|--add option to add a feed to the config file. Correctly handle dates with timezones specified in non-UTC locales (reported by Paul Tomblin and Jon Lasser). When a feed's URL changes, as indicated by a permanent HTTP redirect, automatically update the config file and state. - rawdog 1.13 Handle OverflowError with parsed dates (patch from Matthew Scott). - rawdog 1.12 Add "sortbyfeeddate" option for planet pages (requested by David Dorward). Add "currentonly" option (patch from Chris Cutler). Handle nested CDATA blocks in feed XML and HTML correctly in feedparser. - rawdog 1.11 Add __num_items__ and __num_feeds__ to the page template, and __url__ to the item template (patch from Chris Cutler). Add "daysections" and "timesections" options to control whether to split items up by day and time (based on patch from Chris Cutler). Add "tidyhtml" option to use mx.Tidy to clean feed-provided HTML. Remove the
wrapping __description__ from the default item template, and make rawdog add
...
around the description only if it doesn't start with a block-level element (which isn't perfect, but covers the majority of problem cases). If you have a custom item template and want rawdog to generate a better approximation to valid HTML, you should change "__description__
" to "__description__". HTML metacharacters in links are now encoded correctly in generated HTML ("foo?a=b&c=d" as "foo?a=b&c=d"). Content type selection is now performed for all elements returned from the feed, since some Blogger v5 feeds cause feedparser to return multiple versions of the title and link (reported by Eric Cronin). - rawdog 1.10 Add "ignoretimeouts" option to silently ignore timeout errors. Fix SSL and socket timeouts on Python 2.3 (reported by Tim Bishop). Fix entity encoding problem with HTML sanitisation that was causing rawdog to throw an exception upon writing with feeds containing non-US-ASCII characters in attribute values (reported by David Dorward, Dmitry Mark and Steve Pomeroy). Include MANIFEST.in in the distribution (reported by Chris Cutler). - rawdog 1.9 Add "clear: both;" to item, time and date styles, so that items with floated images in don't extend into the items below them. Changed how rawdog selects the feeds to update; --verbose now shows only the feeds being updated. rawdog now uses feedparser 2.7.6, which adds date parsing and limited sanitisation of feed-provided HTML; I've removed rawdog's own date-parsing (including iso8601.py) and relative-link-fixing code in favour of the more-capable feedparser equivalents. The persister module in rawdoglib is now licensed under the LGPL (requested by Giles Radford). Made the error messages that listed the state dir reflect the -b setting (patch from Antonin Kral). Treat empty titles, links or descriptions as if they weren't supplied at all, to cope with broken feeds that specify "/" as a SHORTTAG; this workaround from # feedparser. html = re.sub(r'(\S)/>', r'\1 />', html) # sgmllib is fragile with broken processing instructions (e.g. # ""); just remove them all. html = re.sub(r']*>', '', html) html = feedparser._resolveRelativeURIs(html, baseurl, "UTF-8", type) p = feedparser._HTMLSanitizer("UTF-8", type) p.feed(html) html = p.output() if not inline and config["blocklevelhtml"]: # If we're after some block-level HTML and the HTML doesn't # start with a block-level element, then insert a
tag # before it. This still fails when the HTML contains text, then # a block-level element, then more text, but it's better than # nothing. if block_level_re.match(html) is None: html = "
" + html if config["tidyhtml"]: args = {"numeric_entities": 1, "output_html": 1, "output_xhtml": 0, "output_xml": 0, "wrap": 0} plugins.call_hook("mxtidy_args", config, args, baseurl, inline) plugins.call_hook("tidy_args", config, args, baseurl, inline) if tidylib is not None: # Disable PyTidyLib's somewhat unhelpful defaults. tidylib.BASE_OPTIONS = {} output = tidylib.tidy_document(html, args)[0] elif mxtidy is not None: output = mxtidy.tidy(html, None, None, **args)[2] else: # No Tidy bindings installed -- do nothing. output = "
" + html + "" html = output[output.find("") + 6 : output.rfind("")].strip() html = html.decode("UTF-8") box = plugins.Box(html) plugins.call_hook("clean_html", config, box, baseurl, inline) return box.value def select_detail(details): """Pick the preferred type of detail from a list of details. (If the argument isn't a list, treat it as a list of one.)""" TYPES = {"text/html": 30, "application/xhtml+xml": 20, "text/plain": 10} if details is None: return None if type(details) is not list: details = [details] ds = [] for detail in details: ctype = detail.get("type", None) if ctype is None: continue if TYPES.has_key(ctype): score = TYPES[ctype] else: score = 0 if detail["value"] != "": ds.append((score, detail)) ds.sort() if len(ds) == 0: return None else: return ds[-1][1] def detail_to_html(details, inline, config, force_preformatted=False): """Convert a detail hash or list of detail hashes as returned by feedparser into HTML.""" detail = select_detail(details) if detail is None: return None if force_preformatted: html = "" + cgi.escape(detail["value"]) + "" elif detail["type"] == "text/plain": html = cgi.escape(detail["value"]) else: html = detail["value"] return sanitise_html(html, detail["base"], inline, config) def author_to_html(entry, feedurl, config): """Convert feedparser author information to HTML.""" author_detail = entry.get("author_detail") if author_detail is not None and author_detail.has_key("name"): name = author_detail["name"] else: name = entry.get("author") url = None fallback = "author" if author_detail is not None: if author_detail.has_key("href"): url = author_detail["href"] elif author_detail.has_key("email") and author_detail["email"] is not None: url = "mailto:" + author_detail["email"] if author_detail.has_key("email") and author_detail["email"] is not None: fallback = author_detail["email"] elif author_detail.has_key("href") and author_detail["href"] is not None: fallback = author_detail["href"] if name == "": name = fallback if url is None: html = name else: html = "" + cgi.escape(name) + "" # We shouldn't need a base URL here anyway. return sanitise_html(html, feedurl, True, config) def string_to_html(s, config): """Convert a string to HTML.""" return sanitise_html(cgi.escape(s), "", True, config) template_re = re.compile(r'(__[^_].*?__)') def fill_template(template, bits): """Expand a template, replacing __x__ with bits["x"], and only including sections bracketed by __if_x__ .. [__else__ ..] __endif__ if bits["x"] is not "". If not bits.has_key("x"), __x__ expands to "".""" result = plugins.Box() plugins.call_hook("fill_template", template, bits, result) if result.value is not None: return result.value encoding = get_system_encoding() f = StringIO() if_stack = [] def write(s): if not False in if_stack: f.write(s) for part in template_re.split(template): if part.startswith("__") and part.endswith("__"): key = part[2:-2] if key.startswith("if_"): k = key[3:] if_stack.append(bits.has_key(k) and bits[k] != "") elif key == "endif": if if_stack != []: if_stack.pop() elif key == "else": if if_stack != []: if_stack.append(not if_stack.pop()) elif bits.has_key(key): if type(bits[key]) == types.UnicodeType: write(bits[key].encode(encoding)) else: write(bits[key]) else: write(part) v = f.getvalue() f.close() return v file_cache = {} def load_file(name): """Read the contents of a template file, caching the result so we don't have to read the file multiple times. The file is assumed to be in the system encoding; the result will be an ASCII string.""" if not file_cache.has_key(name): try: f = open(name) data = f.read() f.close() except IOError: raise ConfigError("Can't read template file: " + name) try: data = data.decode(get_system_encoding()) except UnicodeDecodeError, e: raise ConfigError("Character encoding problem in template file: " + name + ": " + str(e)) data = encode_references(data) file_cache[name] = data.encode(get_system_encoding()) return file_cache[name] def write_ascii(f, s, config): """Write the string s, which should only contain ASCII characters, to file f; if it isn't encodable in ASCII, then print a warning message and write UTF-8.""" try: f.write(s) except UnicodeEncodeError, e: config.bug("Error encoding output as ASCII; UTF-8 has been written instead.\n", e) f.write(s.encode("UTF-8")) def short_hash(s): """Return a human-manipulatable 'short hash' of a string.""" return hashlib.sha1(s).hexdigest()[-8:] def ensure_unicode(value, encoding): """Convert a structure returned by feedparser into an equivalent where all strings are represented as fully-decoded unicode objects.""" if isinstance(value, str): try: return value.decode(encoding) except: # If the encoding's invalid, at least preserve # the byte stream. return value.decode("ISO-8859-1") elif isinstance(value, unicode) and type(value) is not unicode: # This is a subclass of unicode (e.g. BeautifulSoup's # NavigableString, which is unpickleable in some versions of # the library), so force it to be a real unicode object. return unicode(value) elif isinstance(value, dict): d = {} for (k, v) in value.items(): d[k] = ensure_unicode(v, encoding) return d elif isinstance(value, list): return [ensure_unicode(v, encoding) for v in value] else: return value class BasicAuthProcessor(urllib2.BaseHandler): """urllib2 handler that does HTTP basic authentication or proxy authentication with a fixed username and password. (Unlike the classes to do this in urllib2, this doesn't wait for a 401/407 response first.)""" def __init__(self, user, password, proxy=False): self.auth = base64.b64encode(user + ":" + password) if proxy: self.header = "Proxy-Authorization" else: self.header = "Authorization" def http_request(self, req): req.add_header(self.header, "Basic " + self.auth) return req https_request = http_request class DisableIMProcessor(urllib2.BaseHandler): """urllib2 handler that disables RFC 3229 for a request.""" def http_request(self, req): # Request doesn't provide a method for removing headers -- # so overwrite the header instead. req.add_header("A-IM", "identity") return req https_request = http_request class ResponseLogProcessor(urllib2.BaseHandler): """urllib2 handler that maintains a log of HTTP responses.""" # Run after anything that's mangling headers (usually 500 or less), but # before HTTPErrorProcessor (1000). handler_order = 900 def __init__(self): self.log = [] def http_response(self, req, response): entry = { "url": req.get_full_url(), "status": response.getcode(), } location = response.info().get("Location") if location is not None: entry["location"] = location self.log.append(entry) return response https_response = http_response def get_log(self): return self.log non_alphanumeric_re = re.compile(r'<[^>]*>|\&[^\;]*\;|[^a-z0-9]') class Feed: """An RSS feed.""" def __init__(self, url): self.url = url self.period = 30 * 60 self.args = {} self.etag = None self.modified = None self.last_update = 0 self.feed_info = {} def needs_update(self, now): """Return True if it's time to update this feed, or False if its update period has not yet elapsed.""" return ((now - self.last_update) >= self.period) def get_state_filename(self): return "feeds/%s.state" % (short_hash(self.url),) def fetch(self, rawdog, config): """Fetch the current set of articles from the feed.""" handlers = [] logger = ResponseLogProcessor() handlers.append(logger) proxies = {} for name, value in self.args.items(): if name.endswith("_proxy"): proxies[name[:-6]] = value if len(proxies) != 0: handlers.append(urllib2.ProxyHandler(proxies)) if self.args.has_key("proxyuser") and self.args.has_key("proxypassword"): handlers.append(BasicAuthProcessor(self.args["proxyuser"], self.args["proxypassword"], proxy=True)) if self.args.has_key("user") and self.args.has_key("password"): handlers.append(BasicAuthProcessor(self.args["user"], self.args["password"])) if self.get_keepmin(config) == 0 or config["currentonly"]: # If RFC 3229 and "A-IM: feed" is used, then there's # no way to tell when an article has been removed. # So if we only want to keep articles that are still # being published by the feed, we have to turn it off. handlers.append(DisableIMProcessor()) plugins.call_hook("add_urllib2_handlers", rawdog, config, self, handlers) url = self.url # Turn plain filenames into file: URLs. (feedparser will open # plain filenames itself, but we want it to open the file with # urllib2 so we get a URLError if something goes wrong.) if not ":" in url: url = "file:" + url try: result = feedparser.parse(url, etag=self.etag, modified=self.modified, agent=HTTP_AGENT, handlers=handlers) except Exception, e: result = { "rawdog_exception": e, "rawdog_traceback": sys.exc_info()[2], } result["rawdog_responses"] = logger.get_log() return result def update(self, rawdog, now, config, articles, p): """Add new articles from a feed to the collection. Returns True if any articles were read, False otherwise.""" # Note that feedparser might have thrown an exception -- # so until we print the error message and return, we # can't assume that p contains any particular field. responses = p.get("rawdog_responses") if len(responses) > 0: last_status = responses[-1]["status"] elif len(p.get("feed", [])) != 0: # Some protocol other than HTTP -- assume it's OK, # since we got some content. last_status = 200 else: # Timeout, or empty response from non-HTTP. last_status = 0 version = p.get("version") if version is None: version = "" self.last_update = now errors = [] fatal = False old_url = self.url if "rawdog_exception" in p: errors.append("Error fetching or parsing feed:") errors.append(str(p["rawdog_exception"])) if config["showtracebacks"]: from traceback import format_tb errors.append("".join(format_tb(p["rawdog_traceback"]))) errors.append("") fatal = True if len(responses) != 0 and responses[0]["status"] == 301: # Permanent redirect(s). Find the new location. i = 0 while i < len(responses) and responses[i]["status"] == 301: i += 1 location = responses[i - 1].get("location") if location is None: errors.append("The feed returned a permanent redirect, but without a new location.") else: errors.append("New URL: " + location) errors.append("The feed has moved permanently to a new URL.") if config["changeconfig"]: rawdog.change_feed_url(self.url, location, config) errors.append("The config file has been updated automatically.") else: errors.append("You should update its entry in your config file.") errors.append("") bozo_exception = p.get("bozo_exception") got_urlerror = isinstance(bozo_exception, urllib2.URLError) got_timeout = isinstance(bozo_exception, socket.timeout) if got_urlerror or got_timeout: # urllib2 reported an error when fetching the feed. # Check to see if it was a timeout. if not (got_timeout or str(bozo_exception).endswith("timed out>")): errors.append("Error while fetching feed:") errors.append(str(bozo_exception)) errors.append("") fatal = True elif config["ignoretimeouts"]: return False else: errors.append("Timeout while reading feed.") errors.append("") fatal = True elif last_status == 304: # The feed hasn't changed. Return False to indicate # that we shouldn't do expiry. return False elif last_status in [403, 410]: # The feed is disallowed or gone. The feed should be # unsubscribed. errors.append("The feed has gone.") errors.append("You should remove it from your config file.") errors.append("") fatal = True elif last_status / 100 != 2: # Some sort of client or server error. The feed may # need unsubscribing. errors.append("The feed returned an error.") errors.append("If this condition persists, you should remove it from your config file.") errors.append("") fatal = True elif version == "" and len(p.get("entries", [])) == 0: # feedparser couldn't detect the type of this feed or # retrieve any entries from it. errors.append("The data retrieved from this URL could not be understood as a feed.") errors.append("You should check whether the feed has changed URLs or been removed.") errors.append("") fatal = True old_error = "\n".join(errors) plugins.call_hook("feed_fetched", rawdog, config, self, p, old_error, not fatal) if len(errors) != 0: print >>sys.stderr, "Feed: " + old_url if last_status != 0: print >>sys.stderr, "HTTP Status: " + str(last_status) for line in errors: print >>sys.stderr, line if fatal: return False # From here, we can assume that we've got a complete feedparser # response. p = ensure_unicode(p, p.get("encoding") or "UTF-8") # No entries means the feed hasn't changed, but for some reason # we didn't get a 304 response. Handle it the same way. if len(p["entries"]) == 0: return False self.etag = p.get("etag") self.modified = p.get("modified") self.feed_info = p["feed"] feed = self.url article_ids = {} if config["useids"]: # Find IDs for existing articles. for (hash, a) in articles.items(): id = a.entry_info.get("id") if a.feed == feed and id is not None: article_ids[id] = a seen_articles = set() sequence = 0 for entry_info in p["entries"]: article = Article(feed, entry_info, now, sequence) ignore = plugins.Box(False) plugins.call_hook("article_seen", rawdog, config, article, ignore) if ignore.value: continue seen_articles.add(article.hash) sequence += 1 id = entry_info.get("id") if id in article_ids: existing_article = article_ids[id] elif article.hash in articles: existing_article = articles[article.hash] else: existing_article = None if existing_article is not None: existing_article.update_from(article, now) plugins.call_hook("article_updated", rawdog, config, existing_article, now) else: articles[article.hash] = article plugins.call_hook("article_added", rawdog, config, article, now) if config["currentonly"]: for (hash, a) in articles.items(): if a.feed == feed and hash not in seen_articles: del articles[hash] return True def get_html_name(self, config): if self.feed_info.has_key("title_detail"): r = detail_to_html(self.feed_info["title_detail"], True, config) elif self.feed_info.has_key("link"): r = string_to_html(self.feed_info["link"], config) else: r = string_to_html(self.url, config) if r is None: r = "" return r def get_html_link(self, config): s = self.get_html_name(config) if self.feed_info.has_key("link"): return '' + s + '' else: return s def get_id(self, config): if self.args.has_key("id"): return self.args["id"] else: r = self.get_html_name(config).lower() return non_alphanumeric_re.sub('', r) def get_keepmin(self, config): return self.args.get("keepmin", config["keepmin"]) class Article: """An article retrieved from an RSS feed.""" def __init__(self, feed, entry_info, now, sequence): self.feed = feed self.entry_info = entry_info self.sequence = sequence self.date = None parsed = entry_info.get("updated_parsed") if parsed is None: parsed = entry_info.get("published_parsed") if parsed is None: parsed = entry_info.get("created_parsed") if parsed is not None: try: self.date = calendar.timegm(parsed) except OverflowError: pass self.hash = self.compute_initial_hash() self.last_seen = now self.added = now def compute_initial_hash(self): """Compute an initial unique hash for an article. The generated hash must be unique amongst all articles in the system (i.e. it can't just be the article ID, because that would collide if more than one feed included the same article).""" h = hashlib.sha1() def add_hash(s): h.update(s.encode("UTF-8")) add_hash(self.feed) entry_info = self.entry_info if entry_info.has_key("title"): add_hash(entry_info["title"]) if entry_info.has_key("link"): add_hash(entry_info["link"]) if entry_info.has_key("content"): for content in entry_info["content"]: add_hash(content["value"]) if entry_info.has_key("summary_detail"): add_hash(entry_info["summary_detail"]["value"]) return h.hexdigest() def update_from(self, new_article, now): """Update this article's contents from a newer article that's been identified to be the same.""" self.entry_info = new_article.entry_info self.sequence = new_article.sequence self.date = new_article.date self.last_seen = now def can_expire(self, now, config): return ((now - self.last_seen) > config["expireage"]) def get_sort_date(self, config): if config["sortbyfeeddate"]: return self.date or self.added else: return self.added class DayWriter: """Utility class for writing day sections into a series of articles.""" def __init__(self, file, config): self.lasttime = [] self.file = file self.counter = 0 self.config = config def start_day(self, tm): print >>self.file, '
'
day = safe_ftime(self.config["dayformat"], tm)
print >>self.file, 'rawdog
"""
elif name == "feedlist":
return """
"""
elif name == "feeditem":
return """
__feed_title__
__feed_icon__
__feed_last_update__
__feed_next_update__
"""
else:
raise KeyError("Unknown template name: " + name)
def show_template(self, name, config):
"""Show the contents of a template, as currently configured."""
try:
print self.get_template(config, name),
except KeyError:
print >>sys.stderr, "Unknown template name: " + a
def write_article(self, f, article, config):
"""Write an article to the given file."""
feed = self.feeds[article.feed]
entry_info = article.entry_info
link = entry_info.get("link")
if link == "":
link = None
guid = entry_info.get("id")
if guid == "":
guid = None
itembits = self.get_feed_bits(config, feed)
for name, value in feed.args.items():
if name.startswith("define_"):
itembits[name[7:]] = sanitise_html(value, "", True, config)
title = detail_to_html(entry_info.get("title_detail"), True, config)
key = None
for k in ["content", "summary_detail"]:
if entry_info.has_key(k):
key = k
break
if key is None:
description = None
else:
force_preformatted = (feed.args.get("format", "default") == "text")
description = detail_to_html(entry_info[key], False, config, force_preformatted)
date = article.date
if title is None:
if link is None:
title = "Article"
else:
title = "Link"
itembits["title_no_link"] = title
if link is not None:
itembits["url"] = string_to_html(link, config)
else:
itembits["url"] = ""
if guid is not None:
itembits["guid"] = string_to_html(guid, config)
else:
itembits["guid"] = ""
if link is None:
itembits["title"] = title
else:
itembits["title"] = '' + title + ''
itembits["hash"] = short_hash(article.hash)
if description is not None:
itembits["description"] = description
else:
itembits["description"] = ""
author = author_to_html(entry_info, feed.url, config)
if author is not None:
itembits["author"] = author
else:
itembits["author"] = ""
itembits["added"] = format_time(article.added, config)
if date is not None:
itembits["date"] = format_time(date, config)
else:
itembits["date"] = ""
plugins.call_hook("output_item_bits", self, config, feed, article, itembits)
itemtemplate = self.get_template(config, "item")
f.write(fill_template(itemtemplate, itembits))
def write_remove_dups(self, articles, config, now):
"""Filter the list of articles to remove articles that are too
old or are duplicates."""
kept_articles = []
seen_links = set()
seen_guids = set()
dup_count = 0
for article in articles:
feed = self.feeds[article.feed]
age = now - article.added
maxage = feed.args.get("maxage", config["maxage"])
if maxage != 0 and age > maxage:
continue
entry_info = article.entry_info
link = entry_info.get("link")
if link == "":
link = None
guid = entry_info.get("id")
if guid == "":
guid = None
if not feed.args.get("allowduplicates", False):
is_dup = False
for key in config["hideduplicates"]:
if key == "id" and guid is not None:
if guid in seen_guids:
is_dup = True
seen_guids.add(guid)
elif key == "link" and link is not None:
if link in seen_links:
is_dup = True
seen_links.add(link)
if is_dup:
dup_count += 1
continue
kept_articles.append(article)
return (kept_articles, dup_count)
def get_feed_bits(self, config, feed):
"""Get the bits that are used to describe a feed."""
bits = {}
bits["feed_id"] = feed.get_id(config)
bits["feed_hash"] = short_hash(feed.url)
bits["feed_title"] = feed.get_html_link(config)
bits["feed_title_no_link"] = detail_to_html(feed.feed_info.get("title_detail"), True, config)
bits["feed_url"] = string_to_html(feed.url, config)
bits["feed_icon"] = 'XML'
bits["feed_last_update"] = format_time(feed.last_update, config)
bits["feed_next_update"] = format_time(feed.last_update + feed.period, config)
return bits
def write_feeditem(self, f, feed, config):
"""Write a feed list item."""
bits = self.get_feed_bits(config, feed)
f.write(fill_template(self.get_template(config, "feeditem"), bits))
def write_feedlist(self, f, config):
"""Write the feed list."""
bits = {}
feeds = [(feed.get_html_name(config).lower(), feed)
for feed in self.feeds.values()]
feeds.sort()
feeditems = StringIO()
for key, feed in feeds:
self.write_feeditem(feeditems, feed, config)
bits["feeditems"] = feeditems.getvalue()
feeditems.close()
f.write(fill_template(self.get_template(config, "feedlist"), bits))
def get_main_template_bits(self, config):
"""Get the bits that are used in the default main template,
with the exception of items and num_items."""
bits = { "version" : VERSION }
bits.update(config["defines"])
refresh = config["expireage"]
for feed in self.feeds.values():
if feed.period < refresh: refresh = feed.period
bits["refresh"] = """"""
f = StringIO()
self.write_feedlist(f, config)
bits["feeds"] = f.getvalue()
f.close()
bits["num_feeds"] = str(len(self.feeds))
return bits
def write_output_file(self, articles, article_dates, config):
"""Write a regular rawdog HTML output file."""
f = StringIO()
dw = DayWriter(f, config)
plugins.call_hook("output_items_begin", self, config, f)
for article in articles:
if not plugins.call_hook("output_items_heading", self, config, f, article, article_dates[article]):
dw.time(article_dates[article])
self.write_article(f, article, config)
dw.close()
plugins.call_hook("output_items_end", self, config, f)
bits = self.get_main_template_bits(config)
bits["items"] = f.getvalue()
f.close()
bits["num_items"] = str(len(articles))
plugins.call_hook("output_bits", self, config, bits)
s = fill_template(self.get_template(config, "page"), bits)
outputfile = config["outputfile"]
if outputfile == "-":
write_ascii(sys.stdout, s, config)
else:
config.log("Writing output file: ", outputfile)
f = open(outputfile + ".new", "w")
write_ascii(f, s, config)
f.close()
os.rename(outputfile + ".new", outputfile)
def write(self, config):
"""Perform the write action: write articles to the output
file."""
config.log("Starting write")
now = time.time()
def list_articles(articles):
return [(-a.get_sort_date(config), a.feed, a.sequence, a.hash) for a in articles.values()]
if config["splitstate"]:
article_list = []
for feed in self.feeds.values():
with persister.get(FeedState, feed.get_state_filename()) as feedstate:
article_list += list_articles(feedstate.articles)
else:
article_list = list_articles(self.articles)
numarticles = len(article_list)
if not plugins.call_hook("output_sort_articles", self, config, article_list):
article_list.sort()
if config["maxarticles"] != 0:
article_list = article_list[:config["maxarticles"]]
if config["splitstate"]:
wanted = {}
for (date, feed_url, seq, hash) in article_list:
if not feed_url in self.feeds:
# This can happen if you've managed to
# kill rawdog between it updating a
# split state file and the main state
# -- so just ignore the article and
# it'll expire eventually.
continue
wanted.setdefault(feed_url, []).append(hash)
found = {}
for (feed_url, article_hashes) in wanted.items():
feed = self.feeds[feed_url]
with persister.get(FeedState, feed.get_state_filename()) as feedstate:
for hash in article_hashes:
found[hash] = feedstate.articles[hash]
else:
found = self.articles
articles = []
article_dates = {}
for (date, feed, seq, hash) in article_list:
a = found.get(hash)
if a is not None:
articles.append(a)
article_dates[a] = -date
plugins.call_hook("output_write", self, config, articles)
if not plugins.call_hook("output_sorted_filter", self, config, articles):
(articles, dup_count) = self.write_remove_dups(articles, config, now)
else:
dup_count = 0
config.log("Selected ", len(articles), " of ", numarticles, " articles to write; ignored ", dup_count, " duplicates")
if not plugins.call_hook("output_write_files", self, config, articles, article_dates):
self.write_output_file(articles, article_dates, config)
config.log("Finished write")
def usage():
"""Display usage information."""
print """rawdog, version """ + VERSION + """
Usage: rawdog [OPTION]...
General options (use only once):
-d|--dir DIR Use DIR instead of ~/.rawdog
-N, --no-locking Do not lock the state file
-v, --verbose Print more detailed status information
-V|--log FILE Append detailed status information to FILE
-W, --no-lock-wait Exit silently if state file is locked
Actions (performed in order given):
-a|--add URL Try to find a feed associated with URL and
add it to the config file
-c|--config FILE Read additional config file FILE
-f|--update-feed URL Force an update on the single feed URL
-l, --list List feeds known at time of last update
-r|--remove URL Remove feed URL from the config file
-s|--show TEMPLATE Show the contents of a template
(TEMPLATE may be: page item feedlist feeditem)
-u, --update Fetch data from feeds and store it
-w, --write Write out HTML output
Special actions (all other options are ignored if one of these is specified):
--dump URL Show what rawdog's parser returns for URL
--help Display this help and exit
Report bugs to ."""
def main(argv):
"""The command-line interface to the aggregator."""
locale.setlocale(locale.LC_ALL, "")
# This is quite expensive and not threadsafe, so we do it on
# startup and cache the result.
global system_encoding
system_encoding = locale.getpreferredencoding()
try:
SHORTOPTS = "a:c:d:f:lNr:s:tTuvV:wW"
LONGOPTS = [
"add=",
"config=",
"dir=",
"dump=",
"help",
"list",
"log=",
"no-lock-wait",
"no-locking",
"remove=",
"show=",
"show-itemtemplate",
"show-template",
"update",
"update-feed=",
"verbose",
"write",
]
(optlist, args) = getopt.getopt(argv, SHORTOPTS, LONGOPTS)
except getopt.GetoptError, s:
print s
usage()
return 1
if len(args) != 0:
usage()
return 1
if "HOME" in os.environ:
statedir = os.environ["HOME"] + "/.rawdog"
else:
statedir = None
verbose = False
logfile_name = None
locking = True
no_lock_wait = False
for o, a in optlist:
if o == "--dump":
import pprint
pprint.pprint(feedparser.parse(a, agent=HTTP_AGENT))
return 0
elif o == "--help":
usage()
return 0
elif o in ("-d", "--dir"):
statedir = a
elif o in ("-N", "--no-locking"):
locking = False
elif o in ("-v", "--verbose"):
verbose = True
elif o in ("-V", "--log"):
logfile_name = a
elif o in ("-W", "--no-lock-wait"):
no_lock_wait = True
if statedir is None:
print "$HOME not set and state dir not explicitly specified; please use -d/--dir"
return 1
try:
os.chdir(statedir)
except OSError:
print "No " + statedir + " directory"
return 1
sys.path.append(".")
config = Config(locking, logfile_name)
def load_config(fn):
try:
config.load(fn)
except ConfigError, err:
print >>sys.stderr, "In " + fn + ":"
print >>sys.stderr, err
return 1
if verbose:
config["verbose"] = True
load_config("config")
global persister
persister = Persister(config)
rawdog_p = persister.get(Rawdog, "state")
rawdog = rawdog_p.open(no_block=no_lock_wait)
if rawdog is None:
return 0
if not rawdog.check_state_version():
print "The state file " + statedir + "/state was created by an older"
print "version of rawdog, and cannot be read by this version."
print "Removing the state file will fix it."
return 1
rawdog.sync_from_config(config)
plugins.call_hook("startup", rawdog, config)
for o, a in optlist:
if o in ("-a", "--add"):
add_feed("config", a, rawdog, config)
config.reload()
rawdog.sync_from_config(config)
elif o in ("-c", "--config"):
load_config(a)
rawdog.sync_from_config(config)
elif o in ("-f", "--update-feed"):
rawdog.update(config, a)
elif o in ("-l", "--list"):
rawdog.list(config)
elif o in ("-r", "--remove"):
remove_feed("config", a, config)
config.reload()
rawdog.sync_from_config(config)
elif o in ("-s", "--show"):
rawdog.show_template(a, config)
elif o in ("-t", "--show-template"):
rawdog.show_template("page", config)
elif o in ("-T", "--show-itemtemplate"):
rawdog.show_template("item", config)
elif o in ("-u", "--update"):
rawdog.update(config)
elif o in ("-w", "--write"):
rawdog.write(config)
plugins.call_hook("shutdown", rawdog, config)
rawdog_p.close()
return 0
rawdog-2.19/rawdoglib/plugins.py 0000644 0004715 0004715 00000004531 12167103316 016350 0 ustar ats ats 0000000 0000000 # plugins: handle add-on modules for rawdog.
# Copyright 2004, 2005, 2013 Adam Sampson
#
# rawdog is free software; you can redistribute and/or modify it
# under the terms of that license as published by the Free Software
# Foundation; either version 2 of the License, or (at your option)
# any later version.
#
# rawdog is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with rawdog; see the file COPYING. If not, write to the Free
# Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
# MA 02110-1301, USA, or see http://www.gnu.org/.
# The design of rawdog's plugin API was inspired by Stuart Langridge's
# Vellum weblog system:
# http://www.kryogenix.org/code/vellum/
import os, imp
class Box:
"""Utility class that holds a mutable value. Useful for passing
immutable types by reference."""
def __init__(self, value=None):
self.value = value
plugin_count = 0
def load_plugins(dir, config):
global plugin_count
try:
files = os.listdir(dir)
except OSError:
# Ignore directories that can't be read.
return
for file in files:
if file == "" or file[0] == ".":
continue
desc = None
for d in imp.get_suffixes():
if file.endswith(d[0]) and d[2] == imp.PY_SOURCE:
desc = d
if desc is None:
continue
fn = os.path.join(dir, file)
config.log("Loading plugin ", fn)
f = open(fn, "r")
imp.load_module("plugin%d" % (plugin_count,), f, fn, desc)
plugin_count += 1
f.close()
attached = {}
def attach_hook(hookname, func):
"""Attach a function to a hook. The function should take the
appropriate arguments for the hook, and should return either True or
False to indicate whether further functions should be processed."""
attached.setdefault(hookname, []).append(func)
def call_hook(hookname, *args):
"""Call all the functions attached to a hook with the given
arguments, in the order they were added, stopping if a hook function
returns False. Returns True if any hook function returned False (i.e.
returns True if any hook function handled the request)."""
for func in attached.get(hookname, []):
if not func(*args):
return True
return False
rawdog-2.19/setup.py 0000755 0004715 0004715 00000001432 12273445020 014054 0 ustar ats ats 0000000 0000000 #!/usr/bin/env python
from distutils.core import setup
import sys
if sys.version_info < (2, 6) or sys.version_info >= (3,):
print("rawdog requires Python 2.6 or later, and not Python 3.")
sys.exit(1)
setup(name = "rawdog",
version = "2.19",
description = "RSS Aggregator Without Delusions Of Grandeur",
author = "Adam Sampson",
author_email = "ats@offog.org",
url = "http://offog.org/code/rawdog/",
scripts = ['rawdog'],
data_files = [('share/man/man1', ['rawdog.1'])],
packages = ['rawdoglib'],
classifiers = [
"Development Status :: 5 - Production/Stable",
"Environment :: Console",
"License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)",
"Operating System :: POSIX",
"Programming Language :: Python :: 2",
"Topic :: Internet :: WWW/HTTP",
])
rawdog-2.19/PLUGINS 0000644 0004715 0004715 00000030233 12176717227 013421 0 ustar ats ats 0000000 0000000 # Writing rawdog plugins
## Introduction
As provided, rawdog provides a fairly small set of features. In order to
make it do more complex jobs, rawdog can be extended using plugin
modules written in Python. This document is intended for developers who
want to extend rawdog by writing plugins.
Extensions work by registering hook functions which are called by
various bits of rawdog's core as it runs. These functions can modify
rawdog's internal state in various interesting ways. An arbitrary number
of functions can be attached to each hook; they are called in the order
they were attached. Hook functions take various arguments depending on
where they're called from, and returns a boolean value indicating
whether further functions attached to the same hook should be called.
The "plugindirs" config option gives a list of directories to search for
plugins; all Python modules found in those directories will be loaded by
rawdog. In practice, this means that you need to call your file
something ending in ".py" to have it recognised as a plugin.
## The plugins module
All plugins should import the `rawdoglib.plugins` module, which provides
the functions for registering and calling hooks, along with some
utilities for plugins. Many plugins will also want to import the
`rawdoglib.rawdog` module, which contains rawdog's core functionality,
much of which is reusable.
### rawdoglib.plugins.attach_hook(hook_name, function)
The attach_hook function adds a hook function to the hook of the given
name.
### rawdoglib.plugins.Box
The Box class is used to pass immutable types by reference to hook
functions; this allows several plugins to modify a value. It contains a
single `value` attribute for the value it is holding.
## Plugin storage
Since some plugins will need to keep state between runs, the Rawdog
object that most hook functions are provided with has a
`get_plugin_storage` method, which when called with a plugin identifier
for your plugin as an argument will give you a reference to a dictionary
which will be persisted in the rawdog state file. The dictionary is empty to
start with; you may store any pickleable objects you like in it. Plugin
identifiers should be strings based on your email address, in order to be
globally unique -- for example, `org.offog.ats.archive`.
After changing a plugin storage dictionary, you must call "rawdog.modified()"
to ensure that rawdog will write out its state file.
## Hooks
Most hook functions are called with "rawdog" and "config" as their first
two arguments; these are references to the aggregator's Rawdog and
Config objects.
If you need a hook that doesn't currently exist, please contact me.
The following hooks are supported:
### startup(rawdog, config)
Run when rawdog starts up, after the state file and config file have
been loaded, but before rawdog starts processing command-line arguments.
### shutdown(rawdog, config)
Run just before rawdog saves the state file and exits.
### config_option(config, name, value)
* name: the option name
* value: the option value
Called when rawdog encounters a config file option that it doesn't
recognise. The rawdoglib.rawdog.parse_* functions will probably be
useful when dealing with config options. You can raise ValueError to
have rawdog print an appropriate error message. You should return False
from this hook if name is an option you recognise.
Note that using config.log in this hook will probably not do what you
want, because the verbose flag may not yet have been turned on.
### config_option_arglines(config, name, value, arglines)
* name: the option name
* value: the option value
* arglines: a list of extra indented lines given after the option (which
can be used to supply extra arguments for the option)
As config_option for options that can handle extra argument lines.
If the options you are implementing should not have extra arguments,
then use the config_option hook instead.
### output_sort_articles(rawdog, config, articles)
* articles: the mutable list of (date, feed_url, sequence_number,
article_hash) tuples
Called to sort the list of articles to write. The default action here is
to just call the list's sort method; if you sort the list in a different
way, you should return False from this hook to prevent rawdog from
resorting it afterwards.
Later versions of rawdog may add more items at the end of the tuple;
bear this in mind when you're manipulating the items.
### output_write(rawdog, config, articles)
* articles: the mutable list of Article objects
Called immediately before output_sorted_filter; this hook is here for
backwards compatibility, and should not be used in new plugins.
### output_sorted_filter(rawdog, config, articles)
* articles: the mutable list of Article objects
Called after rawdog sorts the list of articles to write, but before it
removes duplicate and excessively old articles. This hook can be used to
implement alternate duplicate-filtering methods. If you return False
from this hook, then rawdog will not do its usual duplicate-removing
filter pass.
### output_write_files(rawdog, config, articles, article_dates)
* articles: the mutable list of Article objects
* article_dates: a dictionary mapping Article objects to the dates that
were used to sort them
Called when rawdog is about to write its output to files. This hook can
be used to implement alternative output methods.
If you return False from this hook, then rawdog will not write any
output itself (and the later output_ hooks will thus not be called). I
would suggest not returning False here unless you plan to call the
rawdog.write_output_file method from your hook implementation; failure
to do so will most likely break other plugins.
### output_items_begin(rawdog, config, f)
* f: a writable file object (__items__)
Called before rawdog starts expanding the items template. This set of
hooks can be used to implement alternative date (or other section)
headings.
### output_items_heading(rawdog, config, f, article, date)
* f: a writable file object (__items__)
* article: the Article object about to be written
* date: the Article's date for sorting purposes
Called before each item is written. If you return False from this hook,
then rawdog's normal time-based section headings will not be written.
### output_items_end(rawdog, config, f)
* f: a writable file object (__items__)
Called after all items are written.
### output_bits(rawdog, config, bits)
* bits: a dictionary of template parameters
Called before expanding the page template. This hook can be used to add
extra template parameters.
Note that template parameters should be valid HTML, with entities
escaped, even if they're URLs or similar. You can use rawdog's
`rawdoglib.rawdog.string_to_html` function to do this for you:
the_thing = "This can contain arbitary text & stuff"
bits["thing"] = string_to_html(the_thing, config)
It's also good idea for template parameter names to be valid Python
identifiers, so that plugins that replace the template system with
something smarter can make them into local variables.
### output_item_bits(rawdog, config, feed, article, bits)
* feed: the Feed containing this article
* article: the Article being templated
* bits: a dictionary of template parameters
Called before expanding the item template for an article. This hook can
be used to add extra template parameters.
(See the documentation for `output_bits` for some advice on adding
template parameters.)
### pre_update_feed(rawdog, config, feed)
* feed: the Feed about to be updated
Called before a feed's content is fetched. This hook can be used to
perform extra actions before fetching a feed. Note that if `usethreads`
is set to a positive number in the config file, this hook may be called
from a worker thread.
### mid_update_feed(rawdog, config, feed, content)
* feed: the Feed being updated
* content: the feedparser output from the feed (may be None)
Called after a feed's content has been fetched, but before rawdog's
internal state has been updated. This hook can be used to modify
feedparser's output.
### post_update_feed(rawdog, config, feed, seen_articles)
* feed: the Feed that has been updated
* seen_articles: a boolean indicating whether any articles were read
from the feed
Called after a feed is updated.
### article_seen(rawdog, config, article, ignore)
* article: the Article that has been received
* ignore: a Boxed boolean indicating whether to ignore the article
Called when an article is received from a feed. This hook can be used to
modify or ignore incoming articles.
### article_updated(rawdog, config, article, now)
* article: the Article that has been updated
* now: the current time
Called after an article has been updated (when rawdog receives an
article from a feed that it already has).
### article_added(rawdog, config, article, now)
* article: the Article that has been added
* now: the current time
Called after a new article has been added.
### article_expired(rawdog, config, article, now)
* article: the Article that will be expired
* now: the current time
Called before an article is expired.
### fill_template(template, bits, result)
* template: the template string to fill
* bits: a dictionary of template arguments
* result: a Boxed Unicode string for the result of template expansion
Called whenever template expansion is performed. If you set the value
inside result to something other than None, then rawdog will treat that
value as the result of template expansion (rather than performing its
normal expansion process); you can thus use this hook either for
manipulating template parameters, or for replacing the template system
entirely.
### tidy_args(config, args, baseurl, inline)
* args: a dictionary of keyword arguments for Tidy
* baseurl: the URL at which the HTML was originally found
* inline: a boolean indicating whether the output should be inline HTML
or a block element
When HTML is being sanitised by rawdog and the "tidyhtml" option is
enabled, this hook will be called just before Tidy is run (either via
PyTidyLib or via mx.Tidy). It can be used to add or modify Tidy options;
for example, to make it produce XHTML output.
### clean_html(config, html, baseurl, inline)
* html: a Boxed Unicode string containing the HTML being cleaned
* baseurl: the URL at which the HTML was originally found
* inline: a boolean indicating whether the output should be inline HTML
or a block element
Called whenever HTML is being sanitised by rawdog (after its existing
HTML sanitisation processes). You can use this to implement extra
sanitisation passes. You'll need to update the boxed value with the new,
cleaned string.
### add_urllib2_handlers(rawdog, config, feed, handlers)
* feed: the Feed to which the request will be made
* handlers: the mutable list of urllib2 *Handler objects that will be
passed to feedparser
Called before feedparser is used to fetch feed content. This hook can be
used to add additional urllib2 handlers to cope with unusual protocol
requirements; use `handlers.append` to add extra handlers.
### feed_fetched(rawdog, config, feed, feed_data, error, non_fatal)
* feed: the Feed that has just been fetched
* feed_data: the data returned from feedparser.parse
* error: the error string if an error occurred, or None if no error
occurred
* non_fatal: if error is not None, a boolean indicating whether the
error was fatal
Called after feedparser has been called to fetch the feed. This hook can
be used to manipulate the received feed data or implement custom error
handling.
## Obsolete hooks
The following hooks existed in previous versions of rawdog, but are no
longer supported:
* output_filter (since rawdog 2.12); use output_sorted_filter instead
* output_sort (since rawdog 2.12); use output_sort_articles instead
## Examples
### backwards.py
This is probably the simplest useful example plugin: it reverses the
sort order of the output.
import rawdoglib.plugins
def backwards(rawdog, config, articles):
articles.sort()
articles.reverse()
return False
rawdoglib.plugins.attach_hook("output_sort_articles", backwards)
### option.py
This plugin shows how to handle a config file option.
import rawdoglib.plugins
def option(config, name, value):
if name == "myoption":
print "Test plugin option:", value
return False
else:
return True
rawdoglib.plugins.attach_hook("config_option", option)
rawdog-2.19/testserver.py 0000644 0004715 0004715 00000016055 12177453766 015151 0 ustar ats ats 0000000 0000000 # testserver: servers for rawdog's test suite.
# Copyright 2013 Adam Sampson
#
# rawdog is free software; you can redistribute and/or modify it
# under the terms of that license as published by the Free Software
# Foundation; either version 2 of the License, or (at your option)
# any later version.
#
# rawdog is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with rawdog; see the file COPYING. If not, write to the Free
# Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
# MA 02110-1301, USA, or see http://www.gnu.org/.
import BaseHTTPServer
import SimpleHTTPServer
import SocketServer
import base64
import cStringIO
import gzip
import hashlib
import os
import re
import sys
import threading
import time
class TimeoutRequestHandler(SocketServer.BaseRequestHandler):
"""Request handler for a server that just does nothing for a few
seconds, then disconnects. This is used for testing timeout handling."""
def handle(self):
time.sleep(5)
class TimeoutServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
"""Timeout server for rawdog's test suite."""
pass
class HTTPRequestHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
"""HTTP request handler for rawdog's test suite."""
# do_GET/do_HEAD are copied from SimpleHTTPServer because send_head isn't
# part of the API.
def do_GET(self):
f = self.send_head()
if f:
self.copyfile(f, self.wfile)
f.close()
def do_HEAD(self):
f = self.send_head()
if f:
f.close()
def send_head(self):
# Look for lines of the form "/oldpath /newpath" in .rewrites.
try:
f = open(os.path.join(self.server.files_dir, ".rewrites"))
for line in f.readlines():
(old, new) = line.split(None, 1)
if self.path == old:
self.path = new
f.close()
except IOError:
pass
m = re.match(r'^/auth-([^/-]+)-([^/]+)(/.*)$', self.path)
if m:
# Require basic authentication.
auth = "Basic " + base64.b64encode(m.group(1) + ":" + m.group(2))
if self.headers.get("Authorization") != auth:
self.send_response(401)
self.end_headers()
return None
self.path = m.group(3)
m = re.match(r'^/digest-([^/-]+)-([^/]+)(/.*)$', self.path)
if m:
# Require digest authentication. (Not a good implementation!)
realm = "rawdog test server"
nonce = "0123456789abcdef"
a1 = m.group(1) + ":" + realm + ":" + m.group(2)
a2 = "GET:" + self.path
def h(s):
return hashlib.md5(s).hexdigest()
response = h(h(a1) + ":" + nonce + ":" + h(a2))
mr = re.search(r'response="([^"]*)"',
self.headers.get("Authorization", ""))
if mr is None or mr.group(1) != response:
self.send_response(401)
self.send_header("WWW-Authenticate",
'Digest realm="%s", nonce="%s"'
% (realm, nonce))
self.end_headers()
return None
self.path = m.group(3)
m = re.match(r'^/(\d\d\d)(/.*)?$', self.path)
if m:
# Request for a particular response code.
code = int(m.group(1))
self.send_response(code)
if m.group(2):
self.send_header("Location", self.server.base_url + m.group(2))
self.end_headers()
return None
encoding = None
m = re.match(r'^/(gzip)(/.*)$', self.path)
if m:
# Request for a content encoding.
encoding = m.group(1)
self.path = m.group(2)
m = re.match(r'^/([^/]+)$', self.path)
if m:
# Request for a file.
filename = os.path.join(self.server.files_dir, m.group(1))
try:
f = open(filename, "rb")
except IOError:
self.send_response(404)
self.end_headers()
return None
# Use the SHA1 hash as an ETag.
etag = '"' + hashlib.sha1(f.read()).hexdigest() + '"'
f.seek(0)
# Oversimplistic, but matches what feedparser sends.
if self.headers.get("If-None-Match", "") == etag:
self.send_response(304)
self.end_headers()
return None
size = os.fstat(f.fileno()).st_size
mime_type = "text/plain"
if filename.endswith(".rss") or filename.endswith(".rss2"):
mime_type = "application/rss+xml"
elif filename.endswith(".rdf"):
mime_type = "application/rdf+xml"
elif filename.endswith(".atom"):
mime_type = "application/atom+xml"
elif filename.endswith(".html"):
mime_type = "text/html"
self.send_response(200)
if encoding:
self.send_header("Content-Encoding", encoding)
if encoding == "gzip":
data = f.read()
f.close()
f = cStringIO.StringIO()
g = gzip.GzipFile(fileobj=f, mode="wb")
g.write(data)
g.close()
size = f.tell()
f.seek(0)
self.send_header("Content-Length", size)
self.send_header("Content-Type", mime_type)
self.send_header("ETag", etag)
self.end_headers()
return f
# A request we can't handle.
self.send_response(500)
self.end_headers()
return None
def log_message(self, fmt, *args):
f = open(self.server.files_dir + "/.log", "a")
f.write(fmt % args + "\n")
f.close()
class HTTPServer(BaseHTTPServer.HTTPServer):
"""HTTP server for rawdog's test suite."""
def __init__(self, base_url, files_dir, *args, **kwargs):
self.base_url = base_url
self.files_dir = files_dir
BaseHTTPServer.HTTPServer.__init__(self, *args, **kwargs)
def main(args):
if len(args) < 3:
print "Usage: testserver.py HOSTNAME TIMEOUT-PORT HTTP-PORT FILES-DIR"
sys.exit(1)
hostname = args[0]
timeout_port = int(args[1])
http_port = int(args[2])
files_dir = args[3]
timeoutd = TimeoutServer((hostname, timeout_port), TimeoutRequestHandler)
t = threading.Thread(target=timeoutd.serve_forever)
t.daemon = True
t.start()
base_url = "http://" + hostname + ":" + str(http_port)
httpd = HTTPServer(base_url, files_dir, (hostname, http_port), HTTPRequestHandler)
httpd.serve_forever()
if __name__ == "__main__":
main(sys.argv[1:])
rawdog-2.19/COPYING 0000644 0004715 0004715 00000043135 10404407140 013373 0 ustar ats ats 0000000 0000000 GNU GENERAL PUBLIC LICENSE
Version 2, June 1991
Copyright (C) 1989, 1991 Free Software Foundation, Inc.
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change free
software--to make sure the software is free for all its users. This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it. (Some other Free Software Foundation software is covered by
the GNU Library General Public License instead.) You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.
To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have. You must make sure that they, too, receive or can get the
source code. And you must show them these terms so they know their
rights.
We protect your rights with two steps: (1) copyright the software, and
(2) offer you this license which gives you legal permission to copy,
distribute and/or modify the software.
Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software. If the software is modified by someone else and passed on, we
want its recipients to know that what they have is not the original, so
that any problems introduced by others will not reflect on the original
authors' reputations.
Finally, any free program is threatened constantly by software
patents. We wish to avoid the danger that redistributors of a free
program will individually obtain patent licenses, in effect making the
program proprietary. To prevent this, we have made it clear that any
patent must be licensed for everyone's free use or not licensed at all.
The precise terms and conditions for copying, distribution and
modification follow.
GNU GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License applies to any program or other work which contains
a notice placed by the copyright holder saying it may be distributed
under the terms of this General Public License. The "Program", below,
refers to any such program or work, and a "work based on the Program"
means either the Program or any derivative work under copyright law:
that is to say, a work containing the Program or a portion of it,
either verbatim or with modifications and/or translated into another
language. (Hereinafter, translation is included without limitation in
the term "modification".) Each licensee is addressed as "you".
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of
running the Program is not restricted, and the output from the Program
is covered only if its contents constitute a work based on the
Program (independent of having been made by running the Program).
Whether that is true depends on what the Program does.
1. You may copy and distribute verbatim copies of the Program's
source code as you receive it, in any medium, provided that you
conspicuously and appropriately publish on each copy an appropriate
copyright notice and disclaimer of warranty; keep intact all the
notices that refer to this License and to the absence of any warranty;
and give any other recipients of the Program a copy of this License
along with the Program.
You may charge a fee for the physical act of transferring a copy, and
you may at your option offer warranty protection in exchange for a fee.
2. You may modify your copy or copies of the Program or any portion
of it, thus forming a work based on the Program, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:
a) You must cause the modified files to carry prominent notices
stating that you changed the files and the date of any change.
b) You must cause any work that you distribute or publish, that in
whole or in part contains or is derived from the Program or any
part thereof, to be licensed as a whole at no charge to all third
parties under the terms of this License.
c) If the modified program normally reads commands interactively
when run, you must cause it, when started running for such
interactive use in the most ordinary way, to print or display an
announcement including an appropriate copyright notice and a
notice that there is no warranty (or else, saying that you provide
a warranty) and that users may redistribute the program under
these conditions, and telling the user how to view a copy of this
License. (Exception: if the Program itself is interactive but
does not normally print such an announcement, your work based on
the Program is not required to print an announcement.)
These requirements apply to the modified work as a whole. If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works. But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.
Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Program.
In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.
3. You may copy and distribute the Program (or a work based on it,
under Section 2) in object code or executable form under the terms of
Sections 1 and 2 above provided that you also do one of the following:
a) Accompany it with the complete corresponding machine-readable
source code, which must be distributed under the terms of Sections
1 and 2 above on a medium customarily used for software interchange; or,
b) Accompany it with a written offer, valid for at least three
years, to give any third party, for a charge no more than your
cost of physically performing source distribution, a complete
machine-readable copy of the corresponding source code, to be
distributed under the terms of Sections 1 and 2 above on a medium
customarily used for software interchange; or,
c) Accompany it with the information you received as to the offer
to distribute corresponding source code. (This alternative is
allowed only for noncommercial distribution and only if you
received the program in object code or executable form with such
an offer, in accord with Subsection b above.)
The source code for a work means the preferred form of the work for
making modifications to it. For an executable work, complete source
code means all the source code for all modules it contains, plus any
associated interface definition files, plus the scripts used to
control compilation and installation of the executable. However, as a
special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary
form) with the major components (compiler, kernel, and so on) of the
operating system on which the executable runs, unless that component
itself accompanies the executable.
If distribution of executable or object code is made by offering
access to copy from a designated place, then offering equivalent
access to copy the source code from the same place counts as
distribution of the source code, even though third parties are not
compelled to copy the source along with the object code.
4. You may not copy, modify, sublicense, or distribute the Program
except as expressly provided under this License. Any attempt
otherwise to copy, modify, sublicense or distribute the Program is
void, and will automatically terminate your rights under this License.
However, parties who have received copies, or rights, from you under
this License will not have their licenses terminated so long as such
parties remain in full compliance.
5. You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify or
distribute the Program or its derivative works. These actions are
prohibited by law if you do not accept this License. Therefore, by
modifying or distributing the Program (or any work based on the
Program), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Program or works based on it.
6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the
original licensor to copy, distribute or modify the Program subject to
these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.
7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Program at all. For example, if a patent
license would not permit royalty-free redistribution of the Program by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Program.
If any portion of this section is held invalid or unenforceable under
any particular circumstance, the balance of the section is intended to
apply and the section as a whole is intended to apply in other
circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system, which is
implemented by public license practices. Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.
This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.
8. If the distribution and/or use of the Program is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Program under this License
may add an explicit geographical distribution limitation excluding
those countries, so that distribution is permitted only in or among
countries not thus excluded. In such case, this License incorporates
the limitation as if written in the body of this License.
9. The Free Software Foundation may publish revised and/or new versions
of the General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation. If the Program does not specify a version number of
this License, you may choose any version ever published by the Free Software
Foundation.
10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission. For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this. Our decision will be guided by the two goals
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.
NO WARRANTY
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
Copyright (C)
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
Also add information on how to contact you by electronic and paper mail.
If the program is interactive, make it output a short notice like this
when it starts in an interactive mode:
Gnomovision version 69, Copyright (C) year name of author
Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, the commands you use may
be called something other than `show w' and `show c'; they could even be
mouse-clicks or menu items--whatever suits your program.
You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the program, if
necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the program
`Gnomovision' (which makes passes at compilers) written by James Hacker.
, 1 April 1989
Ty Coon, President of Vice
This General Public License does not permit incorporating your program into
proprietary programs. If your program is a subroutine library, you may
consider it more useful to permit linking proprietary applications with the
library. If this is what you want to do, use the GNU Library General
Public License instead of this License.
' + day + '
' self.counter += 1 def start_time(self, tm): print >>self.file, ''
clock = safe_ftime(self.config["timeformat"], tm)
print >>self.file, '
"
self.counter -= 1
def parse_time(value, default="m"):
"""Parse a time period with optional units (s, m, h, d, w) into a time
in seconds. If no unit is specified, use minutes by default; specify
the default argument to change this. Raises ValueError if the format
isn't recognised."""
units = { "s" : 1, "m" : 60, "h" : 3600, "d" : 86400, "w" : 604800 }
for unit, size in units.items():
if value.endswith(unit):
return int(value[:-len(unit)]) * size
return int(value) * units[default]
def parse_bool(value):
"""Parse a boolean value (0, 1, false or true). Raise ValueError if
the value isn't recognised."""
value = value.strip().lower()
if value == "0" or value == "false":
return False
elif value == "1" or value == "true":
return True
else:
raise ValueError("Bad boolean value: " + value)
def parse_list(value):
"""Parse a list of keywords separated by whitespace."""
return value.strip().split(None)
def parse_feed_args(argparams, arglines):
"""Parse a list of feed arguments. Raise ConfigError if the syntax is
invalid, or ValueError if an argument value can't be parsed."""
args = {}
for p in argparams:
ps = p.split("=", 1)
if len(ps) != 2:
raise ConfigError("Bad feed argument in config: " + p)
args[ps[0]] = ps[1]
for p in arglines:
ps = p.split(None, 1)
if len(ps) != 2:
raise ConfigError("Bad argument line in config: " + p)
args[ps[0]] = ps[1]
for name, value in args.items():
if name == "allowduplicates":
args[name] = parse_bool(value)
elif name == "keepmin":
args[name] = int(value)
elif name == "maxage":
args[name] = parse_time(value)
return args
class ConfigError(Exception): pass
class Config:
"""The aggregator's configuration."""
def __init__(self, locking=True, logfile_name=None):
self.locking = locking
self.files_loaded = []
self.loglock = threading.Lock()
self.logfile = None
if logfile_name:
self.logfile = open(logfile_name, "a")
self.reset()
def reset(self):
# Note that these default values are *not* the same as
# in the supplied config file. The idea is that someone
# who has an old config file shouldn't notice a difference
# in behaviour on upgrade -- so new options generally
# default to False here, and True in the sample file.
self.config = {
"feedslist" : [],
"feeddefaults" : {},
"defines" : {},
"outputfile" : "output.html",
"maxarticles" : 200,
"maxage" : 0,
"expireage" : 24 * 60 * 60,
"keepmin" : 0,
"dayformat" : "%A, %d %B %Y",
"timeformat" : "%I:%M %p",
"datetimeformat" : None,
"userefresh" : False,
"showfeeds" : True,
"timeout" : 30,
"pagetemplate" : "default",
"itemtemplate" : "default",
"feedlisttemplate" : "default",
"feeditemtemplate" : "default",
"verbose" : False,
"ignoretimeouts" : False,
"showtracebacks" : False,
"daysections" : True,
"timesections" : True,
"blocklevelhtml" : True,
"tidyhtml" : False,
"sortbyfeeddate" : False,
"currentonly" : False,
"hideduplicates" : [],
"newfeedperiod" : "3h",
"changeconfig": False,
"numthreads": 1,
"splitstate": False,
"useids": False,
}
def __getitem__(self, key): return self.config[key]
def get(self, key, default=None): return self.config.get(key, default)
def __setitem__(self, key, value): self.config[key] = value
def reload(self):
self.log("Reloading config files")
self.reset()
for filename in self.files_loaded:
self.load(filename, False)
def load(self, filename, explicitly_loaded=True):
"""Load configuration from a config file."""
if explicitly_loaded:
self.files_loaded.append(filename)
lines = []
try:
f = open(filename, "r")
for line in f.xreadlines():
try:
line = line.decode(get_system_encoding())
except UnicodeDecodeError, e:
raise ConfigError("Character encoding problem in config file: " + filename + ": " + str(e))
stripped = line.strip()
if stripped == "" or stripped[0] == "#":
continue
if line[0] in string.whitespace:
if lines == []:
raise ConfigError("First line in config cannot be an argument")
lines[-1][1].append(stripped)
else:
lines.append((stripped, []))
f.close()
except IOError:
raise ConfigError("Can't read config file: " + filename)
for line, arglines in lines:
try:
self.load_line(line, arglines)
except ValueError:
raise ConfigError("Bad value in config: " + line)
def load_line(self, line, arglines):
"""Process a configuration directive."""
l = line.split(None, 1)
if len(l) == 1 and l[0] == "feeddefaults":
l.append("")
elif len(l) != 2:
raise ConfigError("Bad line in config: " + line)
# Load template files immediately, so we produce an error now
# rather than later if anything goes wrong.
if l[0].endswith("template") and l[1] != "default":
load_file(l[1])
handled_arglines = False
if l[0] == "feed":
l = l[1].split(None)
if len(l) < 2:
raise ConfigError("Bad line in config: " + line)
self["feedslist"].append((l[1], parse_time(l[0]), parse_feed_args(l[2:], arglines)))
handled_arglines = True
elif l[0] == "feeddefaults":
self["feeddefaults"] = parse_feed_args(l[1].split(None), arglines)
handled_arglines = True
elif l[0] == "define":
l = l[1].split(None, 1)
if len(l) != 2:
raise ConfigError("Bad line in config: " + line)
self["defines"][l[0]] = l[1]
elif l[0] == "plugindirs":
for dir in parse_list(l[1]):
plugins.load_plugins(dir, self)
elif l[0] == "outputfile":
self["outputfile"] = l[1]
elif l[0] == "maxarticles":
self["maxarticles"] = int(l[1])
elif l[0] == "maxage":
self["maxage"] = parse_time(l[1])
elif l[0] == "expireage":
self["expireage"] = parse_time(l[1])
elif l[0] == "keepmin":
self["keepmin"] = int(l[1])
elif l[0] == "dayformat":
self["dayformat"] = l[1]
elif l[0] == "timeformat":
self["timeformat"] = l[1]
elif l[0] == "datetimeformat":
self["datetimeformat"] = l[1]
elif l[0] == "userefresh":
self["userefresh"] = parse_bool(l[1])
elif l[0] == "showfeeds":
self["showfeeds"] = parse_bool(l[1])
elif l[0] == "timeout":
self["timeout"] = parse_time(l[1], "s")
elif l[0] in ("template", "pagetemplate"):
self["pagetemplate"] = l[1]
elif l[0] == "itemtemplate":
self["itemtemplate"] = l[1]
elif l[0] == "feedlisttemplate":
self["feedlisttemplate"] = l[1]
elif l[0] == "feeditemtemplate":
self["feeditemtemplate"] = l[1]
elif l[0] == "verbose":
self["verbose"] = parse_bool(l[1])
elif l[0] == "ignoretimeouts":
self["ignoretimeouts"] = parse_bool(l[1])
elif l[0] == "showtracebacks":
self["showtracebacks"] = parse_bool(l[1])
elif l[0] == "daysections":
self["daysections"] = parse_bool(l[1])
elif l[0] == "timesections":
self["timesections"] = parse_bool(l[1])
elif l[0] == "blocklevelhtml":
self["blocklevelhtml"] = parse_bool(l[1])
elif l[0] == "tidyhtml":
self["tidyhtml"] = parse_bool(l[1])
elif l[0] == "sortbyfeeddate":
self["sortbyfeeddate"] = parse_bool(l[1])
elif l[0] == "currentonly":
self["currentonly"] = parse_bool(l[1])
elif l[0] == "hideduplicates":
self["hideduplicates"] = parse_list(l[1])
elif l[0] == "newfeedperiod":
self["newfeedperiod"] = l[1]
elif l[0] == "changeconfig":
self["changeconfig"] = parse_bool(l[1])
elif l[0] == "numthreads":
self["numthreads"] = int(l[1])
elif l[0] == "splitstate":
self["splitstate"] = parse_bool(l[1])
elif l[0] == "useids":
self["useids"] = parse_bool(l[1])
elif l[0] == "include":
self.load(l[1], False)
elif plugins.call_hook("config_option_arglines", self, l[0], l[1], arglines):
handled_arglines = True
elif plugins.call_hook("config_option", self, l[0], l[1]):
pass
else:
raise ConfigError("Unknown config command: " + l[0])
if arglines != [] and not handled_arglines:
raise ConfigError("Bad argument lines in config after: " + line)
def log(self, *args):
"""Print a status message. If running in verbose mode, write
the message to stderr; if using a logfile, write it to the
logfile."""
if self["verbose"]:
with self.loglock:
print >>sys.stderr, "".join(map(str, args))
if self.logfile is not None:
with self.loglock:
print >>self.logfile, "".join(map(str, args))
self.logfile.flush()
def bug(self, *args):
"""Report detection of a bug in rawdog."""
print >>sys.stderr, "Internal error detected in rawdog:"
print >>sys.stderr, "".join(map(str, args))
print >>sys.stderr, "This could be caused by a bug in rawdog itself or in a plugin."
print >>sys.stderr, "Please send this error message and your config file to the rawdog author."
def edit_file(filename, editfunc):
"""Edit a file in place: for each line in the input file, call
editfunc(line, outputfile), then rename the output file over the input
file."""
newname = "%s.new-%d" % (filename, os.getpid())
oldfile = open(filename, "r")
newfile = open(newname, "w")
editfunc(oldfile, newfile)
newfile.close()
oldfile.close()
os.rename(newname, filename)
class AddFeedEditor:
def __init__(self, feedline):
self.feedline = feedline
def edit(self, inputfile, outputfile):
d = inputfile.read()
outputfile.write(d)
if not d.endswith("\n"):
outputfile.write("\n")
outputfile.write(self.feedline)
def add_feed(filename, url, rawdog, config):
"""Try to add a feed to the config file."""
feeds = feedscanner.feeds(url)
if feeds == []:
print >>sys.stderr, "Cannot find any feeds in " + url
return
feed = feeds[0]
if feed in rawdog.feeds:
print >>sys.stderr, "Feed " + feed + " is already in the config file"
return
print >>sys.stderr, "Adding feed " + feed
feedline = "feed %s %s\n" % (config["newfeedperiod"], feed)
edit_file(filename, AddFeedEditor(feedline).edit)
class ChangeFeedEditor:
def __init__(self, oldurl, newurl):
self.oldurl = oldurl
self.newurl = newurl
def edit(self, inputfile, outputfile):
for line in inputfile.xreadlines():
ls = line.strip().split(None)
if len(ls) > 2 and ls[0] == "feed" and ls[2] == self.oldurl:
line = line.replace(self.oldurl, self.newurl, 1)
outputfile.write(line)
class RemoveFeedEditor:
def __init__(self, url):
self.url = url
def edit(self, inputfile, outputfile):
while True:
l = inputfile.readline()
if l == "":
break
ls = l.strip().split(None)
if len(ls) > 2 and ls[0] == "feed" and ls[2] == self.url:
while True:
l = inputfile.readline()
if l == "":
break
elif l[0] == "#":
outputfile.write(l)
elif l[0] not in string.whitespace:
outputfile.write(l)
break
else:
outputfile.write(l)
def remove_feed(filename, url, config):
"""Try to remove a feed from the config file."""
if url not in [f[0] for f in config["feedslist"]]:
print >>sys.stderr, "Feed " + url + " is not in the config file"
else:
print >>sys.stderr, "Removing feed " + url
edit_file(filename, RemoveFeedEditor(url).edit)
class FeedFetcher:
"""Class that will handle fetching a set of feeds in parallel."""
def __init__(self, rawdog, feedlist, config):
self.rawdog = rawdog
self.config = config
self.lock = threading.Lock()
self.jobs = set(feedlist)
self.results = {}
def worker(self, num):
rawdog = self.rawdog
config = self.config
while True:
with self.lock:
try:
job = self.jobs.pop()
except KeyError:
# No jobs left.
break
config.log("[", num, "] Fetching feed: ", job)
feed = rawdog.feeds[job]
plugins.call_hook("pre_update_feed", rawdog, config, feed)
result = feed.fetch(rawdog, config)
with self.lock:
self.results[job] = result
def run(self, max_workers):
max_workers = max(max_workers, 1)
num_workers = min(max_workers, len(self.jobs))
self.config.log("Fetching ", len(self.jobs), " feeds using ",
num_workers, " threads")
workers = []
for i in range(1, num_workers):
t = threading.Thread(target=self.worker, args=(i,))
t.start()
workers.append(t)
self.worker(0)
for worker in workers:
worker.join()
self.config.log("Fetch complete")
return self.results
class FeedState(Persistable):
"""The collection of articles in a feed."""
def __init__(self):
Persistable.__init__(self)
self.articles = {}
class Rawdog(Persistable):
"""The aggregator itself."""
def __init__(self):
Persistable.__init__(self)
self.feeds = {}
self.articles = {}
self.plugin_storage = {}
self.state_version = STATE_VERSION
self.using_splitstate = None
def get_plugin_storage(self, plugin):
try:
st = self.plugin_storage.setdefault(plugin, {})
except AttributeError:
# rawdog before 2.5 didn't have plugin storage.
st = {}
self.plugin_storage = {plugin: st}
return st
def check_state_version(self):
"""Check the version of the state file."""
try:
version = self.state_version
except AttributeError:
# rawdog 1.x didn't keep track of this.
version = 1
return version == STATE_VERSION
def change_feed_url(self, oldurl, newurl, config):
"""Change the URL of a feed."""
assert self.feeds.has_key(oldurl)
if self.feeds.has_key(newurl):
print >>sys.stderr, "Error: New feed URL is already subscribed; please remove the old one"
print >>sys.stderr, "from the config file by hand."
return
edit_file("config", ChangeFeedEditor(oldurl, newurl).edit)
feed = self.feeds[oldurl]
# Changing the URL will change the state filename as well,
# so we need to save the old name to load from.
old_state = feed.get_state_filename()
feed.url = newurl
del self.feeds[oldurl]
self.feeds[newurl] = feed
if config["splitstate"]:
feedstate_p = persister.get(FeedState, old_state)
feedstate_p.rename(feed.get_state_filename())
with feedstate_p as feedstate:
for article in feedstate.articles.values():
article.feed = newurl
feedstate.modified()
else:
for article in self.articles.values():
if article.feed == oldurl:
article.feed = newurl
print >>sys.stderr, "Feed URL automatically changed."
def list(self, config):
"""List the configured feeds."""
for url, feed in self.feeds.items():
feed_info = feed.feed_info
print url
print " ID:", feed.get_id(config)
print " Hash:", short_hash(url)
print " Title:", feed.get_html_name(config)
print " Link:", feed_info.get("link")
def sync_from_config(self, config):
"""Update rawdog's internal state to match the
configuration."""
# Make sure the splitstate directory exists.
if config["splitstate"]:
try:
os.mkdir("feeds")
except OSError:
# Most likely it already exists.
pass
# Convert to or from splitstate if necessary.
try:
u = self.using_splitstate
except AttributeError:
# We were last run with a version of rawdog that didn't
# have this variable -- so we must have a single state
# file.
u = False
if u is None:
self.using_splitstate = config["splitstate"]
elif u != config["splitstate"]:
if config["splitstate"]:
config.log("Converting to split state files")
for feed_hash, feed in self.feeds.items():
with persister.get(FeedState, feed.get_state_filename()) as feedstate:
feedstate.articles = {}
for article_hash, article in self.articles.items():
if article.feed == feed_hash:
feedstate.articles[article_hash] = article
feedstate.modified()
self.articles = {}
else:
config.log("Converting to single state file")
self.articles = {}
for feed_hash, feed in self.feeds.items():
with persister.get(FeedState, feed.get_state_filename()) as feedstate:
for article_hash, article in feedstate.articles.items():
self.articles[article_hash] = article
feedstate.articles = {}
feedstate.modified()
persister.delete(feed.get_state_filename())
self.modified()
self.using_splitstate = config["splitstate"]
seen_feeds = set()
for (url, period, args) in config["feedslist"]:
seen_feeds.add(url)
if not self.feeds.has_key(url):
config.log("Adding new feed: ", url)
self.feeds[url] = Feed(url)
self.modified()
feed = self.feeds[url]
if feed.period != period:
config.log("Changed feed period: ", url)
feed.period = period
self.modified()
newargs = {}
newargs.update(config["feeddefaults"])
newargs.update(args)
if feed.args != newargs:
config.log("Changed feed options: ", url)
feed.args = newargs
self.modified()
for url in self.feeds.keys():
if url not in seen_feeds:
config.log("Removing feed: ", url)
if config["splitstate"]:
persister.delete(self.feeds[url].get_state_filename())
else:
for key, article in self.articles.items():
if article.feed == url:
del self.articles[key]
del self.feeds[url]
self.modified()
def update(self, config, feedurl=None):
"""Perform the update action: check feeds for new articles, and
expire old ones."""
config.log("Starting update")
now = time.time()
socket.setdefaulttimeout(config["timeout"])
if feedurl is None:
update_feeds = [url for url in self.feeds.keys()
if self.feeds[url].needs_update(now)]
elif self.feeds.has_key(feedurl):
update_feeds = [feedurl]
self.feeds[feedurl].etag = None
self.feeds[feedurl].modified = None
else:
print "No such feed: " + feedurl
update_feeds = []
numfeeds = len(update_feeds)
config.log("Will update ", numfeeds, " feeds")
fetcher = FeedFetcher(self, update_feeds, config)
fetched = fetcher.run(config["numthreads"])
seen_some_items = set()
def do_expiry(articles):
"""Expire articles from a list. Return True if any
articles were expired."""
feedcounts = {}
for key, article in articles.items():
url = article.feed
feedcounts[url] = feedcounts.get(url, 0) + 1
expiry_list = []
feedcounts = {}
for key, article in articles.items():
url = article.feed
feedcounts[url] = feedcounts.get(url, 0) + 1
expiry_list.append((article.added, article.sequence, key, article))
expiry_list.sort()
count = 0
for date, seq, key, article in expiry_list:
url = article.feed
if url not in self.feeds:
config.log("Expired article for nonexistent feed: ", url)
count += 1
del articles[key]
continue
if (url in seen_some_items
and self.feeds.has_key(url)
and article.can_expire(now, config)
and feedcounts[url] > self.feeds[url].get_keepmin(config)):
plugins.call_hook("article_expired", self, config, article, now)
count += 1
feedcounts[url] -= 1
del articles[key]
config.log("Expired ", count, " articles, leaving ", len(articles))
return (count > 0)
count = 0
for url in update_feeds:
count += 1
config.log("Updating feed ", count, " of " , numfeeds, ": ", url)
feed = self.feeds[url]
if config["splitstate"]:
feedstate_p = persister.get(FeedState, feed.get_state_filename())
feedstate = feedstate_p.open()
articles = feedstate.articles
else:
articles = self.articles
content = fetched[url]
plugins.call_hook("mid_update_feed", self, config, feed, content)
rc = feed.update(self, now, config, articles, content)
url = feed.url
plugins.call_hook("post_update_feed", self, config, feed, rc)
if rc:
seen_some_items.add(url)
if config["splitstate"]:
feedstate.modified()
if config["splitstate"]:
if do_expiry(articles):
feedstate.modified()
feedstate_p.close()
if config["splitstate"]:
self.articles = {}
else:
do_expiry(self.articles)
self.modified()
config.log("Finished update")
def get_template(self, config, name="page"):
"""Return the contents of a template."""
filename = config.get(name + "template", "default")
if filename != "default":
return load_file(filename)
if name == "page":
template = """
"""
if config["userefresh"]:
template += """__refresh__
"""
template += """
' + clock + '
' self.counter += 1 def time(self, s): tm = time.localtime(s) if tm[:3] != self.lasttime[:3] and self.config["daysections"]: self.close(0) self.start_day(tm) if tm[:6] != self.lasttime[:6] and self.config["timesections"]: if self.config["daysections"]: self.close(1) else: self.close(0) self.start_time(tm) self.lasttime = tm def close(self, n=0): while self.counter > n: print >>self.file, "rawdog
__items__
"""
if config["showfeeds"]:
template += """Feeds
__feeds__
"""
template += """
"""
return template
elif name == "item":
return """__title__ [__feed_title__]
__if_description__
__description__
__endif__
| Feed | RSS | Last fetched | Next fetched after |
|---|