%(msg)s" severity: error ancient-metadata: text: The AppStream metadata should be updated to follow a more recent version of the specification.
Please consult the XML quickstart guide for more information. severity: info svgz-decompress-error: text: "Unable to decompress SVGZ icon '%(icon_fname)s'. Error: %(error)s" severity: error icon-format-unsupported: text: "Icon file '%(icon_fname)s' uses an unsupported image file format." severity: error icon-not-found: text: > The icon '%(icon_fname)s' was not found in the archive. This issue can have multiple reasons:
- The icon is not present in the archive.
- The icon is in a wrong directory.
- The icon is not available in a suitable size (at least 64x64px)
/usr/share/icons/hicolor/<size>/apps and ensure the Icon= value
of the .desktop file is set correctly.
severity: error
icon-open-failed:
text: >
Unable to open icon file '%(icon_fname)s'. Error: %(error)sThis means the generator could not render the icon to its appropriate size, and the icon has therefore been ignored. severity: error deb-filelist-error: text: Could not determine file list for '%(pkg_fname)s'. This could be an error in the archive, dpkg, apt_pkg or the DEP-11 generator.
If you think this error is in the generator, please file a bug. severity: error deb-extract-error: text: "Could not extract file '%(fname)s' from package '%(pkg_fname)s'. Error: %(error)s" severity: error deb-empty-file: text: "File '%(fname)s' from package '%(pkg_fname)s' appeared to be empty." severity: error metainfo-no-id: text: Could not determine an id for this component. The AppStream upstream metadata likely lacks an
<id/> tag.The identifier tag is absolutely essential for AppStream metadata, and must never be missing. severity: error missing-desktop-file: text: Found an AppStream upstream XML file, but the associated .desktop file is missing. This often happens when the .desktop file is renamed, but the
<id/> tag of the AppStream metainfo file is not adepted as well, or if the metainfo file is located in a different package than the .desktop file.Please fix the packaging or work with upstream to resolve this issue. severity: error gui-app-without-icon: text: The GUI application (application which has a .desktop file for the XDG menu and
Type=Application) with AppStream-ID '%(cid)s' has been found,
but we could not find a matching icon.
severity: error
screenshot-download-error:
text: >
Error while downloading screenshot from '%(url)s' for component '%(cpt_id)s': %(error)sThis might be a temporary server issue. severity: warning screenshot-read-error: text: > Error while reading screenshot data for '%(url)s' of component '%(cpt_id)s': %(error)s
Maybe the server returned an invalid image (e.g. an error page), or the image data is broken. severity: warning metainfo-no-type: text: Component has no type defined. A component type (desktop, input-method, generic, ...) is essential. This issue is likely a mistyped component typename in the metainfo file, or an internal bug in the data extractor (componenty without
Type= count as generic components automatically,
so unless there is a typo somewhere, no component should ever have no type).
severity: error
metainfo-no-name:
text: Component has no name specified. Ensure that the AppStream upstream metadata or the .desktop file (if there is any) specify a component name.
severity: error
metainfo-no-package:
text: Component has no package defined. A package must be associated with a component. This is likely a bug in the generator.
severity: error
metainfo-no-summary:
text: >
Component does not contain a short summary. Ensure that the components metainfo file has a summary tag, or that its .desktop file
has a Comment= field set. Especially applications by KDE often use the GenericName field for a short summary,
instead of the Comment field (which is not its intended use case).More information can be found in the Desktop Entry specification. severity: error metainfo-localized-field-without-template: text: The field '%(field_id)s' has no untranslated template entry ("C"). This is usually an error in the accompanying .desktop file, or the translation template in the AppStream XML is missing. severity: error metainfo-duplicate-id: text: The component-id '%(cid)s' already appeared in package '%(pkgname)s'. AppStream-IDs must be unique, please resolve which package will be providing this component by default.
This issue may happen temporarily when metadata is moved from one package to another. In that case, ignore this issue, it will vanish soon. severity: error appstream-dep11-0.4.0/data/templates/ 0000775 0000000 0000000 00000000000 12617473137 0017341 5 ustar 00root root 0000000 0000000 appstream-dep11-0.4.0/data/templates/default/ 0000775 0000000 0000000 00000000000 12617473137 0020765 5 ustar 00root root 0000000 0000000 appstream-dep11-0.4.0/data/templates/default/base.html 0000664 0000000 0000000 00000002036 12617473137 0022566 0 ustar 00root root 0000000 0000000 {% block head %}
{% block content %}{% endblock %}
appstream-dep11-0.4.0/data/templates/default/issues_index.html 0000664 0000000 0000000 00000002412 12617473137 0024354 0 ustar 00root root 0000000 0000000 {% extends "base.html" %} {% block title %}Hints summary for {{suite}}/{{section}}{% endblock %} {% block header_content %} ⇦ | Hints summary for {{suite}}/{{section}} {% endblock %} {% block content %}
Metadata processing hints found for {{suite}}/{{section}}
{% for maintainer, summary in package_summaries.items() %}{{maintainer|e}}
-
{% for pkgname, hints in summary.items() %}
- {{pkgname}} {% if hints.info_count %} Infos: {{hints.info_count}} {% endif %} {% if hints.warning_count %} Warnings: {{hints.warning_count}} {% endif %} {% if hints.error_count %} Errors: {{hints.error_count}} {% endif %} {% endfor %}
Hints for {{package_name}} in {{suite}}
{% for entry in entries %}
{% endif %}
{% if entry.warnings|length %}
{% endif %}
{% if entry.infos|length %}
{% endif %}
{% endfor %}
{% endblock %}
{% block float_right %}
Last updated on: {{time}}
{% endblock %}
appstream-dep11-0.4.0/data/templates/default/metainfo_index.html 0000664 0000000 0000000 00000001743 12617473137 0024651 0 ustar 00root root 0000000 0000000 {% extends "base.html" %}
{% block title %}DEP-11 components for {{suite}}/{{section}}{% endblock %}
{% block header_content %}
⇦ |
DEP-11 components summary for {{suite}}/{{section}}
{% endblock %}
{% block content %}
{{entry.identifier}} {% for arch in entry.archs %} ⚙ {{arch}} {% endfor %}
{% if entry.errors|length %}Errors
-
{% for tag in entry.errors %}
-
{{tag.tag_name}}
{{tag.description}}
{% endfor %}
Warnings
-
{% for tag in entry.warnings %}
-
{{tag.tag_name}}
{{tag.description}}
{% endfor %}
Hints
-
{% for tag in entry.infos %}
-
{{tag.tag_name}}
{{tag.description}}
{% endfor %}
Metadata found for {{suite}}/{{section}}
{% for maintainer, summary in package_summaries.items() %}{{maintainer|e}}
-
{% for pkgname, info in summary.items() %}
-
{{pkgname}}
-
{% for cid in info.cids %}
- {{cid}} {% endfor %}
{% endfor %}
DEP-11 metadata for {{package_name}} in {{suite}}
{% for cpt in cpts %}
{% endblock %}
{% block float_right %}
Last updated on: {{time}}
{% endblock %}
appstream-dep11-0.4.0/data/templates/default/section_overview.html 0000664 0000000 0000000 00000003377 12617473137 0025257 0 ustar 00root root 0000000 0000000 {% extends "base.html" %}
{% block title %}DEP-11 data for {{distro}}: {{suite}} suite{% endblock %}
{% block header_content %}
⇦ |
DEP-11 data for {{distro}}/{{suite}}
{% endblock %}
{% block content %}
{{cpt.cid}} {% for arch in cpt.archs %} ⚙ {{arch}} {% endfor %}
{{cpt.mdata}}
{% endfor %}
Overview for {{section}}
Data
Issues - Issues found while extracting the data
Metainfo - Per-package view of the generated data
Health
Issue overview
- {{metainfo_count}} valid components
- {{error_count}} errors
- {{warning_count}} warnings
- {{info_count}} infos/hints
Global data validation result
{{validate_result}}
Select an archive section
Sections
{% for section in sections %}{{section}}
{% endfor %}Health of suite "{{suite}}"
- {{metainfo_count}} valid components
- {{error_count}} errors
- {{warning_count}} warnings
- {{info_count}} infos/hints
X7%9]`Җ'l$Qn`2M+ǣU 4sLTV(cnCi%+" Jma? |S>$WI2KOXk 8( }" "$E EȩPǿ OYB- 3W '@Jc1NğYCjUi,h jO]I$^A~C<Aq4ha_ #G|< QOU)#r {X:a-|m+!qU8i ;[-x-xl-U)b)hfY͢_)"!dƒ} IENDB` appstream-dep11-0.4.0/data/templates/default/suites_index.html 0000664 0000000 0000000 00000004607 12617473137 0024365 0 ustar 00root root 0000000 0000000 {% extends "base.html" %} {% block title %}DEP-11 data for {{distro}}: Suites overview{% endblock %} {% block header_content %} DEP-11 data hints for {{distro}} {% endblock %} {% block content %}
Welcome!
Welcome to the DEP-11 generator HTML pages!
These pages exist to provide a user-friendly view on the issues discovered by the DEP-11 generator while extracting metadata from packages in the {{distro}} archive. They can also be used to take a look at the raw metadata, to spot possible problems with the data itself or the generation process.
Select a suite
{% for suite in suites %}{{suite}}
{% endfor %}What is DEP-11 and AppStream?
AppStream is a cross-distro XML format to provide metadata for software components and to assign unique identifiers to software.
In Debian, we parse all XML provided by upstream projects as well as other metadata (.desktop-files, ...), and compile a single YAML
metadata file from it, which is then shipped to users via APT.
While the official AppStream specification is based on XML, Debian uses a YAML version of the format for easier use in existing scripts and for better archive integration. This format is called DEP-11, and initially had a much wider scope in enhancing archive metadata than AppStream had. Today AppStream covers that as well.
The generated metadata can for example be used by software centers like GNOME-Software or KDE Discover to display a user-friendly application-centric way on the package archive.
More information
See AppStream @ wiki.d.o for information about AppStream integration and usage in Debian.
The offical AppStream specification can be found at freedesktop.org, a description
of the DEP-11 YAML format is hosted there as well.
You can find the source-code of the DEP-11 generator here.
/usr/share/icons/hicolor/)."})
return False
cpt.icon = icon_name
if icon_name_orig.endswith(".svg"):
svgicon = True
elif icon_name_orig.endswith(".svgz"):
svgicon = True
try:
icon_data = zlib.decompress(bytes(icon_data), 15+32)
except Exception as e:
cpt.add_hint("svgz-decompress-error", {'icon_fname': icon_name, 'error': str(e)})
return False
if not os.path.exists(path):
os.makedirs(path)
if svgicon:
# render the SVG to a bitmap
self._render_svg_to_png(icon_data, icon_store_location, int(size), int(size))
return True
else:
# we don't trust upstream to have the right icon size present, and therefore
# always adjust the icon to the right size
stream = BytesIO(icon_data)
stream.seek(0)
img = None
try:
img = Image.open(stream)
except Exception as e:
cpt.add_hint("icon-open-failed", {'icon_fname': icon_name, 'error': str(e)})
return False
newimg = img.resize((int(size), int(size)), Image.ANTIALIAS)
newimg.save(icon_store_location)
return True
return False
def _match_icon_on_filelist(self, cpt, filelist, icon_name, size):
if size == "scalable":
size_str = "scalable"
else:
size_str = str(size)
icon_path = "usr/share/icons/hicolor/%s/apps/%s" % (size_str, icon_name)
filtered = fnmatch.filter(filelist, icon_path)
if not filtered:
return None
return filtered[0]
def _match_and_store_icon(self, pkg_fname, cpt, cpt_export_path, filelist, icon_name, size):
success = False
matched_icon = self._match_icon_on_filelist(cpt, filelist, icon_name, size)
if not matched_icon:
return False
if not size in self._icon_sizes:
# scale icons to allowed sizes
for asize in self._icon_sizes:
success = self._store_icon(pkg_fname, cpt, cpt_export_path, matched_icon, asize) or success
else:
success = self._store_icon(pkg_fname, cpt, cpt_export_path, matched_icon, size)
return success
def _fetch_icon(self, cpt, cpt_export_path, pkg_fname, filelist):
'''
Searches for icon if absolute path to an icon
is not given. Component with invalid icons are ignored
'''
if not cpt.icon:
# if we don't know an icon-name or path, just return without error
return True
icon_str = cpt.icon
cpt.icon = None
all_icon_sizes = self._icon_sizes[:]
all_icon_sizes.extend(self._large_icon_sizes)
success = False
if icon_str.startswith("/"):
if icon_str[1:] in filelist:
return self._store_icon(pkg_fname, cpt, cpt_export_path, icon_str[1:], IconSize(64))
else:
ret = False
icon_str = os.path.basename (icon_str)
# check if there is some kind of file-extension.
# if there is none, the referenced icon is likely a stock icon, and we assume .png
if "." in icon_str:
icon_name_ext = icon_str
else:
icon_name_ext = icon_str + ".png"
found_sizes = list()
for size in self._icon_sizes:
ret = self._match_and_store_icon(pkg_fname, cpt, cpt_export_path, filelist, icon_name_ext, size)
if ret:
found_sizes.append(size)
success = ret or success
# try if we can add missing icon sizes by scaling down things
# this also ensures that we also have an 64x64 sized icon
if set(found_sizes) != set(self._icon_sizes):
for size in self._icon_sizes:
if size in found_sizes:
continue
for asize in all_icon_sizes:
if asize < size:
continue
icon_fname = self._match_icon_on_filelist(cpt, filelist, icon_name_ext, asize)
if not icon_fname:
continue
ret = self._store_icon(pkg_fname, cpt, cpt_export_path, icon_fname, size)
if ret:
found_sizes.append(size)
success = ret or success
break
# a 64x64 icon is required, so double-check if we have one
if success and not IconSize(64) in found_sizes:
success = False
if not success:
# we cheat and test for larger icons as well, which can be scaled down
# first check for a scalable graphic
success = self._match_and_store_icon(pkg_fname, cpt, cpt_export_path, filelist, icon_str + ".svg", "scalable")
if not success:
success = self._match_and_store_icon(pkg_fname, cpt, cpt_export_path, filelist, icon_str + ".svgz", "scalable")
# then try to scale down larger graphics
if not success:
for size in self._large_icon_sizes:
success = self._match_and_store_icon(pkg_fname, cpt, cpt_export_path, filelist, icon_name_ext, size) or success
if not success:
last_pixmap = None
# handle stuff in the pixmaps directory
for path in filelist:
if path.startswith("usr/share/pixmaps"):
file_basename = os.path.basename(path)
if ((file_basename == icon_str) or (os.path.splitext(file_basename)[0] == icon_str)):
# the pixmap dir can contain icons in multiple formats, and store_icon() fails in case
# the icon format is not allowed. We therefore only exit here, if the icon has a valid format
if self._icon_allowed(path):
return self._store_icon(pkg_fname, cpt, cpt_export_path, path, IconSize(64))
last_pixmap = path
if last_pixmap:
# we don't do a global icon search anymore, since we've found an (unsuitable) icon
# already
cpt.add_hint("icon-format-unsupported", {'icon_fname': os.path.basename(last_pixmap)})
return False
icon_dict = self._icon_finder.find_icons(cpt.pkgname, icon_str, all_icon_sizes)
success = False
if icon_dict:
for size in self._icon_sizes:
if not size in icon_dict:
continue
success = self._store_icon(icon_dict[size]['deb_fname'],
cpt,
cpt_export_path,
icon_dict[size]['icon_fname'],
size) or success
if not success:
for size in self._large_icon_sizes:
if not size in icon_dict:
continue
for asize in self._icon_sizes:
success = self._store_icon(icon_dict[size]['deb_fname'],
cpt,
cpt_export_path,
icon_dict[size]['icon_fname'],
asize) or success
return success
if ("." in icon_str) and (not self._icon_allowed(icon_str)):
cpt.add_hint("icon-format-unsupported", {'icon_fname': icon_str})
else:
cpt.add_hint("icon-not-found", {'icon_fname': icon_str})
return False
return success
def process(self, pkgname, pkg_fname, pkgid=None, metainfo_files=None):
'''
Reads the metadata from the xml file and the desktop files.
And returns a list of DEP11Component objects.
'''
deb = None
try:
deb = DebFile(pkg_fname)
except Exception as e:
log.error("Error reading deb file '%s': %s" % (pkg_fname, e))
if not deb:
return list()
try:
filelist = self._get_deb_filelist(deb)
except:
log.error("List of files for '%s' could not be read" % (pkg_fname))
filelist = None
if not filelist:
cpt = DEP11Component(self._suite_name, self._archive_component, pkgname, pkgid)
cpt.add_hint("deb-filelist-error", {'pkg_fname': os.path.basename(pkg_fname)})
return [cpt]
if not pkgid:
# we didn't get an identifier, so start guessing one.
idname, ext = os.path.splitext(os.path.basename(pkg_fname))
if not idname:
idname = os.path.basename(pkg_fname)
pkgid = idname
export_path = "%s/%s" % (self._export_dir, self._archive_component)
component_dict = dict()
# if we don't have an explicit list of interesting files, we simply scan all
if not metainfo_files:
metainfo_files = filelist
# first cache all additional metadata (.desktop/.pc/etc.) files
mdata_raw = dict()
for meta_file in metainfo_files:
if meta_file.endswith(".desktop") and meta_file.startswith("usr/share/applications"):
# We have a .desktop file
dcontent = None
cpt_id = os.path.basename(meta_file)
error = None
try:
dcontent = str(deb.data.extractdata(meta_file), 'utf-8')
except Exception as e:
error = {'tag': "deb-extract-error",
'params': {'fname': cpt_id, 'pkg_fname': os.path.basename(pkg_fname), 'error': str(e)}}
if not dcontent and not error:
error = {'tag': "deb-empty-file",
'params': {'fname': cpt_id, 'pkg_fname': os.path.basename(pkg_fname)}}
mdata_raw[cpt_id] = {'error': error, 'data': dcontent}
# process all AppStream XML files
for meta_file in metainfo_files:
if meta_file.endswith(".xml") and meta_file.startswith("usr/share/appdata"):
xml_content = None
cpt = DEP11Component(self._suite_name, self._archive_component, pkgname, pkgid)
try:
xml_content = str(deb.data.extractdata(meta_file), 'utf-8')
except Exception as e:
# inability to read an AppStream XML file is a valid reason to skip the whole package
cpt.add_hint("deb-extract-error", {'fname': meta_file, 'pkg_fname': os.path.basename(pkg_fname), 'error': str(e)})
return [cpt]
if not xml_content:
continue
read_appstream_upstream_xml(cpt, xml_content)
component_dict[cpt.cid] = cpt
# Reads the desktop files associated with the xml file
if not cpt.cid:
# if there is no ID at all, we dump this component, since we cannot do anything with it at all
cpt.add_hint("metainfo-no-id")
continue
cpt.set_srcdata_checksum_from_data(xml_content)
if cpt.kind == "desktop-app":
data = mdata_raw.get(cpt.cid)
if not data:
cpt.add_hint("missing-desktop-file")
continue
if data['error']:
# add a non-fatal hint that we couldn't process the .desktop file
cpt.add_hint(data['error']['tag'], data['error']['params'])
else:
# we have a .desktop component, extend it with the associated .desktop data
read_desktop_data(cpt, data['data'])
cpt.set_srcdata_checksum_from_data(xml_content+data['data'])
del mdata_raw[cpt.cid]
# now process the remaining metadata files, which have not been processed together with the XML
for mid, mdata in mdata_raw.items():
if mid.endswith(".desktop"):
# We have a .desktop file
cpt = DEP11Component(self._suite_name, self._archive_component, pkgname, pkgid)
cpt.cid = mid
if mdata['error']:
# add a fatal hint that we couldn't process this file
cpt.add_hint(mdata['error']['tag'], mdata['error']['params'])
component_dict[cpt.cid] = cpt
else:
ret = read_desktop_data(cpt, mdata['data'])
if ret or not cpt.has_ignore_reason():
component_dict[cpt.cid] = cpt
cpt.set_srcdata_checksum_from_data(mdata['data'])
else:
# this means that reading the .desktop file failed and we should
# silently ignore this issue (since the file was marked to be invisible on purpose)
pass
# fetch media (icons/screenshots), if we don't ignore the component already
cpts = component_dict.values()
for cpt in cpts:
if cpt.has_ignore_reason():
continue
if not cpt.global_id:
log.error("Component '%s' from package '%s' has no source-data checksum / global-id." % (cpt.cid, pkg_fname))
continue
# check if we have a component generated from
# this source data in the cache already.
# To account for packages which change their package name, we
# also need to check if the package this component is associated
# with matches ours.
existing_mdata = self._dcache.get_metadata(cpt.global_id)
if existing_mdata:
s = "Package: %s\n" % (pkgname)
if s in existing_mdata:
continue
else:
# the exact same metadata exists in a different package already, raise ab error.
# ATTENTION: This does not cover the case where *different* metadata (as in, different summary etc.)
# but with the *same ID* exists. This kind of issue can only be catched when listing all IDs per
# suite/acomponent combination and checking for dupes (we do that in the DEP-11 validator and display
# the result prominently on the HTML pages)
ecpt = yaml.safe_load(existing_mdata)
cpt.add_hint("metainfo-duplicate-id", {'cid': cpt.cid, 'pkgname': ecpt.get('Package', '')})
continue
self._fetch_icon(cpt, export_path, pkg_fname, filelist)
if cpt.kind == 'desktop-app' and not cpt.icon:
cpt.add_hint("gui-app-without-icon", {'cid': cpt.cid})
else:
self._fetch_screenshots(cpt, export_path)
# write data to cache
if self.write_to_cache:
# write the components we found to the cache
self._dcache.set_components(pkgid, cpts)
return cpts
appstream-dep11-0.4.0/dep11/hints.py 0000664 0000000 0000000 00000006167 12617473137 0017055 0 ustar 00root root 0000000 0000000 #!/usr/bin/env python
#
# Copyright (c) 2015 Matthias Klumpp %s
" % str_enc_dec(prepare_desc_string(usubs.text)) elif usubs.tag == 'ul' or usubs.tag == 'ol': tmp_dict = dict() # find the right locale, or fallback to untranslated for u_usubs in usubs: locale = _get_tag_locale(u_usubs) if not locale in tmp_dict: tmp_dict[locale] = "" if u_usubs.tag == 'li': tmp_dict[locale] += "\n") return pygments.highlight(yml_data, YamlLexer(), HtmlFormatter()) def _expand_hint(self, hint_data): tag_name = hint_data['tag'] tag = get_hint_tag_info(tag_name) desc = "" try: desc = tag['text'] % hint_data['params'] except Exception as e: desc = "Error while expanding hint description: %s" % (str(e)) severity = tag.get('severity') if not severity: log.error("Tag %s has no severity!", tag_name) severity = "info" return {'tag_name': tag_name, 'description': desc, 'severity': severity} def update_html(self): dep11_hintsdir = os.path.join(self._export_dir, "hints") if not os.path.exists(dep11_hintsdir): return dep11_minfodir = os.path.join(self._export_dir, "data") if not os.path.exists(dep11_minfodir): return export_dir = self._html_export_dir media_dir = os.path.join(self._export_dir, "media") noimage_url = os.path.join(self._html_url, "static", "img", "no-image.png") # Render archive suites index page self.render_template("suites_index.html", export_dir, "index.html", suites=self._suites_data.keys()) # TODO: Remove old HTML files for suite_name in self._suites_data: suite = self._suites_data[suite_name] export_dir = os.path.join(self._export_dir, "html", suite_name) suite_error_count = 0 suite_warning_count = 0 suite_info_count = 0 suite_metainfo_count = 0 for component in suite['components']: issue_summaries = dict() mdata_summaries = dict() export_dir_section = os.path.join(self._export_dir, "html", suite_name, component) export_dir_issues = os.path.join(export_dir_section, "issues") export_dir_metainfo = os.path.join(export_dir_section, "metainfo") error_count = 0 warning_count = 0 info_count = 0 metainfo_count = 0 hint_pages = dict() cpt_pages = dict() for arch in suite['architectures']: h_fname = os.path.join(dep11_hintsdir, suite_name, component, "DEP11Hints_%s.yml.gz" % (arch)) hints_data = None if os.path.isfile(h_fname): f = gzip.open(h_fname, 'r') hints_data = yaml.safe_load_all(f.read()) f.close() d_fname = os.path.join(dep11_minfodir, suite_name, component, "Components-%s.yml.gz" % (arch)) dep11_data = None if os.path.isfile(d_fname): f = gzip.open(d_fname, 'r') dep11_data = yaml.safe_load_all(f.read()) f.close() pkg_index = read_packages_dict_from_file(self._archive_root, suite_name, component, arch) if hints_data: for hdata in hints_data: pkg_name = hdata['Package'] pkg_id = hdata.get('PackageID') if not pkg_id: pkg_id = pkg_name pkg = pkg_index.get(pkg_name) maintainer = None if pkg: maintainer = pkg['maintainer'] if not maintainer: maintainer = "Unknown" if not issue_summaries.get(maintainer): issue_summaries[maintainer] = dict() hints_raw = hdata.get('Hints', list()) # expand all hints to show long descriptions errors = list() warnings = list() infos = list() for hint in hints_raw: ehint = self._expand_hint(hint) severity = ehint['severity'] if severity == "info": infos.append(ehint) elif severity == "warning": warnings.append(ehint) else: errors.append(ehint) if not hint_pages.get(pkg_name): hint_pages[pkg_name] = list() # we fold multiple architectures with the same issues into one view pkid_noarch = pkg_id if "/" in pkg_id: pkid_noarch = pkg_id[:pkg_id.rfind("/")] pcid = "" if hdata.get('ID'): pcid = "%s: %s" % (pkid_noarch, hdata.get('ID')) else: pcid = pkid_noarch page_data = {'identifier': pcid, 'errors': errors, 'warnings': warnings, 'infos': infos, 'archs': [arch]} try: l = hint_pages[pkg_name] index = next(i for i, v in enumerate(l) if equal_dicts(v, page_data, ['archs'])) hint_pages[pkg_name][index]['archs'].append(arch) except StopIteration: hint_pages[pkg_name].append(page_data) # add info to global issue count error_count += len(errors) warning_count += len(warnings) info_count += len(infos) # add info for global index if not issue_summaries[maintainer].get(pkg_name): issue_summaries[maintainer][pkg_name] = {'error_count': len(errors), 'warning_count': len(warnings), 'info_count': len(infos)} if dep11_data: for mdata in dep11_data: pkg_name = mdata.get('Package') if not pkg_name: # we probably hit the header continue pkg = pkg_index.get(pkg_name) maintainer = None if pkg: maintainer = pkg['maintainer'] if not maintainer: maintainer = "Unknown" if not mdata_summaries.get(maintainer): mdata_summaries[maintainer] = dict() # ugly hack to have the screenshot entries linked #if mdata.get('Screenshots'): # sshot_baseurl = os.path.join(self._dep11_url, component) # for i in range(len(mdata['Screenshots'])): # url = mdata['Screenshots'][i]['source-image']['url'] # url = "%s" % (os.path.join(sshot_baseurl, url), url) # mdata['Screenshots'][i]['source-image']['url'] = Markup(url) # thumbnails = mdata['Screenshots'][i]['thumbnails'] # for j in range(len(thumbnails)): # url = thumbnails[j]['url'] # url = "%s" % (os.path.join(sshot_baseurl, url), url) # thumbnails[j]['url'] = Markup(url) # mdata['Screenshots'][i]['thumbnails'] = thumbnails mdata_yml = dict_to_dep11_yaml(mdata) mdata_yml = self._highlight_yaml(mdata_yml) cid = mdata.get('ID') # try to find an icon for this component (if it's a GUI app) icon_url = None if mdata['Type'] == 'desktop-app' or mdata['Type'] == "web-app": icon_name = mdata['Icon'].get("cached") cptgid = build_cpt_global_id(cid, mdata.get('X-Source-Checksum')) if icon_name and cptgid: icon_fname = os.path.join(component, cptgid, "icons", "64x64", icon_name) if os.path.isfile(os.path.join(media_dir, icon_fname)): icon_url = os.path.join(self._dep11_url, icon_fname) else: icon_url = noimage_url else: icon_url = noimage_url else: icon_url = os.path.join(self._html_url, "static", "img", "cpt-nogui.png") if not cpt_pages.get(pkg_name): cpt_pages[pkg_name] = list() page_data = {'cid': cid, 'mdata': mdata_yml, 'icon_url': icon_url, 'archs': [arch]} try: l = cpt_pages[pkg_name] index = next(i for i, v in enumerate(l) if equal_dicts(v, page_data, ['archs'])) cpt_pages[pkg_name][index]['archs'].append(arch) except StopIteration: cpt_pages[pkg_name].append(page_data) # increase valid metainfo count metainfo_count += 1 # check if we had this package, and add to summary pksum = mdata_summaries[maintainer].get(pkg_name) if not pksum: pksum = dict() if pksum.get('cids'): if not cid in pksum['cids']: pksum['cids'].append(cid) else: pksum['cids'] = [cid] mdata_summaries[maintainer][pkg_name] = pksum if not dep11_data and not hints_data: log.warning("Suite %s/%s/%s does not contain DEP-11 data or issue hints.", suite_name, component, arch) # now write the HTML pages with the previously collected & transformed issue data for pkg_name, entry_list in hint_pages.items(): # render issues page self.render_template("issues_page.html", export_dir_issues, "%s.html" % (pkg_name), package_name=pkg_name, entries=entry_list, suite=suite_name, section=component) # render page with all components found in a package for pkg_name, cptlist in cpt_pages.items(): # render metainfo page self.render_template("metainfo_page.html", export_dir_metainfo, "%s.html" % (pkg_name), package_name=pkg_name, cpts=cptlist, suite=suite_name, section=component) # Now render our issue index page self.render_template("issues_index.html", export_dir_issues, "index.html", package_summaries=issue_summaries, suite=suite_name, section=component) # ... and the metainfo index page self.render_template("metainfo_index.html", export_dir_metainfo, "index.html", package_summaries=mdata_summaries, suite=suite_name, section=component) validate_result = "Validation was not performed." if dep11_data: # do format validation validator = DEP11Validator() ret = validator.validate_file(d_fname) if ret: validate_result = "No errors found." else: validate_result = "" for issue in validator.issue_list: validate_result += issue.replace("FATAL", "FATAL")+"
\n" # sum up counts for suite statistics suite_metainfo_count += metainfo_count suite_error_count += error_count suite_warning_count += warning_count suite_info_count += info_count # calculate statistics for this component count = metainfo_count + error_count + warning_count + info_count valid_perc = 100/count*metainfo_count if count > 0 else 0 error_perc = 100/count*error_count if count > 0 else 0 warning_perc = 100/count*warning_count if count > 0 else 0 info_perc = 100/count*info_count if count > 0 else 0 # Render our overview page self.render_template("section_overview.html", export_dir_section, "index.html", suite=suite_name, section=component, valid_percentage=valid_perc, error_percentage=error_perc, warning_percentage=warning_perc, info_percentage=info_perc, metainfo_count=metainfo_count, error_count=error_count, warning_count=warning_count, info_count=info_count, validate_result=validate_result) # calculate statistics for this suite count = suite_metainfo_count + suite_error_count + suite_warning_count + suite_info_count valid_perc = 100/count*suite_metainfo_count if count > 0 else 0 error_perc = 100/count*suite_error_count if count > 0 else 0 warning_perc = 100/count*suite_warning_count if count > 0 else 0 info_perc = 100/count*suite_info_count if count > 0 else 0 # Render archive components index/overview page self.render_template("sections_index.html", export_dir, "index.html", sections=suite['components'], suite=suite_name, valid_percentage=valid_perc, error_percentage=error_perc, warning_percentage=warning_perc, info_percentage=info_perc, metainfo_count=suite_metainfo_count, error_count=suite_error_count, warning_count=suite_warning_count, info_count=suite_info_count) # Copy the static files target_static_dir = os.path.join(self._export_dir, "html", "static") shutil.rmtree(target_static_dir, ignore_errors=True) shutil.copytree(os.path.join(self._template_dir, "static"), target_static_dir) def main(): parser = OptionParser() (options, args) = parser.parse_args() if len(args) == 0: print("You need to specify a command!") sys.exit(1) command = args[0] # configure logging log_level = log.INFO if os.environ.get("DEBUG"): log_level = log.DEBUG log.basicConfig(format='%(asctime)s - %(levelname)s: %(message)s', level=log_level) if command == "process": if len(args) != 3: print("Invalid number of arguments: You need to specify a DEP-11 data dir and a suite.") sys.exit(1) gen = DEP11Generator() ret = gen.initialize(args[1]) if not ret: print("Initialization failed, can not continue.") sys.exit(2) gen.process_suite(args[2]) elif command == "cleanup": if len(args) != 2: print("Invalid number of arguments: You need to specify a DEP-11 data dir.") sys.exit(1) gen = DEP11Generator() ret = gen.initialize(args[1]) if not ret: print("Initialization failed, can not continue.") sys.exit(2) gen.expire_cache() elif command == "update-html": if len(args) != 2: print("Invalid number of arguments: You need to specify a DEP-11 data dir.") sys.exit(1) hgen = HTMLGenerator() ret = hgen.initialize(args[1]) if not ret: print("Initialization failed, can not continue.") sys.exit(2) hgen.update_html() elif command == "remove-processed": if len(args) != 3: print("Invalid number of arguments: You need to specify a DEP-11 data dir and suite.") sys.exit(1) gen = DEP11Generator() ret = gen.initialize(args[1]) if not ret: print("Initialization failed, can not continue.") sys.exit(2) gen.remove_processed(args[2]) else: print("Run with --help for a list of available command-line options!") if __name__ == "__main__": apt_pkg.init() main() appstream-dep11-0.4.0/scripts/dep11-validate 0000775 0000000 0000000 00000003056 12617473137 0020554 0 ustar 00root root 0000000 0000000 #!/usr/bin/env python3 # # Copyright (C) 2014-2015 Matthias Klumpp