Text
Page: 1
An experiment about
personalized front-end of
bugs.debian.org
E-mail archives + UDD + Simple web front-end =
Kentaro Hayashi
ClearCode Inc.
DebConf20 online August 25, 2020
Page: 2
NOTE: Presentation slide is
published
This presentation slide is available via Rabbit Slide Show
https://slide.rabbit-shocker.org/authors/kenhys/debconf2020-online/
“An experiment about personalized front-end of bugs.debian.org”
Page: 3
Personal profile
Debian Maintainer (@kenhys)
Trackpoint(soft dome) and Wasa beef(Yamayoshi Wasabi Potato Chips) fan
Working for ClearCode Inc.
Page: 4
ClearCode Inc.
https://www.clear-code.com/
Free software is important in ClearCode Inc.
We develop/support software with our free software development experiences
We feedback our business experiences to free software
Page: 5
Not talk about
Improving bugs.debian.org (debbugs) itself
Explaining bugs.debian.org (debbugs) internal in details
Page: 6
Agenda
Why so curious about bugs.d.o?
The troublesome cases about bugs.d.o
How to solve this situation?
Demonstrate a front-end
Conclusion
Page: 8
Usage of bugs.debian.org -
prerequisite
Communicate each other by E-mail
Change the bug status by sending E-mail
Send E-mail to control@bugs.d.o
Use control command in E-mail reply
Page: 9
Why so curious about bugs.d.o?
I’ve received deprecated notification bugs
Bug# 956770 growl-for-linux: Depends on deprecated libappindicator
Bug# 955899 growl-for-linux: Depends on deprecated dbus-glib
Page: 10
How did you fixed?
Bug# 956770 growl-for-linux: Depends on deprecated libappindicator
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=956770
Migrated to libayatana-indicator
It is easy because there are enough compatibility
I’ve fixed pkg-config target and include path
Page: 11
How did you fixed? (again)
Bug# 955899 growl-for-linux: Depends on deprecated dbus-glib
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=955899
Migrated to GDBus
Use d-feet to detect interface change (Rhythmbox plugin)
Fixed a interface to follow GDBus API
Send PR to upstream https://github.com/mattn/growl-for-linux/
Fixed a potential Rhythmbox related bug, too
Page: 12
growl-for-linux was fixed
But…What about other packages?
Page: 13
What about other packages?
libappindicator: deprecated in Debian; AppIndicator based applications,
please switch to Ayatana (App)Indicator(s)
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=895037
libdbus-glib-1-dev: is deprecated
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=895291
Page: 14
libappindicator: deprecated in
Debian
Many blocked bugs
Page: 15
libdbus-glib-1-dev: is deprecated
in Debian
Too many blocked bugs (Need to scroll!)
Page: 16
How to track many blocked?
In this case, UserTags is recommended
https://wiki.debian.org/bugs.debian.org/usertags
https://udd.debian.org/cgi-bin/bts-usertags.cgi
Page: 17
Does it work with libappindicator?
UserTag is: ayatana-appindicator
Page: 18
Does it work with libdbus-glib1-
dev?
It is a case that UserTag is not available
It should be tagged…
Page: 19
The troublesome cases about
bugs.d.o
UserTag is not always used
It may be difficult to see many blocked bugs
How to solve this situation?
Page: 20
Use udd.debian.org!
UDD can track usertags
UDD can track blocking bugs
Page: 21
In my experience through
contribution, I need
I need easy to:
Track a specified bug (tagged/not tagged)
Find a bug that no one working on
Send control E-mail
Find affected bugs
Page: 22
In my experience through
contribution (again)
I need easy to:
Track a specified bug (tagged/not tagged) Use UDD!
Find a bug that no one working on
Send control E-mail
Find affected bugs
Page: 23
How to solve rest issues?
Find a bug that no one working on
It’s a good attitude to fix a bug
Send control E-mail easily
It’s a good attitude to triage a bug
Find affected bugs
Remember misery bugs #932855, #965164
Page: 24
Supplementary explanation: Find
affected bugs
Sometimes grave/critical bug was found
2019, critical, e2fsprogs 1.45.3-1 breaks initramfs-tools-core <=0.133
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=932855
2020, grave, policykit-1: fails to install
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=965164
Page: 25
What about existing solution?
https://wiki.debian.org/Services
Developer Horizon - A dashboard for developers
http://horizon.debian.net/ Not available
Page: 26
What about existing solution?
Fabre - fulltext search service
http://fabre.debian.net Around 2005, Discontinued
Undocumented Debian “debbugs internal” written in Japanese
https://tokyodebian-team.pages.debian.net/pdf2005/
debianmeetingresume2005-fuyu.pdf
Page: 27
What about existing solution?
Debbugs Enhancement Suite
https://salsa.debian.org/lamby/debbugs-enhancement-suite
A Chrome extension to enhance using Debbugs
Tidy up each bug rendering, it’s cool (Thanks @lolamby FYI:)
…But It is a bit different what I want
Page: 28
What about existing solution?
Debian Popularity Contest
http://popcon.debian.org/
It provides a method to collect installed packages information
Public UDD Mirror
https://udd-mirror.debian.net/
It provides a statistical information about packages
Page: 29
Starting personal project
Concept
Make “unstable life” comfortable
Fix a bug (by finding a bug that no one working on)
Triage a bug (by sending control E-mail easily)
Avoid affected bugs (by finding affected important bugs)
Page: 30
Starting personal project (again)
Concept
Make “unstable life” comfortable
Fix a bug (by finding a bug that no one working on) Count comment in
bug#NNNNNN
Triage a bug (by sending control E-mail easily) Add mailto: link explicitly
Avoid affected bugs (by finding affected important bugs) Alert recent grave
bugs
Page: 31
How to realize concept?
Collect bug information regularly (E-mail archives)
Collect Popcon data (Fix a bug with your familiar packages)
Collect your installed packages information (popularity-contest)
Process collected data and make it accessible!
Page: 32
Collect bug information
Get E-mail archive with rsync
https://salsa.debian.org/debbugs-team/debbugs
rsync -av rsync://bugs-mirror.debian.org/bts-spool-db/
It requires 15GiB!
Use *.log and *.summary
Page: 33
Extract .log and .summary
Use perl module which is used for bugs.d.o (Debbugs)
Debbugs::Log, Debbugs::Status, Debbugs::MIME and so on
*.log contains multiple raw E-mail content with control sequence
*.summary contains metadata of each bug
Page: 34
How to parse .log
my $log = Debbugs::Log->new(log_name => $path);
my @records = $log->read_all_records();
for my $record (@records) {
...
my $entity = Debbugs::MIME::parse_to_mime_entity($record);
print "From: " . $entity->head->get("From");
print "To: " . $entity->head->get("To");
print "Subject: " . get_field($entity, "Subject");
print "Date: " . $entity->head->get("Date");
}
Page: 35
How to parse .summary
Format-Version: 3
Submitter: Nis Martensen <nis.martensen@web.de>
Subject: libexo-helpers: starting thunderbird fails for some mailto URIs
Found-In: exo/0.12.4-1
Tags: patch
Package: libexo-helpers
Message-Id: <159439678897.3833.10774950364480956704.reportbug@siamos.nes.net>
Severity: normal
Date: 1594396982
my $summary = Debbugs::Status::read_bug(summary => $path);
print $summary->{"tags"}
Page: 36
Collect Popcon data and so on
UDD: UltimateDebianDatabase
https://wiki.debian.org/UltimateDebianDatabase
You can access with:
psql –host=udd-mirror.debian.net –user=udd-mirror udd –password
Schema is available: https://udd.debian.org/schema/udd.html
Import popcon and package data
Page: 37
Collect installed packages
information
POPULARITY-CONTEST-0 TIME:1596711793 ID:XXX ARCH:amd64 POPCONVER:1.70 VENDOR:Debian
1596672000 1570622400 libgail-common /usr/lib/x86_64-linux-gnu/gtk-2.0/modules/libgail.so
1596672000 1596585600 libavcodec58 /usr/lib/x86_64-linux-gnu/libavcodec.so.58.91.100 <RECENT-CTIME>
1596672000 1586736000 libgoa-1.0-0b /usr/lib/x86_64-linux-gnu/libgoa-1.0.so.0.0.0
...
1596672000 1586217600 libzvbi0 /usr/lib/x86_64-linux-gnu/libzvbi.so.0.13.2
1596672000 1590624000 udisks2 /usr/share/locale/ja/LC_MESSAGES/udisks2.mo
END-POPULARITY-CONTEST-0 TIME:1596711817
Just upload popcon file
/usr/sbin/popularity-contest –su-nobody
Page: 38
Process collected data and make
it accessible!
Polling subscribed debian-bugs-dist@lists.debian.org
Importing latest E-mail archives from bugs-mirror.debian.org
Merging metadata + Web front-end
Page: 39
Prepared server specs for
experiment
Start with small VPS instance (about 3.26 USD/mo)
1vCPU
1GiB memory
20GiB disk space
Max 100Mbps bandwidth
Page: 41
Named my project as Fabre
cite: https://en.wikipedia.org/wiki/Jean-Henri_Fabre
Page: 42
Why Fabre?
Fabre is “The father of modern entomology” - the study of insects(bugs)
Just reuse previous (maybe) similar project (fabre.debian.net)
Page: 43
Concept of Fabre (again)
Make “unstable life” comfortable
Fix a bug (by finding a bug that no one working on)
Triage a bug (by sending control E-mail easily)
Avoid affected bugs (by checking installed packages)
Page: 44
Finding a bug that no one
working on
Show each number of comments (orange background color means no one
working on yet)
Page: 45
Sending control E-mail easily
Show each mailto: button to control bugs
Page: 46
Easy to view blocked bugs
List blocked bugs appropriately
Page: 47
Finding affected important bugs
Page: 48
How Fabre is good enough to
work?
Now ready to DEMO
Page: 49
Current status of Fabre
Tracked bugs: about 89,000 (UDD: 90,780)
Archived bugs: 720,000 (UDD: 85,0331)
Import bugs: every 1 hour
Update affected bugs: every 1 day
Disk usage (Database): about 1.5 GiB
Page: 50
Weak point about Fabre
Sometimes Killed by OOM, need more memory!
Page: 51
Conclusion
There are plenty room of improvements to develop
In my experiment, mashup some data sources may improve UX
bug report (E-mail archives)
UDD metadata (Popcon and package metadata)
installed packages list (popularity-contest)
Page: 52
Any questions?
Could you speak slowly if you have a question, please?
