Rabbit Slide Show

DebConf20

2020-08-25

Description

Text

Page: 1

An experiment about
personalized front-end of
bugs.debian.org
E-mail archives + UDD + Simple web front-end =
Kentaro Hayashi
ClearCode Inc.
DebConf20 online August 25, 2020

Page: 2

NOTE: Presentation slide is
published
This presentation slide is available via Rabbit Slide Show
https://slide.rabbit-shocker.org/authors/kenhys/debconf2020-online/
“An experiment about personalized front-end of bugs.debian.org”

Page: 3

Personal profile
Debian Maintainer (@kenhys)
Trackpoint(soft dome) and Wasa beef(Yamayoshi Wasabi Potato Chips) fan
Working for ClearCode Inc.

Page: 4

ClearCode Inc.
https://www.clear-code.com/
Free software is important in ClearCode Inc.
We develop/support software with our free software development experiences
We feedback our business experiences to free software

Page: 5

Not talk about
Improving bugs.debian.org (debbugs) itself
Explaining bugs.debian.org (debbugs) internal in details

Page: 6

Agenda
Why so curious about bugs.d.o?
The troublesome cases about bugs.d.o
How to solve this situation?
Demonstrate a front-end
Conclusion

Page: 7

We use bugs.d.o

Page: 8

Usage of bugs.debian.org -
prerequisite
Communicate each other by E-mail
Change the bug status by sending E-mail
Send E-mail to control@bugs.d.o
Use control command in E-mail reply

Page: 9

Why so curious about bugs.d.o?
I’ve received deprecated notification bugs
Bug# 956770 growl-for-linux: Depends on deprecated libappindicator
Bug# 955899 growl-for-linux: Depends on deprecated dbus-glib

Page: 10

How did you fixed?
Bug# 956770 growl-for-linux: Depends on deprecated libappindicator
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=956770
Migrated to libayatana-indicator
It is easy because there are enough compatibility
I’ve fixed pkg-config target and include path

Page: 11

How did you fixed? (again)
Bug# 955899 growl-for-linux: Depends on deprecated dbus-glib
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=955899
Migrated to GDBus
Use d-feet to detect interface change (Rhythmbox plugin)
Fixed a interface to follow GDBus API
Send PR to upstream https://github.com/mattn/growl-for-linux/
Fixed a potential Rhythmbox related bug, too

Page: 12

growl-for-linux was fixed
But…What about other packages?

Page: 13

What about other packages?
libappindicator: deprecated in Debian; AppIndicator based applications,
please switch to Ayatana (App)Indicator(s)
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=895037
libdbus-glib-1-dev: is deprecated
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=895291

Page: 14

libappindicator: deprecated in
Debian
Many blocked bugs

Page: 15

libdbus-glib-1-dev: is deprecated
in Debian
Too many blocked bugs (Need to scroll!)

Page: 16

How to track many blocked?
In this case, UserTags is recommended
https://wiki.debian.org/bugs.debian.org/usertags
https://udd.debian.org/cgi-bin/bts-usertags.cgi

Page: 17

Does it work with libappindicator?
UserTag is: ayatana-appindicator

Page: 18

Does it work with libdbus-glib1-
dev?
It is a case that UserTag is not available
It should be tagged…

Page: 19

The troublesome cases about
bugs.d.o
UserTag is not always used
It may be difficult to see many blocked bugs
How to solve this situation?

Page: 20

Use udd.debian.org!
UDD can track usertags
UDD can track blocking bugs

Page: 21

In my experience through
contribution, I need
I need easy to:
Track a specified bug (tagged/not tagged)
Find a bug that no one working on
Send control E-mail
Find affected bugs

Page: 22

In my experience through
contribution (again)
I need easy to:
Track a specified bug (tagged/not tagged) Use UDD!
Find a bug that no one working on
Send control E-mail
Find affected bugs

Page: 23

How to solve rest issues?
Find a bug that no one working on
It’s a good attitude to fix a bug
Send control E-mail easily
It’s a good attitude to triage a bug
Find affected bugs
Remember misery bugs #932855, #965164

Page: 24

Supplementary explanation: Find
affected bugs
Sometimes grave/critical bug was found
2019, critical, e2fsprogs 1.45.3-1 breaks initramfs-tools-core <=0.133
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=932855
2020, grave, policykit-1: fails to install
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=965164

Page: 25

What about existing solution?
https://wiki.debian.org/Services
Developer Horizon - A dashboard for developers
http://horizon.debian.net/ Not available

Page: 26

What about existing solution?
Fabre - fulltext search service
http://fabre.debian.net Around 2005, Discontinued
Undocumented Debian “debbugs internal” written in Japanese
https://tokyodebian-team.pages.debian.net/pdf2005/
debianmeetingresume2005-fuyu.pdf

Page: 27

What about existing solution?
Debbugs Enhancement Suite
https://salsa.debian.org/lamby/debbugs-enhancement-suite
A Chrome extension to enhance using Debbugs
Tidy up each bug rendering, it’s cool (Thanks @lolamby FYI:)
…But It is a bit different what I want

Page: 28

What about existing solution?
Debian Popularity Contest
http://popcon.debian.org/
It provides a method to collect installed packages information
Public UDD Mirror
https://udd-mirror.debian.net/
It provides a statistical information about packages

Page: 29

Starting personal project
Concept
Make “unstable life” comfortable
Fix a bug (by finding a bug that no one working on)
Triage a bug (by sending control E-mail easily)
Avoid affected bugs (by finding affected important bugs)

Page: 30

Starting personal project (again)
Concept
Make “unstable life” comfortable
Fix a bug (by finding a bug that no one working on) Count comment in
bug#NNNNNN
Triage a bug (by sending control E-mail easily) Add mailto: link explicitly
Avoid affected bugs (by finding affected important bugs) Alert recent grave
bugs

Page: 31

How to realize concept?
Collect bug information regularly (E-mail archives)
Collect Popcon data (Fix a bug with your familiar packages)
Collect your installed packages information (popularity-contest)
Process collected data and make it accessible!

Page: 32

Collect bug information
Get E-mail archive with rsync
https://salsa.debian.org/debbugs-team/debbugs
rsync -av rsync://bugs-mirror.debian.org/bts-spool-db/
It requires 15GiB!
Use *.log and *.summary

Page: 33

Extract .log and .summary
Use perl module which is used for bugs.d.o (Debbugs)
Debbugs::Log, Debbugs::Status, Debbugs::MIME and so on
*.log contains multiple raw E-mail content with control sequence
*.summary contains metadata of each bug

Page: 34

How to parse .log
my $log = Debbugs::Log->new(log_name => $path);
my @records = $log->read_all_records();
for my $record (@records) {
...
my $entity = Debbugs::MIME::parse_to_mime_entity($record);
print "From: " . $entity->head->get("From");
print "To: " . $entity->head->get("To");
print "Subject: " . get_field($entity, "Subject");
print "Date: " . $entity->head->get("Date");
}

Page: 35

How to parse .summary
Format-Version: 3
Submitter: Nis Martensen <nis.martensen@web.de>
Subject: libexo-helpers: starting thunderbird fails for some mailto URIs
Found-In: exo/0.12.4-1
Tags: patch
Package: libexo-helpers
Message-Id: <159439678897.3833.10774950364480956704.reportbug@siamos.nes.net>
Severity: normal
Date: 1594396982
my $summary = Debbugs::Status::read_bug(summary => $path);
print $summary->{"tags"}

Page: 36

Collect Popcon data and so on
UDD: UltimateDebianDatabase
https://wiki.debian.org/UltimateDebianDatabase
You can access with:
psql –host=udd-mirror.debian.net –user=udd-mirror udd –password
Schema is available: https://udd.debian.org/schema/udd.html
Import popcon and package data

Page: 37

Collect installed packages
information
POPULARITY-CONTEST-0 TIME:1596711793 ID:XXX ARCH:amd64 POPCONVER:1.70 VENDOR:Debian
1596672000 1570622400 libgail-common /usr/lib/x86_64-linux-gnu/gtk-2.0/modules/libgail.so
1596672000 1596585600 libavcodec58 /usr/lib/x86_64-linux-gnu/libavcodec.so.58.91.100 <RECENT-CTIME>
1596672000 1586736000 libgoa-1.0-0b /usr/lib/x86_64-linux-gnu/libgoa-1.0.so.0.0.0
...
1596672000 1586217600 libzvbi0 /usr/lib/x86_64-linux-gnu/libzvbi.so.0.13.2
1596672000 1590624000 udisks2 /usr/share/locale/ja/LC_MESSAGES/udisks2.mo
END-POPULARITY-CONTEST-0 TIME:1596711817
Just upload popcon file
/usr/sbin/popularity-contest –su-nobody

Page: 38

Process collected data and make
it accessible!
Polling subscribed debian-bugs-dist@lists.debian.org
Importing latest E-mail archives from bugs-mirror.debian.org
Merging metadata + Web front-end

Page: 39

Prepared server specs for
experiment
Start with small VPS instance (about 3.26 USD/mo)
1vCPU
1GiB memory
20GiB disk space
Max 100Mbps bandwidth

Page: 40

System diagrams

Page: 41

Named my project as Fabre
cite: https://en.wikipedia.org/wiki/Jean-Henri_Fabre

Page: 42

Why Fabre?
Fabre is “The father of modern entomology” - the study of insects(bugs)
Just reuse previous (maybe) similar project (fabre.debian.net)

Page: 43

Concept of Fabre (again)
Make “unstable life” comfortable
Fix a bug (by finding a bug that no one working on)
Triage a bug (by sending control E-mail easily)
Avoid affected bugs (by checking installed packages)

Page: 44

Finding a bug that no one
working on
Show each number of comments (orange background color means no one
working on yet)

Page: 45

Sending control E-mail easily
Show each mailto: button to control bugs

Page: 46

Easy to view blocked bugs
List blocked bugs appropriately

Page: 47

Finding affected important bugs

Page: 48

How Fabre is good enough to
work?
Now ready to DEMO

Page: 49

Current status of Fabre
Tracked bugs: about 89,000 (UDD: 90,780)
Archived bugs: 720,000 (UDD: 85,0331)
Import bugs: every 1 hour
Update affected bugs: every 1 day
Disk usage (Database): about 1.5 GiB

Page: 50

Weak point about Fabre
Sometimes Killed by OOM, need more memory!

Page: 51

Conclusion
There are plenty room of improvements to develop
In my experiment, mashup some data sources may improve UX
bug report (E-mail archives)
UDD metadata (Popcon and package metadata)
installed packages list (popularity-contest)

Page: 52

Any questions?
Could you speak slowly if you have a question, please?

Other slides