Lore & lei
What’s the problem ?
I started working on the kernel code since a month and I cannot seem to wrap my head around the email based kernel workflow - since my experience is always with fancy tools like gerrit/git PR for openbmc application development.In this post we will look at how one can avoid subscribing to various kernel mailing lists to grab the patches of interest. So lets delve into a bit of background.
What’s LKML ?
The Linux kernel mailing list (LKML) is the main electronic mailing list for Linux kernel development, where the majority of the announcements, discussions, debates, and flame wars over the kernel take place.Many other mailing lists exist to discuss the different subsystems and ports of the Linux kernel, but LKML is the principal communication channel among Linux kernel developers. It is a very high-volume list, usually receiving about 1,000 messages each day, most of which are kernel code patches.
Linux utilizes a workflow governed by LKML, which is the “bazaar” where kernel development takes place.
In his book Linux Kernel Development, Robert Love notes:
If the Linux kernel community had to exist somewhere physically, it would call the Linux Kernel Mailing List home.
The moment I registered for this mailing list, my inbox is flooded with patches and its really exhausting to follow what’s going on (or) look for patches of interest.
Is there a way to get the patches / discussion of interest without needing to register to the LKML ? Yes - The answer is lore
lore’s search syntax
Public-inbox uses Xapian behind the scenes, which allows to narrowly tailor the keyword database to very specific needs.
For example, did you know that you can search lore.kernel.org for patches that touch specific files? Here’s every patch that touched the MAINTAINERS file:
https://lore.kernel.org/all/?q=dfn%3AMAINTAINERS
How about every patch that modifies a function that starts with floppy_:
https://lore.kernel.org/all/?q=dfhh%3Afloppy_*
Say you’re the floppy driver maintainer and wanted to find all mail that touches drivers/block/floppy.c and modifies any function that starts with floppy_ or has floppy in the subject and maybe any other mail that mentions floppy and has the words bug or regression? And maybe limit the results to just the past month.
Here’s the query:
1
2
3
(dfn:drivers/block/floppy.c OR dfhh:floppy_* OR s:floppy
OR ((nq:bug OR nq:regression) AND nq:floppy))
AND rt:1.month.ago..
And here are the results:
Now, how about getting that straight into your mailbox, so you don’t have to subscribe to the (very busy) linux-block list, if you are the floppy maintainer?
More about lore
The LWN website frequently covers discussion on the LKML, and the newsletter Kernel Traffic covered the activities of the LKML.Many internet websites include archives of the mailing list, such as lore.kernel.org/lkml, lkml.org, mail-archive.com.
Out of all the mail archiving platform mentioned above, lore.kernel.org is looked as best mail distributed mailing list archival framework with powerful search capabilities.
Even though it started out as merely a list archival service, it quickly became obvious that lore could be used for a lot more. Many developers ended up using its search features to quickly locate emails of interest, which in turn raised a simple question — what if there was a way to “save a search” and have it deliver all new incoming mail matching certain parameters straight to the developers’ inbox?
You can now do this with lei
On ubuntu systems , lei can be installed by following command:
1
sudo apt install lei
lore meets lei
Let’s say we want all the emails that talked about eSPI kernel device driver in openbmc. we could leverage lei-q options and frame the command like this :
1
2
3
4
lei q -o ~/Mail/openbmc \n
--threads --dedupe=mid \n
-I https://lore.kernel.org/openbmc/ \n
-t 's:eSPI'
Before you run it, let’s understand what it’s going to do:
-I https://lore.kernel.org/openbmc/ will query the aggregated index that contains information about all mailing lists archived on lore.kernel.org/openbmc.
-o ~/Mail/openbmc will create a new Maildir folder and put the search results there. Make sure that this folder doesn’t already exist, or lei will clobber anything already present there (unless you use –augment, but I haven’t tested this very extensively yet, so best to start with a clean slate).
--threads will deliver entire threads even if the match is somewhere in the middle of the discussion. This is handy if, for example, someone says “this sounds like a bug in the floppy subsystem” somewhere in the middle of a conversation and –threads will automatically get you the entire conversation context.
--dedupe=mid will deduplicate results based on the message-id header. The default behavior is to dedupe based on the body contents, but with so many lists still adding junky “sent to the foo list” footers, this tends to result in too many duplicated results. Passing –dedupe=mid is less safe (someone could sneak in a bogus message with an identical message-id and have it delivered to you instead), but more convenient. YMMV, BYOB.
As always, backslashes and newlines are there just for readability — you don’t need to use them.
After the command completes, you should get something similar to what is below:
1
2
3
4
5
6
7
# /home/manojeda/.local/share/lei/store 0/0
# /usr/bin/curl -Sf -s -d '' https://lore.kernel.org/openbmc/?x=m&t=1&q=s%3AeSPI
# https://lore.kernel.org/openbmc/ 19/?
# https://lore.kernel.org/openbmc/ 35/?
# https://lore.kernel.org/openbmc/ 63/?
# https://lore.kernel.org/openbmc/ 130/130
# 130 written to /home/manojeda/Mail/openbmc/ (130 matches)
A few things to notice here:
The command actually executes a curl call and retrieves the results as an mbox file and wrote 130 messages into the maildir.
That’s all about lore and lei, in my next post we will explore various options of lei and how to load results obtained from lei into something interesting called mutt.