Kenneth Jenkins

Android APK patching

Tue, 18 Apr 2023 10:30:00 -0700

Recently I’ve been job hunting, and so I’ve been reflecting on my past work experience. One position I applied to had a required “portfolio” link, and at first I thought that must have been a mistake in the application form. But upon further reflection, I thought, for a front-end engineer, why not? This got me wondering if I could dig up screenshots/recordings of any of the features I’ve worked on in previous roles.

Specifically, I was thinking about my time working on the Android Google Maps app. During my time on the team, we went through not one but two major app redesigns, so a lot of the UI I had worked on back then has since been completely replaced.

Archived builds

Fortunately, APKMirror has tons of archived Google Maps APKs. Their coverage gets pretty spotty before 2014, but still includes a decent selection. Most relevant for my nostalgia, they have a build of Google Maps 7.0. This was the major redesign that was underway when I first joined the team.

I wasn’t sure if these old builds would work with modern Android device images, but I was glad to see that Android Studio still provides emulator system images for devices from around the same era.

After I had downloaded Android Studio, fired up Device Manager, selected what I thought to be an appropriately-ancient Pixel 2 device image with Android 5.0 (Lollipop), installed adb, and then finally installed the old Google Maps APK, I was greeted with this:

D’oh!

I had completely forgotten about the server-controlled kill switch built in to the app. The Google Maps team put a strong emphasis on backwards-compatibility, but also realized there would be cases where we might need to force users to stop using a particular released version.

Is there any way to bypass this update prompt?

`apktool` to the rescue

We can disassemble the archived Google Maps APKs using the open-source apktool:

$ apktool d -r com.google.android.apps.maps_7.0.2-700022123_minAPI18\(320dpi\)_apkmirror.com.apk -o maps-7.0
I: Using Apktool 2.7.0 on com.google.android.apps.maps_7.0.2-700022123_minAPI18(320dpi)_apkmirror.com.apk
I: Copying raw resources...
I: Baksmaling classes.dex...
I: Copying assets and libs...
I: Copying unknown files...
I: Copying original files...

(The -r option avoids decoding resources, as apktool seemed to have some issues with the resources.)

The resulting output includes a folder named “smali” containing all of the disassembled Java class bytecode. Many of the class names have been obfuscated, but we’re in luck: there’s a class named KillSwitchFragment.

We can see one of its methods finds and initializes a WebView, so it certainly looks like we’re on the right track:

.method private d()Landroid/view/View;
    .locals 2

    .prologue
    .line 83
    sget v0, Lcom/google/android/apps/gmm/i;->dv:I

    const/4 v1, 0x0

    invoke-virtual {p0, v0, v1}, Lcom/google/android/apps/gmm/terms/KillSwitchFragment;->a(ILandroid/view/ViewGroup;)Landroid/view/View;

    move-result-object v0

    .line 84
    sget v1, Lcom/google/android/apps/gmm/g;->gN:I

    invoke-virtual {v0, v1}, Landroid/view/View;->findViewById(I)Landroid/view/View;

    move-result-object v0

    check-cast v0, Landroid/webkit/WebView;

    .line 86
    new-instance v1, Lcom/google/android/apps/gmm/terms/a;

    invoke-direct {v1, p0}, Lcom/google/android/apps/gmm/terms/a;-><init>(Lcom/google/android/apps/gmm/terms/KillSwitchFragment;)V

    invoke-virtual {v0, v1}, Landroid/webkit/WebView;->setWebViewClient(Landroid/webkit/WebViewClient;)V

    .line 98
    iget-object v1, p0, Lcom/google/android/apps/gmm/terms/KillSwitchFragment;->b:Ljava/lang/String;

    invoke-virtual {v0, v1}, Landroid/webkit/WebView;->loadUrl(Ljava/lang/String;)V

    .line 100
    return-object v0
.end method

This isn’t the easiest to read. A quick primer to help make sense of it:

Method parameter values are referred to as p0, p1, p2, etc.
For an instance method, parameter p0 is the this reference.
Local variables are referred to as v0, v1, v2, etc.
Java classes are referenced with an L, followed by their fully-qualified class name (with all . characters replaced by / characters), followed by a ; (e.g. android.webkit.WebView becomes Landroid/webkit/WebView;)
Other types include I (int), Z (boolean), and V (void).
The invoke- mnemonics represent a method call.
Method references consist of the class name followed by ->, the method name, the types of the method parameters (in parentheses), and the method return type.
Constructors are referred to like a void method with the name <init>.

So this method may have looked something like this in the source code:

  private View createView() {
    View view = this.inflateLayout(some_layout_id, null);
    WebView webView = (WebView) view.findViewById(some_view_id);
    WebViewClient client = new KillSwitchWebViewClient(this);
    webView.setWebViewClient(client);
    webView.loadUrl(this.url);
    return webView;
  }

(I’m guessing at some of the method names based on the signatures.)

We can search through all the disassembled classes to look for references to KillSwitchFragment, and after tracing a few method call chains, I came to this line in a class named GmmActivity:

    invoke-static {p0, v0}, Lcom/google/android/apps/gmm/terms/KillSwitchFragment;->a(Lcom/google/android/apps/gmm/base/activities/GmmActivity;Z)V

Let’s just comment that line out with a # and see what happens….

Building a patched APK

We can reassemble an APK by running:

$ apktool b maps-7.0
I: Using Apktool 2.7.0
I: Checking whether sources has changed...
I: Smaling smali folder into classes.dex...
I: Checking whether resources has changed...
I: Copying raw resources...
I: Building apk file...
I: Copying unknown files/dir...
I: Built apk into: maps-7.0/dist/com.google.android.apps.maps_7.0.2-700022123_minAPI18(320dpi)_apkmirror.com.apk

However in order to install to a device (or emulator), we need to sign the APK. We can generate a dummy signing key for this with keytool (part of the Java runtime):

$ keytool -genkey -keystore dummy.keystore -alias dummy -keyalg RSA -keysize 2048 -validity 100

After following through the prompts, we can then use apksigner (from the Android SDK build tools) to sign our patched APK with the dummy key:

$ apksigner sign --ks dummy.keystore maps-7.0/dist/com.google.android.apps.maps_7.0.2-700022123_minAPI18\(320dpi\)_apkmirror.com.apk

And then we can launch a device emulator and finally install our patched APK using adb:

$ adb install -r maps-7.0/dist/com.google.android.apps.maps_7.0.2-700022123_minAPI18\(320dpi\)_apkmirror.com.apk

And we get this…

Video format not supported on this browser.

Unfortunately, Maps has stopped.

Hmm. Progress?

More patching

Well, at least we can use adb logcat to look for crash information, and what do you know:

FATAL EXCEPTION: main
Process: com.google.android.apps.maps, PID: 3012
java.lang.SecurityException: GoogleCertificatesRslt: not whitelisted: pkg=com.google.android.apps.maps, sha1=9459608ef2083805c6f6075779366c5831df2f8f, atk=true, ver=202414010.true (go/gsrlt)

Well yes, our patched APK would fail whatever hash-based verification it might be subjected to, wouldn’t it.

The logcat output also has a stack trace to work with, and there’s only one stack frame matching the com.google.android.apps.gmm package name:

--------- beginning of crash
E/AndroidRuntime( 3012): FATAL EXCEPTION: main
E/AndroidRuntime( 3012): Process: com.google.android.apps.maps, PID: 3012
E/AndroidRuntime( 3012): java.lang.SecurityException: GoogleCertificatesRslt: not whitelisted: pkg=com.google.android.apps.maps, sha1=9459608ef2083805c6f6075779366c5831df2f8f, atk=true, ver=202414010.true (go/gsrlt)
E/AndroidRuntime( 3012): 	at android.os.Parcel.readException(Parcel.java:1540)
E/AndroidRuntime( 3012): 	at android.os.Parcel.readException(Parcel.java:1493)
E/AndroidRuntime( 3012): 	at com.google.android.gms.internal.v.c(Unknown Source)
E/AndroidRuntime( 3012): 	at com.google.android.gms.internal.aS.a(Unknown Source)
E/AndroidRuntime( 3012): 	at com.google.android.gms.internal.g.b(Unknown Source)
E/AndroidRuntime( 3012): 	at com.google.android.gms.internal.k.onServiceConnected(Unknown Source)
E/AndroidRuntime( 3012): 	at com.google.android.gms.internal.m.a(Unknown Source)
E/AndroidRuntime( 3012): 	at com.google.android.gms.internal.g.c(Unknown Source)
E/AndroidRuntime( 3012): 	at com.google.android.gms.location.reporting.c.a(Unknown Source)
E/AndroidRuntime( 3012): 	at com.google.android.apps.gmm.ulr.UlrPromoFragment.onResume(SourceFile:89)
E/AndroidRuntime( 3012): 	at android.app.Fragment.performResume(Fragment.java:2096)
E/AndroidRuntime( 3012): 	at android.app.FragmentManagerImpl.moveToState(FragmentManager.java:928)
E/AndroidRuntime( 3012): 	at android.app.FragmentManagerImpl.moveToState(FragmentManager.java:1067)
E/AndroidRuntime( 3012): 	at android.app.BackStackRecord.popFromBackStack(BackStackRecord.java:1595)
E/AndroidRuntime( 3012): 	at android.app.FragmentManagerImpl.popBackStackState(FragmentManager.java:1504)
E/AndroidRuntime( 3012): 	at android.app.FragmentManagerImpl$2.run(FragmentManager.java:490)
E/AndroidRuntime( 3012): 	at android.app.FragmentManagerImpl.execPendingActions(FragmentManager.java:1452)
E/AndroidRuntime( 3012): 	at android.app.FragmentManagerImpl$1.run(FragmentManager.java:447)
E/AndroidRuntime( 3012): 	at android.os.Handler.handleCallback(Handler.java:739)
E/AndroidRuntime( 3012): 	at android.os.Handler.dispatchMessage(Handler.java:95)
E/AndroidRuntime( 3012): 	at android.os.Looper.loop(Looper.java:135)
E/AndroidRuntime( 3012): 	at android.app.ActivityThread.main(ActivityThread.java:5221)
E/AndroidRuntime( 3012): 	at java.lang.reflect.Method.invoke(Native Method)
E/AndroidRuntime( 3012): 	at java.lang.reflect.Method.invoke(Method.java:372)
E/AndroidRuntime( 3012): 	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:899)
E/AndroidRuntime( 3012): 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:694)
W/ActivityManager( 1178):   Force finishing activity com.google.android.apps.maps/com.google.android.maps.MapsActivity

Let’s see if we can do without the UlrPromoFragment as well. After some more spelunking through the disassembled classes, I found this section in the class com/google/android/apps/gmm/base/activities/d:

    invoke-static {v0}, Lcom/google/android/apps/gmm/ulr/UlrPromoFragment;->a(Lcom/google/android/apps/gmm/base/activities/GmmActivity;)Z

    move-result v0

    if-eqz v0, :cond_0

    .line 566
    iget-object v0, p0, Lcom/google/android/apps/gmm/base/activities/d;->a:Lcom/google/android/apps/gmm/base/activities/GmmActivity;

    new-instance v1, Lcom/google/android/apps/gmm/ulr/UlrPromoFragment;

    invoke-direct {v1}, Lcom/google/android/apps/gmm/ulr/UlrPromoFragment;-><init>()V

    invoke-virtual {v0, v1}, Lcom/google/android/apps/gmm/base/activities/GmmActivity;->a(Lcom/google/android/apps/gmm/base/fragments/GmmActivityFragment;)V

    .line 568
    :cond_0
    iget-object v0, p0, Lcom/google/android/apps/gmm/base/activities/d;->a:Lcom/google/android/apps/gmm/base/activities/GmmActivity;

This is a call to a static method on UlrPromoFragment, and then depending on the return value from that static method, we may or may not initialize a UlrPromoFragment instance and do something with it.

Let’s just make that an unconditional jump to :cond_0 instead, so we never create a UlrPromoFragment instance here:

    invoke-static {v0}, Lcom/google/android/apps/gmm/ulr/UlrPromoFragment;->a(Lcom/google/android/apps/gmm/base/activities/GmmActivity;)Z

    move-result v0

    goto :cond_0

    .line 566
    iget-object v0, p0, Lcom/google/android/apps/gmm/base/activities/d;->a:Lcom/google/android/apps/gmm/base/activities/GmmActivity;

    new-instance v1, Lcom/google/android/apps/gmm/ulr/UlrPromoFragment;

    invoke-direct {v1}, Lcom/google/android/apps/gmm/ulr/UlrPromoFragment;-><init>()V

    invoke-virtual {v0, v1}, Lcom/google/android/apps/gmm/base/activities/GmmActivity;->a(Lcom/google/android/apps/gmm/base/fragments/GmmActivityFragment;)V

    .line 568
    :cond_0
    iget-object v0, p0, Lcom/google/android/apps/gmm/base/activities/d;->a:Lcom/google/android/apps/gmm/base/activities/GmmActivity;

Repeat the same dance as before: apktool b, apksigner sign, uninstall and then adb install, and then…

Video format not supported on this browser.

Searching for restaurants in Mountain View like it's 2013

The map renders! Search works! The Street View thumbnails don’t seem to load anymore, but other than that it seems totally functional.

Reference

Adventures in consumer electronics repair: GE clock radio

Tue, 11 Apr 2023 16:20:35 -0700

GE clock radio, model number 7-4613B. Likely manufactured in 1994.

I’ve had this GE clock radio ever since I was little. I took it with me to college, and then after I graduated I brought it with me across the country.

It’s been a faithful companion over the years, but it’s starting to show its age. The main slider for selecting between off / on / alarm / music doesn’t work reliably. The radio also has a 9-volt battery backup for keeping the time when there’s a power interruption, but lately it seems to run fast when it’s on backup power.

We’d been having power outages fairly often, unfortunately. After the latest outage I tried to reset it to the current time, but to my dismay I found the “MIN” button no longer functioned at all.

Having developed quite the sentimental attraction to this clock, I thought I’d take it apart and see if I couldn’t fix it.

After removing some screws and pulling off some knobs, I was finally able to coax the cover off. What did I find, but a set of 6mm tactile switches!

Clock radio internals. Note the four push-button switches in the lower left.

Rummaging through an old shoebox of electronics doodads, I found I had a little baggy full of identical-looking switches. Judging from the part number, I must have bought these from Adafruit, probably when I was messing around with Arduino stuff.

Lifting out the circuit board to examine the underside, I found that only three of the switch’s four pins were soldered. (And one of the three that is soldered doesn’t appear to connect to any traces, so I assume that was just for structural purposes.)

Luckily for me, that made desoldering a bit easier. I managed it with some desoldering wick and the pencil tip that was already in my soldering iron, but it was a bit clumsy.

Underside of the circuit board (leads of the switch in question highlighted in magenta). There’s pretty clearly been some water damage, though I don’t have a distinct memory of how or when this might have happened.

Using a multimeter in continuity mode and one of the other working switches, I confirmed that the switches I had on hand behaved the same as the ones on the board (normally open, momentary).

As I’m finding is often the case with repair work, installing the new part was much easier than removing the old.

Success! With the new switch installed, I could once again advance the clock’s minute value.

The only difference I noticed was that the new switch is much “click”-ier than the old ones. I’m not sure if that’s a difference in specification, or something to do with wear over time. For something that I interact with infrequently, I’m not bothered by it.

Maybe one of these days I will tackle replacing that main slider switch.

Advent of Code 2020: Day 19 reflection

Tue, 30 Nov 2021 20:55:00 -0800

Advent of Code 2021 starts tonight! But before then, I have one more reflection from last year’s challenge to share.

This one is about the day 19 puzzle, which, to be perfectly honest, I did not particularly enjoy.

From the puzzle description:

The rules for valid messages (the top part of your puzzle input) are numbered and build upon each other. For example:
0: 1 2
1: "a"
2: 1 3 | 3 1
3: "b"
Some rules, like 3: "b", simply match a single character (in this case, b).

The remaining rules list the sub-rules that must be followed; for example, the rule 0: 1 2 means that to match rule 0, the text being checked must match rule 1, and the text after the part that matched rule 1 must then match rule 2.

Some of the rules have multiple lists of sub-rules separated by a pipe (|). This means that at least one list of sub-rules must match. (The ones that match might be different each time the rule is encountered.) For example, the rule 2: 1 3 | 3 1 means that to match rule 2, the text being checked must match rule 1 followed by rule 3 or it must match rule 3 followed by rule 1.

Fortunately, there are no loops in the rules, so the list of possible matches will be finite.

[…]

The received messages (the bottom part of your puzzle input) need to be checked against the rules so you can determine which are valid and which are corrupted.

[…]

Your goal is to determine the number of messages that completely match rule 0.

(from https://adventofcode.com/2020/day/19)

Dear reader, the "a" and "b" from the example were not just an example. In my puzzle input file, there were in fact only "a" and "b" character rules, followed by an unwieldy number of “messages” that looked like lots of ababbbab and aabaaaba and baaabbaa.

I thought this was a bit too much.

My original approach

In my first attempt at this puzzle, I thought, “I know, I’ll use regular expressions.”

The input rules format seemed half way to regular expression syntax anyway, so I thought, why not translate rule 0 into one giant regular expression and run all the messages through it?

So the example rule set would turn into something like this:

rule 1: /a/
rule 3: /b/
rule 2: /(ab|ba)/
rule 0: /a(ab|ba)/

This strategy worked just fine for part 1 of the puzzle, but looking back, I will concede that it was unnecessarily complicated.

(It would probably have been better even to build a set of all possible valid messages, and use that to query whether each message in the input file was present in the set of valid messages.)

But when it came to part 2…

As you look over the list of messages, you realize your matching rules aren’t quite right. To fix them, completely replace rules 8: 42 and 11: 42 31 with the following:
8: 42 | 42 8
11: 42 31 | 42 11 31
This small change has a big impact: now, the rules do contain loops, and the list of messages they could hypothetically match is infinite. You’ll need to determine how these changes affect which messages are valid.

Fortunately, many of the rules are unaffected by this change; it might help to start by looking at which rules always match the same set of values and how those rules (especially rules 42 and 31) are used by the new versions of rules 8 and 11.

(Remember, you only need to handle the rules you have; building a solution that could handle any hypothetical combination of rules would be significantly more difficult.)

(from https://adventofcode.com/2020/day/19)

The new rule 8 can still be expressed as a regular expression: it’s just “1 or more” copies of rule 42 (we can use the regex + symbol for this).

It’s rule 11 that throws a monkey wrench into the regular expression strategy. In words, we could describe this rule as “one or more matches of rule 42, followed by an equal number of matches of rule 31”. Sadly this “equal number of matches” can’t be expressed as a regular expression.

In fact, this is literally the first non-example of a regular language in the wikipedia article:

A simple example of a language that is not regular is the set of strings $ \{ a^nb^n \mid n ≥ 0 \} $. Intuitively, it cannot be recognized with a finite automaton, since a finite automaton has finite memory and it cannot remember the exact number of a’s.

(from https://en.wikipedia.org/wiki/Regular_language)

I still hacked together a solution out of my existing work for part 1, but it relied on a couple important details:

Neither rule 42 nor rule 31 (or any rules referenced by them) contained any reference to rule 8 or 11.
As it turned out, all of the possible matches for both rule 42 and rule 31 had the same length (8 characters).

The other crucial detail is that the actual rule 0 (at least in my puzzle input) was this: 0: 8 11.

Combining rules 0, 8, and 11 then, essentially what I was looking for was “two or more matches of rule 42, followed by some smaller (but non-zero) number of matches of rule 31”.

Putting all this together, it meant I could examine a message in chunks of 8 characters. If a message wasn’t a multiple of 8 characters, I could discard it right away; otherwise, I could see how many chunks (starting from the beginning) would match rule 42, and how many chunks (now starting from the end) would match rule 31. Let’s say the whole message was $ n $ chunks in length. If we had $ x $ chunks matching rule 42, and $ y $ chunks matching rule 31, then as long as there was some $ k $ that could satisfy all of

$$ \begin{gather*} 2 \le k \le x \\ 1 \le (n - k) \le y \\ k > n - k \\ \end{gather*} $$

this would indicate a valid match.

Like I said, I eventually got this to work, after a fair bit of trial and error.

But this all left me wondering, what’s the “right way” to do something like this, in the general case? Is there some existing parsing library that’s already solved this problem for us?

GLL parsing to the rescue

I recently came across the packrattle parsing library, which can do just this! (I’m sure there are others that would work too, but this was one in a language I felt reasonably confident that I could use.)

The underlying parsing algorithm is fairly magical to me, but the library is simple to use.

The core of my packrattle-based solution looks like this:

const packrattle = require('packrattle');

function parseRule(p, line) {
  var [id, rule] = line.split(': ');
  id = parseInt(id);
  if (rule[0] === '"') {
    p[id] = packrattle.string(rule[1]);
  } else {
    p[id] = packrattle.alt(...rule.split(' | ').map(
      (e) => packrattle.seq(...e.split(' ').map(
        (i) => {
          i = parseInt(i);
          return () => p[i];
        }))));
  }
}

This function can be used to build up an object p of rule parsers (keyed by the rule ID number), using these packrattle functions:

string() matches a fixed input string (in our case, either "a" or "b")
alt() matches any one of a list of alternatives
seq() matches a list of things in sequence

The rest of the solution reads from standard input, parsing rules until it reaches a blank line, and then switches to feeding input lines to the final p[0] parser, keeping a running count of valid matches.

This certainly works, but can we go a little meta? Why are we messing around with split() to parse the input rules? Why not construct a parser that itself returns parsers?

Behold:

const ruleParser = (function() {
  const ruleId = packrattle.regex(/\d+/).map(m => parseInt(m[0]));
  const character = packrattle.regex(/"(.)"/).map(m => m[1]);
  const ruleSeq = packrattle.repeatSeparated(ruleId, ' ');
  const ruleAlts = packrattle.repeatSeparated(ruleSeq, ' | ');
  const ruleDelim = packrattle.string(': ').drop();
  return packrattle.seq(ruleId, ruleDelim, packrattle.alt(character, ruleAlts));
})();

function parseRule(p, line) {
  var [id, rule] = ruleParser.run(line);
  if (typeof rule === 'string') {
    p[id] = packrattle.string(rule);
  } else {
    p[id] = packrattle.alt(...rule.map(
      ids => packrattle.seq(...ids.map(
        id => () => p[id]))));
  }
}

Is this any more readable that the last version? Not really, no.

But this was a fun exercise in using a parsing library!

Advent of Code 2020: Day 18 reflection

Sun, 28 Nov 2021 20:10:00 -0800

The day 18 puzzle from last year’s Advent of Code involved parsing and evaluating arithmetic expressions, but with a twist:

The homework (your puzzle input) consists of a series of expressions that consist of addition (+), multiplication (*), and parentheses ((...)). Just like normal math, parentheses indicate that the expression inside must be evaluated before it can be used by the surrounding expression. Addition still finds the sum of the numbers on both sides of the operator, and multiplication still finds the product.

However, the rules of operator precedence have changed. Rather than evaluating multiplication before addition, the operators have the same precedence, and are evaluated left-to-right regardless of the order in which they appear.

These expressions consisted of integers, addition, multiplication, and parentheses to group sub-expressions.

(from https://adventofcode.com/2020/day/18)

Part 2 is similar, but instead of having the same precedence, addition is evaluated before multiplication.

When I first went about solving this puzzle, I wrote some Python code to look for parentheses, and split expressions at operators, and then evaluate those expressions recursively.

But afterwards I got curious… this seems like an awful lot of work. Can we find a lazier approach?

What if we could use a programming language where we can directly redefine the + and * functions to have a different precedence?

Operator precedence

There may be other languages we could use for this, but the one that came to mind for me was Haskell. Haskell allows you to define your own operators, and to specify their precedence levels.

We can use GHC’s interactive environment to check the default operator precedence for + and *, using the :info command:

ken$ ghci
GHCi, version 8.8.2: https://www.haskell.org/ghc/  :? for help
Prelude> :info +
class Num a where
  (+) :: a -> a -> a
  ...
  	-- Defined in ‘GHC.Num’
infixl 6 +
Prelude> :info *
class Num a where
  ...
  (*) :: a -> a -> a
  ...
  	-- Defined in ‘GHC.Num’
infixl 7 *

The line infixl 6 + tells us that the + operator has precedence level 6.¹ Likewise, the * operator has precedence level 7. Because * has a higher precedence level, it will be evaluated before +.

So, we can just redefine the precedence for + so that it is the same as for *:

Prelude> infixl 7 +

<interactive>:1:10: error:
    The fixity signature for ‘+’ lacks an accompanying binding
      (The fixity signature must be given where ‘+’ is declared)

Well, OK, not quite like that… it looks like we have to redefine the + operator itself at the same time if we want to assign a new operator precedence. We can do that with a no-op definition, using a qualified reference to the original + operator:

Prelude> (+) = (Prelude.+); infixl 7 +
Prelude> 1 + 2 * 3
9

But hey, look at that! Now 1 + 2 * 3 equals 9!

A complete solution

The puzzle asks for the sum of all the expressions in the input file (which has one expression per line).

So, if we take our input file, stick the redefinition of + at the top, and run the whole lot through ghci, we can evaluate all of the expressions in the input.

Something like this command line:

echo "(+) = (Prelude.+); infixl 7 +
$(cat input18.txt)" | ghci -v0

The -v0 option allows us to suppress the prompt output from ghci, so we get just the results of each expression as our output, one per line.

Once we have the expressions evaluated, we still need to get the sum of all the values. We could use some awk for this:

echo "(+) = (Prelude.+); infixl 7 +
$(cat input18.txt)" | ghci -v0 | awk '{sum+=$1} END {print sum}'

We can do a very similar thing for part 2, where we need to swap the precedence of addition and multiplication:

echo "(+) = (Prelude.+); infixl 7 +; (*) = (Prelude.*); infixl 6 *
$(cat input18.txt)" | ghci -v0 | awk '{sum+=$1} END {print sum}'

And there you have it! No need to balance parentheses or parse expressions.

If there were a contest for “most oddball solution”, this would be my entry.

This line is known as a fixity declaration. The l in infixl specifies that the + operator is left associative. There’s also infixr for right associativity, and a plain infix for non-associative operators. (See §4.4.2 of The Haskell 2010 Language for more info.) ↩︎

Advent of Code 2020: Day 13 reflection

Fri, 26 Nov 2021 23:15:00 -0800

The day 13 puzzle from last year’s Advent of Code was the first one that left me really flummoxed. Part 1 (the warm-up) was straight-forward enough, but part 2 required left me feeling like there was some really useful math that I didn’t know.

The puzzle was about a set of buses that all take different amounts of time to complete their routes, and keep looping around forever.

After getting past all the flavor text, part 2 of the puzzle boiled down to this: given a set of tuples $ \{(k_i, n_i)\} $, find the smallest non-negative $ t $ such that \[ t \equiv -k_i \pmod{n_i} \] for all $i$. The problem also gave the hint that $t$ would exceed $10^{14}$.

(I think the intention of this hint was to discourage anyone from trying the brute-force solution of checking all possible $ t $ values one by one. Or maybe just a hint that using 32-bit integers would not be sufficient?)

The puzzle description gave a few example problems (with solutions, thankfully). The first one was the set

\[ \{(0,7), (1,13), (4,59), (6,31), (7,19)\} \]

and gave the solution as $ t = 1068781 $.

The modular arithmetic stirred a vague memory in my mind of Fermat’s little theorem, which also made me wonder if all the $n_i$ in my puzzle input were prime. The problem didn’t specify that this would be the case, but glancing through my input file it seemed that this was indeed true (at least for the puzzle input that I got). But I spent much too long thinking about whether this was useful for the problem, because I couldn’t see that it helped at all.

I didn’t really make any progress until I started looking for ways to reduce the problem to a smaller one.

If the input set consisted of just one tuple, that would be easy enough to solve! If we had just the first tuple $ (0, 7) $ to worry about, I think it’s simple to see that $ t = 0 $ is the solution.

And even adding in the second tuple $ (1, 13) $, it’s not too bad: as long as $ t $ is a multiple of $ 7 $, it will satisfy the condition from the first tuple. So we could start checking all the multiples of $ 7 $, looking for one where $ t \equiv -1 \equiv 12 \pmod{13} $:

$$ \begin{align*} 7 \equiv 7 \pmod{13} \\ 14 \equiv 1 \pmod{13} \\ 21 \equiv 8 \pmod{13} \\ 28 \equiv 2 \pmod{13} \\ ⋮ \qquad \qquad \\ 70 \equiv 5 \pmod{13} \\ 77 \equiv 12 \pmod{13} \\ \end{align*} $$

But here I got stuck. How do we add in the third tuple $ (4, 59) $? How can we find candidate solutions that still satisfy the constraints from the first two tuples?

I think I ended up confusing myself here. I was too fixated on the pattern of looking at multiples of the previous candidate solution. But I couldn’t just start looking at multiples of $ 77 $, as they wouldn’t all have the same remainders modulo $ 13 $ any more.

With the benefit of hindsight, obviously I should have focused on addition rather than multiplication.

We can start adding multiples of $ 91 $ — because $ 91 $ is divisible by both $ 7 $ and $ 13 $, this won’t affect congruence $ \gdef\Mod#1{(\mathrm{mod}\ #1)} \Mod{7} $ or $ \Mod{13} $. So we can continue our search like so, looking for a $ t \equiv -4 \equiv 55 \pmod{59} $:

$$ \begin{align*} 77 \equiv 18 \pmod{59} \\ 168 \equiv 50 \pmod{59} \\ 259 \equiv 23 \pmod{59} \\ 350 \equiv 55 \pmod{59} \\ \end{align*} $$

So $ t = 350 $ satisfies all of the first three constraints.

And we can continue on like this. In Python, this approach could look something like this:

def search(start, increment, target, modulus):
    t = start
    while t % modulus != target:
        t += increment
    return t

def solve(tuples):
    ans = 0
    increment = 1
    for k, n in tuples:
        ans = search(ans, increment, -k % n, n)
        increment *= n
    return ans

Note that the increment *= n line probably isn’t correct in general. I think this works only if the set of $ n_i $s are relatively prime? This should probably be something like increment = math.lcm(increment, n) in the general case.

Otherwise, this appears to work just fine for all of the examples in the puzzle description!

Too clever by half

For some reason I couldn’t come up with this solution when I was first working through this problem. Perhaps I had convinced myself this approach wouldn’t be efficient enough?

I don’t know why; my puzzle input had a total of 9 tuples with a maximum $ n $ value of $ 787 $. That should put us well under 10,000 iterations of the inner while loop in the worst case, which should be no problem (especially if all the values fit in one 64-bit integer, so that there’s no slowdown as we start doing arithmetic with larger and larger numbers).

But no, after much struggle I eventually ended up with a much more complicated solution. I’ll include the core of it here, but I don’t even remember now exactly what I was thinking:

def solve(tuples):
    accum = []
    fs = [(-k % n, n) for k, n in tuples]
    for i in range(len(fs)):
        accum.append(fs[i])
        fs = [(factor(fs[i][1], b[1], (b[0] - fs[i][0]) % b[1]), b[1]) for b in fs]
    return evaluate(accum)

def factor(a, d, q):
    for i in range(1, d+1):
        if a * i % d == q:
            return i

def evaluate(accum):
    if not accum:
        return 0
    return accum[0][0] + accum[0][1] * evaluate(accum[1:])

I think I was trying to express the solution in the form

\[ t = a + n_1(b + n_2(c + n_3(…))) \]

I start with values for the $ a, b, c, … $ coefficients based on the $ k_i $ values from the input tuples, and then keep adjusting them for each of the $ n_i $.

Compared to the simpler solution from above, the factor() method is still doing a linear search for a specific remainder $ \Mod{n_i} $, so it’s not really any more efficient. The only possible advantage I can think of is that we’re working with smaller numbers until the very end, when the evaluate() method computes a final answer based on all the final coefficients. But, it turns out that doesn’t really matter, because for this puzzle as the values fit into 64-bit integers just fine.

Is there a moral to this story? I’m not sure.

Perhaps it’s this: sometimes it’s crucial to have a specific insight for a problem, and you can’t really force insight. It’s fine to take a break and come back to the problem later.

Advent of Code 2020: Day 5 reflection

Wed, 24 Nov 2021 20:00:00 -0800

The day 5 puzzle from last year’s Advent of Code talks about airplane seat numbering using “binary space partitioning”.

From the puzzle description:

Instead of zones or groups, this airline uses binary space partitioning to seat people. A seat might be specified like FBFBBFFRLR, where F means “front”, B means “back”, L means “left”, and R means “right”.

The first 7 characters will either be F or B; these specify exactly one of the 128 rows on the plane (numbered 0 through 127). Each letter tells you which half of a region the given seat is in. Start with the whole list of rows; the first letter indicates whether the seat is in the front (0 through 63) or the back (64 through 127). The next letter indicates which half of that region the seat is in, and so on until you’re left with exactly one row.

[…]

The last three characters will be either L or R; these specify exactly one of the 8 columns of seats on the plane (numbered 0 through 7). The same process as above proceeds again, this time with only three steps. L means to keep the lower half, while R means to keep the upper half.

[…]

Every seat also has a unique seat ID: multiply the row by 8, then add the column.

(from https://adventofcode.com/2020/day/5)

After absorbing all of that explanation, it turns out that parsing a “seat specification” is way less complicated than it might appear. Converting to row and column is just like parsing a binary number: the first character is just like the most significant bit of a 7-bit binary number (the 64s place, if you will). As the rows are numbered with 0 in the front and 127 in the back, an F is like a 0 bit and a B like a 1.

Similarly, the last three characters are the binary representation of the column number, with L like a 0 and R like a 1.

Let’s take the example FBFBBFFRLR. The row and column break down like so:

And then the final seat ID is 44 × 8 + 5 = 357.

But multiplying by 8 is the same as shifting the binary digits left by three places. (The 1s place turns into the 8s place, the 2s place into the 16s place, and so on.)

So there’s no need to distinguish between row and column at all! We can just replace all the Fs and Ls with 0s, and all the Bs and Rs with 1s, and then parse the whole thing as an 11 bit binary number.

(And in Python, parsing a string s representation of a binary number is as simple as int(s, 2).)

So my quick Python code to parse a list of seat descriptions looked something like this:

def parse(lines):
    return [int(l.replace('F', '0').replace('B', '1')
                 .replace('L', '0').replace('R', '1'), 2) for l in lines]

This worked just fine, but all those replace() calls feel a little silly. I thought that there must be a less-verbose way to do this.

And sure enough, there is!

`translate()` and `maketrans()`

The Python str.translate() method does just this:

str.translate(table)

Return a copy of the string in which each character has been mapped through the given translation table.

[…]

You can use str.maketrans() to create a translation map from character-to-character mappings in different formats.

(from https://docs.python.org/3/library/stdtypes.html#str.translate)

And from the description of str.maketrans():

static str.maketrans(x[, y[, z]])

This static method returns a translation table usable for str.translate().

[…]

If there are two arguments, they must be strings of equal length, and in the resulting dictionary, each character in x will be mapped to the character at the same position in y.

(from https://docs.python.org/3/library/stdtypes.html#str.maketrans)

We can use these methods to simplify our parse() method like so:

t = str.maketrans('FBLR', '0101')

def parse(lines):
    return [int(l.translate(t), 2) for l in lines]

Then part 1 of the puzzle (finding the highest seat ID in the input file) is trivial:

def part1(ids):
    return max(ids)

But the puzzle isn’t really the point of this post.

It’s more that I hadn’t ever used Python’s translate() method before, so I wanted to share in case it’s new to others as well.

Addendum

Fun tidbit: the Python translate() method is basically what the Unix tr command does. We can solve part 1 of the puzzle (find the highest seat ID out of all the input seat descriptions) in a one-liner like this:

$ <input05.txt tr FBLR 0101 | sort | tail -n1 | echo "ibase=2; $(cat)" | bc

Because all the seat descriptions are the same length, we can sort them as text strings and we’ll get the same order as if we sorted as numbers.¹ Then we can use tail -n1 to take just the last line (the maximum value), and run it through bc (the basic calculator) to convert from binary to decimal. (We just need to prepend an ibase=2 to get bc to parse its input as binary.)

If we tried to use plain sort after converting from binary to decimal, we could run into problems like 99 sorting after 100. But, we can use the -n option to sort the input numerically, like so:
```
$ <input05.txt tr FBLR 0101 | echo "ibase=2; $(cat)" | bc | sort -n | tail -n1
```
↩︎

Advent of Code 2020: Introduction and Day 4 reflection

Mon, 22 Nov 2021 22:10:00 -0800

This year’s Advent of Code is just around the corner! I had a lot of fun with these puzzles last year, so in honor of the occasion I thought I would share some of the things I learned over the course of the challenge.

For the unfamiliar, Advent of Code is an Advent calendar where a new programming puzzle unlocks each day of December (from the 1st through the 25th). Each puzzle has a part 1 and a part 2, where part 2 is usually a more complicated variation on the first part. Puzzles involve writing some code to compute an answer (usually based on some input file data).

I don’t intend to talk about every single puzzle, or even to give complete solutions to a few, but rather to share a few things I found interesting.

I used Python for all of last year’s puzzles. As the high-level structure of each puzzle was fairly similar (read input, compute something for part 1, compute something else for part 2), I got in the habit of starting each day from a skeleton file that looked something like this:

import sys

def parse(lines):
  pass

def part1(p):
  pass

def part2(p):
  pass

def main():
  parsed_input = parse(sys.stdin.readlines())
  print(part1(parsed_input))
  print(part2(parsed_input))

Usually, I would want to take the raw input and translate it into some other format that would be more convenient, so this is what the parse() method was for: maybe I needed to convert from strings to numbers, or maybe build a map of values indexed by (X, Y) coordinates.

When I got to Day 4, I stumbled a bit with the parse() method. Most of the input files so far involved a list of items, with one item per line. This puzzle’s input, however, groups of key:value pairs that could span multiple lines, with a blank line in between separate items.

This was the example input:

ecl:gry pid:860033327 eyr:2020 hcl:#fffffd
byr:1937 iyr:2017 cid:147 hgt:183cm

iyr:2013 ecl:amb cid:350 eyr:2023 pid:028048884
hcl:#cfa07d byr:1929

hcl:#ae17e1 iyr:2013
eyr:2024
ecl:brn pid:760753108 byr:1931
hgt:179cm

hcl:#cfa07d eyr:2025 pid:166559648
iyr:2011 ecl:brn hgt:59in

(These were supposed to represent passport data fields: ecl was “eye color”, pid was “passport ID number”, and so on.)

To make the actual puzzle solving code easier, I wanted to represent each passport as a dictionary, mapping keys (e.g. 'ecl', 'pid') to their values ('gry', '860033327').

My parse() method started with a list of input lines, but I needed to make sure to group these lines together up to the next blank line. I found myself wanting to use something like Python’s string split() method. This method takes a string and “splits” it at any whitespace, returning a new list of strings.

For example:

>>> 'Well, hello there'.split()
['Well,', 'hello', 'there']

I wanted to take my list of input lines and “split” it at every blank line; something like lines.split('\n'). But lines was a list, not a string, and I didn’t see any method like this in the Python standard library.

After some Googling, I found this Stack Overflow question wondering the same thing: is there a split() for lists?

One of the answers pointed out that we can use itertools.groupby() for this, something like this:

def listsplit(l, delim):
    return [list(group) for is_delim, group in
            itertools.groupby(l, key=lambda e: e == delim) if not is_delim]

It took me a minute to understand how this worked, so I thought I’d try to explain it here.

What does `itertools.groupby()` do?

The Python documentation has this to say:

itertools.groupby(iterable, key=None)

Make an iterator that returns consecutive keys and groups from the iterable. The key is a function computing a key value for each element. If not specified or is None, key defaults to an identity function and returns the element unchanged. Generally, the iterable needs to already be sorted on the same key function.

The operation of groupby() is similar to the uniq filter in Unix. It generates a break or new group every time the value of the key function changes (which is why it is usually necessary to have sorted the data using the same key function). That behavior differs from SQL’s GROUP BY which aggregates common elements regardless of their input order.

The returned group is itself an iterator that shares the underlying iterable with groupby(). Because the source is shared, when the groupby() object is advanced, the previous group is no longer visible. So, if that data is needed later, it should be stored as a list.

I found this a little hard to follow, especially because for our use case we don’t need or want to sort the input.

If we ignore the key parameter for now, the groupby() method essentially takes a sequence of elements and looks for consecutive elements that are the same. It returns an iterator of these individual groups.

A diagram might help. Say we start with a list with some repeated elements. These could be strings, numbers, or what have you, so I’ll use labeled boxes to represent the elements:

Calling groupby() with this list would group the elements like this:

(Here I’ve used angle brackets to represent an iterator that will return the enclosed items.)

Note that even though B appears twice, because they aren’t consecutive in the input, they are part of separate groups in the output.

However, this isn’t quite an accurate picture. The groupby() method doesn’t just return an iterator of each group of elements. It actually returns an iterator of tuples, where each tuple contains the grouped element followed by the group iterator itself. So I guess the output diagram should really look something like this:

That’s a lot of brackets….

But let’s come back to that key function. This function lets us group elements in different ways. Let’s say we had a function that returned 1 for a C element and 0 for any other element. If we called groupby() with this key function, it would apply this function to each element and then group according to the results. We could represent that something like this:

Now the first B is grouped with all of the A’s, because the key function gave the same result for all of these elements. (And note that the first element of each tuple is now the 0 or 1 from the key function.)

Back to `listsplit()`

For the list-splitting case, the key insight (ha!) is that we can use a key function that returns True if an element matches our delimiter, and False otherwise. This way groupby() will return all consecutive non-delimiter elements in one chunk.

Unfortunately, all of our delimiters get returned as an output group too. But, we can filter out these group by checking the key function output values.

Finally, we also need to capture all of the elements from the group iterators, which we can do by making a new list out of each iterator.

Putting it all together, this brings us to the listsplit() method from before:

And then using this method, the passport parsing routine is a snap:

def parse(lines):
    return [parse_passport(' '.join(p)) for p in listsplit(lines, '\n')]

def parse_passport(p):
    return dict(kv.split(':') for kv in p.split())

Now we have a list of dictionaries that are easy to query for individual data fields.

Addendum

The real moral of this story might be that it’s important not to get too focused on one approach.

As I was writing up this reflection, I realized that I don’t really even need a listsplit() method at all.

If I hadn’t had readlines() in my solution skeleton already, I might have seen that we can go straight to the input chunks I wanted by reading the entire file at once and using split('\n\n') to divide the input at blank lines.

I could replace the whole parse() method with something like this:

def parse(input):
    return [dict(kv.split(':') for kv in p.split()) for p in input.split('\n\n')]

A misleading error when using gRPC with Go and nginx

Mon, 28 Dec 2020 15:57:00 -0800

I wanted to document an issue I ran into while trying to write a gRPC service in Go, in case someone else just so happens to run into the same problem.

The scenario was this:

I had a web service written in Go
serving both HTTP traffic and gRPC traffic in plaintext on the same port
using nginx as a reverse proxy to provide TLS.

This might seem a bit contrived, but it came about fairly naturally. I was working on a web service written in Go, and then I wanted to add a gRPC API to it as well, for use by a mobile client. Rather than setting up a new sub-domain or messing with firewall rules to expose another port, I had the Go service accept gRPC requests on the same port that it was already listening on. Then, some time later I decided it would be easier to let nginx handle TLS rather than figuring out how to set up certificate rotation in the Go service itself, and so I had nginx terminate the TLS connection and proxy requests to the Go service over plain HTTP.

The error message

However, when I attempted this, all my gRPC requests stopped working, and I started seeing errors something like this in the nginx logs:

2020/12/13 01:14:04 [error] 29#29: *3 upstream sent too large http2 frame:
4740180 while reading response header from upstream, client: 172.21.0.1,
server: , request: "POST /Service/Method HTTP/2.0", upstream:
"grpc://172.21.0.2:8080", host: "localhost:8080"

My attempts to Google this error message didn’t turn up much. I found a blog post about a similar-sounding error, but I wasn’t using Kubernetes, and changing the proxy_buffers settings in nginx (the recommended fix) didn’t seem to have any effect.

So what was going on here?

I didn’t make much progress towards understanding this, until I came across the httputil.DumpRequest() method. When I used this method to log an incoming request in my main handler method, I saw this:

PRI * HTTP/2.0
Connection: close

I’d never heard of the PRI method, but at least this was something easy enough to search for online. I found that this comes from the HTTP/2 “client connection preface”. The preface is sent by the client at the start of an HTTP/2 connection, and it begins with a fixed byte sequence consisting of:

PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n

(where \r and \n indicate the carriage return and newline characters, respectively). This byte sequence was specifically chosen to look like an invalid HTTP/1.1 request, so that servers which don’t understand HTTP/2 will error out right away.

But why did my Go service start treating this as a request in its own right?

HTTP/2 and Go

When I added the gRPC service on the same port as the rest of the web service, I followed a technique mentioned in the grpc-go documentation. It comes down to a check that the incoming request is using HTTP/2 (as gRPC is implemented using HTTP/2), and has a Content-Type header whose value begins with application/grpc:

To share one port (such as 443 for https) between gRPC and an existing http.Handler, use a root http.Handler such as:
if r.ProtoMajor == 2 && strings.HasPrefix(
	r.Header.Get("Content-Type"), "application/grpc") {
	grpcServer.ServeHTTP(w, r)
} else {
	yourMux.ServeHTTP(w, r)
}

(from https://pkg.go.dev/google.golang.org/grpc#Server.ServeHTTP)

Unfortunately, I either skipped right over the previous paragraph in the documentation, or I simply forgot about it by the time I tried to move the TLS handling into the nginx reverse proxy:

The provided HTTP request must have arrived on an HTTP/2 connection. When using the Go standard library’s server, practically this means that the Request must also have arrived over TLS.

This is the core of the issue. The HTTP server in Go’s standard library does not support plaintext HTTP/2 by default.

(Why doesn’t Go support this by default? Well, HTTP/2 on the web effectively requires encryption, in the sense that currently all major web browsers will use HTTP/2 only with https:// URLs.¹ If Go’s standard library HTTP server is mainly used for writing web services, then this seems like a reasonable design decision.)

But what does it mean to “support plaintext HTTP/2”? Why does the HTTP/2 protocol care if the underlying transport is encrypted or not?

Starting an HTTP/2 connection

Before all of this, I was only vaguely aware that HTTP/2 existed. I had a basic understanding of HTTP/1 from a networking course in college, but HTTP/2 is quite different. To begin with, while HTTP/1 is a text-based protocol, HTTP/2 is a binary protocol. One implication of this is that HTTP/2 is not backwards-compatible with HTTP/1, and so clients and servers need some way to signal to each other that they support HTTP/2. There are three main ways to do this.

The first way is specific to https. As part of any https connection, the client and server first engage in something called a “TLS handshake” to agree on encryption parameters and other connection settings. When HTTP/2 was standardized, an extension was also added to the TLS protocol called “Application-Layer Protocol Negotiation” (ALPN) to let the client and server decide which protocol to use for the connection. The client can use this extension to indicate that it supports both HTTP/1 and HTTP/2, and then the server will indicate back to the client which of these options it has decided to use.

However, for plaintext http, there is no corresponding negotiation step when starting a connection. This brings us to the second way to signal support for HTTP/2: using the connection “upgrade” feature of HTTP/1.1. A client can send an HTTP/1 request that includes the header Upgrade: h2c (along with a few other required headers²) to signal that it can speak HTTP/2.

The third way to use HTTP/2 is for a client to have “prior knowledge” that a particular server supports it. Essentially, if some out-of-band signaling mechanism exists, a client can start speaking HTTP/2 directly.

This last method is the one that nginx uses for gRPC requests. As the gRPC protocol is implemented using HTTP/2, this constitutes “prior knowledge” that any gRPC endpoint must support HTTP/2. When nginx connects to the Go service, it immediately begins speaking HTTP/2, starting with the client connection preface.

However, because the Go standard library HTTP supports only the first method (ALPN), when it receives the HTTP/2 client connection preface, it parses this as an HTTP/1 request, and responds with an HTTP/1 response. When nginx receives this HTTP/1 response, this leads to the original error.

But why does the error message read “upstream sent too large http2 frame”? What’s that about?

HTTP/2 message framing

To answer that question, we need to understand just a little bit more about the HTTP/2 protocol:

HTTP/2 messages consist of a number of “frames.”
Each frame begin with a fixed-length header.
The first 3 bytes of this header is a Length field.
Unless otherwise indicated, the maximum value of this Length field is 16,384. (cf. RFC 7540 §4.1)

When nginx starts reading the erroneous HTTP/1 response from the Go service, it will try to start parsing it as an HTTP/2 message frame. Now, the first line of an HTTP/1 response starts contains the HTTP version and status code, for example:

HTTP/1.1 400 Bad Request

Trying to parse these bytes as the start of an HTTP/2 frame results in reading the first three bytes (“HTT”) as the frame length. In hexadecimal, these bytes are 0x485454, or 4,740,180 in decimal. Obviously this exceeds the maximum frame length of 16,384, and so this explains why nginx thinks the frame is too large.

The number 4740180 even shows up in the error message, and this could have been an important clue, but I did not realize its significance at the time. (This is the same exact idea as the magic number 1213486160, but with only 3 bytes instead of all 4 of “HTTP”.)

Putting it all together

To recap:

When nginx connects to an upstream gRPC server, it begins speaking HTTP/2 directly, starting with the HTTP/2 client connection preface. (The use of gRPC is the “prior knowledge” that the server must support HTTP/2.)
The connection preface begins with a fixed byte sequence chosen to look like like an HTTP/1 request, but with an invalid method and version.
The Go http standard library package does not recognize plaintext HTTP/2 requests, and so the Go service misinterprets the client connection preface as the start of an HTTP/1 request, and responds with an HTTP/1 response.
Then nginx attempts to parse the HTTP/1 response as an HTTP/2 frame, which leads to the original error message.

The nginx error message is misleading because it talks about an HTTP/2 frame being too large, but the real problem is that the Go service wasn’t speaking HTTP/2 at all.

So what is the fix?

Luckily, there’s an easy way to add plaintext HTTP/2 support to the Go standard library HTTP server, using the golang.org/x/net/http2/h2c package.

If the default http2.Server options are acceptable, this can be as simple as replacing something like this:

http.ListenAndServe(addr, handler)

with this:

http.ListenAndServe(addr, h2c.NewHandler(handler, &http2.Server{}))

Of course, another option would be to move the gRPC portion of the Go service onto a different port, and use the gRPC library’s built-in server (which does recognize plaintext HTTP/2 requests just fine). If I had simply done that to begin with, I wouldn’t have run in to this issue.

Kenneth Jenkins

Android APK patching

Archived builds

apktool to the rescue

Building a patched APK

More patching

Reference

Adventures in consumer electronics repair: GE clock radio

Advent of Code 2020: Day 19 reflection

My original approach

GLL parsing to the rescue

Advent of Code 2020: Day 18 reflection

Operator precedence

A complete solution

Advent of Code 2020: Day 13 reflection

Too clever by half

Advent of Code 2020: Day 5 reflection

translate() and maketrans()

Addendum

Advent of Code 2020: Introduction and Day 4 reflection

What does itertools.groupby() do?

Back to listsplit()

Addendum

A misleading error when using gRPC with Go and nginx

The error message

HTTP/2 and Go

Starting an HTTP/2 connection

HTTP/2 message framing

Putting it all together

`apktool` to the rescue

`translate()` and `maketrans()`

What does `itertools.groupby()` do?

Back to `listsplit()`